test: containerized Firefox end-to-end harness (closes #184)
All checks were successful
check / check (push) Successful in 34s

Drives the real popup in a real Firefox with dist/firefox/ installed as an
unpacked MV2 temporary add-on via geckodriver. make test-e2e-firefox, outside
make check like the Chrome suite. Zero npm dependencies: plain fetch and
child_process against geckodriver's HTTP API. Base image, Firefox tarball and
geckodriver are each pinned by digest and verified at build time.

Error capture reads the privileged console service through Marionette's chrome
context, not WebDriver BiDi. BiDi delivers nothing at all for extension pages,
so a BiDi-based harness would observe zero events and report success -- the
vacuous-check shape this repo has shipped twice. Both the driver and the README
say so where someone would be tempted to simplify.

Demonstrated to discriminate: a background page that throws at the top of the
file, a missing import, and an async throw where every UI assertion still
passes each fail the run.

Three limits are measured and documented rather than papered over: capture is
poll-based so an error is attributed to a step, not a moment; the console ring
buffer holds 250 messages and evicts the oldest, measured against a clean-run
peak of 4; and the drained window ends roughly 1.5s after the last step, with
observed jitter rather than a hard boundary. Content-script capture is marked
unverified because --network none leaves no page to inject into, and that same
choice inverts coverage of network-dependent code.
This commit was merged in pull request #256.
This commit is contained in:
2026-08-12 12:20:14 +02:00
parent 918e581ef3
commit 52c7c1b060
7 changed files with 993 additions and 16 deletions

103
README.md
View File

@@ -83,7 +83,10 @@ provide:
git pre-commit hook
- `script/projectname` — print the project name (used for the Docker image tag)
- `script/test` — run the test suite (jest)
- `script/test-e2e` — run the browser end-to-end suite (docker required; see
- `script/test-e2e` — run the Chrome browser end-to-end suite (docker required;
see [End-to-End Tests](#end-to-end-tests))
- `script/test-e2e-firefox` — run the Firefox browser end-to-end suite (docker
required; builds its own pinned image, see
[End-to-End Tests](#end-to-end-tests))
- `script/lint` — run the linter
- `script/fmt` — format all files (writes)
@@ -123,6 +126,14 @@ The Makefile shims to those. It also carries a few targets that have no
## End-to-End Tests
There are two suites, one per browser, and they share no code. Chrome runs on
Playwright; Firefox has its own WebDriver client, because Playwright cannot
observe errors on a Firefox extension page at all — see
[Firefox](#firefox-make-test-e2e-firefox) below. Both require docker, and both
are outside `make check`.
### Chrome (`make test-e2e`)
`make test-e2e` builds `dist/chrome/` and drives the **real popup in a real
Chrome**, loaded as an unpacked MV3 extension inside a pinned
`mcr.microsoft.com/playwright` container (pinned by digest in `script/test-e2e`;
@@ -200,12 +211,90 @@ a `ReferenceError` from a used-but-not-imported identifier is invisible to
`make check` (`script/lint` is only `prettier --check`) but fatal in a browser,
and this suite exists because exactly that class of bug shipped twice.
`make test-e2e` is deliberately **not** part of `make check` or `make test`.
`REPO_POLICIES.md` caps `make test` at 20 seconds and a browser suite does not
fit; nothing in `tests/e2e/` is named `*.test.js`, so jest cannot pick it up
either. It is also not wired into the Gitea workflow yet — docker-in-docker in
CI is a separate question. Run it locally before changing anything under
`src/popup/views/`.
### Firefox (`make test-e2e-firefox`)
`make test-e2e-firefox` builds `dist/firefox/` and drives the **real popup in a
real Firefox**, installed as an unpacked MV2 temporary add-on via geckodriver.
It covers popup load, wallet creation through the UI, and the Add Token screen.
The suite lives in `tests/e2e/firefox/` and has **no npm dependencies at all**:
it is a small WebDriver client built on global `fetch` and `child_process`
against geckodriver's HTTP API.
Unlike the Chrome suite it builds its own container image rather than pulling a
published one, because no published image carries both a pinned Firefox and a
matching geckodriver. `tests/e2e/firefox/Dockerfile` pins all three external
artifacts by digest — the `node` base image, the Firefox 153.0.3 tarball, and
geckodriver 0.36.0 — and the Firefox version in particular must not float:
`-remote-allow-system-access` is **mandatory** on 153 and was not on 142.
Without that flag, both navigating to `moz-extension://` and running
chrome-context script fail with `unsupported operation`. The flag grants the
driver full chrome privileges over that browser, which is acceptable only
because it is a throwaway container.
The popup's `moz-extension://` uuid is **pinned, not discovered**: the profile
pref `extensions.webextensions.uuids` maps the extension id that
`manifest/firefox.json` already declares to a fixed uuid, so the popup URL is
deterministic. Navigation uses **classic** WebDriver `POST /session/{id}/url`,
because BiDi's `browsingContext.navigate` refuses `moz-extension://` outright.
**Any uncaught error from a `moz-extension://` source fails the run**, including
errors from the background page, which the suite never navigates to: a `throw`
at the top of `src/background/index.js` kills the background page and fails
step 1. Content-script errors should arrive by the same route, but this suite
does not exercise it and does not claim it — with `--network none` there is no
`http://` page for a content script to be injected into. Errors from add-on
install and background startup are folded into step 1 rather than discarded.
Errors are read from the privileged `nsIConsoleService` in Marionette's chrome
context and filtered to non-warning entries whose `sourceName` is the extension
origin. That mechanism is not a stylistic choice. WebDriver BiDi's
`log.entryAdded` delivers **nothing** for extension pages: on a plain `http://`
page it reports uncaught errors with stack traces, and on the `moz-extension://`
popup it reports zero events, because Firefox's remote agent excludes extension
browsing contexts from BiDi observation. Any harness built on Playwright-BiDi or
Puppeteer-BiDi would therefore see nothing and report success, which is exactly
the vacuous check this repo has already shipped twice. Do not migrate this suite
to BiDi.
Two limits are worth knowing, both real differences from the Chrome suite:
- **Error capture is poll-based, not event-streamed.** The console is drained at
each step boundary, so an error is attributed to the step it was drained
after, not to a moment within it. The window that is drained runs from add-on
install to **≈1.5s** after the last step returns — a 500ms settle, a 1000ms
tail sleep and two drain round trips — and then the browser is torn down. That
cut-off is not a hard boundary: with throws scheduled at fixed offsets, three
runs reported everything up to +1.5s and one of the three also reported +1.6s,
so an error landing near it may or may not be seen, and anything well past it
is not. Inside the window there is no race — each drain reads and clears the
console in a single chrome round trip, so an error logged mid-drain lands in
that batch or the next rather than being destroyed unread — but there is a
**capacity limit**: `nsIConsoleService` keeps a ring buffer of 250 messages
and silently evicts the oldest, so more than 250 console messages between two
drains destroys the excess unread. 400 throws inside one step are reported as
exactly the newest 250, three runs running. That buffer is shared with
Firefox's own console noise; a clean run peaks at 4 of 250 at the install
drain and 0 at every later drain, so the three steps here have wide headroom,
but a step that logs heavily could evict unread errors. What poll-based costs
is location, not coverage: an error cannot be placed within a step the way the
Chrome suite's `pageerror` events place it.
- **Nothing is stubbed, which inverts the coverage of network-dependent code.**
There is no fixture layer; the container runs with `--network none` instead,
so the run is offline and deterministic and no request can escape. The
extension swallows its own fetch failures, so the flows are unaffected — but
every network call fails, so only the _failure_ branches of code that depends
on one are ever executed. A `ReferenceError` in the success path of
`renderTransactions`, or of price or balance rendering, passes this suite
green. The offline run is also weaker than the Chrome suite's interception: it
proves nothing got out, but it cannot report which requests were attempted.
Closing that gap needs a fixture layer, deliberately out of scope for this
harness.
Neither `make test-e2e` nor `make test-e2e-firefox` is part of `make check` or
`make test`. `REPO_POLICIES.md` caps `make test` at 20 seconds and a browser
suite does not fit; nothing in `tests/e2e/` is named `*.test.js`, so jest cannot
pick it up either. Neither is wired into the Gitea workflow yet —
docker-in-docker in CI is a separate question. Run them locally before changing
anything under `src/popup/views/`.
## Rationale