test: containerized Firefox end-to-end harness (closes #184)
All checks were successful
check / check (push) Successful in 1m18s
All checks were successful
check / check (push) Successful in 1m18s
Drives the real popup in a real Firefox with dist/firefox/ installed as an unpacked MV2 temporary add-on, via geckodriver. Covers popup load, wallet creation through the UI, and the Add Token screen. Outside make check, like the Chrome suite. Zero npm dependencies: tests/e2e/firefox/driver.js is a WebDriver client over global fetch and child_process against geckodriver's HTTP API. The Dockerfile pins the node base image, the Firefox 153.0.3 tarball and geckodriver 0.36.0 by digest. Errors are read from the privileged nsIConsoleService in Marionette's chrome context, filtered to non-warning entries whose sourceName is the extension origin. BiDi log.entryAdded delivers nothing at all for extension pages, so a Playwright-BiDi or Puppeteer-BiDi harness would see nothing and report success; the code says so where someone would be tempted to simplify it. Errors logged during add-on install and background startup are drained and folded into step 1, never discarded: a throw at the top of src/background/index.js kills the background page and fails the run. Content-script capture is left as unverified, because --network none leaves no http:// page for a content script to be injected into. Each drain reads the console and clears it in ONE chrome script. Splitting the read from Services.console.reset() left a window between the two round trips in which an error was logged into a buffer about to be discarded, and destroyed unread rather than deferred to the next drain; a probe of 100 sequenced throws at 20ms spacing lost one. With the drain atomic the same probe accounts for every throw that falls inside the observed window, on two consecutive runs. No driver layer is shared with the Chrome suite and the three UI steps are written twice deliberately: the two backends have no common substrate, and three steps do not pay for a shim. Two limits are documented rather than papered over. Error capture is poll-based, so an error is attributed to a step and not to a moment within it, and the observed window ends ~1.5s after the last step returns, measured: errors at +0.5s, +1.0s and +1.5s are reported and +1.6s and later never are, because the browser is torn down. Nothing is stubbed; the container runs with --network none instead, which proves no request escaped, cannot report which were attempted, and runs only the failure branches of network-dependent code.
This commit is contained in:
95
README.md
95
README.md
@@ -83,7 +83,10 @@ provide:
|
||||
git pre-commit hook
|
||||
- `script/projectname` — print the project name (used for the Docker image tag)
|
||||
- `script/test` — run the test suite (jest)
|
||||
- `script/test-e2e` — run the browser end-to-end suite (docker required; see
|
||||
- `script/test-e2e` — run the Chrome browser end-to-end suite (docker required;
|
||||
see [End-to-End Tests](#end-to-end-tests))
|
||||
- `script/test-e2e-firefox` — run the Firefox browser end-to-end suite (docker
|
||||
required; builds its own pinned image, see
|
||||
[End-to-End Tests](#end-to-end-tests))
|
||||
- `script/lint` — run the linter
|
||||
- `script/fmt` — format all files (writes)
|
||||
@@ -123,6 +126,14 @@ The Makefile shims to those. It also carries a few targets that have no
|
||||
|
||||
## End-to-End Tests
|
||||
|
||||
There are two suites, one per browser, and they share no code. Chrome runs on
|
||||
Playwright; Firefox has its own WebDriver client, because Playwright cannot
|
||||
observe errors on a Firefox extension page at all — see
|
||||
[Firefox](#firefox-make-test-e2e-firefox) below. Both require docker, and both
|
||||
are outside `make check`.
|
||||
|
||||
### Chrome (`make test-e2e`)
|
||||
|
||||
`make test-e2e` builds `dist/chrome/` and drives the **real popup in a real
|
||||
Chrome**, loaded as an unpacked MV3 extension inside a pinned
|
||||
`mcr.microsoft.com/playwright` container (pinned by digest in `script/test-e2e`;
|
||||
@@ -183,12 +194,82 @@ a `ReferenceError` from a used-but-not-imported identifier is invisible to
|
||||
`make check` (`script/lint` is only `prettier --check`) but fatal in a browser,
|
||||
and this suite exists because exactly that class of bug shipped twice.
|
||||
|
||||
`make test-e2e` is deliberately **not** part of `make check` or `make test`.
|
||||
`REPO_POLICIES.md` caps `make test` at 20 seconds and a browser suite does not
|
||||
fit; nothing in `tests/e2e/` is named `*.test.js`, so jest cannot pick it up
|
||||
either. It is also not wired into the Gitea workflow yet — docker-in-docker in
|
||||
CI is a separate question. Run it locally before changing anything under
|
||||
`src/popup/views/`.
|
||||
### Firefox (`make test-e2e-firefox`)
|
||||
|
||||
`make test-e2e-firefox` builds `dist/firefox/` and drives the **real popup in a
|
||||
real Firefox**, installed as an unpacked MV2 temporary add-on via geckodriver.
|
||||
It covers popup load, wallet creation through the UI, and the Add Token screen.
|
||||
The suite lives in `tests/e2e/firefox/` and has **no npm dependencies at all**:
|
||||
it is a small WebDriver client built on global `fetch` and `child_process`
|
||||
against geckodriver's HTTP API.
|
||||
|
||||
Unlike the Chrome suite it builds its own container image rather than pulling a
|
||||
published one, because no published image carries both a pinned Firefox and a
|
||||
matching geckodriver. `tests/e2e/firefox/Dockerfile` pins all three external
|
||||
artifacts by digest — the `node` base image, the Firefox 153.0.3 tarball, and
|
||||
geckodriver 0.36.0 — and the Firefox version in particular must not float:
|
||||
`-remote-allow-system-access` is **mandatory** on 153 and was not on 142.
|
||||
Without that flag, both navigating to `moz-extension://` and running
|
||||
chrome-context script fail with `unsupported operation`. The flag grants the
|
||||
driver full chrome privileges over that browser, which is acceptable only
|
||||
because it is a throwaway container.
|
||||
|
||||
The popup's `moz-extension://` uuid is **pinned, not discovered**: the profile
|
||||
pref `extensions.webextensions.uuids` maps the extension id that
|
||||
`manifest/firefox.json` already declares to a fixed uuid, so the popup URL is
|
||||
deterministic. Navigation uses **classic** WebDriver `POST /session/{id}/url`,
|
||||
because BiDi's `browsingContext.navigate` refuses `moz-extension://` outright.
|
||||
|
||||
**Any uncaught error from a `moz-extension://` source fails the run**, including
|
||||
errors from the background page, which the suite never navigates to: a `throw`
|
||||
at the top of `src/background/index.js` kills the background page and fails
|
||||
step 1. Content-script errors should arrive by the same route, but this suite
|
||||
does not exercise it and does not claim it — with `--network none` there is no
|
||||
`http://` page for a content script to be injected into. Errors from add-on
|
||||
install and background startup are folded into step 1 rather than discarded.
|
||||
Errors are read from the privileged `nsIConsoleService` in Marionette's chrome
|
||||
context and filtered to non-warning entries whose `sourceName` is the extension
|
||||
origin. That mechanism is not a stylistic choice. WebDriver BiDi's
|
||||
`log.entryAdded` delivers **nothing** for extension pages: on a plain `http://`
|
||||
page it reports uncaught errors with stack traces, and on the `moz-extension://`
|
||||
popup it reports zero events, because Firefox's remote agent excludes extension
|
||||
browsing contexts from BiDi observation. Any harness built on Playwright-BiDi or
|
||||
Puppeteer-BiDi would therefore see nothing and report success, which is exactly
|
||||
the vacuous check this repo has already shipped twice. Do not migrate this suite
|
||||
to BiDi.
|
||||
|
||||
Two limits are worth knowing, both real differences from the Chrome suite:
|
||||
|
||||
- **Error capture is poll-based, not event-streamed.** The console is drained at
|
||||
each step boundary, so an error is attributed to the step it was drained
|
||||
after, not to a moment within it. The window that is drained runs from add-on
|
||||
install to **≈1.5s** after the last step returns, then the browser is torn
|
||||
down; measured with throws scheduled at fixed offsets, errors at +0.5s, +1.0s
|
||||
and +1.5s are reported and +1.6s and later never are. Within that window
|
||||
nothing is dropped — each drain reads and clears the console in a single
|
||||
chrome round trip, so an error logged mid-drain lands in that batch or the
|
||||
next one rather than being destroyed unread; a probe of 100 throws at 20ms
|
||||
spacing accounts for every one that falls inside the window, twice running.
|
||||
What poll-based costs is location, not coverage: an error cannot be placed
|
||||
within a step the way the Chrome suite's `pageerror` events place it.
|
||||
- **Nothing is stubbed, which inverts the coverage of network-dependent code.**
|
||||
There is no fixture layer; the container runs with `--network none` instead,
|
||||
so the run is offline and deterministic and no request can escape. The
|
||||
extension swallows its own fetch failures, so the flows are unaffected — but
|
||||
every network call fails, so only the _failure_ branches of code that depends
|
||||
on one are ever executed. A `ReferenceError` in the success path of
|
||||
`renderTransactions`, or of price or balance rendering, passes this suite
|
||||
green. The offline run is also weaker than the Chrome suite's interception: it
|
||||
proves nothing got out, but it cannot report which requests were attempted.
|
||||
Closing that gap needs a fixture layer, deliberately out of scope for this
|
||||
harness.
|
||||
|
||||
Neither `make test-e2e` nor `make test-e2e-firefox` is part of `make check` or
|
||||
`make test`. `REPO_POLICIES.md` caps `make test` at 20 seconds and a browser
|
||||
suite does not fit; nothing in `tests/e2e/` is named `*.test.js`, so jest cannot
|
||||
pick it up either. Neither is wired into the Gitea workflow yet —
|
||||
docker-in-docker in CI is a separate question. Run them locally before changing
|
||||
anything under `src/popup/views/`.
|
||||
|
||||
## Rationale
|
||||
|
||||
|
||||
Reference in New Issue
Block a user