test: containerized Firefox end-to-end harness (geckodriver, MV2 temporary add-on) #184

New Issue

2026-08-09T16:27:58+02:00

clawbot commented

2026-08-09 16:27:58 +02:00

Companion to the Chrome harness in
#181. Firefox needs its own harness
because Chrome cannot verify what
#153 is about.

Feasibility was probed rather than assumed, and the verdict is that this
works today, with a negative control. Details below are measured.

What was proven

geckodriver 0.36.0 driving stock Firefox 153.0.3 headless in a container.
POST /session/{id}/moz/addon/install with
{path: "<unpacked dist/firefox dir>", temporary: true} installs the MV2
build directly. No XPI, no signing, no xpinstall.signatures.required
fiddling - temporary installs bypass signature checks even on release builds.
The popup uuid is pinned, not discovered, by setting the profile pref
before launch:
```
"extensions.webextensions.uuids":
    {"autistmask@sneak.berlin": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"}
```
This works because manifest/firefox.json already declares
browser_specific_settings.gecko.id, so the popup URL is deterministic.
The full wallet-creation flow runs under Firefox with zero console errors.

Two traps, both verified the hard way

-remote-allow-system-access is mandatory on Firefox 153. Without it,
navigating to moz-extension:// and executing chrome-context script both
fail with unsupported operation. Firefox 142 did not need it. Pin the
Firefox version or this will rot.
WebDriver BiDi log.entryAdded does not work for extension pages. This
is the important one. Verified on FF 142 and 153 with a same-session control:
on a plain http:// page BiDi delivers uncaught errors with stack traces;
on the moz-extension:// popup it delivers zero events. Firefox's
remote agent excludes extension browsing contexts from BiDi observation.

Any harness built on Playwright-BiDi or Puppeteer-BiDi will silently see
nothing and report success. That is precisely the vacuous-check failure
mode this repo has already hit twice. Do not go that route.

What works instead is the privileged console service via Marionette's chrome
context - Services.console.getMessageArray(), filtered to
sourceName.startsWith("moz-extension://") and non-warning, with
Services.console.reset() between steps. Same semantics as Playwright's
pageerror (uncaught only), and it captures background-page and
content-script errors for free.

Demonstrated to fail

Against dist/firefox/ built from current main, driving the real UI to the
Add Token screen - exit 1:

view-after-add-token: view-address        &lt;-- screen did not change
errors-after-add-token: [{
  "msg": "ReferenceError: showView is not defined",
  "src": "moz-extension://aaaa.../src/popup/index.js",
  "line": 13, "cat": "content javascript"
}]

Negative control, same harness and image against a build with the one-line
import fix applied - exit 0, view-add-token reached, errors: [].

Different exit codes on identical inputs modulo one line. The harness
discriminates, which is the only property that makes it worth having.

Definition of done

tests/e2e/firefox/ contains the WebDriver client, parameterized on
FIREFOX_BIN / GECKODRIVER with the extension directory as an argument.
script/test-e2e-firefox entrypoint plus a make test-e2e-firefox shim,
building dist/firefox/ first. Outside make check, same as the Chrome
harness.
A pinned tests/e2e/firefox/Dockerfile. Pin all three by digest, with
human-readable versions in comments above each:
- base node@sha256:d649c27dae7ba0137b3cef5dd75baa422c08dc3d9e3fc0c23dfb172dc3cc6436 (node:22-bookworm-slim)
- Firefox 153.0.3 linux-x86_64 tarball, sha256:22b312280900bfb174b685ece32c7b3c6d72e7f8e53d6d30f21ac41a8dc500a2
- geckodriver v0.36.0 linux64, sha256:0bde38707eb0a686a20c6bd50f4adcc7d60d4f73c60eb83ee9e0db8f65823e04
Any uncaught error originating from a moz-extension:// source fails the
run.
Coverage at minimum: popup loads clean, wallet creation through the real UI,
Add Token screen opens.
Demonstrated failing and then passing, with captured output in the PR
body. A green run is not evidence.
README Entrypoints section documents the new target and its container
requirement.

Implementation requirements

Zero npm dependencies. The probe harness is plain Node using global
fetch and child_process against geckodriver's HTTP API - no Selenium, no
Playwright, no ws. Keep it that way; it suits this repo's stated
minimal-dependency ethos and it is genuinely less code than a driver library.
Run the container with --shm-size=1g. No --privileged needed; the
CanCreateUserNamespace() EPERM sandbox warning is cosmetic.
Navigate with classic POST /session/{id}/url. BiDi
browsingContext.navigate refuses moz-extension:// outright.
Accept the duplication. The BiDi dead end means Chrome and Firefox cannot
share a driver layer - Chrome stays on Playwright, Firefox is a separate
WebDriver client of roughly 200 lines. Do not build a premature abstraction
over two backends to avoid writing UI steps twice; wait until the Firefox
suite actually has enough shared steps to justify a shim, and say so in the
PR if you think it has.
Note in the code that -remote-allow-system-access grants the driver full
chrome privileges. Acceptable for a throwaway CI container, and it should
never be pointed at anything else.
Error capture is poll-based (per step), not event-streamed. That is fine for
step-boundary assertions but cannot attribute an error to a precise moment
within a step. Document the limitation rather than implying parity with the
Chrome harness.

Companion to the Chrome harness in https://git.eeqj.de/sneak/AutistMask/issues/181. Firefox needs its own harness because Chrome cannot verify what https://git.eeqj.de/sneak/AutistMask/issues/153 is about. Feasibility was probed rather than assumed, and the verdict is that this **works today**, with a negative control. Details below are measured. ## What was proven - `geckodriver 0.36.0` driving stock Firefox 153.0.3 headless in a container. - `POST /session/{id}/moz/addon/install` with `{path: "<unpacked dist/firefox dir>", temporary: true}` installs the MV2 build directly. No XPI, no signing, no `xpinstall.signatures.required` fiddling - temporary installs bypass signature checks even on release builds. - The popup uuid is **pinned, not discovered**, by setting the profile pref before launch: ``` "extensions.webextensions.uuids": {"autistmask@sneak.berlin": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"} ``` This works because `manifest/firefox.json` already declares `browser_specific_settings.gecko.id`, so the popup URL is deterministic. - The full wallet-creation flow runs under Firefox with zero console errors. ## Two traps, both verified the hard way 1. **`-remote-allow-system-access` is mandatory on Firefox 153.** Without it, navigating to `moz-extension://` and executing chrome-context script both fail with `unsupported operation`. Firefox 142 did not need it. Pin the Firefox version or this will rot. 2. **WebDriver BiDi `log.entryAdded` does not work for extension pages.** This is the important one. Verified on FF 142 and 153 with a same-session control: on a plain `http://` page BiDi delivers uncaught errors with stack traces; on the `moz-extension://` popup it delivers **zero** events. Firefox's remote agent excludes extension browsing contexts from BiDi observation. **Any harness built on Playwright-BiDi or Puppeteer-BiDi will silently see nothing and report success.** That is precisely the vacuous-check failure mode this repo has already hit twice. Do not go that route. What works instead is the privileged console service via Marionette's chrome context - `Services.console.getMessageArray()`, filtered to `sourceName.startsWith("moz-extension://")` and non-warning, with `Services.console.reset()` between steps. Same semantics as Playwright's `pageerror` (uncaught only), and it captures background-page and content-script errors for free. ## Demonstrated to fail Against `dist/firefox/` built from current `main`, driving the real UI to the Add Token screen - exit 1: ``` view-after-add-token: view-address <-- screen did not change errors-after-add-token: [{ "msg": "ReferenceError: showView is not defined", "src": "moz-extension://aaaa.../src/popup/index.js", "line": 13, "cat": "content javascript" }] ``` Negative control, same harness and image against a build with the one-line import fix applied - exit 0, `view-add-token` reached, `errors: []`. Different exit codes on identical inputs modulo one line. The harness discriminates, which is the only property that makes it worth having. ## Definition of done 1. `tests/e2e/firefox/` contains the WebDriver client, parameterized on `FIREFOX_BIN` / `GECKODRIVER` with the extension directory as an argument. 2. `script/test-e2e-firefox` entrypoint plus a `make test-e2e-firefox` shim, building `dist/firefox/` first. Outside `make check`, same as the Chrome harness. 3. A pinned `tests/e2e/firefox/Dockerfile`. Pin **all three** by digest, with human-readable versions in comments above each: - base `node@sha256:d649c27dae7ba0137b3cef5dd75baa422c08dc3d9e3fc0c23dfb172dc3cc6436` (`node:22-bookworm-slim`) - Firefox 153.0.3 linux-x86_64 tarball, `sha256:22b312280900bfb174b685ece32c7b3c6d72e7f8e53d6d30f21ac41a8dc500a2` - geckodriver v0.36.0 linux64, `sha256:0bde38707eb0a686a20c6bd50f4adcc7d60d4f73c60eb83ee9e0db8f65823e04` 4. Any uncaught error originating from a `moz-extension://` source fails the run. 5. Coverage at minimum: popup loads clean, wallet creation through the real UI, Add Token screen opens. 6. **Demonstrated failing and then passing**, with captured output in the PR body. A green run is not evidence. 7. README Entrypoints section documents the new target and its container requirement. ## Implementation requirements - **Zero npm dependencies.** The probe harness is plain Node using global `fetch` and `child_process` against geckodriver's HTTP API - no Selenium, no Playwright, no `ws`. Keep it that way; it suits this repo's stated minimal-dependency ethos and it is genuinely less code than a driver library. - Run the container with `--shm-size=1g`. No `--privileged` needed; the `CanCreateUserNamespace() EPERM` sandbox warning is cosmetic. - Navigate with **classic** `POST /session/{id}/url`. BiDi `browsingContext.navigate` refuses `moz-extension://` outright. - Accept the duplication. The BiDi dead end means Chrome and Firefox cannot share a driver layer - Chrome stays on Playwright, Firefox is a separate WebDriver client of roughly 200 lines. Do not build a premature abstraction over two backends to avoid writing UI steps twice; wait until the Firefox suite actually has enough shared steps to justify a shim, and say so in the PR if you think it has. - Note in the code that `-remote-allow-system-access` grants the driver full chrome privileges. Acceptable for a throwaway CI container, and it should never be pointed at anything else. - Error capture is poll-based (per step), not event-streamed. That is fine for step-boundary assertions but cannot attribute an error to a precise moment within a step. Document the limitation rather than implying parity with the Chrome harness.

clawbot added this to the 1.0.0 milestone 2026-08-09 16:27:58 +02:00

clawbot referenced this issue

2026-08-09 16:28:23 +02:00

fix: Firefox target is non-functional — Chrome callback APIs used against the promise-only browser namespace #153

clawbot referenced this issue

2026-08-09 16:28:45 +02:00

decision: no browser in the agent environment, so several 1.0.0 items cannot have their DoD verified #173

clawbot commented

2026-08-12 10:32:35 +02:00

Plan, following the measured facts in the body rather than re-probing them.

Layout: tests/e2e/firefox/driver.js (WebDriver client over fetch +
child_process, no deps), tests/e2e/firefox/run.js (steps), and the pinned
tests/e2e/firefox/Dockerfile. script/test-e2e-firefox builds dist/firefox/,
builds the image from that Dockerfile, and runs it with --shm-size=1g. Not
wired into script/check.

Error capture is the chrome-context console service exactly as recorded —
Services.console.getMessageArray(), sourceName.startsWith("moz-extension://"),
warning flag excluded, Services.console.reset() at each step boundary. No BiDi
anywhere in the harness.

Two things not settled by the body that I will decide by measurement:

Network. The Chrome suite stubs every request from fixtures; that machinery
is Playwright's and does not port. I intend --network none on the container
instead — hard offline, nothing to keep in sync — and will check whether the
resulting failed fetches produce moz-extension:// console errors of their
own. If they do I will report what they are rather than widening the filter
to hide them.
Step granularity. Poll-based capture attributes an error to a step, not to a
moment, so steps stay small enough that the attribution is still useful. That
limitation gets documented, not papered over.

Scope is the harness. If it surfaces
#153 I will record what it
shows and leave it unfixed.

Plan, following the measured facts in the body rather than re-probing them. Layout: `tests/e2e/firefox/driver.js` (WebDriver client over `fetch` + `child_process`, no deps), `tests/e2e/firefox/run.js` (steps), and the pinned `tests/e2e/firefox/Dockerfile`. `script/test-e2e-firefox` builds `dist/firefox/`, builds the image from that Dockerfile, and runs it with `--shm-size=1g`. Not wired into `script/check`. Error capture is the chrome-context console service exactly as recorded — `Services.console.getMessageArray()`, `sourceName.startsWith("moz-extension://")`, warning flag excluded, `Services.console.reset()` at each step boundary. No BiDi anywhere in the harness. Two things not settled by the body that I will decide by measurement: 1. Network. The Chrome suite stubs every request from fixtures; that machinery is Playwright's and does not port. I intend `--network none` on the container instead — hard offline, nothing to keep in sync — and will check whether the resulting failed fetches produce `moz-extension://` console errors of their own. If they do I will report what they are rather than widening the filter to hide them. 2. Step granularity. Poll-based capture attributes an error to a step, not to a moment, so steps stay small enough that the attribution is still useful. That limitation gets documented, not papered over. Scope is the harness. If it surfaces [#153](https://git.eeqj.de/sneak/AutistMask/issues/153) I will record what it shows and leave it unfixed.

clawbot referenced this issue from a commit

2026-08-12 10:43:47 +02:00

test: containerized Firefox end-to-end harness (closes #184)

clawbot referenced a pull request that will close this issue

2026-08-12 10:44:16 +02:00

test: containerized Firefox end-to-end harness (closes #184) #256

clawbot commented

2026-08-12 10:44:35 +02:00

Built and pushed as
#256; the full record is in
the PR body.

Everything measured in this issue held up exactly as written — both digests
matched the URLs I fetched, -remote-allow-system-access was needed, and the
pinned uuid worked first try. One detail cost time and is worth adding to the
record for whoever writes the next WebDriver client here: the W3C web element
key is element-6066-11e4-a52e-4f735466cecf, ending cecf. Getting the last
character wrong yields an element reference of undefined and a thoroughly
misleading no such element: The element with the reference undefined is not known in the current browsing context from geckodriver. findElement() now
checks for the key and reports the actual response instead.

Verification. make check green (19 suites, 416 tests), re-run after the rebase
onto next. make test-e2e-firefox exits 0 with 3/3.

The discrimination demo is the one recorded here, reproduced against
src/popup/views/addToken.js rather than src/popup/index.js: dropping
showView from its import puts the failure at step 3 instead of step 1, so
steps 1 and 2 still run and the flow is exercised. Output matches what is in
this issue almost verbatim, index.js:13 and content javascript included —
exit 1 with ReferenceError: showView is not defined and current view is view-address, exit 0 with the import restored.

I also checked the "captures background-page errors for free" claim rather than
repeating it: a deliberate throw in src/background/index.js failed step 1,
even though the suite never navigates to the background page.

Two decisions I flagged in the plan, now settled by measurement. Network is
--network none on the container, and the resulting failed fetches produce no
moz-extension:// console errors, so no filter widening was needed. Error
capture stayed poll-based and the limitation is documented in run.js and the
README rather than glossed.

Nothing about
#153 surfaced, which is what
I would expect — its breakage is in the content-script relay, the approval
popups and the background windows/tabs calls, and none of those are on the
three popup-only paths covered here. Whoever takes it will need steps that
actually drive a dApp connection; the harness will not otherwise reach it.

Built and pushed as [#256](https://git.eeqj.de/sneak/AutistMask/pulls/256); the full record is in the PR body. Everything measured in this issue held up exactly as written — both digests matched the URLs I fetched, `-remote-allow-system-access` was needed, and the pinned uuid worked first try. One detail cost time and is worth adding to the record for whoever writes the next WebDriver client here: the W3C web element key is `element-6066-11e4-a52e-4f735466cecf`, ending `cecf`. Getting the last character wrong yields an element reference of `undefined` and a thoroughly misleading `no such element: The element with the reference undefined is not known in the current browsing context` from geckodriver. `findElement()` now checks for the key and reports the actual response instead. Verification. `make check` green (19 suites, 416 tests), re-run after the rebase onto `next`. `make test-e2e-firefox` exits 0 with 3/3. The discrimination demo is the one recorded here, reproduced against `src/popup/views/addToken.js` rather than `src/popup/index.js`: dropping `showView` from its import puts the failure at step 3 instead of step 1, so steps 1 and 2 still run and the flow is exercised. Output matches what is in this issue almost verbatim, `index.js:13` and `content javascript` included — exit 1 with `ReferenceError: showView is not defined` and `current view is view-address`, exit 0 with the import restored. I also checked the "captures background-page errors for free" claim rather than repeating it: a deliberate throw in `src/background/index.js` failed step 1, even though the suite never navigates to the background page. Two decisions I flagged in the plan, now settled by measurement. Network is `--network none` on the container, and the resulting failed fetches produce no `moz-extension://` console errors, so no filter widening was needed. Error capture stayed poll-based and the limitation is documented in `run.js` and the README rather than glossed. Nothing about [#153](https://git.eeqj.de/sneak/AutistMask/issues/153) surfaced, which is what I would expect — its breakage is in the content-script relay, the approval popups and the background `windows`/`tabs` calls, and none of those are on the three popup-only paths covered here. Whoever takes it will need steps that actually drive a dApp connection; the harness will not otherwise reach it.

clawbot referenced this issue

2026-08-12 10:53:38 +02:00

test: containerized Firefox end-to-end harness (closes #184) #256

clawbot referenced this issue

2026-08-12 10:56:28 +02:00

build: nothing automatic ever runs the e2e suites, so every browser-level guarantee is manual-only #259

clawbot referenced this issue from a commit

2026-08-12 11:01:14 +02:00

test: containerized Firefox end-to-end harness (closes #184)

clawbot referenced this issue

2026-08-12 11:02:54 +02:00

test: containerized Firefox end-to-end harness (closes #184) #256