sneak/AutistMask
1
0
Fork 0
Code Issues 42 Pull Requests 8 Actions Packages Projects Releases Wiki Activity

test: containerized Firefox end-to-end harness (geckodriver, MV2 temporary add-on) #184

New Issue
Closed
opened 2026-08-09 16:27:58 +02:00 by clawbot · 2 comments
clawbot commented 2026-08-09 16:27:58 +02:00
Collaborator
Copy Link

Companion to the Chrome harness in
#181. Firefox needs its own harness
because Chrome cannot verify what
#153 is about.

Feasibility was probed rather than assumed, and the verdict is that this
works today, with a negative control. Details below are measured.

What was proven

  • geckodriver 0.36.0 driving stock Firefox 153.0.3 headless in a container.

  • POST /session/{id}/moz/addon/install with
    {path: "<unpacked dist/firefox dir>", temporary: true} installs the MV2
    build directly. No XPI, no signing, no xpinstall.signatures.required
    fiddling - temporary installs bypass signature checks even on release builds.

  • The popup uuid is pinned, not discovered, by setting the profile pref
    before launch:

    "extensions.webextensions.uuids":
    {"autistmask@sneak.berlin": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"}

    This works because manifest/firefox.json already declares
    browser_specific_settings.gecko.id, so the popup URL is deterministic.

  • The full wallet-creation flow runs under Firefox with zero console errors.

Two traps, both verified the hard way

  1. -remote-allow-system-access is mandatory on Firefox 153. Without it,
    navigating to moz-extension:// and executing chrome-context script both
    fail with unsupported operation. Firefox 142 did not need it. Pin the
    Firefox version or this will rot.

  2. WebDriver BiDi log.entryAdded does not work for extension pages. This
    is the important one. Verified on FF 142 and 153 with a same-session control:
    on a plain http:// page BiDi delivers uncaught errors with stack traces;
    on the moz-extension:// popup it delivers zero events. Firefox's
    remote agent excludes extension browsing contexts from BiDi observation.

    Any harness built on Playwright-BiDi or Puppeteer-BiDi will silently see
    nothing and report success.     That is precisely the vacuous-check failure
    mode this repo has already hit twice. Do not go that route.

    What works instead is the privileged console service via Marionette's chrome
    context - Services.console.getMessageArray(), filtered to
    sourceName.startsWith("moz-extension://") and non-warning, with
    Services.console.reset() between steps. Same semantics as Playwright's
    pageerror (uncaught only), and it captures background-page and
    content-script errors for free.

Demonstrated to fail

Against dist/firefox/ built from current main, driving the real UI to the
Add Token screen - exit 1:

view-after-add-token: view-address        <-- screen did not change
errors-after-add-token: [{
  "msg": "ReferenceError: showView is not defined",
  "src": "moz-extension://aaaa.../src/popup/index.js",
  "line": 13, "cat": "content javascript"
}]

Negative control, same harness and image against a build with the one-line
import fix applied - exit 0, view-add-token reached, errors: [].

Different exit codes on identical inputs modulo one line. The harness
discriminates, which is the only property that makes it worth having.

Definition of done

  1. tests/e2e/firefox/ contains the WebDriver client, parameterized on
    FIREFOX_BIN / GECKODRIVER with the extension directory as an argument.
  2. script/test-e2e-firefox entrypoint plus a make test-e2e-firefox shim,
    building dist/firefox/ first. Outside make check, same as the Chrome
    harness.
  3. A pinned tests/e2e/firefox/Dockerfile. Pin all three by digest, with
    human-readable versions in comments above each:
    • base node@sha256:d649c27dae7ba0137b3cef5dd75baa422c08dc3d9e3fc0c23dfb172dc3cc6436 (node:22-bookworm-slim)
    • Firefox 153.0.3 linux-x86_64 tarball, sha256:22b312280900bfb174b685ece32c7b3c6d72e7f8e53d6d30f21ac41a8dc500a2
    • geckodriver v0.36.0 linux64, sha256:0bde38707eb0a686a20c6bd50f4adcc7d60d4f73c60eb83ee9e0db8f65823e04
  4. Any uncaught error originating from a moz-extension:// source fails the
    run.
  5. Coverage at minimum: popup loads clean, wallet creation through the real UI,
    Add Token screen opens.
  6. Demonstrated failing and then passing, with captured output in the PR
    body. A green run is not evidence.
  7. README Entrypoints section documents the new target and its container
    requirement.

Implementation requirements

  • Zero npm dependencies. The probe harness is plain Node using global
    fetch and child_process against geckodriver's HTTP API - no Selenium, no
    Playwright, no ws. Keep it that way; it suits this repo's stated
    minimal-dependency ethos and it is genuinely less code than a driver library.
  • Run the container with --shm-size=1g. No --privileged needed; the
    CanCreateUserNamespace() EPERM sandbox warning is cosmetic.
  • Navigate with classic POST /session/{id}/url. BiDi
    browsingContext.navigate refuses moz-extension:// outright.
  • Accept the duplication. The BiDi dead end means Chrome and Firefox cannot
    share a driver layer - Chrome stays on Playwright, Firefox is a separate
    WebDriver client of roughly 200 lines. Do not build a premature abstraction
    over two backends to avoid writing UI steps twice; wait until the Firefox
    suite actually has enough shared steps to justify a shim, and say so in the
    PR if you think it has.
  • Note in the code that -remote-allow-system-access grants the driver full
    chrome privileges. Acceptable for a throwaway CI container, and it should
    never be pointed at anything else.
  • Error capture is poll-based (per step), not event-streamed. That is fine for
    step-boundary assertions but cannot attribute an error to a precise moment
    within a step. Document the limitation rather than implying parity with the
    Chrome harness.
Companion to the Chrome harness in https://git.eeqj.de/sneak/AutistMask/issues/181. Firefox needs its own harness because Chrome cannot verify what https://git.eeqj.de/sneak/AutistMask/issues/153 is about. Feasibility was probed rather than assumed, and the verdict is that this **works today**, with a negative control. Details below are measured. ## What was proven - `geckodriver 0.36.0` driving stock Firefox 153.0.3 headless in a container. - `POST /session/{id}/moz/addon/install` with `{path: "<unpacked dist/firefox dir>", temporary: true}` installs the MV2 build directly. No XPI, no signing, no `xpinstall.signatures.required` fiddling - temporary installs bypass signature checks even on release builds. - The popup uuid is **pinned, not discovered**, by setting the profile pref before launch: ``` "extensions.webextensions.uuids": {"autistmask@sneak.berlin": "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"} ``` This works because `manifest/firefox.json` already declares `browser_specific_settings.gecko.id`, so the popup URL is deterministic. - The full wallet-creation flow runs under Firefox with zero console errors. ## Two traps, both verified the hard way 1. **`-remote-allow-system-access` is mandatory on Firefox 153.** Without it, navigating to `moz-extension://` and executing chrome-context script both fail with `unsupported operation`. Firefox 142 did not need it. Pin the Firefox version or this will rot. 2. **WebDriver BiDi `log.entryAdded` does not work for extension pages.** This is the important one. Verified on FF 142 and 153 with a same-session control: on a plain `http://` page BiDi delivers uncaught errors with stack traces; on the `moz-extension://` popup it delivers **zero** events. Firefox's remote agent excludes extension browsing contexts from BiDi observation. **Any harness built on Playwright-BiDi or Puppeteer-BiDi will silently see nothing and report success.** That is precisely the vacuous-check failure mode this repo has already hit twice. Do not go that route. What works instead is the privileged console service via Marionette's chrome context - `Services.console.getMessageArray()`, filtered to `sourceName.startsWith("moz-extension://")` and non-warning, with `Services.console.reset()` between steps. Same semantics as Playwright's `pageerror` (uncaught only), and it captures background-page and content-script errors for free. ## Demonstrated to fail Against `dist/firefox/` built from current `main`, driving the real UI to the Add Token screen - exit 1: ``` view-after-add-token: view-address <-- screen did not change errors-after-add-token: [{ "msg": "ReferenceError: showView is not defined", "src": "moz-extension://aaaa.../src/popup/index.js", "line": 13, "cat": "content javascript" }] ``` Negative control, same harness and image against a build with the one-line import fix applied - exit 0, `view-add-token` reached, `errors: []`. Different exit codes on identical inputs modulo one line. The harness discriminates, which is the only property that makes it worth having. ## Definition of done 1. `tests/e2e/firefox/` contains the WebDriver client, parameterized on `FIREFOX_BIN` / `GECKODRIVER` with the extension directory as an argument. 2. `script/test-e2e-firefox` entrypoint plus a `make test-e2e-firefox` shim, building `dist/firefox/` first. Outside `make check`, same as the Chrome harness. 3. A pinned `tests/e2e/firefox/Dockerfile`. Pin **all three** by digest, with human-readable versions in comments above each: - base `node@sha256:d649c27dae7ba0137b3cef5dd75baa422c08dc3d9e3fc0c23dfb172dc3cc6436` (`node:22-bookworm-slim`) - Firefox 153.0.3 linux-x86_64 tarball, `sha256:22b312280900bfb174b685ece32c7b3c6d72e7f8e53d6d30f21ac41a8dc500a2` - geckodriver v0.36.0 linux64, `sha256:0bde38707eb0a686a20c6bd50f4adcc7d60d4f73c60eb83ee9e0db8f65823e04` 4. Any uncaught error originating from a `moz-extension://` source fails the run. 5. Coverage at minimum: popup loads clean, wallet creation through the real UI, Add Token screen opens. 6. **Demonstrated failing and then passing**, with captured output in the PR body. A green run is not evidence. 7. README Entrypoints section documents the new target and its container requirement. ## Implementation requirements - **Zero npm dependencies.** The probe harness is plain Node using global `fetch` and `child_process` against geckodriver's HTTP API - no Selenium, no Playwright, no `ws`. Keep it that way; it suits this repo's stated minimal-dependency ethos and it is genuinely less code than a driver library. - Run the container with `--shm-size=1g`. No `--privileged` needed; the `CanCreateUserNamespace() EPERM` sandbox warning is cosmetic. - Navigate with **classic** `POST /session/{id}/url`. BiDi `browsingContext.navigate` refuses `moz-extension://` outright. - Accept the duplication. The BiDi dead end means Chrome and Firefox cannot share a driver layer - Chrome stays on Playwright, Firefox is a separate WebDriver client of roughly 200 lines. Do not build a premature abstraction over two backends to avoid writing UI steps twice; wait until the Firefox suite actually has enough shared steps to justify a shim, and say so in the PR if you think it has. - Note in the code that `-remote-allow-system-access` grants the driver full chrome privileges. Acceptable for a throwaway CI container, and it should never be pointed at anything else. - Error capture is poll-based (per step), not event-streamed. That is fine for step-boundary assertions but cannot attribute an error to a precise moment within a step. Document the limitation rather than implying parity with the Chrome harness.
clawbot added this to the 1.0.0 milestone 2026-08-09 16:27:58 +02:00
clawbot commented 2026-08-12 10:32:35 +02:00
Author
Collaborator
Copy Link

Plan, following the measured facts in the body rather than re-probing them.

Layout: tests/e2e/firefox/driver.js (WebDriver client over fetch +
child_process, no deps), tests/e2e/firefox/run.js (steps), and the pinned
tests/e2e/firefox/Dockerfile. script/test-e2e-firefox builds dist/firefox/,
builds the image from that Dockerfile, and runs it with --shm-size=1g. Not
wired into script/check.

Error capture is the chrome-context console service exactly as recorded —
Services.console.getMessageArray(), sourceName.startsWith("moz-extension://"),
warning flag excluded, Services.console.reset() at each step boundary. No BiDi
anywhere in the harness.

Two things not settled by the body that I will decide by measurement:

  1. Network. The Chrome suite stubs every request from fixtures; that machinery
    is Playwright's and does not port. I intend --network none on the container
    instead — hard offline, nothing to keep in sync — and will check whether the
    resulting failed fetches produce moz-extension:// console errors of their
    own. If they do I will report what they are rather than widening the filter
    to hide them.
  2. Step granularity. Poll-based capture attributes an error to a step, not to a
    moment, so steps stay small enough that the attribution is still useful. That
    limitation gets documented, not papered over.

Scope is the harness. If it surfaces
#153 I will record what it
shows and leave it unfixed.

Plan, following the measured facts in the body rather than re-probing them. Layout: `tests/e2e/firefox/driver.js` (WebDriver client over `fetch` + `child_process`, no deps), `tests/e2e/firefox/run.js` (steps), and the pinned `tests/e2e/firefox/Dockerfile`. `script/test-e2e-firefox` builds `dist/firefox/`, builds the image from that Dockerfile, and runs it with `--shm-size=1g`. Not wired into `script/check`. Error capture is the chrome-context console service exactly as recorded — `Services.console.getMessageArray()`, `sourceName.startsWith("moz-extension://")`, warning flag excluded, `Services.console.reset()` at each step boundary. No BiDi anywhere in the harness. Two things not settled by the body that I will decide by measurement: 1. Network. The Chrome suite stubs every request from fixtures; that machinery is Playwright's and does not port. I intend `--network none` on the container instead — hard offline, nothing to keep in sync — and will check whether the resulting failed fetches produce `moz-extension://` console errors of their own. If they do I will report what they are rather than widening the filter to hide them. 2. Step granularity. Poll-based capture attributes an error to a step, not to a moment, so steps stay small enough that the attribution is still useful. That limitation gets documented, not papered over. Scope is the harness. If it surfaces [#153](https://git.eeqj.de/sneak/AutistMask/issues/153) I will record what it shows and leave it unfixed.
clawbot commented 2026-08-12 10:44:35 +02:00
Author
Collaborator
Copy Link

Built and pushed as
#256; the full record is in
the PR body.

Everything measured in this issue held up exactly as written — both digests
matched the URLs I fetched, -remote-allow-system-access was needed, and the
pinned uuid worked first try. One detail cost time and is worth adding to the
record for whoever writes the next WebDriver client here: the W3C web element
key is element-6066-11e4-a52e-4f735466cecf, ending cecf. Getting the last
character wrong yields an element reference of undefined and a thoroughly
misleading no such element: The element with the reference undefined is not known in the current browsing context from geckodriver. findElement() now
checks for the key and reports the actual response instead.

Verification. make check green (19 suites, 416 tests), re-run after the rebase
onto next. make test-e2e-firefox exits 0 with 3/3.

The discrimination demo is the one recorded here, reproduced against
src/popup/views/addToken.js rather than src/popup/index.js: dropping
showView from its import puts the failure at step 3 instead of step 1, so
steps 1 and 2 still run and the flow is exercised. Output matches what is in
this issue almost verbatim, index.js:13 and content javascript included —
exit 1 with ReferenceError: showView is not defined and current view is view-address, exit 0 with the import restored.

I also checked the "captures background-page errors for free" claim rather than
repeating it: a deliberate throw in src/background/index.js failed step 1,
even though the suite never navigates to the background page.

Two decisions I flagged in the plan, now settled by measurement. Network is
--network none on the container, and the resulting failed fetches produce no
moz-extension:// console errors, so no filter widening was needed. Error
capture stayed poll-based and the limitation is documented in run.js and the
README rather than glossed.

Nothing about
#153 surfaced, which is what
I would expect — its breakage is in the content-script relay, the approval
popups and the background windows/tabs calls, and none of those are on the
three popup-only paths covered here. Whoever takes it will need steps that
actually drive a dApp connection; the harness will not otherwise reach it.

Built and pushed as [#256](https://git.eeqj.de/sneak/AutistMask/pulls/256); the full record is in the PR body. Everything measured in this issue held up exactly as written — both digests matched the URLs I fetched, `-remote-allow-system-access` was needed, and the pinned uuid worked first try. One detail cost time and is worth adding to the record for whoever writes the next WebDriver client here: the W3C web element key is `element-6066-11e4-a52e-4f735466cecf`, ending `cecf`. Getting the last character wrong yields an element reference of `undefined` and a thoroughly misleading `no such element: The element with the reference undefined is not known in the current browsing context` from geckodriver. `findElement()` now checks for the key and reports the actual response instead. Verification. `make check` green (19 suites, 416 tests), re-run after the rebase onto `next`. `make test-e2e-firefox` exits 0 with 3/3. The discrimination demo is the one recorded here, reproduced against `src/popup/views/addToken.js` rather than `src/popup/index.js`: dropping `showView` from its import puts the failure at step 3 instead of step 1, so steps 1 and 2 still run and the flow is exercised. Output matches what is in this issue almost verbatim, `index.js:13` and `content javascript` included — exit 1 with `ReferenceError: showView is not defined` and `current view is view-address`, exit 0 with the import restored. I also checked the "captures background-page errors for free" claim rather than repeating it: a deliberate throw in `src/background/index.js` failed step 1, even though the suite never navigates to the background page. Two decisions I flagged in the plan, now settled by measurement. Network is `--network none` on the container, and the resulting failed fetches produce no `moz-extension://` console errors, so no filter widening was needed. Error capture stayed poll-based and the limitation is documented in `run.js` and the README rather than glossed. Nothing about [#153](https://git.eeqj.de/sneak/AutistMask/issues/153) surfaced, which is what I would expect — its breakage is in the content-script relay, the approval popups and the background `windows`/`tabs` calls, and none of those are on the three popup-only paths covered here. Whoever takes it will need steps that actually drive a dApp connection; the harness will not otherwise reach it.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
merge-ready
needs-checks
needs-rebase
needs-review
needs-rework
No Label
Milestone
No items
No Milestone 1.0.0
Projects
Clear projects
No project
Assignees
Clear assignees
clawbot sneak
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/AutistMask#184
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?