test: containerized Firefox end-to-end harness (closes #184) #256

Merged
clawbot merged 1 commits from test/issue-184-firefox-e2e-harness into next 2026-08-12 12:20:15 +02:00
7 changed files with 993 additions and 16 deletions

View File

@@ -1,4 +1,4 @@
.PHONY: bootstrap setup install test test-e2e lint fmt fmt-check check docker hooks build build-debug verify-build clean dev .PHONY: bootstrap setup install test test-e2e test-e2e-firefox lint fmt fmt-check check docker hooks build build-debug verify-build clean dev
# Standard targets are thin shims; the implementations live in script/ # Standard targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern (see the Entrypoints section # per the scripts-to-rule-them-all pattern (see the Entrypoints section
@@ -16,10 +16,13 @@ install:
test: test:
@script/test @script/test
# Browser end-to-end suite. Requires docker; not part of check. # Browser end-to-end suites. Both require docker; neither is part of check.
test-e2e: test-e2e:
@script/test-e2e @script/test-e2e
test-e2e-firefox:
@script/test-e2e-firefox
lint: lint:
@script/lint @script/lint

103
README.md
View File

@@ -83,7 +83,10 @@ provide:
git pre-commit hook git pre-commit hook
- `script/projectname` — print the project name (used for the Docker image tag) - `script/projectname` — print the project name (used for the Docker image tag)
- `script/test` — run the test suite (jest) - `script/test` — run the test suite (jest)
- `script/test-e2e` — run the browser end-to-end suite (docker required; see - `script/test-e2e` — run the Chrome browser end-to-end suite (docker required;
see [End-to-End Tests](#end-to-end-tests))
- `script/test-e2e-firefox` — run the Firefox browser end-to-end suite (docker
required; builds its own pinned image, see
[End-to-End Tests](#end-to-end-tests)) [End-to-End Tests](#end-to-end-tests))
- `script/lint` — run the linter - `script/lint` — run the linter
- `script/fmt` — format all files (writes) - `script/fmt` — format all files (writes)
@@ -123,6 +126,14 @@ The Makefile shims to those. It also carries a few targets that have no
## End-to-End Tests ## End-to-End Tests
There are two suites, one per browser, and they share no code. Chrome runs on
Playwright; Firefox has its own WebDriver client, because Playwright cannot
observe errors on a Firefox extension page at all — see
[Firefox](#firefox-make-test-e2e-firefox) below. Both require docker, and both
are outside `make check`.
### Chrome (`make test-e2e`)
`make test-e2e` builds `dist/chrome/` and drives the **real popup in a real `make test-e2e` builds `dist/chrome/` and drives the **real popup in a real
Chrome**, loaded as an unpacked MV3 extension inside a pinned Chrome**, loaded as an unpacked MV3 extension inside a pinned
`mcr.microsoft.com/playwright` container (pinned by digest in `script/test-e2e`; `mcr.microsoft.com/playwright` container (pinned by digest in `script/test-e2e`;
@@ -200,12 +211,90 @@ a `ReferenceError` from a used-but-not-imported identifier is invisible to
`make check` (`script/lint` is only `prettier --check`) but fatal in a browser, `make check` (`script/lint` is only `prettier --check`) but fatal in a browser,
and this suite exists because exactly that class of bug shipped twice. and this suite exists because exactly that class of bug shipped twice.
`make test-e2e` is deliberately **not** part of `make check` or `make test`. ### Firefox (`make test-e2e-firefox`)
`REPO_POLICIES.md` caps `make test` at 20 seconds and a browser suite does not
fit; nothing in `tests/e2e/` is named `*.test.js`, so jest cannot pick it up `make test-e2e-firefox` builds `dist/firefox/` and drives the **real popup in a
either. It is also not wired into the Gitea workflow yet — docker-in-docker in real Firefox**, installed as an unpacked MV2 temporary add-on via geckodriver.
CI is a separate question. Run it locally before changing anything under It covers popup load, wallet creation through the UI, and the Add Token screen.
`src/popup/views/`. The suite lives in `tests/e2e/firefox/` and has **no npm dependencies at all**:
it is a small WebDriver client built on global `fetch` and `child_process`
against geckodriver's HTTP API.
Unlike the Chrome suite it builds its own container image rather than pulling a
published one, because no published image carries both a pinned Firefox and a
matching geckodriver. `tests/e2e/firefox/Dockerfile` pins all three external
artifacts by digest — the `node` base image, the Firefox 153.0.3 tarball, and
geckodriver 0.36.0 — and the Firefox version in particular must not float:
`-remote-allow-system-access` is **mandatory** on 153 and was not on 142.
Without that flag, both navigating to `moz-extension://` and running
chrome-context script fail with `unsupported operation`. The flag grants the
driver full chrome privileges over that browser, which is acceptable only
because it is a throwaway container.
The popup's `moz-extension://` uuid is **pinned, not discovered**: the profile
pref `extensions.webextensions.uuids` maps the extension id that
`manifest/firefox.json` already declares to a fixed uuid, so the popup URL is
deterministic. Navigation uses **classic** WebDriver `POST /session/{id}/url`,
because BiDi's `browsingContext.navigate` refuses `moz-extension://` outright.
**Any uncaught error from a `moz-extension://` source fails the run**, including
errors from the background page, which the suite never navigates to: a `throw`
at the top of `src/background/index.js` kills the background page and fails
step 1. Content-script errors should arrive by the same route, but this suite
does not exercise it and does not claim it — with `--network none` there is no
`http://` page for a content script to be injected into. Errors from add-on
install and background startup are folded into step 1 rather than discarded.
Errors are read from the privileged `nsIConsoleService` in Marionette's chrome
context and filtered to non-warning entries whose `sourceName` is the extension
origin. That mechanism is not a stylistic choice. WebDriver BiDi's
`log.entryAdded` delivers **nothing** for extension pages: on a plain `http://`
page it reports uncaught errors with stack traces, and on the `moz-extension://`
popup it reports zero events, because Firefox's remote agent excludes extension
browsing contexts from BiDi observation. Any harness built on Playwright-BiDi or
Puppeteer-BiDi would therefore see nothing and report success, which is exactly
the vacuous check this repo has already shipped twice. Do not migrate this suite
to BiDi.
Two limits are worth knowing, both real differences from the Chrome suite:
- **Error capture is poll-based, not event-streamed.** The console is drained at
each step boundary, so an error is attributed to the step it was drained
after, not to a moment within it. The window that is drained runs from add-on
install to **≈1.5s** after the last step returns — a 500ms settle, a 1000ms
tail sleep and two drain round trips — and then the browser is torn down. That
cut-off is not a hard boundary: with throws scheduled at fixed offsets, three
runs reported everything up to +1.5s and one of the three also reported +1.6s,
so an error landing near it may or may not be seen, and anything well past it
is not. Inside the window there is no race — each drain reads and clears the
console in a single chrome round trip, so an error logged mid-drain lands in
that batch or the next rather than being destroyed unread — but there is a
**capacity limit**: `nsIConsoleService` keeps a ring buffer of 250 messages
and silently evicts the oldest, so more than 250 console messages between two
drains destroys the excess unread. 400 throws inside one step are reported as
exactly the newest 250, three runs running. That buffer is shared with
Firefox's own console noise; a clean run peaks at 4 of 250 at the install
drain and 0 at every later drain, so the three steps here have wide headroom,
but a step that logs heavily could evict unread errors. What poll-based costs
is location, not coverage: an error cannot be placed within a step the way the
Chrome suite's `pageerror` events place it.
- **Nothing is stubbed, which inverts the coverage of network-dependent code.**
There is no fixture layer; the container runs with `--network none` instead,
so the run is offline and deterministic and no request can escape. The
extension swallows its own fetch failures, so the flows are unaffected — but
every network call fails, so only the _failure_ branches of code that depends
on one are ever executed. A `ReferenceError` in the success path of
`renderTransactions`, or of price or balance rendering, passes this suite
green. The offline run is also weaker than the Chrome suite's interception: it
proves nothing got out, but it cannot report which requests were attempted.
Closing that gap needs a fixture layer, deliberately out of scope for this
harness.
Neither `make test-e2e` nor `make test-e2e-firefox` is part of `make check` or
`make test`. `REPO_POLICIES.md` caps `make test` at 20 seconds and a browser
suite does not fit; nothing in `tests/e2e/` is named `*.test.js`, so jest cannot
pick it up either. Neither is wired into the Gitea workflow yet —
docker-in-docker in CI is a separate question. Run them locally before changing
anything under `src/popup/views/`.
## Rationale ## Rationale

33
TODO.md
View File

@@ -30,9 +30,10 @@ compiled off.
The backlog lives on the The backlog lives on the
[Gitea tracker](https://git.eeqj.de/sneak/AutistMask/issues), which is [Gitea tracker](https://git.eeqj.de/sneak/AutistMask/issues), which is
authoritative; this file does not duplicate it. Full policy file set present. A authoritative; this file does not duplicate it. Full policy file set present.
real-browser end-to-end suite (`make test-e2e`) now sits alongside `make check`, Real-browser end-to-end suites (`make test-e2e` for Chrome,
which cannot see a runtime `ReferenceError` in a popup view. `make test-e2e-firefox` for Firefox) now sit alongside `make check`, which
cannot see a runtime `ReferenceError` in a popup view.
# Next Step # Next Step
@@ -44,6 +45,24 @@ undefined identifiers, which is how
# Completed Steps # Completed Steps
- 2026-08-12: A containerized Firefox end-to-end harness
(`make test-e2e-firefox`) drives the real popup in a real Firefox with the MV2
build installed as a temporary add-on. Zero npm dependencies — a WebDriver
client over `fetch` against geckodriver — with `node`, Firefox 153.0.3 and
geckodriver 0.36.0 all pinned by digest. Uncaught errors are read from the
privileged console service in Marionette's chrome context, because BiDi
`log.entryAdded` reports nothing at all for extension pages; each drain reads
and clears the console in one chrome round trip, so no error is destroyed
unread by the drain itself, and errors logged during add-on install and
background startup are folded into step 1 instead of being cleared. The two
measured limits are documented rather than claimed away: the console ring
buffer holds 250 messages (a clean run peaks at 4), and the drained window
ends ≈1.5s after the last step returns. Demonstrated discriminating by exiting
1 on a `throw` at the top of `src/background/index.js`, on a build with one
import removed, on a `setTimeout` throw whose UI assertions all pass, on an
unhandled `Promise.reject` and on an undefined identifier in `home.js`, and 0
on the branch as it stands
([#184](https://git.eeqj.de/sneak/AutistMask/issues/184)).
- 2026-08-12: The transaction a dApp asks for is now populated in the background - 2026-08-12: The transaction a dApp asks for is now populated in the background
before the approval window opens, so the object the user is shown is the before the approval window opens, so the object the user is shown is the
object the signed artifact is verified against — nonce, gas limit and every object the signed artifact is verified against — nonce, gas limit and every
@@ -279,9 +298,9 @@ tracker.
- Pre-1.0 security review of the extension (key handling, DEBUG mode policy, RPC - Pre-1.0 security review of the extension (key handling, DEBUG mode policy, RPC
input validation) before any 1.0rc tag. Individual filed issues are parts of input validation) before any 1.0rc tag. Individual filed issues are parts of
it, but the review is broader than any of them. it, but the review is broader than any of them.
- Decide whether docker-in-docker makes `make test-e2e` runnable in the Gitea - Decide whether docker-in-docker makes `make test-e2e` and
workflow. Extending the suite itself is tracked as `make test-e2e-firefox` runnable in the Gitea workflow. Extending the Chrome
[#183](https://git.eeqj.de/sneak/AutistMask/issues/183) and suite itself is tracked as
[#184](https://git.eeqj.de/sneak/AutistMask/issues/184). [#183](https://git.eeqj.de/sneak/AutistMask/issues/183).
- Cut 1.0.0 once the milestone is empty, then continue tagging as milestones - Cut 1.0.0 once the milestone is empty, then continue tagging as milestones
land. land.

63
script/test-e2e-firefox Executable file
View File

@@ -0,0 +1,63 @@
#!/bin/sh
# script/test-e2e-firefox: build the extension and drive the real popup in
# a real Firefox inside a pinned container. The Firefox counterpart to
# script/test-e2e. Our own extension to scripts-to-rule-them-all.
#
# Deliberately NOT called by script/check or script/test, for the same
# reason as the Chrome suite: REPO_POLICIES.md caps make test at 20 seconds
# and a browser suite does not fit.
#
# Unlike script/test-e2e this builds its image locally, because no
# published image carries both a pinned Firefox and a matching geckodriver.
# All three external artifacts are pinned by digest inside the Dockerfile;
# see tests/e2e/firefox/Dockerfile.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
IMAGE="$("$SCRIPT_DIR/projectname")-e2e-firefox"
main() {
cd "$ROOT"
if ! command -v docker >/dev/null 2>&1; then
echo "test-e2e-firefox: docker is required to run the e2e suite" >&2
exit 1
fi
echo "Building extension for e2e..."
yarn run build 2>&1
# The build context is tests/e2e/firefox/ and holds nothing but the
# Dockerfile: the harness itself arrives over the bind mount below, so
# editing it never invalidates an image layer.
echo "Building the pinned Firefox e2e image..."
docker build -t "$IMAGE" "$ROOT/tests/e2e/firefox"
echo "Running the Firefox e2e suite..."
# --shm-size=1g: Firefox needs more than the default 64MB /dev/shm.
# --network none: the suite stubs nothing, so this is what keeps the
# run offline and deterministic. The extension swallows its own
# fetch failures, so the popup flows work unchanged; see the
# network note in README.md. Weaker than the Chrome suite's
# fixture interception, and honestly so — it proves no request
# escaped, but it cannot report which ones were attempted.
# --user: keep files the suite touches owned by the caller, not root.
# HOME=/tmp: the mapped uid has no home directory in the image.
#
# No --privileged. Firefox's sandbox logs
# "CanCreateUserNamespace() clone() failure: EPERM" on startup here;
# it is cosmetic and headless Firefox runs fine without it.
docker run --rm \
--shm-size=1g \
--network none \
--user "$(id -u):$(id -g)" \
-e HOME=/tmp \
-v "$ROOT:/work" \
-w /work \
"$IMAGE" \
node tests/e2e/firefox/run.js dist/firefox
}
main "$@"

View File

@@ -0,0 +1,51 @@
# Firefox end-to-end image: stock Firefox plus geckodriver on a node base,
# built by script/test-e2e-firefox. The repo is bind-mounted at /work; the
# harness itself has no dependencies, so nothing is installed for it.
#
# All three external artifacts are pinned by digest. The Firefox version in
# particular must not float: -remote-allow-system-access is mandatory on 153
# and was not on 142, so the flag the harness passes is version-coupled.
# node:22-bookworm-slim, 2026-08-12
FROM node@sha256:d649c27dae7ba0137b3cef5dd75baa422c08dc3d9e3fc0c23dfb172dc3cc6436
ENV DEBIAN_FRONTEND=noninteractive
# Firefox's shared-library dependencies on a slim base, plus the two tools
# needed to fetch and unpack the pinned tarballs.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates \
curl \
libasound2 \
libdbus-glib-1-2 \
libgtk-3-0 \
libx11-xcb1 \
libxt6 \
libxtst6 \
xz-utils \
&& rm -rf /var/lib/apt/lists/*
# Firefox 153.0.3, linux-x86_64, en-US
ARG FIREFOX_URL=https://ftp.mozilla.org/pub/firefox/releases/153.0.3/linux-x86_64/en-US/firefox-153.0.3.tar.xz
ARG FIREFOX_SHA256=22b312280900bfb174b685ece32c7b3c6d72e7f8e53d6d30f21ac41a8dc500a2
RUN curl -fsSL -o /tmp/firefox.tar.xz "$FIREFOX_URL" \
&& echo "$FIREFOX_SHA256 /tmp/firefox.tar.xz" | sha256sum -c - \
&& tar -xJf /tmp/firefox.tar.xz -C /opt \
&& rm /tmp/firefox.tar.xz \
&& /opt/firefox/firefox --version
# geckodriver v0.36.0, linux64
ARG GECKODRIVER_URL=https://github.com/mozilla/geckodriver/releases/download/v0.36.0/geckodriver-v0.36.0-linux64.tar.gz
ARG GECKODRIVER_SHA256=0bde38707eb0a686a20c6bd50f4adcc7d60d4f73c60eb83ee9e0db8f65823e04
RUN curl -fsSL -o /tmp/geckodriver.tar.gz "$GECKODRIVER_URL" \
&& echo "$GECKODRIVER_SHA256 /tmp/geckodriver.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/geckodriver.tar.gz -C /usr/local/bin \
&& rm /tmp/geckodriver.tar.gz \
&& geckodriver --version
ENV FIREFOX_BIN=/opt/firefox/firefox
ENV GECKODRIVER=/usr/local/bin/geckodriver
WORKDIR /work
CMD ["node", "tests/e2e/firefox/run.js", "dist/firefox"]

464
tests/e2e/firefox/driver.js Normal file
View File

@@ -0,0 +1,464 @@
// A minimal WebDriver client for geckodriver, plus the privileged console
// reader the error assertions are built on. No npm dependencies: global
// fetch and child_process against geckodriver's HTTP API is less code than
// a driver library and keeps the harness at zero packages.
//
// Run through script/test-e2e-firefox, which builds dist/firefox/ and the
// pinned container around this. FIREFOX_BIN and GECKODRIVER locate the two
// binaries; the image sets both.
"use strict";
const { spawn } = require("child_process");
const net = require("net");
const FIREFOX_BIN = process.env.FIREFOX_BIN || "firefox";
const GECKODRIVER = process.env.GECKODRIVER || "geckodriver";
// The extension id declared in manifest/firefox.json, and the uuid the
// popup is served from. Firefox normally assigns that uuid randomly per
// profile, which would make the popup URL undiscoverable without querying
// privileged state; setting extensions.webextensions.uuids before launch
// pins it instead. This only works because the manifest declares a fixed
// browser_specific_settings.gecko.id — without one the mapping has no key.
const EXTENSION_ID = "autistmask@sneak.berlin";
const EXTENSION_UUID = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee";
const EXTENSION_ORIGIN = "moz-extension://" + EXTENSION_UUID;
// The W3C web element identifier. Getting the last character wrong yields
// an element reference of "undefined" and a bewildering "element with the
// reference undefined is not known" from geckodriver, so findElement()
// below checks for the key rather than indexing blindly.
const WEB_ELEMENT_KEY = "element-6066-11e4-a52e-4f735466cecf";
const SCRIPT_TIMEOUT_MS = 120000;
const DEFAULT_WAIT_MS = 20000;
const POLL_INTERVAL_MS = 100;
function sleep(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
// An ephemeral port picked by the kernel, then handed to geckodriver.
// There is a race between closing this listener and geckodriver binding,
// but this host runs many sessions at once and a fixed 4444 is a
// guaranteed collision rather than a possible one.
function freePort() {
return new Promise((resolve, reject) => {
const srv = net.createServer();
srv.on("error", reject);
srv.listen(0, "127.0.0.1", () => {
const { port } = srv.address();
srv.close(() => resolve(port));
});
});
}
class WebDriverError extends Error {
constructor(command, body) {
const v = (body && body.value) || {};
super(
command +
" failed: " +
(v.error || "unknown error") +
": " +
(v.message || JSON.stringify(body)),
);
this.name = "WebDriverError";
this.error = v.error;
}
}
class Driver {
constructor(proc, base) {
this.proc = proc;
this.base = base;
this.sessionId = null;
this.context = "content";
}
async send(method, path, body) {
const url = this.base + path;
const res = await fetch(url, {
method,
headers: { "Content-Type": "application/json" },
body: body === undefined ? undefined : JSON.stringify(body),
});
const text = await res.text();
let parsed;
try {
parsed = JSON.parse(text);
} catch (_) {
throw new Error(
method + " " + path + ": non-JSON response: " + text,
);
}
if (!res.ok) throw new WebDriverError(method + " " + path, parsed);
return parsed.value;
}
session(method, path, body) {
return this.send(method, "/session/" + this.sessionId + path, body);
}
// ------------------------------------------------------------ setup
async newSession() {
const prefs = {
// See EXTENSION_UUID above. The pref is a string pref whose
// value is itself JSON.
"extensions.webextensions.uuids": JSON.stringify({
[EXTENSION_ID]: EXTENSION_UUID,
}),
};
const value = await this.send("POST", "/session", {
capabilities: {
alwaysMatch: {
browserName: "firefox",
"moz:firefoxOptions": {
binary: FIREFOX_BIN,
args: [
"-headless",
// Mandatory on Firefox 153: without it,
// navigating to moz-extension:// and running
// chrome-context script both fail with
// "unsupported operation".
//
// It grants the driver FULL CHROME PRIVILEGES
// over this browser. Acceptable only because
// the browser is a throwaway in a CI
// container; never point a session with this
// flag at anything you care about.
"-remote-allow-system-access",
],
prefs,
},
},
},
});
this.sessionId = value.sessionId;
await this.session("POST", "/timeouts", { script: SCRIPT_TIMEOUT_MS });
return value;
}
// Installs the unpacked MV2 build straight from a directory.
// temporary:true bypasses signature checks, so no XPI and no signing
// are involved, and the add-on dies with the profile.
async installAddon(dir) {
return this.session("POST", "/moz/addon/install", {
path: dir,
temporary: true,
});
}
// Classic navigation on purpose. BiDi's browsingContext.navigate
// refuses moz-extension:// URLs outright.
async navigate(url) {
await this.session("POST", "/url", { url });
}
async quit() {
if (this.sessionId) {
await this.session("DELETE", "").catch(() => {});
this.sessionId = null;
}
this.proc.kill("SIGTERM");
}
// ---------------------------------------------------------- scripts
async setContext(context) {
if (this.context === context) return;
await this.session("POST", "/moz/context", { context });
this.context = context;
}
async execute(script, args = []) {
await this.setContext("content");
return this.session("POST", "/execute/sync", { script, args });
}
// Runs in the privileged chrome scope, where Services and Ci exist.
async executeChrome(script, args = []) {
await this.setContext("chrome");
try {
return await this.session("POST", "/execute/sync", {
script,
args,
});
} finally {
await this.setContext("content");
}
}
// ------------------------------------------------------- page waits
// Polls a content-context expression until it returns truthy. Every
// wait in the suite goes through here so a timeout always says which
// condition it was waiting on rather than "timed out".
async waitFor(what, script, args = [], timeout = DEFAULT_WAIT_MS) {
const deadline = Date.now() + timeout;
let last = null;
for (;;) {
try {
const v = await this.execute(script, args);
if (v) return v;
last = null;
} catch (e) {
// A navigation or view swap in flight makes execute
// throw; that is a not-yet, not a failure, until the
// deadline says otherwise.
last = e.message;
}
if (Date.now() >= deadline) {
throw new Error(
"timed out after " +
timeout +
"ms waiting for " +
what +
(last ? " (last error: " + last + ")" : ""),
);
}
await sleep(POLL_INTERVAL_MS);
}
}
// Shown means shown: in the popup a view is switched by toggling a
// "hidden" class, and an element that is present but collapsed is not
// the thing a test means by visible.
async waitVisible(selector, timeout = DEFAULT_WAIT_MS) {
return this.waitFor(
"selector " + selector + " to be visible",
`const el = document.querySelector(arguments[0]);
if (!el) return false;
const r = el.getBoundingClientRect();
return r.width > 0 && r.height > 0;`,
[selector],
timeout,
);
}
async isVisible(selector) {
return this.execute(
`const el = document.querySelector(arguments[0]);
if (!el) return false;
const r = el.getBoundingClientRect();
return r.width > 0 && r.height > 0;`,
[selector],
);
}
async count(selector) {
return this.execute(
"return document.querySelectorAll(arguments[0]).length;",
[selector],
);
}
async text(selector) {
return this.execute(
`const el = document.querySelector(arguments[0]);
return el ? el.textContent : null;`,
[selector],
);
}
async title() {
return this.session("GET", "/title");
}
// The id of the view element currently on top, which is what a
// failing step needs to report: "the screen did not change" is only
// useful if it says which screen it stayed on.
async currentView() {
return this.execute(
`const views = document.querySelectorAll('[id^="view-"]');
for (const v of views) {
const r = v.getBoundingClientRect();
if (r.width > 0 && r.height > 0) return v.id;
}
return null;`,
);
}
// ----------------------------------------------------- interactions
async findElement(selector) {
const value = await this.session("POST", "/element", {
using: "css selector",
value: selector,
});
const ref = value && value[WEB_ELEMENT_KEY];
if (typeof ref !== "string") {
throw new Error(
"no " +
WEB_ELEMENT_KEY +
" in the element response for " +
selector +
": " +
JSON.stringify(value),
);
}
return ref;
}
// Real WebDriver clicks and real key events rather than in-page
// .click() and value assignment: the popup's handlers are wired to
// events, and synthesising them from inside the page would test the
// harness's idea of the UI instead of the UI.
async click(selector) {
await this.waitVisible(selector);
const id = await this.findElement(selector);
await this.session("POST", "/element/" + id + "/click", {});
}
async fill(selector, value) {
await this.waitVisible(selector);
const id = await this.findElement(selector);
await this.session("POST", "/element/" + id + "/clear", {});
await this.session("POST", "/element/" + id + "/value", {
text: String(value),
});
}
async value(selector) {
return this.execute(
`const el = document.querySelector(arguments[0]);
return el ? el.value : null;`,
[selector],
);
}
}
// ------------------------------------------------------- error capture
// Uncaught errors from extension code, read out of the privileged console
// service.
//
// This is not the obvious mechanism, and the obvious one does not work:
// WebDriver BiDi's log.entryAdded delivers NOTHING for extension pages.
// Verified on Firefox 142 and 153 against a same-session control — a plain
// http:// page yields uncaught errors with stack traces, the
// moz-extension:// popup yields zero events, because the remote agent
// excludes extension browsing contexts from BiDi observation. A harness
// built on Playwright-BiDi or Puppeteer-BiDi therefore sees nothing and
// reports success. Do not "simplify" this back to BiDi.
//
// nsIConsoleService is not per-page: it also carries errors from the
// background page, which BiDi would not have covered even if it worked.
// Background-page capture is verified by probe — a throw at the top of
// src/background/index.js, which kills the background page outright, fails
// the run. Content-script errors should arrive by the same route, but that
// is UNVERIFIED here and must not be claimed: the container runs with
// --network none, so there is no http:// page for a content script to be
// injected into and this suite never exercises one.
//
// Warnings are excluded so the semantics match Playwright's pageerror:
// uncaught errors only.
//
// The read and the clear are ONE chrome script on purpose. Splitting them
// into two round trips leaves a blind window between them in which an
// error is logged into a buffer that is about to be discarded, and is
// destroyed unread rather than deferred to the next drain. That was not
// theoretical: with a separate reset() call, a probe of 100 sequenced
// throws at 20ms spacing lost one of them outright.
const DRAIN_ERRORS_SCRIPT = `
const origin = arguments[0];
const out = [];
for (const raw of Services.console.getMessageArray() || []) {
let e;
try {
e = raw.QueryInterface(Ci.nsIScriptError);
} catch (_) {
continue;
}
if (e.flags & Ci.nsIScriptError.warningFlag) continue;
const src = e.sourceName || "";
if (!src.startsWith(origin)) continue;
out.push({
msg: e.errorMessage,
src: src,
line: e.lineNumber,
cat: e.category,
});
}
Services.console.reset();
return out;
`;
class ConsoleErrors {
constructor(driver, originPrefix) {
this.driver = driver;
this.originPrefix = originPrefix;
}
// Everything logged since the last take, read and cleared atomically
// in a single chrome round trip. Poll-based, so an error is attributed
// to the step that was running when it was drained, not to the moment
// inside that step at which it happened — see the limitation note in
// run.js. An error that arrives mid-drain is not lost — it makes this
// batch or the next one — but the console service ring buffer holds
// only 250 messages, so more than that between two takes evicts the
// oldest unread. A clean run peaks at 4.
async take() {
const found = await this.driver.executeChrome(DRAIN_ERRORS_SCRIPT, [
this.originPrefix,
]);
return found || [];
}
}
// ------------------------------------------------------------- startup
async function waitForDriverReady(base, timeoutMs) {
const deadline = Date.now() + timeoutMs;
for (;;) {
try {
const res = await fetch(base + "/status");
if (res.ok) {
const body = await res.json();
if (body && body.value && body.value.ready !== false) return;
}
} catch (_) {
// not listening yet
}
if (Date.now() >= deadline) {
throw new Error(
"geckodriver did not become ready within " + timeoutMs + "ms",
);
}
await sleep(POLL_INTERVAL_MS);
}
}
async function start() {
const port = await freePort();
const proc = spawn(
GECKODRIVER,
["--port", String(port), "--host", "127.0.0.1"],
{ stdio: ["ignore", "inherit", "inherit"] },
);
proc.on("error", (e) => {
console.error("geckodriver failed to spawn: " + e.message);
});
const base = "http://127.0.0.1:" + port;
try {
await waitForDriverReady(base, 30000);
} catch (e) {
proc.kill("SIGKILL");
throw e;
}
return new Driver(proc, base);
}
module.exports = {
ConsoleErrors,
Driver,
EXTENSION_ID,
EXTENSION_ORIGIN,
EXTENSION_UUID,
start,
sleep,
};

288
tests/e2e/firefox/run.js Normal file
View File

@@ -0,0 +1,288 @@
// Firefox end-to-end suite: drives the real popup in a real Firefox with
// the unpacked MV2 build installed as a temporary add-on, and fails the run
// on any uncaught error coming from an extension source.
//
// Run via script/test-e2e-firefox, which builds dist/firefox/ and the pinned
// container. The extension directory is the one argument.
//
// node tests/e2e/firefox/run.js [dist/firefox]
//
// Deliberately not part of script/check, and deliberately not named
// *.test.js: REPO_POLICIES.md caps make test at 20 seconds and a browser
// suite does not fit.
//
// This shares no driver layer with the Chrome suite in tests/e2e/, and the
// UI steps below are written twice on purpose. Chrome runs on Playwright,
// which cannot see extension-page errors in Firefox at all (see the BiDi
// note in driver.js), so the two backends have no common substrate to
// abstract over. Three duplicated steps do not pay for a shim; revisit if
// this suite grows to where they do.
//
// LIMITATION, and the difference from the Chrome suite worth knowing: error
// capture here is POLL-BASED, not event-streamed. The console service is
// drained at each step boundary, so an error is attributed to the step it
// was drained after, never to a moment within that step. What is drained
// covers the whole run from add-on install to the last drain below, which
// lands ~1.5s after the last step returns (500ms settle + 1000ms sleep +
// two drain round trips). That cut-off jitters run to run: three runs of
// throws at fixed offsets reported everything to +1.5s and one of them
// also +1.6s, and past it the browser is torn down first. Inside the
// window there is no race — the drain reads and clears in one chrome
// round trip — but there is a capacity limit: nsIConsoleService keeps
// only the newest 250 messages, so 400 throws in one step report as
// exactly 250. A clean run peaks at 4 of 250, so that is headroom today
// and not a guarantee for a step that logs heavily. The Chrome harness
// receives pageerror events as they happen and can say more. Do not read
// a green Firefox run as the same claim.
"use strict";
const fs = require("fs");
const path = require("path");
const { ConsoleErrors, EXTENSION_ORIGIN, start, sleep } = require("./driver");
const REPO_ROOT = path.resolve(__dirname, "..", "..", "..");
const POPUP_URL = EXTENSION_ORIGIN + "/src/popup/index.html";
const PASSWORD = "e2e-harness-password";
// Firefox installs the add-on and starts its background page asynchronously
// after the install call returns. Nothing observable marks the end of that,
// so the popup's own first render is the signal we wait on instead.
const STEP_TIMEOUT_MS = 120000;
const steps = [];
function step(name, fn) {
steps.push({ name, fn });
}
function assert(cond, message) {
if (!cond) throw new Error(message);
}
function withTimeout(promise, name) {
let timer;
const timeout = new Promise((_, reject) => {
timer = setTimeout(
() =>
reject(
new Error(
name + " timed out after " + STEP_TIMEOUT_MS + "ms",
),
),
STEP_TIMEOUT_MS,
);
});
return Promise.race([promise, timeout]).finally(() => clearTimeout(timer));
}
// ------------------------------------------------------------- steps
step("popup loads and reaches the welcome view", async (env) => {
const d = env.driver;
await d.navigate(POPUP_URL);
await d.waitVisible("#view-welcome", STEP_TIMEOUT_MS);
const title = await d.title();
assert(title === "AutistMask", "unexpected popup title: " + title);
});
step("wallet creation through the UI reaches the main view", async (env) => {
const d = env.driver;
await d.click("#btn-welcome-add");
await d.waitVisible("#view-add-wallet");
await d.click("#btn-generate-phrase");
await d.waitFor(
"a generated recovery phrase of at least 12 words",
`const el = document.getElementById("wallet-mnemonic");
return !!el && el.value.trim().split(/\\s+/).length >= 12;`,
);
env.phrase = (await d.value("#wallet-mnemonic")).trim();
await d.fill("#add-wallet-password", PASSWORD);
await d.fill("#add-wallet-password-confirm", PASSWORD);
await d.click("#btn-add-wallet-confirm");
// Argon2id under libsodium, for real, so this is the slow one.
await d.waitVisible("#view-main", STEP_TIMEOUT_MS);
assert(
env.phrase.split(/\s+/).length >= 12,
"wallet creation did not yield a recovery phrase",
);
const addrs = await d.count("#wallet-list .btn-addr-info");
assert(addrs > 0, "no addresses rendered in the wallet list");
});
step("add token screen opens from address detail", async (env) => {
const d = env.driver;
if (!(await d.isVisible("#view-address"))) {
await d.waitVisible("#view-main");
await d.click("#wallet-list .btn-addr-info");
}
await d.waitVisible("#view-address");
await d.click("#btn-add-token");
// Reported with the view it actually stayed on: a screen that does
// not change is the symptom a missing import produces, and naming
// the screen is what makes that diagnosable.
try {
await d.waitVisible("#view-add-token");
} catch (e) {
throw new Error(
e.message + "; current view is " + (await d.currentView()),
);
}
const picks = await d.count("#common-token-list .common-token");
assert(picks > 0, "no common-token quick-pick buttons rendered");
});
// ------------------------------------------------------------- runner
function formatError(e) {
return (
e.msg + " (" + e.src + ":" + e.line + (e.cat ? ", " + e.cat : "") + ")"
);
}
async function main() {
// A suite that runs nothing must never report success.
if (steps.length === 0) {
console.log("1..0");
console.log("# FAILED: the Firefox e2e suite registered no steps");
process.exitCode = 1;
return;
}
const extDir = path.resolve(REPO_ROOT, process.argv[2] || "dist/firefox");
if (!fs.existsSync(path.join(extDir, "manifest.json"))) {
console.error(
"e2e-firefox: no unpacked build at " +
extDir +
" — run make build first",
);
process.exitCode = 1;
return;
}
let driver;
try {
driver = await start();
await driver.newSession();
await driver.installAddon(extDir);
} catch (e) {
// A browser we cannot start is a failure of the suite, not an
// absent suite. Never skip and report success.
console.error("e2e-firefox: cannot run the suite: " + e.message);
if (driver) await driver.quit().catch(() => {});
process.exitCode = 1;
return;
}
const errors = new ConsoleErrors(driver, EXTENSION_ORIGIN);
const env = { driver, phrase: null };
console.log("# extension origin: " + EXTENSION_ORIGIN);
console.log("1.." + steps.length);
let failed = 0;
let n = 0;
try {
// Drain, never reset: anything the add-on logged while installing
// and starting its background page has no earlier step to belong
// to, so it is folded into step 1 below. Services.console.reset()
// here would DELETE it instead, and a background page that throws
// at the top of the file — a dead background page — would then
// produce a fully green run.
let installErrors = [];
let installFailure = null;
try {
installErrors = await errors.take();
} catch (e) {
installFailure =
"could not read the console after install: " + e.message;
}
for (const s of steps) {
n += 1;
let failure = null;
try {
await withTimeout(s.fn(env), s.name);
} catch (e) {
failure = e.message;
}
// Let anything the step provoked reach the console service
// before draining it. Without this a failure logged on the
// way out of the step lands in the next step's drain, which
// still fails the run but blames the wrong step.
await sleep(500);
let found = [];
try {
found = await errors.take();
} catch (e) {
failure = failure || "could not read the console: " + e.message;
}
if (n === 1) {
found = installErrors.concat(found);
installErrors = [];
failure = failure || installFailure;
installFailure = null;
}
// Any uncaught error from an extension source fails the step
// that provoked it, whether or not its assertions passed.
if (!failure && found.length > 0) {
failure =
n === 1
? "uncaught extension errors during add-on install, " +
"background startup or this step"
: "uncaught extension errors during this step";
}
if (failure) {
failed += 1;
console.log("not ok " + n + " - " + s.name);
console.log(" " + failure);
for (const e of found) console.log(" " + formatError(e));
} else {
console.log("ok " + n + " - " + s.name);
}
}
// The tail: errors logged after the last step returned cannot be
// blamed on any one step, but they are still reported and they
// still fail the run.
await sleep(1000);
const trailing = await errors.take();
console.log(
"# " +
(steps.length - failed) +
"/" +
steps.length +
" steps passed",
);
if (trailing.length > 0) {
console.log(
"# " +
trailing.length +
" extension error(s) recorded after the last step, not " +
"attributable to any single step:",
);
for (const e of trailing) console.log("# " + formatError(e));
}
if (failed > 0 || trailing.length > 0) {
console.log("# FAILED");
process.exitCode = 1;
}
} finally {
await driver.quit().catch(() => {});
}
}
main().catch((e) => {
console.error("e2e-firefox: " + (e && e.stack ? e.stack : e));
process.exitCode = 1;
});