build: run the browser e2e suites in CI (closes #259) #291

clawbot · 2026-08-14T06:26:48+02:00

clawbot commented

2026-08-14 06:26:48 +02:00

Closes #259.

Nothing automatic ran either e2e suite, so every browser-level guarantee in this wallet — WebAssembly under the shipped CSP, the recovery-phrase and private-key DOM wipes, the ConfirmTx spend gate, the dApp approval round trips — held only when a human or an agent remembered to run it by hand.

.gitea/workflows/e2e.yml adds two jobs, e2e-chrome and e2e-firefox, one per browser so a Chrome failure cannot hide the Firefox result. They are separate from the check workflow, make check is byte-for-byte unchanged, and no e2e ran anywhere near script/check.

Can the runner actually run them? Yes — measured, not assumed

The abandoned WIP branch build/issue-259-e2e-in-ci left a probe workflow and one green e2e.yml run behind. clawbot gets 403 user should be the owner of the repo on the Actions API, but job logs are readable unauthenticated at /sneak/AutistMask/actions/runs/<id>/jobs/<n>/logs, which is how the below was read.

Probe run 615 established the shape of the runner:

The job body runs as root inside a container (docker.gitea.com/runner-images:ubuntu-latest) with the host's /run/docker.sock mounted. Nested containers are siblings on the host daemon; Docker-in-Docker is neither needed nor available.
The checkout is a docker named volume mounted at /workspace/sneak/AutistMask, not a host path. The probe ran docker run --rm -v "$PWD:/work" -w /work alpine:3 ls -la /work and got:
```
total 30
drwxr-xr-x    2 root     root          4096 Aug 12 11:35 .
drwxr-xr-x   20 root     root            21 Aug 12 11:35 ..
```
An empty directory. The host daemon resolves the source path against the host, where nothing is. The mount does not error — it silently succeeds and mounts nothing, which is exactly the failure that produces a vacuous green.
The runner image's node is too old to install this repo's dependencies.

Both suites bind-mounted the repo, so neither could work there unchanged. So: the repo is delivered to the daemon as a build context and the extension is built inside the image, which leaves docker as the only prerequisite on a runner or a laptop. Run 619 (the bind-mount attempt) failed in 22s; run 622, with the build-context approach, passed both jobs — e2e-chrome in 1m54s reporting 37/37 tests passed and e2e-firefox in 1m03s reporting 3/3 steps passed.

Nothing in either suite was weakened, reshaped or reordered to make this work. The only change is how the repo reaches the container.

What was salvaged, and what was not

Salvaged from the WIP branch, re-derived as one commit on current next: the two-job workflow, the new tests/e2e/Dockerfile, the Firefox Dockerfile's move to a repo-root build context, and both script rewrites. Everything external stays digest-pinned — the Playwright image, the node base, the Firefox tarball, geckodriver, and actions/checkout by commit — and the Firefox/geckodriver fetch layers stay above the repo copy, so editing a source file re-runs only the two cheap layers.

Both scripts now build with --iidfile and run the image by ID, not by tag: on a host where two clones run a suite at once, the tag can move between build and run, and the suite would then silently test the other checkout.

Discarded: the probe.yml workflow, a README bullet for a script/e2e-container that was never written, and a README paragraph describing the jobs as script/bootstrap plus a make target, which is not what the workflow runs.

The jobs are load-bearing — mutation demonstrated

One assertion was broken in each suite's runner on a scratch branch (assert(title === "AutistMask") changed to "MUTANT", in tests/e2e/run.js and tests/e2e/firefox/run.js) — deliberately in the runners rather than in product code, so check stays green and any red is unambiguously the new jobs.

Locally, both scripts fail and exit non-zero:

not ok 1 - popup loads and reaches the welcome view
# 36/37 tests passed
# FAILED
chrome exit: 1

not ok 1 - popup loads and reaches the welcome view
ok 2 - wallet creation through the UI reaches the main view
ok 3 - add token screen opens from address detail
# 2/3 steps passed
# FAILED
firefox exit: 1

And in CI, on the scratch branch tmp/issue-259-mutation-demo (commit 3075c23), both jobs went red — actions run 630, failure, 3m49s:

# e2e-chrome
1..37
not ok 1 - popup loads and reaches the welcome view
  unexpected popup title: AutistMask
ok 2 - the popup compiles WebAssembly under the shipped CSP (#182)
...
# 36/37 tests passed
# FAILED
Job 'e2e-chrome' failed

# e2e-firefox
# extension origin: moz-extension://aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
1..3
not ok 1 - popup loads and reaches the welcome view
  unexpected popup title: AutistMask
ok 2 - wallet creation through the UI reaches the main view
ok 3 - add token screen opens from address detail
# 2/3 steps passed
# FAILED
Job 'e2e-firefox' failed

check on that same commit — run 629 — reported success in 1m7s. Same commit, check green, both e2e jobs red: the new jobs see something check structurally cannot, and they are the only thing that went red. The scratch branch is deleted.

Disclosure: the Chrome suite is flaky under load

Wiring this up measured something that has to be said rather than absorbed. script/test-e2e was run six times against unmutated code on this loaded machine: two of six failed, identically, in the dApp section.

not ok 31 - personal_sign signs, and the signature recovers to the address (#183)
  page.waitForSelector: Target page, context or browser has been closed
Call log:
  - waiting for locator('#view-approve-sign') to be visible

The follow-on not ok 37 is not a second flake — it is the anti-vacuity guard correctly reporting two approvals where three were signed.

Filed as #287 and deliberately not papered over here: no retry wrapper, no lengthened timeout, no weakened assertion. A suite that reruns until green stops being evidence. README.md and the workflow both say so where a reader meets them.

Gate or report

Report, not gate. Whether a check blocks a merge is Gitea branch protection, which this repo does not configure, so the merge gate remains the independent reviewer. The flake above is the substantive reason too: a required check that fails one run in three teaches people to merge past red. Making e2e-chrome required is the intended end state and is blocked on #287.

Nothing can pass vacuously: no continue-on-error, no || true, and both scripts exit non-zero when docker is missing, when the image build fails, and when the browser fails to start. The Chrome harness aborts the whole suite if its network interception is not in effect.

Cost

Per push, on top of check: about 3m50s of runner time for the two jobs together, which is what run 630 took end to end today with the two jobs serialized. Broken down from the green run 622: about 1m55s for e2e-chrome — most of it the one-time pull of the ~800MB Playwright layer — and about 1m05s for e2e-firefox, both dropping well under a minute once their layers are cached on the runner. check itself is unaffected and still lands in under a minute.

Verification

Everything below ran through make targets and script/ entrypoints only; no direct node, npx, eslint, prettier, jest, npm or yarn invocation.

make check — green on the rebased head: Test Suites: 29 passed, 29 total, Tests: 703 passed, 703 total, test-verify-build: 18 case(s) passed, All matched files use Prettier code style!, exit 0, about 22s wall. Unchanged in content and runtime; the e2e suites are in neither script/check nor script/test.
make fmt — run, and its result is in the commit.
script/test-e2e — 37/37, about 1m39s including a cold image build (plus the flakes disclosed above, 2 of 6 runs).
script/test-e2e-firefox — 3/3, about 1m12s including a cold image build.
Rebased onto current next (0be20d7) immediately before pushing, and make check re-run after the rebase, not before.
Not yet observed: this PR head's own e2e.yml run (636) was still queued behind a runner backlog when this was written, so the CI evidence above is run 630 on the mutation branch and run 622 on the WIP branch — the same two jobs, the same scripts. A comment will follow when 636 executes.

The stale build/issue-259-e2e-in-ci branch is deleted, so it cannot be picked up again, as is the mutation scratch branch.

Also found and filed, not fixed here: #285 — README.md still documents the EIP-1193 code loss that #274 fixed.

Closes https://git.eeqj.de/sneak/AutistMask/issues/259. Nothing automatic ran either e2e suite, so every browser-level guarantee in this wallet — WebAssembly under the shipped CSP, the recovery-phrase and private-key DOM wipes, the ConfirmTx spend gate, the dApp approval round trips — held only when a human or an agent remembered to run it by hand. `.gitea/workflows/e2e.yml` adds two jobs, `e2e-chrome` and `e2e-firefox`, one per browser so a Chrome failure cannot hide the Firefox result. They are separate from the `check` workflow, `make check` is byte-for-byte unchanged, and no e2e ran anywhere near `script/check`. ## Can the runner actually run them? Yes — measured, not assumed The abandoned WIP branch `build/issue-259-e2e-in-ci` left a probe workflow and one green `e2e.yml` run behind. `clawbot` gets `403 user should be the owner of the repo` on the Actions API, but job logs are readable unauthenticated at `/sneak/AutistMask/actions/runs/<id>/jobs/<n>/logs`, which is how the below was read. Probe run 615 established the shape of the runner: - The job body runs **as root inside a container** (`docker.gitea.com/runner-images:ubuntu-latest`) with the **host's** `/run/docker.sock` mounted. Nested containers are siblings on the host daemon; Docker-in-Docker is neither needed nor available. - The checkout is a **docker named volume** mounted at `/workspace/sneak/AutistMask`, not a host path. The probe ran `docker run --rm -v "$PWD:/work" -w /work alpine:3 ls -la /work` and got: total 30 drwxr-xr-x 2 root root 4096 Aug 12 11:35 . drwxr-xr-x 20 root root 21 Aug 12 11:35 .. An **empty directory**. The host daemon resolves the source path against the host, where nothing is. The mount does not error — it silently succeeds and mounts nothing, which is exactly the failure that produces a vacuous green. - The runner image's node is too old to install this repo's dependencies. Both suites bind-mounted the repo, so neither could work there unchanged. So: the repo is delivered to the daemon as a **build context** and the extension is built **inside the image**, which leaves docker as the only prerequisite on a runner or a laptop. Run 619 (the bind-mount attempt) failed in 22s; run 622, with the build-context approach, passed both jobs — `e2e-chrome` in 1m54s reporting `37/37 tests passed` and `e2e-firefox` in 1m03s reporting `3/3 steps passed`. Nothing in either suite was weakened, reshaped or reordered to make this work. The only change is how the repo reaches the container. ## What was salvaged, and what was not Salvaged from the WIP branch, re-derived as one commit on current `next`: the two-job workflow, the new `tests/e2e/Dockerfile`, the Firefox Dockerfile's move to a repo-root build context, and both script rewrites. Everything external stays digest-pinned — the Playwright image, the node base, the Firefox tarball, geckodriver, and `actions/checkout` by commit — and the Firefox/geckodriver fetch layers stay above the repo copy, so editing a source file re-runs only the two cheap layers. Both scripts now build with `--iidfile` and run the image **by ID**, not by tag: on a host where two clones run a suite at once, the tag can move between build and run, and the suite would then silently test the other checkout. Discarded: the `probe.yml` workflow, a README bullet for a `script/e2e-container` that was never written, and a README paragraph describing the jobs as `script/bootstrap` plus a `make` target, which is not what the workflow runs. ## The jobs are load-bearing — mutation demonstrated One assertion was broken in each suite's runner on a scratch branch (`assert(title === "AutistMask")` changed to `"MUTANT"`, in `tests/e2e/run.js` and `tests/e2e/firefox/run.js`) — deliberately in the runners rather than in product code, so `check` stays green and any red is unambiguously the new jobs. Locally, both scripts fail and exit non-zero: not ok 1 - popup loads and reaches the welcome view # 36/37 tests passed # FAILED chrome exit: 1 not ok 1 - popup loads and reaches the welcome view ok 2 - wallet creation through the UI reaches the main view ok 3 - add token screen opens from address detail # 2/3 steps passed # FAILED firefox exit: 1 And in CI, on the scratch branch `tmp/issue-259-mutation-demo` (commit `3075c23`), **both jobs went red** — actions run 630, `failure`, 3m49s: # e2e-chrome 1..37 not ok 1 - popup loads and reaches the welcome view unexpected popup title: AutistMask ok 2 - the popup compiles WebAssembly under the shipped CSP (#182) ... # 36/37 tests passed # FAILED Job 'e2e-chrome' failed # e2e-firefox # extension origin: moz-extension://aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee 1..3 not ok 1 - popup loads and reaches the welcome view unexpected popup title: AutistMask ok 2 - wallet creation through the UI reaches the main view ok 3 - add token screen opens from address detail # 2/3 steps passed # FAILED Job 'e2e-firefox' failed `check` on that same commit — run 629 — reported **success** in 1m7s. Same commit, `check` green, both e2e jobs red: the new jobs see something `check` structurally cannot, and they are the only thing that went red. The scratch branch is deleted. ## Disclosure: the Chrome suite is flaky under load Wiring this up measured something that has to be said rather than absorbed. `script/test-e2e` was run six times against unmutated code on this loaded machine: **two of six failed**, identically, in the dApp section. not ok 31 - personal_sign signs, and the signature recovers to the address (#183) page.waitForSelector: Target page, context or browser has been closed Call log: - waiting for locator('#view-approve-sign') to be visible The follow-on `not ok 37` is not a second flake — it is the anti-vacuity guard correctly reporting two approvals where three were signed. Filed as https://git.eeqj.de/sneak/AutistMask/issues/287 and deliberately **not** papered over here: no retry wrapper, no lengthened timeout, no weakened assertion. A suite that reruns until green stops being evidence. `README.md` and the workflow both say so where a reader meets them. ## Gate or report **Report, not gate.** Whether a check blocks a merge is Gitea branch protection, which this repo does not configure, so the merge gate remains the independent reviewer. The flake above is the substantive reason too: a required check that fails one run in three teaches people to merge past red. Making `e2e-chrome` required is the intended end state and is blocked on https://git.eeqj.de/sneak/AutistMask/issues/287. Nothing can pass vacuously: no `continue-on-error`, no `|| true`, and both scripts exit non-zero when docker is missing, when the image build fails, and when the browser fails to start. The Chrome harness aborts the whole suite if its network interception is not in effect. ## Cost Per push, on top of `check`: **about 3m50s of runner time for the two jobs together**, which is what run 630 took end to end today with the two jobs serialized. Broken down from the green run 622: about **1m55s** for `e2e-chrome` — most of it the one-time pull of the ~800MB Playwright layer — and about **1m05s** for `e2e-firefox`, both dropping well under a minute once their layers are cached on the runner. `check` itself is unaffected and still lands in under a minute. ## Verification Everything below ran through `make` targets and `script/` entrypoints only; no direct `node`, `npx`, `eslint`, `prettier`, `jest`, `npm` or `yarn` invocation. - `make check` — **green** on the rebased head: `Test Suites: 29 passed, 29 total`, `Tests: 703 passed, 703 total`, `test-verify-build: 18 case(s) passed`, `All matched files use Prettier code style!`, exit 0, about 22s wall. Unchanged in content and runtime; the e2e suites are in neither `script/check` nor `script/test`. - `make fmt` — run, and its result is in the commit. - `script/test-e2e` — **37/37**, about 1m39s including a cold image build (plus the flakes disclosed above, 2 of 6 runs). - `script/test-e2e-firefox` — **3/3**, about 1m12s including a cold image build. - Rebased onto current `next` (`0be20d7`) immediately before pushing, and `make check` re-run **after** the rebase, not before. - Not yet observed: this PR head's own `e2e.yml` run (636) was still queued behind a runner backlog when this was written, so the CI evidence above is run 630 on the mutation branch and run 622 on the WIP branch — the same two jobs, the same scripts. A comment will follow when 636 executes. The stale `build/issue-259-e2e-in-ci` branch is deleted, so it cannot be picked up again, as is the mutation scratch branch. Also found and filed, not fixed here: https://git.eeqj.de/sneak/AutistMask/issues/285 — `README.md` still documents the EIP-1193 code loss that https://git.eeqj.de/sneak/AutistMask/issues/274 fixed.

clawbot added the needs-review label 2026-08-14 06:26:48 +02:00

clawbot added 1 commit 2026-08-14 06:26:49 +02:00

build: run the browser e2e suites in CI (closes #259 )

check / check (push) Successful in 33s

Details

e2e / e2e-chrome (push) Successful in 51s

Details

e2e / e2e-firefox (push) Successful in 20s

Details

329f3e1558

Nothing automatic ran either e2e suite, so every browser-level guarantee in this
wallet -- WebAssembly under the shipped CSP, the recovery-phrase and private-key
DOM wipes, the ConfirmTx spend gate, the dApp approval round trips -- held only
when a human or an agent remembered to run it by hand.

.gitea/workflows/e2e.yml adds two jobs, e2e-chrome and e2e-firefox, one per
browser so a Chrome failure cannot hide the Firefox result. They are separate
from the check workflow: REPO_POLICIES.md caps make test at 20 seconds and
script/cibuild is a docker build whose Dockerfile runs make check, so neither the
cap nor the local fast path is touched. make check is byte-for-byte unchanged.

Neither suite could run on the runner as it stood, and the reason is not
docker-in-docker. The runner executes a job inside a container against the HOST's
docker daemon, and the job's checkout lives on a docker volume rather than a host
path, so `docker run -v "$PWD:/work"` is resolved by the host, silently succeeds
and mounts an empty directory -- measured on this runner. The runner image's node
is also too old to install this repo's dependencies. Both suites therefore ship
the repo to the daemon as a build context and build the extension inside the
pinned image, which leaves docker as the only prerequisite on a runner or a
laptop. The suites themselves are unchanged; only how the repo reaches the
container is.

Both scripts now build with --iidfile and run the image by ID rather than by tag,
so two clones running a suite at once on the same host cannot swap it under each
other.

The jobs report, they do not gate. Whether a check blocks a merge is Gitea branch
protection, which this repo does not configure, so a failure is a red mark a
reviewer must account for. Nothing can pass vacuously: no continue-on-error, no
`|| true`, and both scripts exit non-zero when docker is missing, when the image
build fails and when the browser fails to start.

Wiring this up measured something that has to be said rather than absorbed: the
Chrome suite is flaky under load. Two of six runs of unmutated code on a loaded
machine lost the approval popup out from under the dApp signing wait. It is
filed as #287 and not papered over here -- no retry wrapper, no longer timeout,
no weakened assertion -- and it is the reason e2e-chrome cannot become a
required check yet. README and the workflow say so where a reader meets them.

clawbot self-assigned this 2026-08-14 06:26:53 +02:00

check / check (push) Successful in 33s

Details

e2e / e2e-chrome (push) Successful in 51s

Details

e2e / e2e-firefox (push) Successful in 20s

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin issue-259-e2e-in-ci:issue-259-e2e-in-ci

git checkout issue-259-e2e-in-ci

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/AutistMask#291