build: run the browser e2e suites in CI (closes #259) #291
Reference in New Issue
Block a user
Delete Branch "issue-259-e2e-in-ci"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #259.
Nothing automatic ran either e2e suite, so every browser-level guarantee in this wallet — WebAssembly under the shipped CSP, the recovery-phrase and private-key DOM wipes, the ConfirmTx spend gate, the dApp approval round trips — held only when a human or an agent remembered to run it by hand.
.gitea/workflows/e2e.ymladds two jobs,e2e-chromeande2e-firefox, one per browser so a Chrome failure cannot hide the Firefox result. They are separate from thecheckworkflow,make checkis byte-for-byte unchanged, and no e2e ran anywhere nearscript/check.Can the runner actually run them? Yes — measured, not assumed
The abandoned WIP branch
build/issue-259-e2e-in-cileft a probe workflow and one greene2e.ymlrun behind.clawbotgets403 user should be the owner of the repoon the Actions API, but job logs are readable unauthenticated at/sneak/AutistMask/actions/runs/<id>/jobs/<n>/logs, which is how the below was read.Probe run 615 established the shape of the runner:
The job body runs as root inside a container (
docker.gitea.com/runner-images:ubuntu-latest) with the host's/run/docker.sockmounted. Nested containers are siblings on the host daemon; Docker-in-Docker is neither needed nor available.The checkout is a docker named volume mounted at
/workspace/sneak/AutistMask, not a host path. The probe randocker run --rm -v "$PWD:/work" -w /work alpine:3 ls -la /workand got:An empty directory. The host daemon resolves the source path against the host, where nothing is. The mount does not error — it silently succeeds and mounts nothing, which is exactly the failure that produces a vacuous green.
The runner image's node is too old to install this repo's dependencies.
Both suites bind-mounted the repo, so neither could work there unchanged. So: the repo is delivered to the daemon as a build context and the extension is built inside the image, which leaves docker as the only prerequisite on a runner or a laptop. Run 619 (the bind-mount attempt) failed in 22s; run 622, with the build-context approach, passed both jobs —
e2e-chromein 1m54s reporting37/37 tests passedande2e-firefoxin 1m03s reporting3/3 steps passed.Nothing in either suite was weakened, reshaped or reordered to make this work. The only change is how the repo reaches the container.
What was salvaged, and what was not
Salvaged from the WIP branch, re-derived as one commit on current
next: the two-job workflow, the newtests/e2e/Dockerfile, the Firefox Dockerfile's move to a repo-root build context, and both script rewrites. Everything external stays digest-pinned — the Playwright image, the node base, the Firefox tarball, geckodriver, andactions/checkoutby commit — and the Firefox/geckodriver fetch layers stay above the repo copy, so editing a source file re-runs only the two cheap layers.Both scripts now build with
--iidfileand run the image by ID, not by tag: on a host where two clones run a suite at once, the tag can move between build and run, and the suite would then silently test the other checkout.Discarded: the
probe.ymlworkflow, a README bullet for ascript/e2e-containerthat was never written, and a README paragraph describing the jobs asscript/bootstrapplus amaketarget, which is not what the workflow runs.The jobs are load-bearing — mutation demonstrated
One assertion was broken in each suite's runner on a scratch branch (
assert(title === "AutistMask")changed to"MUTANT", intests/e2e/run.jsandtests/e2e/firefox/run.js) — deliberately in the runners rather than in product code, socheckstays green and any red is unambiguously the new jobs.Locally, both scripts fail and exit non-zero:
And in CI, on the scratch branch
tmp/issue-259-mutation-demo(commit3075c23), both jobs went red — actions run 630,failure, 3m49s:checkon that same commit — run 629 — reported success in 1m7s. Same commit,checkgreen, both e2e jobs red: the new jobs see somethingcheckstructurally cannot, and they are the only thing that went red. The scratch branch is deleted.Disclosure: the Chrome suite is flaky under load
Wiring this up measured something that has to be said rather than absorbed.
script/test-e2ewas run six times against unmutated code on this loaded machine: two of six failed, identically, in the dApp section.The follow-on
not ok 37is not a second flake — it is the anti-vacuity guard correctly reporting two approvals where three were signed.Filed as #287 and deliberately not papered over here: no retry wrapper, no lengthened timeout, no weakened assertion. A suite that reruns until green stops being evidence.
README.mdand the workflow both say so where a reader meets them.Gate or report
Report, not gate. Whether a check blocks a merge is Gitea branch protection, which this repo does not configure, so the merge gate remains the independent reviewer. The flake above is the substantive reason too: a required check that fails one run in three teaches people to merge past red. Making
e2e-chromerequired is the intended end state and is blocked on #287.Nothing can pass vacuously: no
continue-on-error, no|| true, and both scripts exit non-zero when docker is missing, when the image build fails, and when the browser fails to start. The Chrome harness aborts the whole suite if its network interception is not in effect.Cost
Per push, on top of
check: about 3m50s of runner time for the two jobs together, which is what run 630 took end to end today with the two jobs serialized. Broken down from the green run 622: about 1m55s fore2e-chrome— most of it the one-time pull of the ~800MB Playwright layer — and about 1m05s fore2e-firefox, both dropping well under a minute once their layers are cached on the runner.checkitself is unaffected and still lands in under a minute.Verification
Everything below ran through
maketargets andscript/entrypoints only; no directnode,npx,eslint,prettier,jest,npmoryarninvocation.make check— green on the rebased head:Test Suites: 29 passed, 29 total,Tests: 703 passed, 703 total,test-verify-build: 18 case(s) passed,All matched files use Prettier code style!, exit 0, about 22s wall. Unchanged in content and runtime; the e2e suites are in neitherscript/checknorscript/test.make fmt— run, and its result is in the commit.script/test-e2e— 37/37, about 1m39s including a cold image build (plus the flakes disclosed above, 2 of 6 runs).script/test-e2e-firefox— 3/3, about 1m12s including a cold image build.next(0be20d7) immediately before pushing, andmake checkre-run after the rebase, not before.e2e.ymlrun (636) was still queued behind a runner backlog when this was written, so the CI evidence above is run 630 on the mutation branch and run 622 on the WIP branch — the same two jobs, the same scripts. A comment will follow when 636 executes.The stale
build/issue-259-e2e-in-cibranch is deleted, so it cannot be picked up again, as is the mutation scratch branch.Also found and filed, not fixed here: #285 —
README.mdstill documents the EIP-1193 code loss that #274 fixed.Review of #291 — needs-rework, one defect.
README.md:253-255— the PR makes this sentence false. It reads "Unlike the Chrome suite it builds its own container image rather than pulling a published one". After this changescript/test-e2eno longer pulls-and-runs a published image; it runsdocker build -f tests/e2e/Dockerfile .and builds its own image too. The same sentence inscript/test-e2e-firefox:9was correctly updated to "builds its base image locally"; the README copy was not. This matters because it is the one paragraph a reader consults to learn how the Chrome image is obtained, and it tells them the pre-PR mechanism — the bind-mount-a-published-image model this PR removed precisely because it silently mounts an empty directory under Actions. Stale README text of exactly this class is already filed as a defect in this repo (#285). Acceptable: state the real distinction — both suites build an image; Firefox's is assembled from anodebase rather than derived from a published browser image, because none carries both a pinned Firefox and a matching geckodriver.Disclosure:
script/test-e2ewas not executed locally (another session was running the Chrome suite on this host and the known flake in #287 makes a concurrent second run poor evidence). That claim rests on the run-636 job log for head329f3e1, where build steps#10 COPY . .and#11 RUN make buildare notCACHED, the in-image build stampscommitHashFull: 329f3e15..., and the suite reports1..40/40/40 tests passed.script/test-e2e-firefoxwas run locally on the head: 3/3, no container or process left behind.329f3e1558to89e35ac359Re-review of #291 — PASS: the rewritten
README.md:250-256Firefox paragraph and its siblings (Chrome paragraph, Entrypoints bullets,### In CI) are each true againstscript/test-e2e,script/test-e2e-firefox,tests/e2e/Dockerfileandtests/e2e/firefox/Dockerfileat89e35ac; the only delta from the previously reviewed329f3e1is that paragraph plus theTODO.mdrebase resolution, which drops nothing from either parent; single commit, author and committer bothclawbot, basenext, fast-forwardable, no attribution trailers;make checkgreen locally (29 suites / 703 tests, no cache markers, prettier clean); and CI on89e35acis green with both e2e jobs genuinely executed —COPY . .andRUN make buildnotCACHED, the in-image build stampingcommitHashFull: 89e35ac3...,40/40 tests passedand3/3 steps passed. Disclosure: per instruction the browser suites were not re-run locally (concurrent runs on this host plus #287 would make that poor evidence), so the suite results rest on the run-642 job logs.89e35ac359to9684eeacfcPASS — #291 satisfies #259; CI on head
9684eea(run 654) genuinely ran both jobs (COPY . .andRUN make builduncached,commitHashFull: 9684eea..., 44/44 and 3/3), and the failure path is real rather than asserted.Disclosures: to prove the failure path I broke the title assertion at
tests/e2e/run.js:84andtests/e2e/firefox/run.js:87in a private clone —script/test-e2eexited 1 (43/44),script/test-e2e-firefoxexited 1 (2/3), adashprobe confirmedtrap cleanup EXITpreserves a non-zero status rather than masking it, and run 630's job logs show both jobs red under the same mutation; clone reverted, nothing pushed, no containers or images of mine left behind. Gates run here:make checkexit 0 (29 suites, 703 tests, prettier clean),script/cibuildexit 0 withRUN make checkexecuted uncached (23.6s), unmutatedscript/test-e2e44/44 andscript/test-e2e-firefox3/3 on the head, merges clean onto currentnext(c06765e) with noTODO.mdentry dropped from either parent. One anomaly, not a defect: the PR body's Verification section is stale after the last two reworks — it cites run 636 at head329f3e1, quotes 37/37, and says the head's own e2e run was not yet observed. The tree, commit message,README.mdandTODO.mdare all accurate; only the body is behind.