sneak/AutistMask
1
0
Fork 0
Code Issues 67 Pull Requests 3 Actions Packages Projects Releases Wiki Activity

build: nothing automatic ever runs the e2e suites, so every browser-level guarantee is manual-only #259

New Issue
Closed
opened 2026-08-12 10:56:28 +02:00 by clawbot · 1 comment
clawbot commented 2026-08-12 10:56:28 +02:00
Collaborator
Copy Link
Copy Source

Traced during the independent review of #243: script/check is test + lint + fmt-check; the Dockerfile runs make check and make build; .gitea/workflows/check.yml runs script/cibuild, which is docker build .; script/precommit runs script/check. Nothing in that chain reaches make test-e2e or make test-e2e-firefox.

That is by design — REPO_POLICIES.md:192 caps make test at 20s and the e2e suites are deliberately outside it. The consequence is worth stating plainly, though: every browser-level guarantee this repo has is enforced only when a human or an agent remembers to run it. That now includes the libsodium WASM-under-CSP check (#182), the recovery-phrase and private-key DOM-wipe checks (#161, #221), the ConfirmTx spend-gate coverage (#238), and the layout-shift assertion added by #243 — several of which are considerably more safety-critical than the 20s budget they were excluded to protect.

make check staying fast is right. The fix is a separate CI job, not folding e2e into make check.

Note this is currently moot in practice for a second reason: no Actions runner is picking up jobs at all (#220), so even the existing check workflow never runs. That issue gates this one.

Implementation requirements

  • Add a CI job that runs make test-e2e (and make test-e2e-firefox once #184 lands) as a SEPARATE job from check, outside the 20s make test cap.
  • Do not add e2e to make check or script/check — the local fast path must stay fast.
  • State the expected wall-clock cost of the new job in the PR body.
  • Decide and document whether the job gates a merge or only reports.

Definition of done

  • A CI job runs the e2e suites on every push, separately from check.
  • make check runtime is unchanged.
  • A deliberately broken e2e assertion fails that job — demonstrated, with captured output.
  • TODO.md updated in the same commit.
  • make check passes.
Traced during the independent review of https://git.eeqj.de/sneak/AutistMask/pulls/243: `script/check` is test + lint + fmt-check; the `Dockerfile` runs `make check` and `make build`; `.gitea/workflows/check.yml` runs `script/cibuild`, which is `docker build .`; `script/precommit` runs `script/check`. Nothing in that chain reaches `make test-e2e` or `make test-e2e-firefox`. That is by design — `REPO_POLICIES.md:192` caps `make test` at 20s and the e2e suites are deliberately outside it. The consequence is worth stating plainly, though: **every browser-level guarantee this repo has is enforced only when a human or an agent remembers to run it.** That now includes the libsodium WASM-under-CSP check (https://git.eeqj.de/sneak/AutistMask/issues/182), the recovery-phrase and private-key DOM-wipe checks (https://git.eeqj.de/sneak/AutistMask/issues/161, https://git.eeqj.de/sneak/AutistMask/issues/221), the ConfirmTx spend-gate coverage (https://git.eeqj.de/sneak/AutistMask/issues/238), and the layout-shift assertion added by https://git.eeqj.de/sneak/AutistMask/pulls/243 — several of which are considerably more safety-critical than the 20s budget they were excluded to protect. `make check` staying fast is right. The fix is a separate CI job, not folding e2e into `make check`. Note this is currently moot in practice for a second reason: no Actions runner is picking up jobs at all (https://git.eeqj.de/sneak/AutistMask/issues/220), so even the existing `check` workflow never runs. That issue gates this one. ## Implementation requirements - Add a CI job that runs `make test-e2e` (and `make test-e2e-firefox` once https://git.eeqj.de/sneak/AutistMask/issues/184 lands) as a SEPARATE job from `check`, outside the 20s `make test` cap. - Do not add e2e to `make check` or `script/check` — the local fast path must stay fast. - State the expected wall-clock cost of the new job in the PR body. - Decide and document whether the job gates a merge or only reports. ## Definition of done - [ ] A CI job runs the e2e suites on every push, separately from `check`. - [ ] `make check` runtime is unchanged. - [ ] A deliberately broken e2e assertion fails that job — demonstrated, with captured output. - [ ] `TODO.md` updated in the same commit. - [ ] `make check` passes.
clawbot commented 2026-08-14 06:06:04 +02:00
Author
Collaborator
Copy Link
Copy Source

Plan, and the runner-capability answer up front: the runner can run both containerized suites. Not a prediction — the abandoned WIP branch build/issue-259-e2e-in-ci already got a green e2e.yml run on it, actions run 622: e2e-chrome 1m54s reporting 37/37 tests passed, e2e-firefox 1m03s reporting 3/3 steps passed. clawbot still gets 403 on the Actions API, but the job logs are readable unauthenticated at /sneak/AutistMask/actions/runs/<id>/jobs/<n>/logs, which is how I read them.

What the probe run (615) established, and what it forces:

  • The job body runs as root inside docker.gitea.com/runner-images:ubuntu-latest with the host's /run/docker.sock mounted, so nested containers are siblings on the host daemon. Docker-in-Docker is not needed and is not available.
  • The checkout lives on a docker named volume at /workspace/sneak/AutistMask, not a host path. docker run -v "$PWD:/work" is therefore resolved by the host daemon and mounts an empty directory — the probe ran exactly that and got an empty ls. Both suites bind-mount the repo today, which is why they cannot work unchanged.
  • The runner image also ships a node too old to install this repo's dependencies.

So the fix is the one the WIP found: deliver the repo to the daemon as a build context and build the extension inside the image, leaving docker as the only prerequisite on the machine that starts a run. That is also better locally.

Salvaging from the WIP, rebased fresh onto current next as one commit:

  • .gitea/workflows/e2e.yml — one job per browser so a Chrome failure does not hide the Firefox result, actions/checkout pinned by commit as in check.yml.
  • New tests/e2e/Dockerfile, and tests/e2e/firefox/Dockerfile moved to a repo-root build context; all base images stay digest-pinned, and the Firefox/geckodriver download layers stay above the repo copy so a source edit does not re-fetch them.
  • script/test-e2e / script/test-e2e-firefox build with --iidfile and run the image by ID, so two clones running the suite at once on this host cannot swap tags under each other.

Discarding: the probe.yml workflow (its job is done); a README bullet for a script/e2e-container that was never written; a README "In CI" paragraph describing jobs as script/bootstrap plus a make target, which is not what the workflow runs; and an unsourced "the Chrome suite is flaky under load" line — I will not put a flakiness claim in the tree that I have not measured.

Remaining decisions, both stated in the PR body: the jobs report, they do not gate, because this repo configures no branch protection — the gate is still the independent reviewer; and the cost is roughly 3 minutes of runner occupancy per push, cold-cache, on top of check.

make check is untouched — no e2e in script/check, no new *.test.js. The mutation demonstration will be a deliberately broken assertion pushed to a scratch branch, with the failing job output quoted in the PR body, and the scratch branch deleted afterwards.

Plan, and the runner-capability answer up front: **the runner can run both containerized suites.** Not a prediction — the abandoned WIP branch `build/issue-259-e2e-in-ci` already got a green `e2e.yml` run on it, actions run 622: `e2e-chrome` 1m54s reporting `37/37 tests passed`, `e2e-firefox` 1m03s reporting `3/3 steps passed`. `clawbot` still gets 403 on the Actions API, but the job logs are readable unauthenticated at `/sneak/AutistMask/actions/runs/<id>/jobs/<n>/logs`, which is how I read them. What the probe run (615) established, and what it forces: - The job body runs as root inside `docker.gitea.com/runner-images:ubuntu-latest` with the host's `/run/docker.sock` mounted, so nested containers are siblings on the host daemon. Docker-in-Docker is not needed and is not available. - The checkout lives on a docker **named volume** at `/workspace/sneak/AutistMask`, not a host path. `docker run -v "$PWD:/work"` is therefore resolved by the host daemon and mounts an **empty directory** — the probe ran exactly that and got an empty `ls`. Both suites bind-mount the repo today, which is why they cannot work unchanged. - The runner image also ships a node too old to install this repo's dependencies. So the fix is the one the WIP found: deliver the repo to the daemon as a **build context** and build the extension inside the image, leaving docker as the only prerequisite on the machine that starts a run. That is also better locally. Salvaging from the WIP, rebased fresh onto current `next` as one commit: - `.gitea/workflows/e2e.yml` — one job per browser so a Chrome failure does not hide the Firefox result, `actions/checkout` pinned by commit as in `check.yml`. - New `tests/e2e/Dockerfile`, and `tests/e2e/firefox/Dockerfile` moved to a repo-root build context; all base images stay digest-pinned, and the Firefox/geckodriver download layers stay above the repo copy so a source edit does not re-fetch them. - `script/test-e2e` / `script/test-e2e-firefox` build with `--iidfile` and run the image **by ID**, so two clones running the suite at once on this host cannot swap tags under each other. Discarding: the `probe.yml` workflow (its job is done); a README bullet for a `script/e2e-container` that was never written; a README "In CI" paragraph describing jobs as `script/bootstrap` plus a `make` target, which is not what the workflow runs; and an unsourced "the Chrome suite is flaky under load" line — I will not put a flakiness claim in the tree that I have not measured. Remaining decisions, both stated in the PR body: the jobs **report, they do not gate**, because this repo configures no branch protection — the gate is still the independent reviewer; and the cost is roughly 3 minutes of runner occupancy per push, cold-cache, on top of `check`. `make check` is untouched — no e2e in `script/check`, no new `*.test.js`. The mutation demonstration will be a deliberately broken assertion pushed to a scratch branch, with the failing job output quoted in the PR body, and the scratch branch deleted afterwards.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
merge-ready
needs-checks
needs-rebase
needs-review
needs-rework
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/AutistMask#259