test: make test-e2e is flaky under host load — "the extension opened no approval window within 30000ms" #290

New Issue

clawbot · 2026-08-14T06:26:15+02:00

clawbot commented

2026-08-14 06:26:15 +02:00

The Chrome e2e suite fails intermittently under concurrent load on the build host, with a different test failing each time. The failure is always the same shape: the extension opened no approval window within 30000ms.

Measured across five sequential runs by one reviewer on unrelated branches:

next baseline: 40/40
PR head: 36/37, then 33/37
PR head merged with next: 40/40, then 38/40

The same tree goes both green and red, so this is timing, not a defect in the code under test. Two other workers independently hit it in the same window; one saw eth_sendTransaction rejected time out on its 30s wait in an untouched transaction path while a concurrent make check was running.

Why this matters

next must stay green and mergeable without notice, and this suite is the repo's only browser-level guarantee. A suite that fails randomly is worse than a slow one:

it costs a rework cycle every time a reviewer has to decide whether a red run is real;
it trains readers to re-run until green, which is how a genuine regression gets waved through;
once #259 puts these suites in CI, a flake rate this high makes the pipeline useless.

Distinct from #218, which is a specific loadHomeTxs failed: Failed to fetch on popup reload mid-refresh. This one is the approval-window wait, and it hits several different tests.

Implementation requirements

Find the actual cause before tuning any number. A 30s wait that expires under load usually means the wait is polling for something that is not the real readiness signal, or the extension is genuinely slow to open the window because the service worker was evicted or is still starting. Raising the timeout hides it.
If the service worker is being torn down and restarted mid-test, say so — that is a production-relevant finding, not just a harness one.
Do not make the suite pass by retrying the assertion. A retry that masks a real slow path is the same defect with a longer fuse.
If the fix is genuinely "the host was oversubscribed", then the harness should detect and report that condition explicitly rather than presenting it as a test failure.

Definition of done

The cause is identified and stated, not worked around by a larger timeout.
make test-e2e passes repeatedly under concurrent load — demonstrate with several consecutive runs while the host is busy, with the output captured.
No assertion is weakened or made retry-until-green to achieve it.
TODO.md updated in the same commit.
make check passes.

The Chrome e2e suite fails intermittently under concurrent load on the build host, with a different test failing each time. The failure is always the same shape: `the extension opened no approval window within 30000ms`. Measured across five sequential runs by one reviewer on unrelated branches: - `next` baseline: 40/40 - PR head: 36/37, then 33/37 - PR head merged with `next`: 40/40, then 38/40 The same tree goes both green and red, so this is timing, not a defect in the code under test. Two other workers independently hit it in the same window; one saw `eth_sendTransaction rejected` time out on its 30s wait in an untouched transaction path while a concurrent `make check` was running. ## Why this matters `next` must stay green and mergeable without notice, and this suite is the repo's only browser-level guarantee. A suite that fails randomly is worse than a slow one: - it costs a rework cycle every time a reviewer has to decide whether a red run is real; - it trains readers to re-run until green, which is how a genuine regression gets waved through; - once https://git.eeqj.de/sneak/AutistMask/issues/259 puts these suites in CI, a flake rate this high makes the pipeline useless. Distinct from https://git.eeqj.de/sneak/AutistMask/issues/218, which is a specific `loadHomeTxs failed: Failed to fetch` on popup reload mid-refresh. This one is the approval-window wait, and it hits several different tests. ## Implementation requirements - Find the actual cause before tuning any number. A 30s wait that expires under load usually means the wait is polling for something that is not the real readiness signal, or the extension is genuinely slow to open the window because the service worker was evicted or is still starting. Raising the timeout hides it. - If the service worker is being torn down and restarted mid-test, say so — that is a production-relevant finding, not just a harness one. - Do not make the suite pass by retrying the assertion. A retry that masks a real slow path is the same defect with a longer fuse. - If the fix is genuinely "the host was oversubscribed", then the harness should detect and report that condition explicitly rather than presenting it as a test failure. ## Definition of done - [ ] The cause is identified and stated, not worked around by a larger timeout. - [ ] `make test-e2e` passes repeatedly under concurrent load — demonstrate with several consecutive runs while the host is busy, with the output captured. - [ ] No assertion is weakened or made retry-until-green to achieve it. - [ ] `TODO.md` updated in the same commit. - [ ] `make check` passes.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/AutistMask#290