Tests that fail at random under load #507

Open
opened 2026-10-06 06:06:39 +02:00 by clawbot · 0 comments
Collaborator

Two tests, and one package's run time, fail at random on a busy host. Any of them can turn the build red for a commit that is correct. Seen by the independent reviews of #505, which changes none of these tests:

  • TestAlpineRunsUnderTheSecurityPolicy (internal/server/alpine_browser_test.go) failed once in make test-browser: a page element was gone while the test read its value. It looks like the test reads before the page has finished loading.
  • TestRecoveryAndSweepDoNotDoubleDispatch (internal/delivery/inflight_test.go) failed once in an image build, with nothing dispatched, at host load 100 to 200. It looks like the test checks before dispatch has had time to happen.
  • The internal/handlers package hit the 90-second per-package test timeout (the test phase of the Dockerfile) in 2 of 6 runs at host load 260 to 290. At lower load it took 19 to 50 seconds. Since #505, lint and tests run at the same time in the image build.

Definition of done

  • The browser test waits for the page to finish loading before reading what it checks, and the delivery test waits for dispatch with a deadline instead of a fixed point in time. Each still fails when the behaviour it guards breaks: show this by breaking it once.
  • internal/handlers finishes well inside its timeout under load, made faster without weakening any test. Raising the timeout only to hide slowness is not a fix; if a raise is still needed, say why in the PR.
  • make check and make test-browser each pass five runs in a row on the shared host, through the gate lock.

Model: opus-5-5

Two tests, and one package's run time, fail at random on a busy host. Any of them can turn the build red for a commit that is correct. Seen by the independent reviews of https://git.eeqj.de/sneak/webhooker/pulls/505, which changes none of these tests: - `TestAlpineRunsUnderTheSecurityPolicy` (`internal/server/alpine_browser_test.go`) failed once in `make test-browser`: a page element was gone while the test read its value. It looks like the test reads before the page has finished loading. - `TestRecoveryAndSweepDoNotDoubleDispatch` (`internal/delivery/inflight_test.go`) failed once in an image build, with nothing dispatched, at host load 100 to 200. It looks like the test checks before dispatch has had time to happen. - The `internal/handlers` package hit the 90-second per-package test timeout (the `test` phase of the `Dockerfile`) in 2 of 6 runs at host load 260 to 290. At lower load it took 19 to 50 seconds. Since https://git.eeqj.de/sneak/webhooker/pulls/505, lint and tests run at the same time in the image build. ## Definition of done - The browser test waits for the page to finish loading before reading what it checks, and the delivery test waits for dispatch with a deadline instead of a fixed point in time. Each still fails when the behaviour it guards breaks: show this by breaking it once. - `internal/handlers` finishes well inside its timeout under load, made faster without weakening any test. Raising the timeout only to hide slowness is not a fix; if a raise is still needed, say why in the PR. - `make check` and `make test-browser` each pass five runs in a row on the shared host, through the gate lock. Model: opus-5-5
clawbot self-assigned this 2026-10-06 06:06:39 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/webhooker#507