make test retry masks real data races and timeouts #101

New Issue

clawbot · 2026-08-10T15:46:24+02:00

clawbot commented

2026-08-10 15:46:24 +02:00

Makefile:35 runs go test ... || go test -v .... When the first pass fails, the retry frequently passes, and make test (and therefore make check and CI) reports green. Real failures are being swallowed.

Observed on next with no PR applied: first go test pass failed, retry passed. Observed again while reviewing #96 at f24e33a: docker build --no-cache-filter=lint,builder . run 1 failed with a data race in TestIntegrationTwoClients and then panic: test timed out after 30s in internal/handlers; run 2 passed clean. Build is nondeterministic, so no commit can currently be certified green.

Contributing factor: internal/handlers takes 23-30s against a hard -timeout 30s, leaving no margin.

Definition of done:

make test fails when any test fails. No || retry that can convert a red run into a green one.
The data race(s) in the ircserver integration tests are fixed (not skipped, not -race disabled).
The internal/handlers timeout margin is resolved, either by making the package faster or by a justified timeout that is not near the runtime.
Ten consecutive docker build --no-cache . runs pass on next.

`Makefile:35` runs `go test ... || go test -v ...`. When the first pass fails, the retry frequently passes, and `make test` (and therefore `make check` and CI) reports green. Real failures are being swallowed. Observed on `next` with no PR applied: first `go test` pass failed, retry passed. Observed again while reviewing https://git.eeqj.de/sneak/neoirc/pulls/96 at `f24e33a`: `docker build --no-cache-filter=lint,builder .` run 1 failed with a data race in `TestIntegrationTwoClients` and then `panic: test timed out after 30s` in `internal/handlers`; run 2 passed clean. Build is nondeterministic, so no commit can currently be certified green. Contributing factor: `internal/handlers` takes 23-30s against a hard `-timeout 30s`, leaving no margin. Definition of done: - `make test` fails when any test fails. No `||` retry that can convert a red run into a green one. - The data race(s) in the `ircserver` integration tests are fixed (not skipped, not `-race` disabled). - The `internal/handlers` timeout margin is resolved, either by making the package faster or by a justified timeout that is not near the runtime. - Ten consecutive `docker build --no-cache .` runs pass on `next`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/neoirc#101

make test retry masks real data races and timeouts #101

`make test` retry masks real data races and timeouts #101