Make main and next green: make test fails in the Docker build, and its output is cut off before the failure #414

Closed
opened 2026-10-02 02:26:36 +02:00 by clawbot · 3 comments
Collaborator

Top priority: this comes before all other work in this repo. Owner's words (sneak/project-management#20): "main and next should be green on all managed repos at all times. any divergence is a top priority."

Red: main (f703b72c) and next (1cafaeb9). script/cibuild fails at RUN make test in the Dockerfile: make test exits 1, and every package that shows a result passed. The failing test cannot be seen: script/test always runs go test -v, and the step's output passes the 2 MiB log limit of the Docker build, which cuts it off before the failure. internal/handlers is the package with no result line in either run. The host was busy (three builds at once), which #225 says makes the internal/handlers tests fail.

Reproduce: clone the branch fresh and run script/cibuild, with other builds running at the same time.

Related causes already filed: #225 (tests fail under host load), #198 (make test too slow, mostly internal/handlers), #315 (script/test always verbose), #344 (make test needs up to 6.7 GiB of RAM).

Definition of done:

  • A failing test is visible in the script/cibuild output.
  • The cause is fixed for real: no test skipped, deleted or retried into passing, no check weakened.
  • next: through a reviewed PR to next, a fresh clone's script/cibuild passes several times in a row, including while other builds run.
  • main: the same fix, and nothing else from next, lands on main through its own reviewed PR targeting main, then main passes the same way.

Model: opus-5-5

Top priority: this comes before all other work in this repo. Owner's words (https://git.eeqj.de/sneak/project-management/issues/20): "main and next should be green on all managed repos at all times. any divergence is a top priority." Red: `main` (`f703b72c`) and `next` (`1cafaeb9`). `script/cibuild` fails at `RUN make test` in the `Dockerfile`: `make test` exits 1, and every package that shows a result passed. The failing test cannot be seen: `script/test` always runs `go test -v`, and the step's output passes the 2 MiB log limit of the Docker build, which cuts it off before the failure. `internal/handlers` is the package with no result line in either run. The host was busy (three builds at once), which https://git.eeqj.de/sneak/webhooker/issues/225 says makes the `internal/handlers` tests fail. Reproduce: clone the branch fresh and run `script/cibuild`, with other builds running at the same time. Related causes already filed: https://git.eeqj.de/sneak/webhooker/issues/225 (tests fail under host load), https://git.eeqj.de/sneak/webhooker/issues/198 (`make test` too slow, mostly `internal/handlers`), https://git.eeqj.de/sneak/webhooker/issues/315 (`script/test` always verbose), https://git.eeqj.de/sneak/webhooker/issues/344 (`make test` needs up to 6.7 GiB of RAM). Definition of done: - A failing test is visible in the `script/cibuild` output. - The cause is fixed for real: no test skipped, deleted or retried into passing, no check weakened. - `next`: through a reviewed PR to `next`, a fresh clone's `script/cibuild` passes several times in a row, including while other builds run. - `main`: the same fix, and nothing else from `next`, lands on `main` through its own reviewed PR targeting `main`, then `main` passes the same way. Model: opus-5-5
clawbot self-assigned this 2026-10-02 02:26:36 +02:00
Author
Collaborator

Plan.

Reading: internal/handlers never prints a result line because under host load the package overruns go test's 90-second timeout. Its tests hash the admin password with Argon2id at 64 MB on every database start, many at once under -race, which is also what #344 measures. The failure itself is hidden because script/test runs go test -v, whose output passes the Docker build's 2 MiB log limit first.

  • next: #404 (test-only hashing cost, bounded test parallelism) is the fix for the cause and lands first; it is under review now. This issue's PR to next then makes script/test run without -v, so only failures and per-package results reach the log, and proves the result: a fresh clone's script/cibuild passing several times in a row while other builds run. If the handlers package still fails with 404 in place, that failure is found and fixed in this PR; nothing is skipped, retried or given a longer timeout.
  • main: a branch off main carrying exactly those two changes and nothing else from next, in its own PR targeting main, independently reviewed, then proven the same way.

Model: opus-5-5

Plan. Reading: `internal/handlers` never prints a result line because under host load the package overruns `go test`'s 90-second timeout. Its tests hash the admin password with Argon2id at 64 MB on every database start, many at once under `-race`, which is also what https://git.eeqj.de/sneak/webhooker/issues/344 measures. The failure itself is hidden because `script/test` runs `go test -v`, whose output passes the Docker build's 2 MiB log limit first. - **`next`:** https://git.eeqj.de/sneak/webhooker/pulls/404 (test-only hashing cost, bounded test parallelism) is the fix for the cause and lands first; it is under review now. This issue's PR to `next` then makes `script/test` run without `-v`, so only failures and per-package results reach the log, and proves the result: a fresh clone's `script/cibuild` passing several times in a row while other builds run. If the handlers package still fails with 404 in place, that failure is found and fixed in this PR; nothing is skipped, retried or given a longer timeout. - **`main`:** a branch off `main` carrying exactly those two changes and nothing else from `next`, in its own PR targeting `main`, independently reviewed, then proven the same way. Model: opus-5-5
Author
Collaborator

Done, in two PRs, both waiting for review.

Cause: every test that starts a database hashed the admin password with Argon2id at 64 MB. Under the race detector, with dozens of those tests running in parallel on a busy host, internal/handlers overran its 15-second application start and then the 90-second package timeout. The failure was invisible because script/test ran go test -v, whose output passed the Docker build's 2 MiB log limit before the failure was printed. Once -v was dropped, a loaded build of next without #404 showed those failing tests. With 404, nothing else needed fixing.

  • #415 (to next): script/test runs go test without -v, so the build log shows one result line per package and the output of failing tests only. 404 is already on next.

  • #416 (to main): 404's change and the one in 415, one commit each, and nothing else from next.

  • Deviation: no -v rerun on failure, which REPO_POLICIES.md asks for; that is #315, and a full verbose rerun would pass the log limit again.

  • Not fixed here: under CPU load well beyond other builds running, internal/handlers and internal/delivery can still overrun 90 seconds; that is #225.

  • Still possible: a failing package prints everything its tests wrote, so several packages failing at once can still reach the log limit.

Model: opus-5-5

Done, in two PRs, both waiting for review. Cause: every test that starts a database hashed the admin password with Argon2id at 64 MB. Under the race detector, with dozens of those tests running in parallel on a busy host, `internal/handlers` overran its 15-second application start and then the 90-second package timeout. The failure was invisible because `script/test` ran `go test -v`, whose output passed the Docker build's 2 MiB log limit before the failure was printed. Once `-v` was dropped, a loaded build of `next` without https://git.eeqj.de/sneak/webhooker/pulls/404 showed those failing tests. With 404, nothing else needed fixing. - https://git.eeqj.de/sneak/webhooker/pulls/415 (to `next`): `script/test` runs `go test` without `-v`, so the build log shows one result line per package and the output of failing tests only. 404 is already on `next`. - https://git.eeqj.de/sneak/webhooker/pulls/416 (to `main`): 404's change and the one in 415, one commit each, and nothing else from `next`. - Deviation: no `-v` rerun on failure, which `REPO_POLICIES.md` asks for; that is https://git.eeqj.de/sneak/webhooker/issues/315, and a full verbose rerun would pass the log limit again. - Not fixed here: under CPU load well beyond other builds running, `internal/handlers` and `internal/delivery` can still overrun 90 seconds; that is https://git.eeqj.de/sneak/webhooker/issues/225. - Still possible: a failing package prints everything its tests wrote, so several packages failing at once can still reach the log limit. Model: opus-5-5
clawbot reopened this issue 2026-10-02 06:32:01 +02:00
Author
Collaborator

next is green: #404 (cheaper test hashing) and #415 (go test without -v) are on it. The same two changes for main are #416, reviewed and ready for you to merge; this issue closes with it.

Model: opus-5-5

`next` is green: https://git.eeqj.de/sneak/webhooker/pulls/404 (cheaper test hashing) and https://git.eeqj.de/sneak/webhooker/pulls/415 (`go test` without `-v`) are on it. The same two changes for `main` are https://git.eeqj.de/sneak/webhooker/pulls/416, reviewed and ready for you to merge; this issue closes with it. Model: opus-5-5
sneak closed this issue 2026-10-03 14:08:52 +02:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/webhooker#414