internal/handlers tests are load-fragile: both the fx start budget and the 90s package timeout blow under host load, so the gate reports failures unrelated to the change #225
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Found while reviewing #219, then hit independently by the reviewers of #218, #222 and #224. Reproduced on
nextitself, so it is not attributable to any PR.TWO SEPARATE BUDGETS are being blown, both in
internal/handlers, both only under load:1. fx start timeout. Subtests fail with
Seen on
TestFailedLogin_*,TestLogin_*,TestStoredUsername_*and everyBodiesRoundTripByteIdenticalsubtest.2. The 90s per-package
go testtimeout raised by #194:The second is the more alarming number: on a clean re-run at lower load,
internal/handlerscompleted in 90.203s against a 90s budget — passing by about 0.2%. That is not a margin, it is a coin flip.script/test's own header documents this package at 16.9s unloaded and 67.3s atGOMAXPROCS=4under load; observed load averages during these runs were 122 to 170 on 48 cores.Why this matters rather than being written off as "the host was busy": the authoritative gate is a cache-defeated container build (#119), and that gate now produces FALSE FAILURES under load. A gate that fails at random is the same category of problem #119 fixed — it stops being evidence, and every reviewer has to spend a run on
nextto attribute their own failure. It also meansnextcannot be confidently declared green under load, andnextmust stay green and mergeable without notice.Related but distinct: #198 (
make testwall time, driven by this same package) and #190 (the last wall-clock-dependent test).Definition of done:
internal/handlersno longer sits within a few percent of the per-package timeout on a loaded host — fix the runtime, do not simply raise the number againinternal/handlers tests fail under host load: fx start timeout exceeded, so the gate reports failure for reasons unrelated to the changeto internal/handlers tests are load-fragile: both the fx start budget and the 90s package timeout blow under host load, so the gate reports failures unrelated to the changeRoot cause measured while implementing #67.
Every
newTestAppininternal/handlersseeds an admin user, and that pays an Argon2id hash at 64 MB. The package's runtime is therefore roughly linear in the number of fx applications the tests stand up, not in what they assert.Measurements, all in a cache-defeated container build:
next, baseline: 67.7s, and 82.8s in a separate container run — against a 90s per-package budgetpanic: test timed out after 1m30sSo the package sits at roughly 92% of budget before anyone adds anything, and the marginal cost of a new test is an Argon2id hash rather than the test's own work. That is why it tips under load and why raising the timeout again would only defer it.
Suggested direction for whoever takes this: a test-only seam that skips the Argon2id admin seed for tests that do not exercise login, leaving production hashing untouched. Done-criterion worth aiming at:
internal/handlersbelow 20s in a cache-defeated container build, which would also close out #198.clawbot referenced this issue2026-08-20 08:21:53 +02:00
clawbot referenced this issue2026-08-20 08:34:07 +02:00