test: drive the autosave race test to a condition, not a turn count (closes #36)
TestAutoSaveOnSignalRacesTurnLoop failed intermittently under load. The
captured failure text settles what it was: driveUntilDone's
t.Fatal("the turn loop ran out of turns before the saves were taken")
with no WARNING: DATA RACE anywhere in the log. The handoff is fine; the
test's own drive loop ran out of its fixed 1000-turn budget first.
Confirmed by instrumenting the loop to report the turns it actually
used. The count tracks scheduling pressure and nothing else: about
60-120 turns at host load ~57 with the whole machine to spread over, 418
at GOMAXPROCS=4, 539 and 655 at 2 and 1, and past 1000 under the doubled
load of the verbose rerun the test target performs after a failure. The
turns spent between one save being answered and the next request
arriving are not work, they are the saving goroutine's wake-up latency,
so a fixed turn count is a wall-clock assumption in disguise. Raising it
would hide the flake, not fix it. Each old-code failure took 0.12
seconds - 1000 turns burned in a tenth of a second - which is why every
attempt to reproduce this by loading the host failed: the cap was never
a wall-clock allowance at all.
So the budget is gone rather than larger. driveUntilDone drives until
the saving goroutine finishes and nothing else. Termination still holds,
it just belongs to the code under test: every AutoSaveOnSignal returns
within the timeout it is handed, so the saving goroutine always
finishes. A handoff that has stopped answering costs one autoSaveWait in
total, because g.sigSave is one deep and an unserviced request stays in
the channel for every later call to find full and fail on at once; what
fails is then the real assertion, "saves taken = 0, want 25", rather
than "out of turns". That is not the worst case, and the comment states
the bound that actually holds: a handoff that drains each request but
slower than autoSaveWait costs one timeout per save, wantSaves *
autoSaveWait = 250s, which would run past the 30s package timeout. It
needs about ten seconds of scheduler starvation per save against the
0.12s-per-1000-turns regime above, so it is remote, and a turn cap did
not bound it either.
Removing the cap exposed a second assumption underneath it. testTerm
answers space and newline for ever once its script is exhausted, and
neither key takes a turn, so command(), which loops until the player
consumes one, never returns; the old cap was silently sized to the
script. An uncapped drive wedged inside a single command() call. The two
drive tests now use driveTerm, a headless terminal whose script repeats.
Repeating is necessary and not sufficient, and the comment on driveTerm
says which property is load-bearing: ' ' clears After outright and all
eight movement keys clear it on a refused step, so a script of only
those keys wedges exactly as testTerm's tail did - with the script set
to " " the drive hits the 30s timeout inside command(). What makes the
wedge impossible is that the cycle always contains an unconditional
turn-taker, and these scripts contain two, '.' and 's', neither of which
can be refused by being blocked in all directions, Held, in a bear trap,
or under NoCommand > 0. Trimming both out would bring the wedge back.
The guard is undiminished, shown by mutation and reverted afterwards.
Reverting the fix from the earlier signal-autosave work - AutoSaveOnSignal
replaced by a direct g.autoSave(), encoding on the calling goroutine -
still fails the test with a flood of DATA RACE reports (139 here, 62-110
on another machine; the property is what is pinned, not the number),
the encoder reading what the turn loop writes. Removing
serviceAutoSaveRequest from command() still fails it too, now in 10s
with "saves taken = 0, want 25" instead of by hanging.
Under load: at GOMAXPROCS=2 on a 48-core host at load ~150, with an
unrelated deliberate failure in the tree so every run took the verbose
rerun, the old code failed 8 of 8 runs and the new code 0 of 8. Also
green across 24 concurrent unconstrained runs at load ~120, 10 runs
alongside a spinner load, and 5 runs each at GOMAXPROCS 1, 2 and 4.
No non-test code changed. make check green, lint 0 issues, .golangci.yml
byte-identical.
This commit is contained in:
70
TODO.md
70
TODO.md
@@ -35,6 +35,76 @@ is finished.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-08-09 `TestAutoSaveOnSignalRacesTurnLoop` de-flaked at the cause
|
||||
(`fix/autosave-turn-budget-36`, closes #36). The failure text was captured
|
||||
before anything was changed and it is **not** a data race: the assertion was
|
||||
`driveUntilDone`'s
|
||||
`t.Fatal("the turn loop ran out of turns before the saves were taken")`, with
|
||||
no `WARNING: DATA RACE` anywhere in the log. The handoff fixed in #24 was
|
||||
working; the test's own drive loop was running out of its fixed 1000-turn
|
||||
budget first.
|
||||
|
||||
Confirmed rather than taken on trust. Instrumenting the loop to report the
|
||||
turns it actually used showed the count tracking scheduling pressure and
|
||||
nothing else: about 60-120 turns at host load ~57 with the whole machine to
|
||||
spread over, 418 at `GOMAXPROCS=4`, 539 and 655 at 2 and 1, and past 1000 —
|
||||
the recorded failure — under the doubled load of the verbose rerun that the
|
||||
test target performs after a failure. The turns between one save being
|
||||
answered and the next request arriving are not work; they are the saving
|
||||
goroutine's wake-up latency, so a fixed turn count is a wall-clock
|
||||
assumption in disguise, which is why raising it would have hidden the flake
|
||||
rather than fixed it.
|
||||
|
||||
So the budget is gone rather than larger. `driveUntilDone` now drives until
|
||||
the saving goroutine finishes and nothing else. Termination is not lost, it
|
||||
just belongs to the code under test instead of to the test: every
|
||||
`AutoSaveOnSignal` returns within the timeout it is handed, so the saving
|
||||
goroutine always finishes. A handoff that has stopped answering costs one
|
||||
`autoSaveWait` in total — `g.sigSave` is one deep, so an unserviced request
|
||||
stays in the channel and every later call finds it full and fails at once —
|
||||
and the failure is then the real assertion (`saves taken = 0, want 25`)
|
||||
instead of "out of turns". The worst case is not that one: a handoff that
|
||||
drains each request but slower than `autoSaveWait` costs one timeout per
|
||||
save, `wantSaves × autoSaveWait` = 250s, which would run past the 30s
|
||||
package timeout instead of reaching the assertion. It takes ~10s of
|
||||
scheduler starvation per save against a measured 0.12s per 1000 turns, so it
|
||||
is remote, and the turn cap did not bound it either. The comment in the test
|
||||
states that bound rather than the optimistic one.
|
||||
|
||||
Removing the cap exposed a second assumption underneath it, which is the
|
||||
reason this is not a one-line diff. `testTerm` answers space and newline for
|
||||
ever once its script is exhausted, and neither key takes a turn, so
|
||||
`command()` — which loops until the player consumes one — never returns; the
|
||||
old cap was silently sized to the script (4000 characters, two per turn,
|
||||
against 1000 turns). An uncapped drive wedged inside a single `command()`
|
||||
call. The two drive tests therefore use a new `driveTerm`, a headless
|
||||
terminal whose script repeats. Repeating is necessary but not sufficient,
|
||||
and the test says so: `' '` clears `After` outright and all eight movement
|
||||
keys clear it on a refused step, so a script of only those keys wedges just
|
||||
as `testTerm`'s tail did. What makes the wedge impossible is that the cycle
|
||||
always holds an _unconditional_ turn-taker, and these scripts hold two —
|
||||
`'.'` (empty handler) and `'s'` (`search`, which writes `After` on no path),
|
||||
neither refusable by blocked-in-all-directions, `Held`, a bear trap, or
|
||||
`NoCommand > 0`. Removing both would bring the wedge back.
|
||||
|
||||
Both halves of the definition of done were demonstrated by mutation, with
|
||||
the deliberately-broken tree reverted afterwards and `.golangci.yml` left
|
||||
byte-identical (sha256 `021cc83f...46bcb`). Reverting #24 —
|
||||
`AutoSaveOnSignal` replaced by a direct `g.autoSave()`, encoding on the
|
||||
calling goroutine — still fails the test with 139 `WARNING: DATA RACE`
|
||||
reports naming `snapshotHeader` reading what `executeCommand` writes, so the
|
||||
guard is undiminished. Removing the `serviceAutoSaveRequest` call from
|
||||
`command()` still fails it too, now in 10s with `saves taken = 0, want 25`
|
||||
rather than by hanging.
|
||||
|
||||
Under load, an A/B at `GOMAXPROCS=2` on a 48-core host at load ~150, with an
|
||||
unrelated deliberate failure in the tree so that every run took the verbose
|
||||
rerun: the old code failed 8 of 8 runs with "ran out of turns"; the new code
|
||||
failed 0 of 8, the only failure being the planted one. Also green across 24
|
||||
concurrent unconstrained runs at load ~120, 10 runs alongside a spinner
|
||||
load, and 5 runs each at `GOMAXPROCS` 1, 2 and 4. `make check` green, lint 0
|
||||
issues.
|
||||
|
||||
- 2026-08-09 Wizard commands under test (`test/wizard-coverage`, closes #7): the
|
||||
last of the three thin spots, so the coverage step is now closed rather than
|
||||
narrowed. `game/wizard.go`'s eight functions had no tests of their own, and
|
||||
|
||||
Reference in New Issue
Block a user