Terminal QA harness: drive the real binary under tmux at 80x24 and assert on screen contents #42

New Issue

2026-08-10T14:29:39+02:00

clawbot commented

2026-08-10 14:29:39 +02:00

The release gate #8 was filed as "needs a human" on the premise that nothing here can exercise a real terminal. That premise is wrong, and I verified it rather than assuming: tmux is present on this host, and a detached session created with -x 80 -y 24 gives a real 80x24 terminal with a real PTY, into which tmux send-keys feeds keystrokes and out of which tmux capture-pane -p returns the exact rendered cell grid. A probe placing text with a raw cursor-position escape came back at the expected row and column.

That is enough to test everything the existing suite structurally cannot: real tcell rendering, a real process boundary, and the actual death and win paths, none of which the headless suite reaches because it pins HP/food/exp via fortify().

Definition of done

A harness, driven through a make target, that starts the built binary in a detached 80x24 tmux session, sends keys, and asserts against captured screen contents. It must:

Fail loudly if tmux is absent rather than silently skipping — a skipped terminal test that reports green is worse than no test.
Assert on the rendered grid, not merely on the process exiting 0.
Be deterministic: fixed seed, fixed ROGUEOPTS, fixed geometry, and a bounded wait for each expected screen rather than a fixed sleep.
Clean up its session on failure, including on timeout.

Cases to cover, taken from the gate's own list:

Death path: die, confirm the RIP screen renders and the score is recorded.
Save/restore across a real process boundary: S, quit, relaunch against the save file, confirm state is intact and the save file is removed on restore.
Screen integrity: no corruption, no stuck message line, cursor where it belongs, correct --More-- handling.
ROGUEOPTS handling: name=, terse, jump, fruit=.
Amulet retrieval end to end, landing on the score screen with the entry written. Wizard-mode descent is acceptable for a first structural pass, but say so plainly in the test name and in the harness output — a wizard-assisted run is not evidence that an honest game completes.

Scores are written to a real path in $HOME; point the harness at a private throwaway HOME so it never touches a real ~/.rogue.scores.

Dependency

Blocked on #19 — the harness needs a built binary at a known path, and the make-targets-only rule means it must come from a make build target, which does not exist yet. Do 19 first.

The release gate https://git.eeqj.de/sneak/rgoue/issues/8 was filed as "needs a human" on the premise that nothing here can exercise a real terminal. That premise is wrong, and I verified it rather than assuming: `tmux` is present on this host, and a detached session created with `-x 80 -y 24` gives a real 80x24 terminal with a real PTY, into which `tmux send-keys` feeds keystrokes and out of which `tmux capture-pane -p` returns the exact rendered cell grid. A probe placing text with a raw cursor-position escape came back at the expected row and column. That is enough to test everything the existing suite structurally cannot: real `tcell` rendering, a real process boundary, and the actual death and win paths, none of which the headless suite reaches because it pins HP/food/exp via `fortify()`. ## Definition of done A harness, driven through a `make` target, that starts the built binary in a detached 80x24 `tmux` session, sends keys, and asserts against captured screen contents. It must: - Fail loudly if `tmux` is absent rather than silently skipping — a skipped terminal test that reports green is worse than no test. - Assert on the rendered grid, not merely on the process exiting 0. - Be deterministic: fixed seed, fixed `ROGUEOPTS`, fixed geometry, and a bounded wait for each expected screen rather than a fixed sleep. - Clean up its session on failure, including on timeout. Cases to cover, taken from the gate's own list: 1. Death path: die, confirm the RIP screen renders and the score is recorded. 2. Save/restore across a real process boundary: `S`, quit, relaunch against the save file, confirm state is intact and the save file is removed on restore. 3. Screen integrity: no corruption, no stuck message line, cursor where it belongs, correct `--More--` handling. 4. `ROGUEOPTS` handling: `name=`, `terse`, `jump`, `fruit=`. 5. Amulet retrieval end to end, landing on the score screen with the entry written. Wizard-mode descent is acceptable for a first structural pass, but say so plainly in the test name and in the harness output — a wizard-assisted run is not evidence that an honest game completes. Scores are written to a real path in `$HOME`; point the harness at a private throwaway `HOME` so it never touches a real `~/.rogue.scores`. ## Dependency Blocked on https://git.eeqj.de/sneak/rgoue/issues/19 — the harness needs a built binary at a known path, and the make-targets-only rule means it must come from a `make build` target, which does not exist yet. Do 19 first.

clawbot referenced this issue

2026-08-10 14:29:49 +02:00

Release gate: full Amulet-retrieval playthrough at a real terminal (needs a human) #8

clawbot commented

2026-08-10 15:59:00 +02:00

Unblocked: #19 landed, so make build exists and puts the binary at ./build/rogue (gitignored, under build/).

Implementation requirements settled before dispatch.

Determinism over sleeps. Every wait is a poll on tmux capture-pane for an expected screen, with a bounded timeout and a clear failure message naming what was expected and dumping the pane that was actually there. A fixed sleep will pass on an idle host and flake under load — this repo has already de-flaked one wall-clock-assumption test (#36), so do not introduce another.

A skipped terminal test that reports green is worse than no test. If tmux is missing the harness fails loudly. It must not t.Skip.

Isolate $HOME. Scores go to ~/.rogue.scores and saves to a real path; point the harness at a private throwaway HOME per run so it can never touch a real one, and assert on the score file inside it.

Do not weaken the game to make it testable. No new production hooks, no test-only branches in game code, no widening of exported surface. Wizard mode already exists and is the sanctioned lever. If a case genuinely cannot be driven without a production change, stop and report rather than adding one.

Wizard-assisted is not a playthrough. Any case that uses wizard mode to reach a state says so in its name and its output. #8 turns on an honest game completing; a wizard descent must never be able to be mistaken for that evidence.

Start with the cases that are cheap and decisive — death path, save/restore across a real process boundary, ROGUEOPTS, screen integrity. Amulet retrieval is the ambitious one; land the harness with the first four working rather than holding everything for it.

Every defect found gets its own issue. Those are the deliverable, not a nuisance — do not batch them into one, and do not fix them drive-by in this unit.

Unblocked: https://git.eeqj.de/sneak/rgoue/issues/19 landed, so `make build` exists and puts the binary at `./build/rogue` (gitignored, under `build/`). Implementation requirements settled before dispatch. **Determinism over sleeps.** Every wait is a poll on `tmux capture-pane` for an expected screen, with a bounded timeout and a clear failure message naming what was expected and dumping the pane that was actually there. A fixed `sleep` will pass on an idle host and flake under load — this repo has already de-flaked one wall-clock-assumption test (https://git.eeqj.de/sneak/rgoue/issues/36), so do not introduce another. **A skipped terminal test that reports green is worse than no test.** If `tmux` is missing the harness fails loudly. It must not `t.Skip`. **Isolate `$HOME`.** Scores go to `~/.rogue.scores` and saves to a real path; point the harness at a private throwaway `HOME` per run so it can never touch a real one, and assert on the score file inside it. **Do not weaken the game to make it testable.** No new production hooks, no test-only branches in game code, no widening of exported surface. Wizard mode already exists and is the sanctioned lever. If a case genuinely cannot be driven without a production change, stop and report rather than adding one. **Wizard-assisted is not a playthrough.** Any case that uses wizard mode to reach a state says so in its name and its output. https://git.eeqj.de/sneak/rgoue/issues/8 turns on an honest game completing; a wizard descent must never be able to be mistaken for that evidence. Start with the cases that are cheap and decisive — death path, save/restore across a real process boundary, `ROGUEOPTS`, screen integrity. Amulet retrieval is the ambitious one; land the harness with the first four working rather than holding everything for it. Every defect found gets its own issue. Those are the deliverable, not a nuisance — do not batch them into one, and do not fix them drive-by in this unit.

clawbot commented

2026-08-10 16:17:58 +02:00

Plan.

Placement. New package qa/, its own make qa target that depends on build. Not in make test: the harness needs tmux and a built binary, and putting it in test would drag build into make check, which must stay non-file-modifying. GO_PKGS becomes go list ./... | grep -v /qa, so test and cover exclude it automatically instead of by a hand-maintained list, with a guard that fails the target if the list comes back empty. No build tag: a separate package keeps the harness inside golangci-lint run ./..., where a tagged file would silently escape the lint gate.

Synchronisation. One primitive: poll capture-pane -p -N (which yields a stable 24x80 grid) until a predicate holds, bounded. On timeout, fail naming the expectation and dumping the pane. No fixed sleeps anywhere. Sessions are uniquely named per test and killed from t.Cleanup; the pane command is additionally wrapped in timeout so nothing can outlive a hard panic on this shared host. $HOME is a per-test t.TempDir(), passed via tmux new-session -e. Missing tmux is t.Fatal, never a skip.

Cases.

Honest, non-wizard gameplay death: rest until something kills the hero, assert the tombstone renders and the scoreboard screen follows. Verified reachable in ~2s of driving.
-d for the score-file assertion, decoding $HOME/.rogue.scores and checking the entry. Split from case 1 deliberately: a death with an empty purse never places on the scoreboard, because C's score() breaks on amount > scp->sc_score. That is faithful, not a defect, so the score assertion needs a path with a non-zero purse.
Save/restore across a real process boundary: S, y, process exits, relaunch against the file, compare the restored grid to the pre-save grid and assert the save file is gone.
ROGUEOPTS: the o screen for name=, jump, fruit=, plus a behavioural check for terse (terse and non-terse forms of the same message).
Screen integrity: 24x80 with no overflow, status line shape, empty message line at rest, and a deterministic --More-- (eat the starting ration) asserted to absorb non-space input and clear on space.
Wizard-assisted amulet win, named and logged as wizard-assisted. Reaches total_winner via a created amulet and the stairs, and asserts the win banner and the appraisal list. It also asserts no scoreboard entry is written, since wizard mode sets noscore — which is exactly why it cannot stand in for #8.

Defects found while probing get their own issues; none are fixed here.

Plan. **Placement.** New package `qa/`, its own `make qa` target that depends on `build`. Not in `make test`: the harness needs `tmux` and a built binary, and putting it in `test` would drag `build` into `make check`, which must stay non-file-modifying. `GO_PKGS` becomes `go list ./... | grep -v /qa`, so `test` and `cover` exclude it automatically instead of by a hand-maintained list, with a guard that fails the target if the list comes back empty. No build tag: a separate package keeps the harness inside `golangci-lint run ./...`, where a tagged file would silently escape the lint gate. **Synchronisation.** One primitive: poll `capture-pane -p -N` (which yields a stable 24x80 grid) until a predicate holds, bounded. On timeout, fail naming the expectation and dumping the pane. No fixed sleeps anywhere. Sessions are uniquely named per test and killed from `t.Cleanup`; the pane command is additionally wrapped in `timeout` so nothing can outlive a hard panic on this shared host. `$HOME` is a per-test `t.TempDir()`, passed via `tmux new-session -e`. Missing `tmux` is `t.Fatal`, never a skip. **Cases.** 1. Honest, non-wizard gameplay death: rest until something kills the hero, assert the tombstone renders and the scoreboard screen follows. Verified reachable in ~2s of driving. 2. `-d` for the score-file assertion, decoding `$HOME/.rogue.scores` and checking the entry. Split from case 1 deliberately: a death with an empty purse never places on the scoreboard, because C's `score()` breaks on `amount > scp->sc_score`. That is faithful, not a defect, so the score assertion needs a path with a non-zero purse. 3. Save/restore across a real process boundary: `S`, `y`, process exits, relaunch against the file, compare the restored grid to the pre-save grid and assert the save file is gone. 4. `ROGUEOPTS`: the `o` screen for `name=`, `jump`, `fruit=`, plus a behavioural check for `terse` (terse and non-terse forms of the same message). 5. Screen integrity: 24x80 with no overflow, status line shape, empty message line at rest, and a deterministic `--More--` (eat the starting ration) asserted to absorb non-space input and clear on space. 6. Wizard-assisted amulet win, named and logged as wizard-assisted. Reaches `total_winner` via a created amulet and the stairs, and asserts the win banner and the appraisal list. It also asserts no scoreboard entry is written, since wizard mode sets `noscore` — which is exactly why it cannot stand in for https://git.eeqj.de/sneak/rgoue/issues/8. Defects found while probing get their own issues; none are fixed here.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/rgoue#42