script/test omits -count=1, so Go's test cache can satisfy the gate without running tests #93

Closed
opened 2026-08-09 10:09:55 +02:00 by clawbot · 1 comment
Collaborator

Surfaced by the PR #92 review, which corrected a claim I had been relying
on. This is a fifth distinct way this repo's gate could report a green
it did not earn — the first four being #78, #80, #85, and #88.

The gap

script/test runs:

go test -race -timeout 30s ./...

No -count=1. Go's own test cache is therefore live, and a cached
package prints:

ok  	sneak.berlin/go/vaultik/internal/database	(cached)

That line counts as an ok line. So the evidence signal this repo has
been leaning on — "14 ok lines means the suite really ran" — can be
satisfied by a run in which no test executed at all.

This is one level below the Docker layer cache (#85). Fixing the layer
cache guarantees the RUN make test step re-executes; it does not
guarantee go test inside that step does any work, because GOCACHE
baked into earlier image layers survives into the re-executed step.

Why it has not bitten yet

Every verification recorded during #85, #91, and #92 explicitly counted
(cached) occurrences and found zero, so no landed verdict rests on a
cached test run. That was luck plus discipline, not a property of the
tooling — and relying on every future agent to count (cached) by hand is
the same brittle defence that #88's VOID rule currently depends on.

Also correcting the record

PR #92 argued that differing per-package durations prove real execution
because "a replayed layer reproduces its output byte for byte". That
mechanism is wrong.
Under BUILDKIT_PROGRESS=plain a replayed layer
prints CACHED and no stdout at all, so zero ok lines already rules out
layer replay; durations add nothing there.

Where durations genuinely help is exactly this issue — distinguishing a
real ok pkg 5.8s from a cached ok pkg (cached). The reviewer also
found the primitive is not reliable per-package: internal/pidlock
measured 1.016s on two independently-executed runs, identical to the
millisecond. Applied to the whole vector it is informative; applied to a
single package it is not.

Definition of done

  1. script/test cannot be satisfied by Go's test cache. Preferred:
    add -count=1. If that is judged too costly for the local inner loop,
    the alternative is to keep the cache locally but force -count=1 in
    the containerised/CI path — state which was chosen and why.
  2. Demonstrate it: run make test twice back to back and confirm the
    second run shows zero (cached) markers and real per-package
    durations. Paste both runs.
  3. Confirm the cost. -count=1 disables test-result caching, so warm
    local make test will get slower. Report before/after wall time so
    the trade-off is explicit rather than discovered later.
  4. Check whether the same omission exists in script/check,
    test-coverage, or test-integration (see #69 — that target is
    currently a no-op), and fix consistently or say why not.
  5. script/cibuild exits 0, verified as a genuine run.

Note

Consider this alongside #69, which reviews the -timeout 30s per-package
budget. Both concern script/test, and -count=1 makes every run a cold
test run, which interacts directly with that timeout — doing them
together avoids landing a timeout value tuned against cached runs.

Surfaced by the PR #92 review, which corrected a claim I had been relying on. This is a **fifth** distinct way this repo's gate could report a green it did not earn — the first four being #78, #80, #85, and #88. ## The gap `script/test` runs: ```sh go test -race -timeout 30s ./... ``` No `-count=1`. Go's own test cache is therefore live, and a cached package prints: ``` ok sneak.berlin/go/vaultik/internal/database (cached) ``` That line **counts as an `ok` line**. So the evidence signal this repo has been leaning on — "14 `ok` lines means the suite really ran" — can be satisfied by a run in which no test executed at all. This is one level below the Docker layer cache (#85). Fixing the layer cache guarantees the `RUN make test` step re-executes; it does **not** guarantee `go test` inside that step does any work, because `GOCACHE` baked into earlier image layers survives into the re-executed step. ## Why it has not bitten yet Every verification recorded during #85, #91, and #92 explicitly counted `(cached)` occurrences and found zero, so no landed verdict rests on a cached test run. That was luck plus discipline, not a property of the tooling — and relying on every future agent to count `(cached)` by hand is the same brittle defence that #88's VOID rule currently depends on. ## Also correcting the record PR #92 argued that differing per-package durations prove real execution because "a replayed layer reproduces its output byte for byte". **That mechanism is wrong.** Under `BUILDKIT_PROGRESS=plain` a replayed layer prints `CACHED` and no stdout at all, so zero `ok` lines already rules out layer replay; durations add nothing there. Where durations genuinely help is exactly this issue — distinguishing a real `ok pkg 5.8s` from a cached `ok pkg (cached)`. The reviewer also found the primitive is not reliable per-package: `internal/pidlock` measured 1.016s on two independently-executed runs, identical to the millisecond. Applied to the whole vector it is informative; applied to a single package it is not. ## Definition of done 1. `script/test` cannot be satisfied by Go's test cache. Preferred: add `-count=1`. If that is judged too costly for the local inner loop, the alternative is to keep the cache locally but force `-count=1` in the containerised/CI path — state which was chosen and why. 2. Demonstrate it: run `make test` twice back to back and confirm the second run shows **zero** `(cached)` markers and real per-package durations. Paste both runs. 3. Confirm the cost. `-count=1` disables test-result caching, so warm local `make test` will get slower. Report before/after wall time so the trade-off is explicit rather than discovered later. 4. Check whether the same omission exists in `script/check`, `test-coverage`, or `test-integration` (see #69 — that target is currently a no-op), and fix consistently or say why not. 5. `script/cibuild` exits 0, verified as a genuine run. ## Note Consider this alongside #69, which reviews the `-timeout 30s` per-package budget. Both concern `script/test`, and `-count=1` makes every run a cold test run, which interacts directly with that timeout — doing them together avoids landing a timeout value tuned against cached runs.
clawbot added this to the 1.0.0 milestone 2026-08-09 10:09:55 +02:00
Author
Collaborator

Plan

Implementing this together with #69 as a single PR, since both change
script/test and the timeout in #69 has to be chosen against the
uncached behaviour this issue introduces.

Defect reproduced first

Before changing anything, on main at 3f9c2e5, two back-to-back
make test runs:

run wall time ok lines (cached) markers
1 15.9s 14 0
2 0.42s 14 14

Run 2 produced the full 14-ok evidence signal in 0.42 seconds while
executing no test whatsoever. Exactly as described.

What I will do

  1. Add -count=1 to script/test (DoD item 1, the preferred option) —
    unconditionally, not only in the containerised path. Rationale to be
    confirmed against the measured cost below, but the local gate is the
    one the pre-commit hook runs, so leaving it forgeable locally keeps
    the hole where it is most likely to be hit.
  2. DoD item 4: script/check calls script/test, so it inherits the
    fix; test-coverage is a raw inline go test and gets -count=1
    too; test-integration is resolved by #69.
  3. Timeout review per #69 item 3, against a genuinely cold containerised
    run rather than warm host numbers.

One claim I intend to check rather than inherit

The comment on #69 states that "compilation of a large -race build on
a cold cache is charged against that same 30-second budget". I do not
believe that is how go test -timeout works: the flag is passed to the
built test binary as -test.timeout and its clock starts inside
testing.M.Run, after compilation and linking are already finished. The
sibling-repo failure cited was timeout 30 go test ./... — a shell
timeout wrapping the whole invocation, which genuinely does include
compilation. That is a different mechanism from go test -timeout.

I will settle it empirically: a cold-cache containerised run should
report per-package durations close to the warm host durations if
compilation is excluded, and much larger ones if it is included. The
timeout value will be chosen from whatever that shows, and the PR will
state the result either way.

## Plan Implementing this together with #69 as a single PR, since both change `script/test` and the timeout in #69 has to be chosen against the uncached behaviour this issue introduces. ### Defect reproduced first Before changing anything, on `main` at `3f9c2e5`, two back-to-back `make test` runs: | run | wall time | `ok` lines | `(cached)` markers | | --- | --- | --- | --- | | 1 | 15.9s | 14 | 0 | | 2 | **0.42s** | **14** | **14** | Run 2 produced the full 14-`ok` evidence signal in 0.42 seconds while executing no test whatsoever. Exactly as described. ### What I will do 1. Add `-count=1` to `script/test` (DoD item 1, the preferred option) — unconditionally, not only in the containerised path. Rationale to be confirmed against the measured cost below, but the local gate is the one the pre-commit hook runs, so leaving it forgeable locally keeps the hole where it is most likely to be hit. 2. DoD item 4: `script/check` calls `script/test`, so it inherits the fix; `test-coverage` is a raw inline `go test` and gets `-count=1` too; `test-integration` is resolved by #69. 3. Timeout review per #69 item 3, against a genuinely cold containerised run rather than warm host numbers. ### One claim I intend to check rather than inherit The comment on #69 states that "compilation of a large `-race` build on a cold cache is charged against that same 30-second budget". I do not believe that is how `go test -timeout` works: the flag is passed to the built test binary as `-test.timeout` and its clock starts inside `testing.M.Run`, after compilation and linking are already finished. The sibling-repo failure cited was `timeout 30 go test ./...` — a shell `timeout` wrapping the whole invocation, which genuinely does include compilation. That is a different mechanism from `go test -timeout`. I will settle it empirically: a cold-cache containerised run should report per-package durations close to the warm host durations if compilation is excluded, and much larger ones if it is included. The timeout value will be chosen from whatever that shows, and the PR will state the result either way.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/vaultik#93