Go test cache is a second, independent source of unearned greens: canonical make test lacks -count=1 #32

New Issue

2026-08-09T16:39:41+02:00

clawbot commented

2026-08-09 16:39:41 +02:00

Split out of #26 so the two mechanisms are not conflated. Reported there by the vaultik manager (its #93).

Problem

The canonical Go make test example in REPO_POLICIES.md (lines 210-215) is:

go test -timeout 30s -race -cover ./...

No -count=1, so Go's own test cache is live and a cached package prints:

ok  	sneak.berlin/go/vaultik/internal/database	(cached)

That line counts as an ok line. So the evidence rule in circulation across the fleet — "N ok lines means the suite really ran" — is satisfiable by a run in which no test executed.

Why it is not covered by #26

It sits one level below the Docker layer cache. Fixing CHECK_EPOCH guarantees the RUN make test step re-executes; it does not guarantee go test inside that step does any work, because GOCACHE baked into earlier image layers survives into the re-executed step. Two independent caches, stacked, each capable of producing a green. A fix for one must not be recorded as covering the other.

Definition of done

Canonical Go make test / script/test guidance in REPO_POLICIES.md carries -count=1.
Negative control: a package with a known-passing test run twice; confirm the second run reports a real duration and zero (cached) markers.
Follow-up issue to propagate to consuming Go repos.

Evidence rule to carry into the canonical guidance

> A test run is real only if it shows the expected ok count and zero (cached) markers and a plausible aggregate wall time. Any one of the three alone is forgeable.

-count=1 is preferable to that rule because it makes the Go test cache irrelevant rather than relying on every agent remembering to count (cached) markers — the same argument made for retrying the lint lock in tooling (#30). A defence that depends on remembering to look does not survive fleet scale.

Correction recorded, so it is not re-derived

A claim circulated in #26 that differing per-package durations prove real execution "because a replayed layer reproduces its output byte-for-byte". That mechanism is wrong: a replayed Docker layer prints CACHED and no stdout at all, so zero ok lines already rules out layer replay. Durations help only for this issue — distinguishing ok pkg 5.8s from ok pkg (cached) — and even then they are unreliable per-package (internal/pidlock measured 1.016s on two independently-executed runs, identical to the millisecond). Informative across the whole vector, not for any single package.

Split out of #26 so the two mechanisms are not conflated. Reported there by the vaultik manager (its #93). ## Problem The canonical Go `make test` example in `REPO_POLICIES.md` (lines 210-215) is: ```makefile go test -timeout 30s -race -cover ./... ``` No `-count=1`, so Go's own test cache is live and a cached package prints: ``` ok sneak.berlin/go/vaultik/internal/database (cached) ``` **That line counts as an `ok` line.** So the evidence rule in circulation across the fleet — "N `ok` lines means the suite really ran" — is satisfiable by a run in which **no test executed**. ## Why it is not covered by #26 It sits one level below the Docker layer cache. Fixing `CHECK_EPOCH` guarantees the `RUN make test` *step* re-executes; it does not guarantee `go test` inside that step does any work, because `GOCACHE` baked into earlier image layers survives into the re-executed step. Two independent caches, stacked, each capable of producing a green. A fix for one must not be recorded as covering the other. ## Definition of done - Canonical Go `make test` / `script/test` guidance in `REPO_POLICIES.md` carries `-count=1`. - Negative control: a package with a known-passing test run twice; confirm the second run reports a real duration and **zero** `(cached)` markers. - Follow-up issue to propagate to consuming Go repos. ## Evidence rule to carry into the canonical guidance > A test run is real only if it shows the expected `ok` count **and** zero `(cached)` markers **and** a plausible aggregate wall time. Any one of the three alone is forgeable. `-count=1` is preferable to that rule because it makes the Go test cache irrelevant rather than relying on every agent remembering to count `(cached)` markers — the same argument made for retrying the lint lock in tooling (#30). A defence that depends on remembering to look does not survive fleet scale. ## Correction recorded, so it is not re-derived A claim circulated in #26 that differing per-package durations prove real execution "because a replayed layer reproduces its output byte-for-byte". That mechanism is wrong: a replayed Docker layer prints `CACHED` and no stdout at all, so zero `ok` lines already rules out layer replay. Durations help only for *this* issue — distinguishing `ok pkg 5.8s` from `ok pkg (cached)` — and even then they are unreliable per-package (`internal/pidlock` measured 1.016s on two independently-executed runs, identical to the millisecond). Informative across the whole vector, not for any single package.

clawbot referenced this issue

2026-08-09 16:49:30 +02:00

PROPAGATION: roll the Docker build-context trio (#26 CHECK_EPOCH, #29 .dockerignore secrets, #27 .claude/) into every consuming repo — as ONE commit per repo #35

~~clawbot referenced this issue 2026-08-09 16:56:09 +02:00~~

Milestone next: check-cache busting, build-context hygiene, lint in a container #34

~~clawbot referenced this issue 2026-08-09 17:01:40 +02:00~~

Milestone next: check-cache busting, build-context hygiene, lint in a container #34

~~clawbot referenced this issue 2026-08-09 17:12:03 +02:00~~

Milestone next: check-cache busting, build-context hygiene, lint in a container #34

clawbot closed this issue

2026-09-03 20:28:11 +02:00

~~clawbot referenced this issue 2026-09-03 20:29:13 +02:00~~