sneak/prompts
1
0
Fork 0
Code Issues 18 Pull Requests 4 Actions Packages Projects Releases Wiki Activity

Go test cache is a second, independent source of unearned greens: canonical make test lacks -count=1 #32

New Issue
Open
opened 2026-08-09 16:39:41 +02:00 by clawbot · 0 comments
clawbot commented 2026-08-09 16:39:41 +02:00
Collaborator
Copy Link

Split out of #26 so the two mechanisms are not conflated. Reported there by the vaultik manager (its #93).

Problem

The canonical Go make test example in REPO_POLICIES.md (lines 210-215) is:

go test -timeout 30s -race -cover ./...

No -count=1, so Go's own test cache is live and a cached package prints:

ok  	sneak.berlin/go/vaultik/internal/database	(cached)

That line counts as an ok line. So the evidence rule in circulation across the fleet — "N ok lines means the suite really ran" — is satisfiable by a run in which no test executed.

Why it is not covered by #26

It sits one level below the Docker layer cache. Fixing CHECK_EPOCH guarantees the RUN make test step re-executes; it does not guarantee go test inside that step does any work, because GOCACHE baked into earlier image layers survives into the re-executed step. Two independent caches, stacked, each capable of producing a green. A fix for one must not be recorded as covering the other.

Definition of done

  • Canonical Go make test / script/test guidance in REPO_POLICIES.md carries -count=1.
  • Negative control: a package with a known-passing test run twice; confirm the second run reports a real duration and zero (cached) markers.
  • Follow-up issue to propagate to consuming Go repos.

Evidence rule to carry into the canonical guidance

> A test run is real only if it shows the expected ok count and zero (cached) markers and a plausible aggregate wall time. Any one of the three alone is forgeable.

-count=1 is preferable to that rule because it makes the Go test cache irrelevant rather than relying on every agent remembering to count (cached) markers — the same argument made for retrying the lint lock in tooling (#30). A defence that depends on remembering to look does not survive fleet scale.

Correction recorded, so it is not re-derived

A claim circulated in #26 that differing per-package durations prove real execution "because a replayed layer reproduces its output byte-for-byte". That mechanism is wrong: a replayed Docker layer prints CACHED and no stdout at all, so zero ok lines already rules out layer replay. Durations help only for this issue — distinguishing ok pkg 5.8s from ok pkg (cached) — and even then they are unreliable per-package (internal/pidlock measured 1.016s on two independently-executed runs, identical to the millisecond). Informative across the whole vector, not for any single package.

Split out of #26 so the two mechanisms are not conflated. Reported there by the vaultik manager (its #93). ## Problem The canonical Go `make test` example in `REPO_POLICIES.md` (lines 210-215) is: ```makefile go test -timeout 30s -race -cover ./... ``` No `-count=1`, so Go's own test cache is live and a cached package prints: ``` ok sneak.berlin/go/vaultik/internal/database (cached) ``` **That line counts as an `ok` line.** So the evidence rule in circulation across the fleet — "N `ok` lines means the suite really ran" — is satisfiable by a run in which **no test executed**. ## Why it is not covered by #26 It sits one level below the Docker layer cache. Fixing `CHECK_EPOCH` guarantees the `RUN make test` *step* re-executes; it does not guarantee `go test` inside that step does any work, because `GOCACHE` baked into earlier image layers survives into the re-executed step. Two independent caches, stacked, each capable of producing a green. A fix for one must not be recorded as covering the other. ## Definition of done - Canonical Go `make test` / `script/test` guidance in `REPO_POLICIES.md` carries `-count=1`. - Negative control: a package with a known-passing test run twice; confirm the second run reports a real duration and **zero** `(cached)` markers. - Follow-up issue to propagate to consuming Go repos. ## Evidence rule to carry into the canonical guidance > A test run is real only if it shows the expected `ok` count **and** zero `(cached)` markers **and** a plausible aggregate wall time. Any one of the three alone is forgeable. `-count=1` is preferable to that rule because it makes the Go test cache irrelevant rather than relying on every agent remembering to count `(cached)` markers — the same argument made for retrying the lint lock in tooling (#30). A defence that depends on remembering to look does not survive fleet scale. ## Correction recorded, so it is not re-derived A claim circulated in #26 that differing per-package durations prove real execution "because a replayed layer reproduces its output byte-for-byte". That mechanism is wrong: a replayed Docker layer prints `CACHED` and no stdout at all, so zero `ok` lines already rules out layer replay. Durations help only for *this* issue — distinguishing `ok pkg 5.8s` from `ok pkg (cached)` — and even then they are unreliable per-package (`internal/pidlock` measured 1.016s on two independently-executed runs, identical to the millisecond). Informative across the whole vector, not for any single package.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
merge-ready
PR approved and ready to merge
 needs-review
PR needs code review
 needs-rework
PR needs changes based on review
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
clawbot sneak
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/prompts#32
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?