Canonical Go make test example omits -count=1, shipping a false green to every repo #44

New Issue

clawbot · 2026-08-10T15:49:17+02:00

clawbot commented

2026-08-10 15:49:17 +02:00

REPO_POLICIES.md lines 210-215 give this as the canonical Go test target:

test:
	@go test -timeout 30s -race -cover ./... || \
		{ echo "--- Rerunning with -v for details ---"; \
		  go test -timeout 30s -race -v ./...; exit 1; }

Go caches successful test results and replays them for an unchanged tree, so
this target exits 0 having executed no test. Observed in sneak/cattbox, which
copied it: a script/precommit run exited 0 with
ok sneak.berlin/go/cattbox/cmd/cattbox (cached) for every package, in 0.235s.

That is the same false-green class #40
is eliminating for the linter, one gate over — and the pre-commit hook is the
gate a person leans on most, where a cached pass is indistinguishable from a
real one.

Both invocations need the flag. The rerun especially: without it a failing run
replays from cache instead of reproducing the failure, which is exactly when the
output matters.

Proposed replacement:

test:
	@go test -count=1 -timeout 30s -race -cover ./... || \
		{ echo "--- Rerunning with -v for details ---"; \
		  go test -count=1 -timeout 30s -race -v ./...; exit 1; }

-count=1 defeats the test result cache only; the build cache is untouched,
so this costs the runtime of the suite and no recompilation. Measured on
cattbox: 0.235s replayed against ~1.6s executed, with three consecutive executed
runs flat at 1.668s / 1.708s / 1.556s — a recompile would have made the first
one much longer than the rest.

The generic non-Go template above it (<test-command>) needs no change, but the
Python example is worth a look for the same reason if pytest caching is ever
enabled by default there.

Fixed in cattbox by sneak/cattbox#36. A PR against
this repo follows.

`REPO_POLICIES.md` lines 210-215 give this as the canonical Go `test` target: test: @go test -timeout 30s -race -cover ./... || \ { echo "--- Rerunning with -v for details ---"; \ go test -timeout 30s -race -v ./...; exit 1; } Go caches successful test results and replays them for an unchanged tree, so this target exits 0 having executed no test. Observed in `sneak/cattbox`, which copied it: a `script/precommit` run exited 0 with `ok sneak.berlin/go/cattbox/cmd/cattbox (cached)` for every package, in 0.235s. That is the same false-green class https://git.eeqj.de/sneak/prompts/issues/40 is eliminating for the linter, one gate over — and the pre-commit hook is the gate a person leans on most, where a cached pass is indistinguishable from a real one. Both invocations need the flag. The rerun especially: without it a failing run replays from cache instead of reproducing the failure, which is exactly when the output matters. Proposed replacement: test: @go test -count=1 -timeout 30s -race -cover ./... || \ { echo "--- Rerunning with -v for details ---"; \ go test -count=1 -timeout 30s -race -v ./...; exit 1; } `-count=1` defeats the test **result** cache only; the build cache is untouched, so this costs the runtime of the suite and no recompilation. Measured on cattbox: 0.235s replayed against ~1.6s executed, with three consecutive executed runs flat at 1.668s / 1.708s / 1.556s — a recompile would have made the first one much longer than the rest. The generic non-Go template above it (`<test-command>`) needs no change, but the Python example is worth a look for the same reason if pytest caching is ever enabled by default there. Fixed in cattbox by https://git.eeqj.de/sneak/cattbox/issues/36. A PR against this repo follows.

clawbot referenced this issue from a commit

2026-08-10 15:51:36 +02:00

Add -count=1 to the canonical Go make test example (closes #44)

clawbot referenced this issue

2026-08-10 15:51:50 +02:00

Add -count=1 to the canonical Go make test example #45

clawbot commented

2026-08-10 15:52:27 +02:00

PR: #45.

Two notes from writing it. Root REPO_POLICIES.md is a symlink to
prompts/REPO_POLICIES.md, so one edit covers both. And pytest does not
have this defect — .pytest_cache records which tests failed for --lf/--ff
selection, but selected tests still execute; replaying a stored pass needs a
third-party plugin. So no flag was added to the Python example.

Still open after that PR: propagation. Every Go repo that copied this target
has the false green live right now, and fixing the template does not fix them.
That wants its own tracking issue and a PR per repo, as
#40 does for the linter. I have fixed
sneak/cattbox (sneak/cattbox#36); I do not manage
the others and have not touched them.

PR: https://git.eeqj.de/sneak/prompts/pulls/45. Two notes from writing it. Root `REPO_POLICIES.md` is a symlink to `prompts/REPO_POLICIES.md`, so one edit covers both. And pytest does **not** have this defect — `.pytest_cache` records which tests failed for `--lf`/`--ff` selection, but selected tests still execute; replaying a stored pass needs a third-party plugin. So no flag was added to the Python example. Still open after that PR: **propagation.** Every Go repo that copied this target has the false green live right now, and fixing the template does not fix them. That wants its own tracking issue and a PR per repo, as https://git.eeqj.de/sneak/prompts/issues/40 does for the linter. I have fixed `sneak/cattbox` (https://git.eeqj.de/sneak/cattbox/issues/36); I do not manage the others and have not touched them.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/prompts#44