test: disable Go's test cache so every run queries live DNS (closes #139)
All checks were successful
check / check (push) Successful in 1m34s

`script/test` did not pass `-count=1`, so on an unchanged tree Go
served the whole suite from its test cache: exit 0 in ~0.2s with every
package marked `(cached)` and not one DNS query made. This repo's suite
exists to exercise live resolution on every run (`TESTING.md`), so that
green asserted nothing — and it is exactly the green used as evidence
that a flakiness fix works, since "run it a few times" stops being
runs after the first.

`-count=1` now disables caching on every invocation.

The conditional verbose rerun that `REPO_POLICIES.md` mandates was
missing at the same spot and is added here rather than left broken: the
primary run had been unconditionally `-v`, which is the failure mode
the policy exists to prevent (unreadable CI and `docker build` logs on
success). Tests now run quiet, and only a failure triggers the `-v`
rerun. The rerun carries `-count=1` too, so it cannot replay a cached
copy of the failure it is meant to diagnose, and its exit status is
discarded in favour of a forced 1: the first failure already proved the
suite broken, so a flake that passes the second time must not turn the
build green.

`-timeout 90s` is untouched. It is a deliberate backstop that must
strictly exceed the 60s hard cap on suite duration.

No special-casing for the Docker build, which also reaches this script
via `RUN make test`: a fresh container's test cache is empty, so
`-count=1` changes nothing there and carving out an exception would
only create a second code path that could drift.

Verified: three back-to-back `make test` runs on an unchanged tree,
zero `(cached)` markers, ~4.0-4.5s wall each (was ~0.2s cached),
comfortably inside the 20s target with `-race` and `-cover` both still
working and coverage percentages unchanged. The rerun-and-still-fail
path was exercised against a purpose-built flaky test that fails once
then passes: quiet failure, verbose rerun that genuinely re-executed,
exit 1 regardless. `make check` green.
This commit is contained in:
2026-08-10 13:48:15 +00:00
parent 87bce43f8d
commit 6f6bf3a65b
4 changed files with 44 additions and 2 deletions

View File

@@ -387,7 +387,11 @@ them. We provide:
plus the git pre-commit hook
- `script/projectname` — print the project name (used for the Docker
image tag)
- `script/test` — run the test suite (race detector, coverage)
- `script/test` — run the test suite (race detector, coverage). Caching
is waived for testing, exactly as it is for linting: `-count=1`
forces every invocation to execute, because the suite queries live
DNS and a cached pass queries nothing. Failures are rerun with `-v`
automatically, and the build fails even if that rerun passes.
- `script/lint` — run golangci-lint, always inside Docker: it builds
`Dockerfile.lint`, which COPYs the repo into the digest-pinned
`golangci-lint` image and lints as a build step, so a successful

View File

@@ -31,4 +31,7 @@ real servers ensures the resolver works correctly in production.
exists for unit-testing other packages that consume the resolver)
- **Do not add `-short` flags** to skip slow tests
- **Do not increase `-timeout`** to hide hanging queries
- **Do not remove `-count=1` from `script/test`** — Go's test cache
replays a previous run's output without querying anything, so a
cached pass is not evidence that live resolution works
- **Do not modify linter configuration** to suppress findings

12
TODO.md
View File

@@ -25,6 +25,18 @@ confirm make check still passes.
# Completed Steps
- 2026-08-10: Go's test cache disabled for `script/test` via `-count=1`,
so every invocation actually executes. A cached pass replays an
earlier run's output without querying DNS at all, which in this repo
means the suite's entire premise goes unexercised while the run
reports green in under a second. The conditional verbose rerun that
`REPO_POLICIES.md` mandates was added at the same time (the primary
run had been unconditionally `-v`): quiet first, `-v` only on
failure, `-count=1` on both, and exit 1 forced regardless of the
rerun's result so a flake passing the second time cannot turn the
build green. `-timeout 90s` left alone as the deliberate backstop
above the 60s hard cap. Uncached suite runs ~4s, well inside the 20s
target
- 2026-08-10: live-DNS test flakiness addressed by robustness rather
than gating, per the owner's ruling on #93: new
`internal/resolver/livedns_test.go` adds a package-wide concurrency

View File

@@ -1,12 +1,35 @@
#!/bin/sh
# script/test: run the test suite.
#
# -count=1 disables Go's test cache, and is load-bearing here. This
# suite queries live DNS on every run by policy (TESTING.md); a cached
# result is a replay of an earlier run's output with no query made at
# all. On an unchanged tree the whole suite would return success in
# under a second having resolved nothing, which makes the repeated-run
# green that is used as evidence for flakiness fixes worthless. Do not
# remove it.
#
# Conditional verbose rerun per REPO_POLICIES.md: run quiet first so
# CI and docker build logs stay readable, and rerun with -v only on
# failure. The rerun also carries -count=1 (a cached replay of the
# failure would show nothing new), and the exit status is forced to 1
# no matter how the rerun ends: the first failure already proved the
# suite broken, so a flaky test that passes the second time must not
# turn the build green.
#
# -timeout 90s is a deliberate backstop above the 60s hard cap on
# suite duration. Do not lower it.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
go test -v -race -timeout 90s -cover ./...
go test -count=1 -race -timeout 90s -cover ./... || {
echo "--- Rerunning with -v for details ---" >&2
go test -count=1 -race -timeout 90s -v ./... || true
exit 1
}
}
main "$@"