From 6f6bf3a65bbe3cf0ea42dd256a62a0b7e6f27796 Mon Sep 17 00:00:00 2001 From: sneak Date: Mon, 10 Aug 2026 13:48:15 +0000 Subject: [PATCH] test: disable Go's test cache so every run queries live DNS (closes #139) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `script/test` did not pass `-count=1`, so on an unchanged tree Go served the whole suite from its test cache: exit 0 in ~0.2s with every package marked `(cached)` and not one DNS query made. This repo's suite exists to exercise live resolution on every run (`TESTING.md`), so that green asserted nothing — and it is exactly the green used as evidence that a flakiness fix works, since "run it a few times" stops being runs after the first. `-count=1` now disables caching on every invocation. The conditional verbose rerun that `REPO_POLICIES.md` mandates was missing at the same spot and is added here rather than left broken: the primary run had been unconditionally `-v`, which is the failure mode the policy exists to prevent (unreadable CI and `docker build` logs on success). Tests now run quiet, and only a failure triggers the `-v` rerun. The rerun carries `-count=1` too, so it cannot replay a cached copy of the failure it is meant to diagnose, and its exit status is discarded in favour of a forced 1: the first failure already proved the suite broken, so a flake that passes the second time must not turn the build green. `-timeout 90s` is untouched. It is a deliberate backstop that must strictly exceed the 60s hard cap on suite duration. No special-casing for the Docker build, which also reaches this script via `RUN make test`: a fresh container's test cache is empty, so `-count=1` changes nothing there and carving out an exception would only create a second code path that could drift. Verified: three back-to-back `make test` runs on an unchanged tree, zero `(cached)` markers, ~4.0-4.5s wall each (was ~0.2s cached), comfortably inside the 20s target with `-race` and `-cover` both still working and coverage percentages unchanged. The rerun-and-still-fail path was exercised against a purpose-built flaky test that fails once then passes: quiet failure, verbose rerun that genuinely re-executed, exit 1 regardless. `make check` green. --- README.md | 6 +++++- TESTING.md | 3 +++ TODO.md | 12 ++++++++++++ script/test | 25 ++++++++++++++++++++++++- 4 files changed, 44 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 96141f9..8a5e580 100644 --- a/README.md +++ b/README.md @@ -387,7 +387,11 @@ them. We provide: plus the git pre-commit hook - `script/projectname` — print the project name (used for the Docker image tag) -- `script/test` — run the test suite (race detector, coverage) +- `script/test` — run the test suite (race detector, coverage). Caching + is waived for testing, exactly as it is for linting: `-count=1` + forces every invocation to execute, because the suite queries live + DNS and a cached pass queries nothing. Failures are rerun with `-v` + automatically, and the build fails even if that rerun passes. - `script/lint` — run golangci-lint, always inside Docker: it builds `Dockerfile.lint`, which COPYs the repo into the digest-pinned `golangci-lint` image and lints as a build step, so a successful diff --git a/TESTING.md b/TESTING.md index 0acae53..659db13 100644 --- a/TESTING.md +++ b/TESTING.md @@ -31,4 +31,7 @@ real servers ensures the resolver works correctly in production. exists for unit-testing other packages that consume the resolver) - **Do not add `-short` flags** to skip slow tests - **Do not increase `-timeout`** to hide hanging queries +- **Do not remove `-count=1` from `script/test`** — Go's test cache + replays a previous run's output without querying anything, so a + cached pass is not evidence that live resolution works - **Do not modify linter configuration** to suppress findings diff --git a/TODO.md b/TODO.md index 8905768..97286e7 100644 --- a/TODO.md +++ b/TODO.md @@ -25,6 +25,18 @@ confirm make check still passes. # Completed Steps +- 2026-08-10: Go's test cache disabled for `script/test` via `-count=1`, + so every invocation actually executes. A cached pass replays an + earlier run's output without querying DNS at all, which in this repo + means the suite's entire premise goes unexercised while the run + reports green in under a second. The conditional verbose rerun that + `REPO_POLICIES.md` mandates was added at the same time (the primary + run had been unconditionally `-v`): quiet first, `-v` only on + failure, `-count=1` on both, and exit 1 forced regardless of the + rerun's result so a flake passing the second time cannot turn the + build green. `-timeout 90s` left alone as the deliberate backstop + above the 60s hard cap. Uncached suite runs ~4s, well inside the 20s + target - 2026-08-10: live-DNS test flakiness addressed by robustness rather than gating, per the owner's ruling on #93: new `internal/resolver/livedns_test.go` adds a package-wide concurrency diff --git a/script/test b/script/test index 568e3f5..b4ef492 100755 --- a/script/test +++ b/script/test @@ -1,12 +1,35 @@ #!/bin/sh # script/test: run the test suite. +# +# -count=1 disables Go's test cache, and is load-bearing here. This +# suite queries live DNS on every run by policy (TESTING.md); a cached +# result is a replay of an earlier run's output with no query made at +# all. On an unchanged tree the whole suite would return success in +# under a second having resolved nothing, which makes the repeated-run +# green that is used as evidence for flakiness fixes worthless. Do not +# remove it. +# +# Conditional verbose rerun per REPO_POLICIES.md: run quiet first so +# CI and docker build logs stay readable, and rerun with -v only on +# failure. The rerun also carries -count=1 (a cached replay of the +# failure would show nothing new), and the exit status is forced to 1 +# no matter how the rerun ends: the first failure already proved the +# suite broken, so a flaky test that passes the second time must not +# turn the build green. +# +# -timeout 90s is a deliberate backstop above the 60s hard cap on +# suite duration. Do not lower it. set -eu ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" main() { cd "$ROOT" - go test -v -race -timeout 90s -cover ./... + go test -count=1 -race -timeout 90s -cover ./... || { + echo "--- Rerunning with -v for details ---" >&2 + go test -count=1 -race -timeout 90s -v ./... || true + exit 1 + } } main "$@"