sneak
54c43407d4
All checks were successful
check / check (pull_request) Successful in 2m58s
script/test omitted -count=1, so Go's test result cache could satisfy the gate outright. A cached package prints `ok <pkg> (cached)`, and that is an `ok` line: two back-to-back `make test` runs on the parent commit produced the same 14 `ok` lines, the second in 0.42s with every line marked (cached), having executed no test at all. The evidence signal this repo leans on was therefore forgeable. This sits one level below the Docker layer cache #85 addressed: CHECK_EPOCH forces the `RUN make test` step to re-execute, but a GOCACHE baked into an earlier image layer would survive into the re-executed step, so the step can re-run and still do no work. Add -count=1 unconditionally rather than only in the containerised path. The pre-commit hook runs this same script, and a gate that is honest only in CI is dishonest exactly where people rely on it most. test-coverage gets the same flag -- a coverage profile assembled from cached results describes a run that did not happen -- and script/check inherits it by calling script/test. Delete make test-integration rather than making it real (option (b) of issue #69). No file in the repo carries a build tag, so -tags=integration selected nothing extra and the target was an exact duplicate of make test, while internal/vaultik/integration_test.go ran unconditionally on every make test -- the opposite of what a reader of the Makefile would conclude. Tagging a subset was rejected because the entire suite runs in well under a minute, so gating would save seconds in exchange for a build-tag scheme and a second CI path that must be kept wired up; in a repo that has now found five distinct ways for a gate to report an unearned green, a mechanism whose failure mode is "some tests silently stopped running" is a bad trade. Deleting it also means nothing can drop out of CI coverage, since nothing is conditional. The Makefile and README now say plainly that make test runs everything. Raise -timeout from 30s to 120s, after measuring rather than assuming. The standing claim that cold-cache compilation is charged against -timeout is false: -timeout reaches the test binary as -test.timeout and its clock starts inside testing.M.Run, after compilation and linking. A containerised run with an empty GOCACHE spent 46s compiling and still reported per-package durations within noise of a warm host run. The real exposure was margin, not compilation: against the slowest package's worst observed time, 10.2s for internal/database on a cold run, 30s left only 2.9x, thin for a loaded CI runner, and each fresh measurement of that package has come in above the last. -timeout is a hang backstop, not a performance budget, so it should sit far above the slowest legitimate runtime; 120s leaves about 12x while still bounding a hung package, including the verbose rerun, to a few minutes. 120s is a deliberate divergence from REPO_POLICIES.md:192, which mandates a 30-second timeout, and from that file's canonical Go recipe at :212-214. REPO_POLICIES.md is org-canonical and not editable from this repo, so the divergence and its reasoning are recorded in script/test's comment and in TODO.md, and issue #101 proposes amending the policy text upstream. That issue also carries the trade this surfaces: the verbose rerun makes a hung package pay the timeout twice, which at 120s puts a hang-case Docker build over the same policy's five-minute build limit. Both invocations in script/test now share one run_tests function so the quiet run and the verbose rerun cannot drift apart in flags. Measurements and the full verification are recorded once, on the pull request, and deliberately not restated here or in TODO.md.
70 lines
3.3 KiB
Bash
Executable File
70 lines
3.3 KiB
Bash
Executable File
#!/bin/sh
|
|
# script/test: run the test suite. Quiet on success; on failure, rerun
|
|
# verbosely for full diagnostic output (the exit 1 ensures the rerun
|
|
# never turns a failure into a pass).
|
|
set -eu
|
|
|
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|
|
|
# The flags live in one function so the quiet run and the verbose rerun
|
|
# below cannot drift apart. A rerun that used different flags would
|
|
# diagnose a different program than the one that failed.
|
|
#
|
|
# -count=1 is the documented way to bypass Go's test result cache, and
|
|
# it is not optional here. Without it, a package whose inputs are
|
|
# unchanged prints `ok <pkg> (cached)`, and that line is
|
|
# indistinguishable -- to every check this repo performs -- from a
|
|
# package that actually ran. The whole suite reports its full set of
|
|
# `ok` lines in under half a second having executed nothing. That
|
|
# matters beyond the local inner loop: the Dockerfile's `RUN make test`
|
|
# is forced to re-execute by CHECK_EPOCH, but a GOCACHE baked into an
|
|
# earlier image layer survives into the re-executed step, so the step
|
|
# can re-run and still do no work. It is applied unconditionally rather
|
|
# than only in the containerised path because the pre-commit hook runs
|
|
# this same script; a gate that is honest only in CI is dishonest
|
|
# exactly where people lean on it most.
|
|
#
|
|
# -timeout is a hang backstop, not a performance budget: its job is to
|
|
# turn a deadlocked test into a stack dump instead of a wedged CI job,
|
|
# so it wants to sit far above the slowest legitimate runtime, not just
|
|
# above it. It is per test binary and covers test execution only -- the
|
|
# clock starts inside testing.M.Run, after compilation and linking, so
|
|
# build time is not charged against it. (Measured: a containerised run
|
|
# with an empty GOCACHE reports per-package durations within noise of a
|
|
# warm host run. A shell `timeout 30 go test ./...` would include
|
|
# compilation, but that is a different mechanism from this flag.)
|
|
#
|
|
# The 120s value DELIBERATELY DIVERGES from REPO_POLICIES.md:192, which
|
|
# mandates "Add a 30-second timeout", and from that file's canonical Go
|
|
# recipe at :212-214, which uses -timeout 30s. REPO_POLICIES.md is
|
|
# org-canonical and cannot be amended from this repo, so the divergence
|
|
# is recorded here instead, and issue #101 proposes amending the policy
|
|
# text upstream. Do not revert this to 30s without reading #101 first.
|
|
#
|
|
# Why it diverges: the slowest packages are internal/database and
|
|
# internal/vaultik, observed under -race at about 6.4s warm, 8.1s in a
|
|
# cold containerised run on a contended host, and 10.2s in an
|
|
# independent cold run on this same host. The worst case is not tightly
|
|
# characterised -- each fresh measurement has come in above the last --
|
|
# which is itself an argument for generous headroom. Against the 10.2s
|
|
# observation, 30s is only 2.9x: not a safety margin but a flake
|
|
# waiting for a slow day, whose failure mode is a timeout that looks
|
|
# like a real defect. 120s leaves about 12x while still bounding a hung
|
|
# package -- including the verbose rerun below -- to a few minutes. The
|
|
# cost of that choice, also recorded on #101: because of the rerun, a
|
|
# hung package pays the timeout twice.
|
|
run_tests() {
|
|
go test -race -timeout 120s -count=1 "$@" ./...
|
|
}
|
|
|
|
main() {
|
|
cd "$ROOT"
|
|
run_tests || {
|
|
echo "--- Rerunning with -v for details ---"
|
|
run_tests -v
|
|
exit 1
|
|
}
|
|
}
|
|
|
|
main "$@"
|