From ff66ecc0c9c958e915b713a2f2084b59c20bfca6 Mon Sep 17 00:00:00 2001 From: sneak Date: Sun, 9 Aug 2026 06:04:15 +0000 Subject: [PATCH] ci: re-run make check on every cibuild instead of serving it from the layer cache (closes #115) `script/cibuild` was plain `docker build .`. The Dockerfile does `COPY . .` and then `RUN make check`, and Docker invalidates `COPY . .` only on a content change, so on a byte-identical tree the check layer was reused and the suite never ran. The script's header comment claimed that a successful build implies all checks pass, which was false whenever the cache was warm. Reproduced on this branch's parent: a second consecutive run returned success in 283 ms with `#13 [builder 9/10] RUN make check` reported `CACHED`. That matters more here than in a typical repo. DNS is never mocked in this repository, so the suite queries live DNS and its outcome varies with real-world conditions; caching the verdict of a non-deterministic check replays a stale result in exactly the case where re-running is most valuable. It is also the gate every PR is verified through. Fix: declare `ARG CHECK_EPOCH` immediately above the check step and expand it into the command, with `script/cibuild` passing a fresh `$(date +%s%N)` per invocation. A build argument's value participates in the cache key of later instructions in the stage even when they do not reference it, so a fresh value busts this layer either way; the value is expanded into the command deliberately, which makes the invalidation a property of the command string itself rather than of how a given builder treats unreferenced args, and surfaces the epoch in the build log as a diagnostic. Placing the ARG here and no earlier keeps the pinned toolchain installs and `go mod download` above the invalidation line, so only the check and the steps after it re-run. The epoch is nanosecond granular so that two concurrent invocations starting in the same second cannot share a value. A plain `docker build` without the argument caches as before; nothing outside the CI entrypoint changes behaviour. Verified by experiment, not inspection: - Two consecutive runs on an unchanged tree: 55.2 s and 42.2 s, both exit 0, with distinct epochs. The second run shows `RUN echo "check epoch: ..." && make check` executing for 36.0 s and 216 passing tests across all eight packages, while `apk add`, both pinned `go install` steps, `go mod download`, `COPY go.mod go.sum` and `COPY . .` all report `CACHED`. - Negative control: planted `internal/config/zz_negative_control_test.go` calling `t.Fatal("NEGATIVE-CONTROL-115: planted failure, cache did not serve this layer")`. The build failed in 24.7 s with exit 1, printing that exact message and `--- FAIL: TestNegativeControlIssue115`, and the check step exited with code 2. A cached layer cannot produce a failure predicted in advance, so this establishes the suite ran. The file was then removed, `git status` confirmed clean, and the tree built green again in 48.1 s. - Total build time 42-55 s against the policy's 5-minute ceiling. - `make check` green. No pin touched: the `golang` and `alpine` sha256 digests, golangci-lint `c0d3ddc9`, and goimports `009367f5` are unchanged, and `.golangci.yml` still hashes to `021cc83f4e6f...`. --- Dockerfile | 21 +++++++++++++++++++-- README.md | 5 ++++- TODO.md | 10 ++++++++++ script/cibuild | 11 ++++++++--- 4 files changed, 41 insertions(+), 6 deletions(-) diff --git a/Dockerfile b/Dockerfile index ec33b34..1e7faff 100644 --- a/Dockerfile +++ b/Dockerfile @@ -15,8 +15,25 @@ RUN go mod download COPY . . -# Run all checks - build fails if any check fails -RUN make check +# Run all checks - build fails if any check fails. +# +# CHECK_EPOCH is a cache-busting build argument. Without it, an +# unchanged tree leaves this layer's cache key identical and Docker +# serves the previous verdict instead of re-running the suite, so the +# build reports a green it did not earn. A build argument's value +# participates in the cache key of later instructions in the stage even +# when they do not reference it, so a fresh value busts this layer +# either way. It is expanded into the command deliberately: that makes +# the invalidation a property of the command string itself rather than +# of how a given builder treats unreferenced args, and it surfaces the +# epoch in the build log as a diagnostic. +# +# Placing the ARG here and nowhere earlier keeps everything above it +# (toolchain install, go mod download) cached, so only the check and the +# steps after it re-run. script/cibuild passes a fresh value per run; a +# plain `docker build` without it caches as before. +ARG CHECK_EPOCH +RUN echo "check epoch: ${CHECK_EPOCH}" && make check # Build the binary RUN make build diff --git a/README.md b/README.md index 83f9226..2f83929 100644 --- a/README.md +++ b/README.md @@ -393,7 +393,10 @@ them. We provide: - `script/check` — run test, lint, and fmt-check - `script/docker` — build the Docker image tagged via `script/projectname` -- `script/cibuild` — CI entrypoint: plain `docker build .` +- `script/cibuild` — CI entrypoint: `docker build .` with a fresh + `CHECK_EPOCH` build argument, so the Dockerfile's `make check` layer + is never served from the cache and a green build always means the + checks ran on this invocation - `script/precommit` — run by the git pre-commit hook; `go mod tidy` guard, then `script/check` - `script/install-precommit` — install the git pre-commit hook diff --git a/TODO.md b/TODO.md index bc4c519..e5c59b3 100644 --- a/TODO.md +++ b/TODO.md @@ -25,6 +25,16 @@ confirm make check still passes. # Completed Steps +- 2026-08-09: `script/cibuild` can no longer report a green it did not + earn. The Dockerfile declares `ARG CHECK_EPOCH` immediately above the + check step and expands it into the `RUN` command, and `script/cibuild` + passes a fresh `$(date +%s%N)` per invocation, so the `make check` + layer is always re-executed while the pinned toolchain install and + `go mod download` stay cached. Verified by experiment: before the fix + a second run on an unchanged tree returned in 283 ms with the check + layer `CACHED`; after it the check runs every time, and a deliberately + planted always-failing test made the build fail with exactly that + test's message - 2026-08-07: golangci-lint bumped to v2.12.2 (commit-pinned installs in `Dockerfile` and `script/bootstrap`); `.golangci.yml` set to the org-standard v2-schema config used across the org's repos diff --git a/script/cibuild b/script/cibuild index 966f51d..46abc5c 100755 --- a/script/cibuild +++ b/script/cibuild @@ -1,13 +1,18 @@ #!/bin/sh -# script/cibuild: run the CI build. The Dockerfile runs make check, so -# a successful build implies all checks pass. +# script/cibuild: run the CI build. The Dockerfile runs make check, and +# the CHECK_EPOCH build argument below is fresh on every invocation, so +# the check layer is never served from the Docker layer cache: a +# successful build means the checks were executed and passed on this +# run, not on some earlier one. Only the check step and the steps after +# it are invalidated; the toolchain install and go mod download stay +# cached. set -eu ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" main() { cd "$ROOT" - docker build . + docker build --build-arg CHECK_EPOCH="$(date +%s%N)" . } main "$@" -- 2.49.1