ci: re-run make check on every cibuild instead of serving it from the layer cache (closes #115)
All checks were successful
check / check (push) Successful in 58s
All checks were successful
check / check (push) Successful in 58s
`script/cibuild` was plain `docker build .`. The Dockerfile does
`COPY . .` and then `RUN make check`, and Docker invalidates `COPY . .`
only on a content change, so on a byte-identical tree the check layer
was reused and the suite never ran. The script's header comment claimed
that a successful build implies all checks pass, which was false
whenever the cache was warm. Reproduced on this branch's parent: a
second consecutive run returned success in 283 ms with
`#13 [builder 9/10] RUN make check` reported `CACHED`.
That matters more here than in a typical repo. DNS is never mocked in
this repository, so the suite queries live DNS and its outcome varies
with real-world conditions; caching the verdict of a non-deterministic
check replays a stale result in exactly the case where re-running is
most valuable. It is also the gate every PR is verified through.
Fix: declare `ARG CHECK_EPOCH` immediately above the check step and
expand it into the command, with `script/cibuild` passing a fresh
`$(date +%s)` per invocation. The value is referenced inside the `RUN`
rather than merely declared above it because BuildKit derives an
instruction's cache key from the command string after expansion; a bare
`ARG` would leave `RUN make check` byte-identical and still hit the
cache, reintroducing the bug in a form that looks fixed. Placing the ARG
here and no earlier keeps the pinned toolchain installs and
`go mod download` above the invalidation line, so only the check and the
steps after it re-run.
A plain `docker build` without the argument caches as before; nothing
outside the CI entrypoint changes behaviour.
Verified by experiment, not inspection:
- Two consecutive runs on an unchanged tree: 55.2 s and 42.2 s, both
exit 0, with distinct epochs (1786255109, 1786255167). The second run
shows `RUN echo "check epoch: ..." && make check` executing for 36.0 s
and 216 passing tests across all eight packages, while `apk add`, both
pinned `go install` steps, `go mod download`, `COPY go.mod go.sum` and
`COPY . .` all report `CACHED`.
- Negative control: planted `internal/config/zz_negative_control_test.go`
calling `t.Fatal("NEGATIVE-CONTROL-115: planted failure, cache did not
serve this layer")`. The build failed in 24.7 s with exit 1, printing
that exact message and `--- FAIL: TestNegativeControlIssue115`, and the
check step exited with code 2. A cached layer cannot produce a failure
predicted in advance, so this establishes the suite ran. The file was
then removed, `git status` confirmed clean, and the tree built green
again in 48.1 s.
- Total build time 42-55 s against the policy's 5-minute ceiling.
- `make check` green. No pin touched: the `golang` and `alpine` sha256
digests, golangci-lint `c0d3ddc9`, and goimports `009367f5` are
unchanged, and `.golangci.yml` still hashes to `021cc83f4e6f...`.
This commit is contained in:
18
Dockerfile
18
Dockerfile
@@ -15,8 +15,22 @@ RUN go mod download
|
||||
|
||||
COPY . .
|
||||
|
||||
# Run all checks - build fails if any check fails
|
||||
RUN make check
|
||||
# Run all checks - build fails if any check fails.
|
||||
#
|
||||
# CHECK_EPOCH is a cache-busting build argument. Without it, an
|
||||
# unchanged tree leaves this layer's cache key identical and Docker
|
||||
# serves the previous verdict instead of re-running the suite, so the
|
||||
# build reports a green it did not earn. The value is expanded into the
|
||||
# command itself, not merely declared: BuildKit derives each
|
||||
# instruction's cache key from the command string after expansion, so a
|
||||
# bare ARG above an unchanged RUN would still hit the cache.
|
||||
#
|
||||
# Placing the ARG here and nowhere earlier keeps everything above it
|
||||
# (toolchain install, go mod download) cached, so only the check and the
|
||||
# steps after it re-run. script/cibuild passes a fresh value per run; a
|
||||
# plain `docker build` without it caches as before.
|
||||
ARG CHECK_EPOCH
|
||||
RUN echo "check epoch: ${CHECK_EPOCH}" && make check
|
||||
|
||||
# Build the binary
|
||||
RUN make build
|
||||
|
||||
@@ -393,7 +393,10 @@ them. We provide:
|
||||
- `script/check` — run test, lint, and fmt-check
|
||||
- `script/docker` — build the Docker image tagged via
|
||||
`script/projectname`
|
||||
- `script/cibuild` — CI entrypoint: plain `docker build .`
|
||||
- `script/cibuild` — CI entrypoint: `docker build .` with a fresh
|
||||
`CHECK_EPOCH` build argument, so the Dockerfile's `make check` layer
|
||||
is never served from the cache and a green build always means the
|
||||
checks ran on this invocation
|
||||
- `script/precommit` — run by the git pre-commit hook; `go mod tidy`
|
||||
guard, then `script/check`
|
||||
- `script/install-precommit` — install the git pre-commit hook
|
||||
|
||||
10
TODO.md
10
TODO.md
@@ -25,6 +25,16 @@ confirm make check still passes.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-08-09: `script/cibuild` can no longer report a green it did not
|
||||
earn. The Dockerfile declares `ARG CHECK_EPOCH` immediately above the
|
||||
check step and expands it into the `RUN` command, and `script/cibuild`
|
||||
passes a fresh `$(date +%s)` per invocation, so the `make check` layer
|
||||
is always re-executed while the pinned toolchain install and
|
||||
`go mod download` stay cached. Verified by experiment: before the fix
|
||||
a second run on an unchanged tree returned in 283 ms with the check
|
||||
layer `CACHED`; after it the check runs every time, and a deliberately
|
||||
planted always-failing test made the build fail with exactly that
|
||||
test's message
|
||||
- 2026-08-07: golangci-lint bumped to v2.12.2 (commit-pinned installs
|
||||
in `Dockerfile` and `script/bootstrap`); `.golangci.yml` set to the
|
||||
org-standard v2-schema config used across the org's repos
|
||||
|
||||
@@ -1,13 +1,18 @@
|
||||
#!/bin/sh
|
||||
# script/cibuild: run the CI build. The Dockerfile runs make check, so
|
||||
# a successful build implies all checks pass.
|
||||
# script/cibuild: run the CI build. The Dockerfile runs make check, and
|
||||
# the CHECK_EPOCH build argument below is fresh on every invocation, so
|
||||
# the check layer is never served from the Docker layer cache: a
|
||||
# successful build means the checks were executed and passed on this
|
||||
# run, not on some earlier one. Only the check step and the steps after
|
||||
# it are invalidated; the toolchain install and go mod download stay
|
||||
# cached.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
docker build .
|
||||
docker build --build-arg CHECK_EPOCH="$(date +%s)" .
|
||||
}
|
||||
|
||||
main "$@"
|
||||
|
||||
Reference in New Issue
Block a user