1 Commits

Author SHA1 Message Date
ff66ecc0c9 ci: re-run make check on every cibuild instead of serving it from the layer cache (closes #115)
All checks were successful
check / check (push) Successful in 43s
`script/cibuild` was plain `docker build .`. The Dockerfile does
`COPY . .` and then `RUN make check`, and Docker invalidates `COPY . .`
only on a content change, so on a byte-identical tree the check layer
was reused and the suite never ran. The script's header comment claimed
that a successful build implies all checks pass, which was false
whenever the cache was warm. Reproduced on this branch's parent: a
second consecutive run returned success in 283 ms with
`#13 [builder 9/10] RUN make check` reported `CACHED`.

That matters more here than in a typical repo. DNS is never mocked in
this repository, so the suite queries live DNS and its outcome varies
with real-world conditions; caching the verdict of a non-deterministic
check replays a stale result in exactly the case where re-running is
most valuable. It is also the gate every PR is verified through.

Fix: declare `ARG CHECK_EPOCH` immediately above the check step and
expand it into the command, with `script/cibuild` passing a fresh
`$(date +%s%N)` per invocation. A build argument's value participates in
the cache key of later instructions in the stage even when they do not
reference it, so a fresh value busts this layer either way; the value is
expanded into the command deliberately, which makes the invalidation a
property of the command string itself rather than of how a given builder
treats unreferenced args, and surfaces the epoch in the build log as a
diagnostic. Placing the ARG here and no earlier keeps the pinned
toolchain installs and `go mod download` above the invalidation line, so
only the check and the steps after it re-run. The epoch is nanosecond
granular so that two concurrent invocations starting in the same second
cannot share a value.

A plain `docker build` without the argument caches as before; nothing
outside the CI entrypoint changes behaviour.

Verified by experiment, not inspection:

- Two consecutive runs on an unchanged tree: 55.2 s and 42.2 s, both
  exit 0, with distinct epochs. The second run shows
  `RUN echo "check epoch: ..." && make check` executing for 36.0 s and
  216 passing tests across all eight packages, while `apk add`, both
  pinned `go install` steps, `go mod download`, `COPY go.mod go.sum` and
  `COPY . .` all report `CACHED`.
- Negative control: planted `internal/config/zz_negative_control_test.go`
  calling `t.Fatal("NEGATIVE-CONTROL-115: planted failure, cache did not
  serve this layer")`. The build failed in 24.7 s with exit 1, printing
  that exact message and `--- FAIL: TestNegativeControlIssue115`, and the
  check step exited with code 2. A cached layer cannot produce a failure
  predicted in advance, so this establishes the suite ran. The file was
  then removed, `git status` confirmed clean, and the tree built green
  again in 48.1 s.
- Total build time 42-55 s against the policy's 5-minute ceiling.
- `make check` green. No pin touched: the `golang` and `alpine` sha256
  digests, golangci-lint `c0d3ddc9`, and goimports `009367f5` are
  unchanged, and `.golangci.yml` still hashes to `021cc83f4e6f...`.
2026-08-09 06:22:28 +00:00
5 changed files with 50 additions and 57 deletions

View File

@@ -15,8 +15,25 @@ RUN go mod download
COPY . .
# Run all checks - build fails if any check fails
RUN make check
# Run all checks - build fails if any check fails.
#
# CHECK_EPOCH is a cache-busting build argument. Without it, an
# unchanged tree leaves this layer's cache key identical and Docker
# serves the previous verdict instead of re-running the suite, so the
# build reports a green it did not earn. A build argument's value
# participates in the cache key of later instructions in the stage even
# when they do not reference it, so a fresh value busts this layer
# either way. It is expanded into the command deliberately: that makes
# the invalidation a property of the command string itself rather than
# of how a given builder treats unreferenced args, and it surfaces the
# epoch in the build log as a diagnostic.
#
# Placing the ARG here and nowhere earlier keeps everything above it
# (toolchain install, go mod download) cached, so only the check and the
# steps after it re-run. script/cibuild passes a fresh value per run; a
# plain `docker build` without it caches as before.
ARG CHECK_EPOCH
RUN echo "check epoch: ${CHECK_EPOCH}" && make check
# Build the binary
RUN make build

View File

@@ -393,7 +393,10 @@ them. We provide:
- `script/check` — run test, lint, and fmt-check
- `script/docker` — build the Docker image tagged via
`script/projectname`
- `script/cibuild` — CI entrypoint: plain `docker build .`
- `script/cibuild` — CI entrypoint: `docker build .` with a fresh
`CHECK_EPOCH` build argument, so the Dockerfile's `make check` layer
is never served from the cache and a green build always means the
checks ran on this invocation
- `script/precommit` — run by the git pre-commit hook; `go mod tidy`
guard, then `script/check`
- `script/install-precommit` — install the git pre-commit hook

17
TODO.md
View File

@@ -25,13 +25,16 @@ confirm make check still passes.
# Completed Steps
- 2026-08-09: `script/bootstrap` now installs the pinned `golangci-lint`
and `goimports` unconditionally instead of only when the binary is
absent from `PATH`, so the commit pins actually take effect on
already-provisioned machines; it also warns when `PATH` resolves
either tool to a copy outside the directory `go install` writes to.
The `missing` presence check is retained for `git`, `make`, and `go`
(#117)
- 2026-08-09: `script/cibuild` can no longer report a green it did not
earn. The Dockerfile declares `ARG CHECK_EPOCH` immediately above the
check step and expands it into the `RUN` command, and `script/cibuild`
passes a fresh `$(date +%s%N)` per invocation, so the `make check`
layer is always re-executed while the pinned toolchain install and
`go mod download` stay cached. Verified by experiment: before the fix
a second run on an unchanged tree returned in 283 ms with the check
layer `CACHED`; after it the check runs every time, and a deliberately
planted always-failing test made the build fail with exactly that
test's message
- 2026-08-07: golangci-lint bumped to v2.12.2 (commit-pinned installs
in `Dockerfile` and `script/bootstrap`); `.golangci.yml` set to the
org-standard v2-schema config used across the org's repos

View File

@@ -1,13 +1,10 @@
#!/bin/sh
# script/bootstrap: install all dependencies needed to build and develop
# this repo. Base tooling (git, make, go) comes from nix, apt, brew, or
# apk (detected in that order) and is installed only when absent;
# assumes nothing is present. golangci-lint and goimports are always
# (re)installed via `go install` at the same pinned commits the
# Dockerfile uses (never "latest") -- a presence check cannot tell the
# pinned build from an arbitrary one already on PATH, so guarding them
# would make the pins inert. Idempotent either way: running this twice
# succeeds both times and leaves the same result.
# this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes nothing is present.
# golangci-lint and goimports are installed via `go install` at the same
# pinned commits the Dockerfile uses (never "latest").
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -65,31 +62,6 @@ missing() {
! command -v "$1" >/dev/null 2>&1
}
# go_bin_dir: directory `go install` writes binaries to.
go_bin_dir() {
gobin="$(go env GOBIN)"
if [ -n "$gobin" ]; then
echo "$gobin"
else
echo "$(go env GOPATH)/bin"
fi
}
# warn_if_shadowed <tool> <dir>: the pinned build was just installed
# into <dir>. If PATH resolves <tool> anywhere else, that other copy is
# what `make lint` and `make fmt` will actually run, and it is not the
# pinned version. Warn loudly rather than failing, since the fix is the
# user's PATH and not anything this script can do.
warn_if_shadowed() {
resolved="$(command -v "$1" 2>/dev/null || true)"
if [ "$resolved" != "$2/$1" ]; then
echo "bootstrap: WARNING: installed pinned $1 to $2/$1, but PATH" >&2
echo "bootstrap: WARNING: resolves $1 to ${resolved:-(not on PATH)};" >&2
echo "bootstrap: WARNING: put $2 first on PATH or lint results will" >&2
echo "bootstrap: WARNING: not match CI." >&2
fi
}
main() {
cd "$ROOT"
@@ -97,17 +69,10 @@ main() {
if missing make; then pkg_install gnumake make make make; fi
if missing go; then pkg_install go golang go go; fi
# Lint/format tools, pinned via go install. These are installed
# unconditionally: `command -v` only proves *some* build is on PATH,
# and a wrong golangci-lint either cannot parse our v2-schema
# .golangci.yml at all or silently disagrees with CI. Installing at
# a fixed commit ref is idempotent and cheap with a warm module
# cache, so there is nothing to save by skipping it.
GOBIN_DIR="$(go_bin_dir)"
go install "$GOLANGCI_LINT_REF"
go install "$GOIMPORTS_REF"
warn_if_shadowed golangci-lint "$GOBIN_DIR"
warn_if_shadowed goimports "$GOBIN_DIR"
# Lint/format tools, pinned via go install (installs into
# "$(go env GOPATH)/bin"; ensure that is on your PATH).
if missing golangci-lint; then go install "$GOLANGCI_LINT_REF"; fi
if missing goimports; then go install "$GOIMPORTS_REF"; fi
go mod download

View File

@@ -1,13 +1,18 @@
#!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs make check, so
# a successful build implies all checks pass.
# script/cibuild: run the CI build. The Dockerfile runs make check, and
# the CHECK_EPOCH build argument below is fresh on every invocation, so
# the check layer is never served from the Docker layer cache: a
# successful build means the checks were executed and passed on this
# run, not on some earlier one. Only the check step and the steps after
# it are invalidated; the toolchain install and go mod download stay
# cached.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
docker build .
docker build --build-arg CHECK_EPOCH="$(date +%s%N)" .
}
main "$@"