script/cibuild reports a green it did not earn: make check is served from Docker cache #89

New Issue

clawbot · 2026-08-09T07:39:41+02:00

clawbot commented

2026-08-09 07:39:41 +02:00

Reproduced in this repo, on `3bfbb3f`

script/cibuild runs a bare docker build . with no cache control. The
Dockerfile does COPY . . and then RUN make fmt-check, RUN make lint,
RUN make test. Docker's layer cache keys on the content of the build
context, so on an unchanged tree every check layer is a cache hit, the
suite never executes, and the build still exits 0.

Measured here, back to back, same tree, no changes in between:

first run:   1m10.613s   RUN make test executed   (ok ... mfer 5.658s)
second run:  0m0.624s    exit 0
             #10 [lint 8/8]    RUN make lint    CACHED
             #14 [builder 8/9] RUN make test    CACHED
             #16 [lint 7/8]    RUN make fmt-check CACHED

0.6 seconds, exit code 0, nothing ran. A "green CI" that proves nothing.

Why this matters more here than elsewhere

The Gitea workflow calls script/cibuild, and README.md states the claim
plainly: "a successful build implies all checks pass." That implication is
false as written. Every "docker build green" claim in this repo's PR history
is only as trustworthy as whether the tree happened to differ from a previous
build — which is not something a reviewer can see from the exit code.

It compounds with two open issues:

#67 — script/test runs go test -v --timeout 10s ./... with no
-race and no -cover. A weak test command, cached, is doubly weak.
#62 — the gpg subprocess calls have no enforced deadline, so the
failure this cache masks is precisely the intermittent one.

A cached green over a weak test script is the worst combination: the check
that would have caught the flake is the check that did not run.

Definition of done

script/cibuild cannot report success without actually executing
make fmt-check, make lint, and make test.
Running script/cibuild twice in a row on an unchanged tree executes the
checks both times. Demonstrate it: run it twice, show the second run's
duration and that the check layers are not CACHED.
Dependency layers (base images, go mod download, and the yarn install
in the mdfmt stage once #69 lands) stay cached, so the build does not
regress to a full cold build every time. Docker builds must still complete
in under 5 minutes per policy.
README.md's "a successful build implies all checks pass" is true again,
or reworded to match reality.

Implementation requirements

The known-good fix, filed upstream in the prompts repo as issue #26: add
ARG CHECK_EPOCH in the Dockerfile immediately above the first
check-running RUN, and have script/cibuild pass
--build-arg CHECK_EPOCH="$(date +%s)". The changing ARG invalidates
everything from that point down while leaving the dependency layers above
it cached. Prefer this over --no-cache, which would also discard
go mod download and blow the 5-minute budget.
Place the ARG deliberately. It must sit below go mod download and
above the first RUN make ... in each stage that runs checks. This
Dockerfile has two such stages (lint and builder), and after #69 lands
a third (mdfmt) — all of them need it, or the uninvalidated stage keeps
serving stale greens.
Verify the fix defeats the cache in every check stage, not just the first
one you tested. The failure mode of a partial fix is indistinguishable
from a working one at the exit code.
Do not fix this by disabling the cache wholesale. Measure the build time
after the change and state it in the PR.
Adopt the upstream prompts fix rather than inventing a local variant, so
this repo stays alignable with the canonical template.
Commit title must end with (closes #89).

Note

Do not backfill-invalidate the existing evidence on PR #59 and PR #88 on
account of this. Both were separately verified with cache control
(--no-cache-filter on the check stages for #59, a dedicated cold buildx
builder for #88), and #59's head is being re-verified with a full
--no-cache run before it merges. This issue is about making that rigour
automatic instead of dependent on whoever happens to remember.

## Reproduced in this repo, on `3bfbb3f` `script/cibuild` runs a bare `docker build .` with no cache control. The `Dockerfile` does `COPY . .` and then `RUN make fmt-check`, `RUN make lint`, `RUN make test`. Docker's layer cache keys on the content of the build context, so on an **unchanged tree** every check layer is a cache hit, the suite never executes, and the build still exits 0. Measured here, back to back, same tree, no changes in between: ``` first run: 1m10.613s RUN make test executed (ok ... mfer 5.658s) second run: 0m0.624s exit 0 #10 [lint 8/8] RUN make lint CACHED #14 [builder 8/9] RUN make test CACHED #16 [lint 7/8] RUN make fmt-check CACHED ``` **0.6 seconds, exit code 0, nothing ran.** A "green CI" that proves nothing. ## Why this matters more here than elsewhere The Gitea workflow calls `script/cibuild`, and `README.md` states the claim plainly: "a successful build implies all checks pass." That implication is false as written. Every "docker build green" claim in this repo's PR history is only as trustworthy as whether the tree happened to differ from a previous build — which is not something a reviewer can see from the exit code. It compounds with two open issues: - **#67** — `script/test` runs `go test -v --timeout 10s ./...` with no `-race` and no `-cover`. A weak test command, cached, is doubly weak. - **#62** — the gpg subprocess calls have no enforced deadline, so the failure this cache masks is precisely the intermittent one. A cached green over a weak test script is the worst combination: the check that would have caught the flake is the check that did not run. ## Definition of done - `script/cibuild` cannot report success without actually executing `make fmt-check`, `make lint`, and `make test`. - Running `script/cibuild` twice in a row on an unchanged tree executes the checks **both** times. Demonstrate it: run it twice, show the second run's duration and that the check layers are not `CACHED`. - Dependency layers (base images, `go mod download`, and the `yarn install` in the `mdfmt` stage once #69 lands) stay cached, so the build does not regress to a full cold build every time. Docker builds must still complete in under 5 minutes per policy. - `README.md`'s "a successful build implies all checks pass" is true again, or reworded to match reality. ## Implementation requirements - The known-good fix, filed upstream in the `prompts` repo as issue #26: add `ARG CHECK_EPOCH` in the `Dockerfile` immediately above the first check-running `RUN`, and have `script/cibuild` pass `--build-arg CHECK_EPOCH="$(date +%s)"`. The changing ARG invalidates everything from that point down while leaving the dependency layers above it cached. Prefer this over `--no-cache`, which would also discard `go mod download` and blow the 5-minute budget. - Place the `ARG` deliberately. It must sit **below** `go mod download` and **above** the first `RUN make ...` in each stage that runs checks. This Dockerfile has two such stages (`lint` and `builder`), and after #69 lands a third (`mdfmt`) — all of them need it, or the uninvalidated stage keeps serving stale greens. - Verify the fix defeats the cache in every check stage, not just the first one you tested. The failure mode of a partial fix is indistinguishable from a working one at the exit code. - Do not fix this by disabling the cache wholesale. Measure the build time after the change and state it in the PR. - Adopt the upstream `prompts` fix rather than inventing a local variant, so this repo stays alignable with the canonical template. - Commit title must end with ` (closes #89)`. ## Note Do not backfill-invalidate the existing evidence on PR #59 and PR #88 on account of this. Both were separately verified with cache control (`--no-cache-filter` on the check stages for #59, a dedicated cold buildx builder for #88), and #59's head is being re-verified with a full `--no-cache` run before it merges. This issue is about making that rigour automatic instead of dependent on whoever happens to remember.

clawbot added this to the 1.0.0 milestone 2026-08-09 07:39:41 +02:00

clawbot referenced this issue

2026-08-09 07:41:07 +02:00

Update golangci-lint to v2.12.2 with canonical config (closes #60) #59

clawbot referenced this issue

2026-08-10 16:04:57 +02:00

Update golangci-lint to v2.12.2 with canonical config (closes #60) #59

clawbot referenced this issue

2026-08-10 16:17:01 +02:00

1.0.0 milestone #92

Sign in to join this conversation.