Force check layers to execute on every cibuild run (closes #89) #96

Closed
clawbot wants to merge 1 commits from fix/cibuild-check-cache-89 into next
Collaborator

The problem

script/cibuild ran a bare docker build .. The Dockerfile does COPY . .
and then runs the checks, so on an unchanged tree every check layer was a
cache hit — the suite never executed and the build still exited 0.
Issue #89 measured a second run at
0.624s with every check layer CACHED. A green from script/cibuild did not
mean the checks passed, only that they had passed at some point in the past.

The fix

ARG CHECK_EPOCH is declared in every stage that runs a check, below the
dependency layers and immediately above the first check, and script/cibuild
passes a fresh $(date +%s) on each invocation. A changed build arg
invalidates every layer below its declaration, so the checks always execute
while the base images, go mod download and the yarn install in mdfmt
stay cached.

All three check-running stages are covered. ARG is scoped per stage, so a
stage without its own declaration keeps serving a cached pass and is
indistinguishable from a working fix at the exit code:

Stage Checks it runs
lint make fmt-check-go, make lint
mdfmt script/prettier --check
builder make test

The mdfmt stage already exists on next — it landed with
#69 in 5683d0f — so it is patched
here rather than left for later.

--no-cache was deliberately not used: it would also discard go mod download
and the yarn install, for no additional guarantee.

Evidence: two consecutive runs on an unchanged tree

Cache pre-warmed, then script/cibuild run twice back to back with
git status --porcelain empty between them. Both exited 0.

=== RUN 1 ===          === RUN 2 ===
RUN1_EXIT=0            RUN2_EXIT=0
real 12.99             real 12.92

Neither run completes in 0.624s, and both took essentially identical wall
time — the signature of the checks actually running each time.

Full step inventory for run 2 (the unchanged-tree repeat), derived from
BUILDKIT_PROGRESS=plain. Every check layer is EXECUTED; no check layer is
CACHED:

CACHED    # 10  [lint 3/8] COPY go.mod go.sum ./
CACHED    # 11  [lint 4/8] RUN go mod download
CACHED    # 12  [lint 5/8] COPY . .
CACHED    # 14  [mdfmt 4/6] RUN yarn install --frozen-lockfile
CACHED    # 17  [lint 6/8] RUN touch mfer/mf.pb.go
CACHED    # 18  [mdfmt 5/6] COPY . .
EXECUTED  # 19  [mdfmt 6/6] RUN script/prettier --check
EXECUTED  # 20  [lint 7/8] RUN make fmt-check-go
EXECUTED  # 21  [lint 8/8] RUN make lint
CACHED    # 26  [builder  6/10] RUN go mod download
CACHED    # 27  [builder  7/10] COPY . .
CACHED    # 28  [builder  8/10] RUN touch mfer/mf.pb.go
EXECUTED  # 29  [builder  9/10] RUN make test
EXECUTED  # 30  [builder 10/10] RUN cd cmd/mfer && go build ...

Verbatim proof the checks produced fresh output in both runs:

run1: #19 0.194 Checking formatting...        run2: #19 0.194 Checking formatting...
run1: #19 0.419 Checking formatting...        run2: #19 0.421 Checking formatting...
run1: #21 0.230 level=warning msg="The lin... run2: #21 0.241 level=warning msg="The lin...
run1: #29 3.755 --- PASS: TestAppname        run2: #29 3.775 --- PASS: TestAppname

The dependency layers stayed cached exactly as required: go mod download
(#11, #26) and yarn install --frozen-lockfile (#14).

Evidence: the checks are load-bearing

A gofmt violation was appended to cmd/mfer/main.go and script/cibuild re-run:

NEGATIVE_EXIT=1
#20 0.155 gofmt: files need formatting:
#20 ERROR: process "/bin/sh -c make fmt-check-go" did not complete successfully: exit code: 2

The change was then reverted. script/cibuild cannot exit 0 without the checks
having run and passed.

Build time

Well under the 5 minute budget:

Scenario Time
Warm cache, unchanged tree (the CI repeat case) 12.9s
First build in a fresh clone 24.7s
--no-cache, everything rebuilt 26.0s

Measured with /usr/bin/time -p on linux/arm64, Docker 29.2.1, BuildKit v0.24.0.

Notes

  • .gitea/workflows/check.yml invokes script/cibuild, so CI picks this up
    with no workflow change.
  • No linter was weakened or disabled and no linter config was touched.
  • The evidence on PR #59 and
    PR #88 is not backfill-invalidated;
    both were separately verified with explicit cache control.
  • The README.md edit is confined to the four-line Build Status paragraph, to
    minimise collision with
    PR #95, which is in review on
    fix/gpg-subprocess-timeouts. That branch was not touched.
  • Scope note: script/docker (make docker) still runs a plain docker build
    and so can still serve cached checks. It is a local convenience target, not a
    CI entrypoint; #89 is specifically
    about script/cibuild. Say the word and I will extend it there too.

Closes #89

## The problem `script/cibuild` ran a bare `docker build .`. The Dockerfile does `COPY . .` and then runs the checks, so on an unchanged tree every check layer was a cache hit — the suite never executed and the build still exited 0. [Issue #89](https://git.eeqj.de/sneak/mfer/issues/89) measured a second run at 0.624s with every check layer `CACHED`. A green from `script/cibuild` did not mean the checks passed, only that they had passed at some point in the past. ## The fix `ARG CHECK_EPOCH` is declared in every stage that runs a check, below the dependency layers and immediately above the first check, and `script/cibuild` passes a fresh `$(date +%s)` on each invocation. A changed build arg invalidates every layer below its declaration, so the checks always execute while the base images, `go mod download` and the `yarn install` in `mdfmt` stay cached. **All three check-running stages are covered.** `ARG` is scoped per stage, so a stage without its own declaration keeps serving a cached pass and is indistinguishable from a working fix at the exit code: | Stage | Checks it runs | | --- | --- | | `lint` | `make fmt-check-go`, `make lint` | | `mdfmt` | `script/prettier --check` | | `builder` | `make test` | The `mdfmt` stage already exists on `next` — it landed with [#69](https://git.eeqj.de/sneak/mfer/issues/69) in 5683d0f — so it is patched here rather than left for later. `--no-cache` was deliberately not used: it would also discard `go mod download` and the yarn install, for no additional guarantee. ## Evidence: two consecutive runs on an unchanged tree Cache pre-warmed, then `script/cibuild` run twice back to back with `git status --porcelain` empty between them. Both exited 0. ``` === RUN 1 === === RUN 2 === RUN1_EXIT=0 RUN2_EXIT=0 real 12.99 real 12.92 ``` Neither run completes in 0.624s, and both took essentially identical wall time — the signature of the checks actually running each time. Full step inventory for **run 2** (the unchanged-tree repeat), derived from `BUILDKIT_PROGRESS=plain`. Every check layer is `EXECUTED`; no check layer is `CACHED`: ``` CACHED # 10 [lint 3/8] COPY go.mod go.sum ./ CACHED # 11 [lint 4/8] RUN go mod download CACHED # 12 [lint 5/8] COPY . . CACHED # 14 [mdfmt 4/6] RUN yarn install --frozen-lockfile CACHED # 17 [lint 6/8] RUN touch mfer/mf.pb.go CACHED # 18 [mdfmt 5/6] COPY . . EXECUTED # 19 [mdfmt 6/6] RUN script/prettier --check EXECUTED # 20 [lint 7/8] RUN make fmt-check-go EXECUTED # 21 [lint 8/8] RUN make lint CACHED # 26 [builder 6/10] RUN go mod download CACHED # 27 [builder 7/10] COPY . . CACHED # 28 [builder 8/10] RUN touch mfer/mf.pb.go EXECUTED # 29 [builder 9/10] RUN make test EXECUTED # 30 [builder 10/10] RUN cd cmd/mfer && go build ... ``` Verbatim proof the checks produced fresh output in **both** runs: ``` run1: #19 0.194 Checking formatting... run2: #19 0.194 Checking formatting... run1: #19 0.419 Checking formatting... run2: #19 0.421 Checking formatting... run1: #21 0.230 level=warning msg="The lin... run2: #21 0.241 level=warning msg="The lin... run1: #29 3.755 --- PASS: TestAppname run2: #29 3.775 --- PASS: TestAppname ``` The dependency layers stayed cached exactly as required: `go mod download` (#11, #26) and `yarn install --frozen-lockfile` (#14). ## Evidence: the checks are load-bearing A gofmt violation was appended to `cmd/mfer/main.go` and `script/cibuild` re-run: ``` NEGATIVE_EXIT=1 #20 0.155 gofmt: files need formatting: #20 ERROR: process "/bin/sh -c make fmt-check-go" did not complete successfully: exit code: 2 ``` The change was then reverted. `script/cibuild` cannot exit 0 without the checks having run and passed. ## Build time Well under the 5 minute budget: | Scenario | Time | | --- | --- | | Warm cache, unchanged tree (the CI repeat case) | **12.9s** | | First build in a fresh clone | 24.7s | | `--no-cache`, everything rebuilt | 26.0s | Measured with `/usr/bin/time -p` on linux/arm64, Docker 29.2.1, BuildKit v0.24.0. ## Notes - `.gitea/workflows/check.yml` invokes `script/cibuild`, so CI picks this up with no workflow change. - No linter was weakened or disabled and no linter config was touched. - The evidence on [PR #59](https://git.eeqj.de/sneak/mfer/pulls/59) and [PR #88](https://git.eeqj.de/sneak/mfer/pulls/88) is not backfill-invalidated; both were separately verified with explicit cache control. - The `README.md` edit is confined to the four-line Build Status paragraph, to minimise collision with [PR #95](https://git.eeqj.de/sneak/mfer/pulls/95), which is in review on `fix/gpg-subprocess-timeouts`. That branch was not touched. - Scope note: `script/docker` (`make docker`) still runs a plain `docker build` and so can still serve cached checks. It is a local convenience target, not a CI entrypoint; [#89](https://git.eeqj.de/sneak/mfer/issues/89) is specifically about `script/cibuild`. Say the word and I will extend it there too. Closes #89
clawbot added 1 commit 2026-09-03 22:40:02 +02:00
Force check layers to execute on every cibuild run (closes #89)
All checks were successful
check / check (push) Successful in 29s
b7cb8cd9d0
script/cibuild ran a bare "docker build .". The Dockerfile does "COPY . ."
and then runs the checks, so on an unchanged tree every check layer was a
cache hit: the suite never executed and the build still exited 0. A green
from script/cibuild did not mean the checks had passed, only that they had
passed at some point in the past.

Declare "ARG CHECK_EPOCH" in each stage that runs a check, positioned below
the dependency layers and immediately above the first check, and have
script/cibuild pass a fresh "$(date +%s)" on every invocation. A changed
build arg invalidates every layer below its declaration, so the checks
always execute while the base images, "go mod download" and the "yarn
install" in mdfmt stay cached.

All three check-running stages are covered: lint (fmt-check-go, lint),
mdfmt (prettier --check) and builder (test). ARG is scoped per stage, so a
stage without its own declaration would keep serving a cached pass and be
indistinguishable from a working fix at the exit code.

"--no-cache" was not used: it would also discard "go mod download" and the
yarn install, for no additional guarantee.

The README's build-status claim is accurate again and now says why.
clawbot added the needs-review label 2026-09-03 23:50:20 +02:00
clawbot changed title from Force check layers to execute on every cibuild run to Force check layers to execute on every cibuild run (closes #89) 2026-09-04 03:37:04 +02:00
clawbot added merge-ready and removed needs-review labels 2026-09-04 03:37:05 +02:00
sneak was assigned by clawbot 2026-09-04 03:37:05 +02:00
Author
Collaborator

Closed at sneak's instruction, 2026-09-05: this was opened by an agent running on another machine outside the managed fleet, under superseded rules. Reopen if the content is wanted.

Model: fable-5-1

Closed at sneak's instruction, 2026-09-05: this was opened by an agent running on another machine outside the managed fleet, under superseded rules. Reopen if the content is wanted. Model: fable-5-1
clawbot closed this pull request 2026-09-05 11:32:52 +02:00
All checks were successful
check / check (push) Successful in 29s

Pull request closed

Sign in to join this conversation.