Cache-bust the make check layer via CHECK_EPOCH (closes #23) #30
Reference in New Issue
Block a user
Delete Branch "fix/23-cibuild-check-epoch-v2"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #23.
script/cibuildwas a baredocker build ., and theDockerfiledidCOPY . .thenRUN make check.COPYis keyed on content, so on anunchanged tree Docker served the check layer from cache: the checks never
executed, no Hugo or prettier output appeared, and the build still exited 0.
Baseline, on
mainbefore this changeReproduced here first, so the fix is measured against a demonstrated defect
rather than an assumed one:
exit=0. No Hugo output, no prettier line.The change
immediately below
COPY . ., with bothscript/cibuildandscript/dockergenerating and passing the value identically:
Each element is load-bearing:
cache key — the defect unchanged.
COPY . .. Everything above keeps caching, so thescript/bootstraplayer, which compiles Hugo from source, is not rebuilt.Whole-build
--no-cachewould have discarded it for no benefit.ARGis the empty string, which is also a stablecache key.
RUN. Hardening rather than the fix: thebare unreferenced-
ARGform does work, but expansion makes the cache misscontractual rather than dependent on BuildKit's handling of an unreferenced
ARG, and puts the epoch in the log. The guard also references the value, sothere are two independent invalidation points, not one.
epoch=on its own line, never inlined into the argument list. A commandsubstitution that fails inside an argument does not trip
set -e, so theinline form would quietly pass an empty string and restore the false green.
%Nfor concurrency on this host;$$because busyboxdatedrops%Nsilently and still exits 0.
ARGis stage-scoped and must be redeclared in every stage that runs checks.This image is single-stage, so one declaration is complete.
Guard decision
Kept, and
script/dockerupdated to pass the argument — which is exactlywhat made the earlier
fix/23-cibuild-check-epochattempt unmergeable.The reason is measurement, not taste. An unset
ARGis the empty string andempty is a stable cache key, so without the guard the original defect stays
reachable through a bare
docker build .— the command a reviewer diagnosing abuild is most likely to type by hand, and the one that has already fooled three
reviewers in this repo. Upstream measured a repo that had "landed the fix" and
still got the false green this way, and found documentation-only mitigation
insufficient. The cost is one duplicated pair of lines in
script/docker.Consequence, accepted deliberately: a bare
docker build .now fails. Thatis the point.
Upstream sync
The canonical fix is tracked upstream in the
promptsrepo as its issue #26.It has not merged to that repo's
main—script/cibuildthere is still abare
docker build .. The shape has settled though: it is implemented on thatrepo's
nextbranch and the #26 portion passed independent re-review; that PRremains open only because a later commit on it, for a different issue, needs
rework.
This PR adopts that settled shape verbatim rather than inventing a local
variant. It may need re-syncing once upstream merges. Propagation to
consuming repos is tracked upstream separately.
The upstream change also sweeps prose in
REPO_POLICIES.md,NEW_REPO_CHECKLIST.md,EXISTING_REPO_CHECKLIST.mdandCODE_STYLEGUIDE_GO.md. None of those files exist in this repo yet, so theonly prose target here was
README.md, whose Entrypoints section claimedscript/cibuildisdocker build ..git grep -nF 'docker build'now returnsonly the two canonical
--build-arginvocations plus sites describing the barecommand as failing closed by design.
Verification
All runs are on this branch's tree, from this worktree. No
docker builder prunein any form, and no whole-build--no-cache. Thecounterfactual and guard checks use direct
docker buildinvocations, sincetheir whole purpose is to exercise paths the scripts do not take.
1. Two consecutive
script/cibuildruns, unchanged treeRun 1 — exit 0, 15s wall.
Run 2 — exit 0, 6s wall. Nothing touched in between.
Both runs show two Hugo builds and the prettier line. The epochs differ, and
they differ in the sub-second digits, so
%Nis live on this host.2. Validity control: the bootstrap layer still caches
Run 2 shows
#8 [4/7] RUN script/bootstrap→CACHED, andCOPY . .→
CACHEDas well. This is what makes run 2 evidence rather than noise:had a concurrent eviction landed between the two runs,
script/bootstrapwouldhave recompiled Hugo and the pair would have to be discarded. It also rules out
an accidental whole-build
--no-cache.Note the reading order:
COPY . .beingCACHEDwhile the two layers belowit re-execute is precisely the mechanism working. The content hash is unchanged
— that is the defect — and the epoch below it is what forces the checks.
Wall clock: 15s then 6s. Corroborating only; the layer log is the
evidence.
3. Counterfactual: a constant epoch restores the false green
The literal "revert
script/cibuildtodocker build ." counterfactual is notusable once the guard exists, because that now fails. Passing a constant
instead isolates the varying value as the active ingredient:
Every layer
CACHED, exit 0, no check output — the original defect, reproducedon the fixed
Dockerfile. So it is the per-invocation value doing the work,not merely the fact that the file changed.
4. The guard fails a bare
docker build ., and keeps failingFailed steps are never cached, so this is a hard failure on every invocation,
not just the first. The failing command in the error text names
CHECK_EPOCH,so the diagnosis is in the output.
5. Planted defect: a genuine check failure still fails the build
Appended badly-wrapped text to
README.mdso prettier fails, then ranscript/cibuild:script/cibuildexited 1. Reverted; the tree is clean.6.
make dockerandmake checkmake docker— exit 0,script/bootstrapCACHED, check layer executed(
check epoch: 17862906212619341473565177,All matched files use Prettier code style!),naming to docker.io/library/lora.vegas:latest done. So theguard did not break the local entrypoint.
make checkon the host — green.Not verified
.gitea/workflows/check.ymlinvokesscript/cibuildunchanged, so no workflow edit was needed, but the runnerhas its own builder cache and this evidence is from this host only.
.dockerignorechange lands. That isa follow-up on that work, not something this PR can pre-empt — excluding
.claude/removes a current source of build-context churn, and a fixverified before the context changed is not evidence about the context after.
make checkruns (out of scope), and.dockerignoreis untouched.script/cibuild was a bare `docker build .`, and the Dockerfile did `COPY . .` then `RUN make check`. COPY is keyed on content, so on an unchanged tree Docker served the check layer from cache: the checks never executed, no Hugo or prettier output appeared, and the build still exited 0. A gate that reports success without running is worse than no gate, because it is trusted -- three separate reviewers in this repo have been fooled by it. The Dockerfile now declares `ARG CHECK_EPOCH` immediately below `COPY . .`, guards it, and expands it into the check command: ARG CHECK_EPOCH RUN [ -n "$CHECK_EPOCH" ] || exit 1 RUN echo "check epoch: ${CHECK_EPOCH}" && make check script/cibuild and script/docker both generate the value identically and pass it. Every element is load-bearing: - No default value. A default is a constant, and a constant is a stable cache key -- the defect unchanged. - Placed below `COPY . .`. Everything above keeps caching, so the script/bootstrap layer, which compiles Hugo from source, is not rebuilt. Whole-build `--no-cache` would have discarded it and blown the five-minute budget for no benefit. - The guard. An unset ARG is the empty string, which is also a stable cache key, so without it a bare `docker build .` still collects the false green. Failed steps are never cached, so it fails on every such invocation rather than only the first. This is why script/docker had to be updated too: the guard makes passing the argument mandatory for every entrypoint that builds the image. - The value expanded into the RUN. Hardening rather than the fix: the bare unreferenced-ARG form does work, but expansion makes the cache miss contractual rather than dependent on BuildKit's handling of an unreferenced ARG, and puts the epoch in the build log. The guard also references the value, so there are two independent invalidation points, not one. - `epoch="$(date +%s%N)$$"` on its own line rather than inlined into the argument list. A command substitution that fails inside an argument does not trip `set -e`, so the inline form would quietly pass an empty string and restore the cached false green. `%N` keeps concurrent invocations distinct; `$$` covers busybox date, which drops `%N` silently and still exits 0. ARG is stage-scoped and must be redeclared in every stage that runs checks. This image is single-stage, so one declaration is complete. This is the shape settled upstream in the prompts repo, where it has not merged to main yet, so it may need re-syncing later. Verified: two consecutive script/cibuild runs on an unchanged tree both executed the checks (two Hugo builds and the prettier line in each, 15s then 6s) with `RUN script/bootstrap` and `COPY . .` both CACHED in the second -- the validity control that rules out a cache eviction between them. A constant-epoch counterfactual restored the cached false green, confirming the varying value is what does the work. A bare `docker build .` now fails on the guard, and fails again on immediate repeat. A planted prettier failure failed the build with exit 1. `make docker` and `make check` both pass.Review: PASS
Independent re-verification, this worktree, this branch at
223c520. Nodocker builder prune, no whole-build--no-cache. No blocking findings.The counterfactual reproduces
docker build --build-arg CHECK_EPOCH=<constant> .twice: first exit 0 with thechecks running, second exit 0 in 0.28s with every layer
CACHED—RUN [ -n "..." ] || exit 1CACHED,RUN echo ... && make checkCACHED, noHugo output, no prettier line. The original defect, reproduced on the fixed
Dockerfile. So the per-invocation value is the active ingredient, not the fileedit. The PR's central claim holds.
Two consecutive runs, unchanged tree
Run 1: exit 0, 2.4s. Run 2: exit 0, 4.1s, nothing touched between. Both
show
RUN script/bootstrapCACHED(Hugo not recompiled),COPY . .CACHED,distinct epochs, both Hugo builds, and
All matched files use Prettier code style!. Tree clean before and after.Note this pair is stronger than the one in the PR body: in my run 1 the
COPY . .layer was alreadyCACHED(the author's run 1 rebuilt it at 2.6s), sothe checks executed with the entire context layer served from cache. That is the
exact configuration the defect required, and the check layer still ran.
Everything else verified
docker build .exits 1, and exits 1 again on immediate repeat;the error text names
CHECK_EPOCH.the tree, so the reviewed worktree was never modified) fails the build with
exit 1 at
Dockerfile:45.make dockerexit 0, bootstrapCACHED, check layer executed, image tagged.make checkgreen on the host.git grep -nF 'docker build'returns only thetwo canonical
--build-arginvocations plus three sites describing the barecommand as failing closed.
check.ymlcallsscript/cibuildunchanged;deploy.ymlusesscript/bootstrap+script/testand never touches Docker.successon223c520; mergeable, single commit fast-forward ontomainat
8034fd8; title ends(closes #23);TODO.mdin-commit;make fmtclean; 5 files, no debris; no vendor/attribution references anywhere in the
tree, diff, commit message, trailers, or PR body; terminology clean; scope
respected (
.dockerignoreuntouched,make checkunchanged, single-stage).Disclosures
only. Checked against
sneak/promptsorigin/next:script/cibuild'sepoch=+docker build --build-argpair,script/docker's three lines, andthe
ARG/ guard / checkRUNtriple in theDockerfileare byte-identical.The surrounding comment prose is locally adapted (upstream says
script/check, heremake check; theDockerfilecomment gains twosentences and the
cibuildheader a paragraph). Immaterial, and confirmedthat upstream
origin/mainis still a baredocker build .as the PR says.list_runsreturns 403for this account, so I could only see the aggregate
success(56s, i.e. not asub-second no-op) rather than the runner's per-layer output. This overlaps the
gap the PR already discloses. Structurally there is no longer a path to a
green without execution on the runner:
script/cibuildalways passes theargument, an absent one fails the guard, and a false green would now require
two invocations to collide on both nanosecond and PID.
Makefilehas nocibuildshim, so
script/cibuildis the only invocation path for the CI build. Worthknowing now that
README.mdinstructs building the image through those twoscripts only.
The guard decision is sound and correctly documented in
README.md, theDockerfileheader, and the check comment — the three places someone hitting theexit 1 would look.
PASS accepted.
merge-ready, merging directly —mainis unprotected.This is the PR that makes every other green in this repo mean something, so the
evidence standard mattered more than usual. Two things stand out.
The counterfactual was the author's own idea and it is what actually proves
the fix. My definition of done asked for two consecutive runs both executing
the checks — which, as the author spotted, does not establish causation,
because the tree had just changed. Passing a constant
CHECK_EPOCHtwicereproduced the original false green on the fixed
Dockerfile(0.28s, everylayer
CACHED, no output). That isolates the varying value as the activeingredient. My DoD was weaker than the work; noting that so the next issue of
this kind asks for the counterfactual up front.
The reviewer's reproduction was stronger than the author's. Its first run
had
COPY . .alreadyCACHEDwhere the author's rebuilt it — so the checksran with the entire context layer cached, which is exactly the configuration
the defect required. A reviewer landing in a harder starting state and still
confirming the claim is worth more than a matching re-run.
Accepted, non-blocking:
comment prose is locally adapted (
make checkvs upstream'sscript/check).Immaterial, but the claim is slightly stronger than the fact.
clawbotgets 403 on theActions logs API. Aggregate
successonly. A runner false green would nowrequire two invocations colliding on both nanosecond and PID.
make cibuildshim, soscript/cibuildis the only path tothe CI build. More visible now that
README.mddirects people to build theimage only through those two scripts. Filing separately.
Next: #8's
.dockerignorework, whose definition of done will explicitlyrequire re-running this issue's two-consecutive-run proof afterwards. Excluding
.claude/removes a live source of build-context churn, so it changes thecache behaviour just validated here. That re-verification is the step the
ordering comment flags as most likely to be skipped.