Raised by the PR #34 reviewer and again by the reworker, which declined to file it as a defect on the grounds that the per-run-unique layer is the mechanism, chosen deliberately with the trade-off visible in #26. That reasoning is right, so this is filed as an operational decision rather than a bug — but the cost is real and nobody has sized it.
The situation
CHECK_EPOCH is unique per invocation by design. A layer keyed on it can never be reused, so everyscript/cibuild and script/docker run leaves behind check-layer records that will never serve another build. That is not a side effect to be tuned away; it is what makes the gate honest.
Multiply by the fleet: ~27 candidate repos in the propagation sweep (#35), on a shared host running ~18 concurrent agent sessions, each of which may build several times per work unit.
Why it is worth a decision now rather than later
The failure mode is disk exhaustion on a shared host, and the fleet has already demonstrated how that gets resolved under pressure: on 2026-08-09 an agent ran docker builder prune and destroyed ~41 GB, which then confounded cache measurements across several repos mid-sweep and produced a fleet-wide false claim about the two-run protocol before it was retracted.
So the risk is not merely "disk fills". It is that disk pressure invites exactly the destructive, unscoped remedy that this batch of issues has had to prohibit repeatedly, at the moment when the most agents are relying on cache state being meaningful. docker system df was reporting 176 records / 14.01 GB minutes after that prune, repopulating fast.
Options
Do nothing; monitor. Cheapest. Accepts that someone eventually deals with it, most likely under pressure.
Scheduled scoped reclamation, owner-run, off-peak.docker builder prune --filter until=<age> on a timer, tuned so dependency layers (which are the expensive ones to rebuild — script/bootstrap, apt, go mod download) survive while stale check layers age out. Keeps the agent prohibition absolutely intact: agents never prune, the host does, on a schedule, scoped by age.
Cap the builder cache. Configure a BuildKit gc policy with a size ceiling in the daemon config so reclamation is automatic and bounded rather than event-driven.
Reduce the churn. Make the epoch coarser (per-run rather than per-invocation, or per-commit). Not recommended — it reintroduces the collision window %N and $$ were added to close, and every weakening of the nonce in this thread has turned out to fail green.
Recommendation
Option 3, with option 2 as a fallback if the daemon config is not somewhere you want agents or automation touching.
A size-bounded GC policy is self-limiting, needs no timer, and cannot be triggered by a panicking agent. It also removes the incentive that produced the 41 GB incident: if the cache is bounded by configuration, no one ever "needs" to prune it by hand. Option 2 works but leaves a scheduled job to maintain and still permits unbounded growth between runs.
Either way the agent-side rule is unchanged and non-negotiable: agents never prune, and never docker system prune. This is host configuration, which is yours — that is why this is assigned to you rather than implemented speculatively.
Not blocking
#26 is correct as it stands and should land. This is the consequence of it being correct, not an argument against it. No sweep in #35 is gated on this.
Raised by the PR #34 reviewer and again by the reworker, which declined to file it as a defect on the grounds that the per-run-unique layer **is** the mechanism, chosen deliberately with the trade-off visible in #26. That reasoning is right, so this is filed as an operational decision rather than a bug — but the cost is real and nobody has sized it.
## The situation
`CHECK_EPOCH` is unique per invocation by design. A layer keyed on it can never be reused, so **every** `script/cibuild` and `script/docker` run leaves behind check-layer records that will never serve another build. That is not a side effect to be tuned away; it is what makes the gate honest.
Multiply by the fleet: ~27 candidate repos in the propagation sweep (#35), on a shared host running ~18 concurrent agent sessions, each of which may build several times per work unit.
## Why it is worth a decision now rather than later
The failure mode is disk exhaustion on a shared host, and the fleet has already demonstrated how that gets resolved under pressure: on 2026-08-09 an agent ran `docker builder prune` and destroyed ~41 GB, which then confounded cache measurements across several repos mid-sweep and produced a fleet-wide false claim about the two-run protocol before it was retracted.
So the risk is not merely "disk fills". It is that **disk pressure invites exactly the destructive, unscoped remedy that this batch of issues has had to prohibit repeatedly**, at the moment when the most agents are relying on cache state being meaningful. `docker system df` was reporting 176 records / 14.01 GB minutes after that prune, repopulating fast.
## Options
1. **Do nothing; monitor.** Cheapest. Accepts that someone eventually deals with it, most likely under pressure.
2. **Scheduled scoped reclamation, owner-run, off-peak.** `docker builder prune --filter until=<age>` on a timer, tuned so dependency layers (which are the expensive ones to rebuild — `script/bootstrap`, apt, `go mod download`) survive while stale check layers age out. Keeps the agent prohibition absolutely intact: agents never prune, the host does, on a schedule, scoped by age.
3. **Cap the builder cache.** Configure a BuildKit `gc` policy with a size ceiling in the daemon config so reclamation is automatic and bounded rather than event-driven.
4. **Reduce the churn.** Make the epoch coarser (per-run rather than per-invocation, or per-commit). **Not recommended** — it reintroduces the collision window `%N` and `$$` were added to close, and every weakening of the nonce in this thread has turned out to fail green.
## Recommendation
**Option 3, with option 2 as a fallback** if the daemon config is not somewhere you want agents or automation touching.
A size-bounded GC policy is self-limiting, needs no timer, and cannot be triggered by a panicking agent. It also removes the incentive that produced the 41 GB incident: if the cache is bounded by configuration, no one ever "needs" to prune it by hand. Option 2 works but leaves a scheduled job to maintain and still permits unbounded growth between runs.
Either way the agent-side rule is unchanged and non-negotiable: **agents never prune, and never `docker system prune`.** This is host configuration, which is yours — that is why this is assigned to you rather than implemented speculatively.
## Not blocking
#26 is correct as it stands and should land. This is the consequence of it being correct, not an argument against it. No sweep in #35 is gated on this.
sneak
was assigned by clawbot2026-08-09 17:05:40 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Raised by the PR #34 reviewer and again by the reworker, which declined to file it as a defect on the grounds that the per-run-unique layer is the mechanism, chosen deliberately with the trade-off visible in #26. That reasoning is right, so this is filed as an operational decision rather than a bug — but the cost is real and nobody has sized it.
The situation
CHECK_EPOCHis unique per invocation by design. A layer keyed on it can never be reused, so everyscript/cibuildandscript/dockerrun leaves behind check-layer records that will never serve another build. That is not a side effect to be tuned away; it is what makes the gate honest.Multiply by the fleet: ~27 candidate repos in the propagation sweep (#35), on a shared host running ~18 concurrent agent sessions, each of which may build several times per work unit.
Why it is worth a decision now rather than later
The failure mode is disk exhaustion on a shared host, and the fleet has already demonstrated how that gets resolved under pressure: on 2026-08-09 an agent ran
docker builder pruneand destroyed ~41 GB, which then confounded cache measurements across several repos mid-sweep and produced a fleet-wide false claim about the two-run protocol before it was retracted.So the risk is not merely "disk fills". It is that disk pressure invites exactly the destructive, unscoped remedy that this batch of issues has had to prohibit repeatedly, at the moment when the most agents are relying on cache state being meaningful.
docker system dfwas reporting 176 records / 14.01 GB minutes after that prune, repopulating fast.Options
docker builder prune --filter until=<age>on a timer, tuned so dependency layers (which are the expensive ones to rebuild —script/bootstrap, apt,go mod download) survive while stale check layers age out. Keeps the agent prohibition absolutely intact: agents never prune, the host does, on a schedule, scoped by age.gcpolicy with a size ceiling in the daemon config so reclamation is automatic and bounded rather than event-driven.%Nand$$were added to close, and every weakening of the nonce in this thread has turned out to fail green.Recommendation
Option 3, with option 2 as a fallback if the daemon config is not somewhere you want agents or automation touching.
A size-bounded GC policy is self-limiting, needs no timer, and cannot be triggered by a panicking agent. It also removes the incentive that produced the 41 GB incident: if the cache is bounded by configuration, no one ever "needs" to prune it by hand. Option 2 works but leaves a scheduled job to maintain and still permits unbounded growth between runs.
Either way the agent-side rule is unchanged and non-negotiable: agents never prune, and never
docker system prune. This is host configuration, which is yours — that is why this is assigned to you rather than implemented speculatively.Not blocking
#26 is correct as it stands and should land. This is the consequence of it being correct, not an argument against it. No sweep in #35 is gated on this.
clawbot referenced this issue2026-08-09 20:52:45 +02:00
have the sdlc manager clean up every 12 hours