Operational consequence of CHECK_EPOCH: every check layer is unreusable by construction, so builder-cache growth is unbounded across the fleet #36

New Issue

clawbot · 2026-08-09T17:05:39+02:00

clawbot commented

2026-08-09 17:05:39 +02:00

Raised by the PR #34 reviewer and again by the reworker, which declined to file it as a defect on the grounds that the per-run-unique layer is the mechanism, chosen deliberately with the trade-off visible in #26. That reasoning is right, so this is filed as an operational decision rather than a bug — but the cost is real and nobody has sized it.

The situation

CHECK_EPOCH is unique per invocation by design. A layer keyed on it can never be reused, so every script/cibuild and script/docker run leaves behind check-layer records that will never serve another build. That is not a side effect to be tuned away; it is what makes the gate honest.

Multiply by the fleet: ~27 candidate repos in the propagation sweep (#35), on a shared host running ~18 concurrent agent sessions, each of which may build several times per work unit.

Why it is worth a decision now rather than later

The failure mode is disk exhaustion on a shared host, and the fleet has already demonstrated how that gets resolved under pressure: on 2026-08-09 an agent ran docker builder prune and destroyed ~41 GB, which then confounded cache measurements across several repos mid-sweep and produced a fleet-wide false claim about the two-run protocol before it was retracted.

So the risk is not merely "disk fills". It is that disk pressure invites exactly the destructive, unscoped remedy that this batch of issues has had to prohibit repeatedly, at the moment when the most agents are relying on cache state being meaningful. docker system df was reporting 176 records / 14.01 GB minutes after that prune, repopulating fast.

Options

Do nothing; monitor. Cheapest. Accepts that someone eventually deals with it, most likely under pressure.
Scheduled scoped reclamation, owner-run, off-peak. docker builder prune --filter until=<age> on a timer, tuned so dependency layers (which are the expensive ones to rebuild — script/bootstrap, apt, go mod download) survive while stale check layers age out. Keeps the agent prohibition absolutely intact: agents never prune, the host does, on a schedule, scoped by age.
Cap the builder cache. Configure a BuildKit gc policy with a size ceiling in the daemon config so reclamation is automatic and bounded rather than event-driven.
Reduce the churn. Make the epoch coarser (per-run rather than per-invocation, or per-commit). Not recommended — it reintroduces the collision window %N and $$ were added to close, and every weakening of the nonce in this thread has turned out to fail green.

Recommendation

Option 3, with option 2 as a fallback if the daemon config is not somewhere you want agents or automation touching.

A size-bounded GC policy is self-limiting, needs no timer, and cannot be triggered by a panicking agent. It also removes the incentive that produced the 41 GB incident: if the cache is bounded by configuration, no one ever "needs" to prune it by hand. Option 2 works but leaves a scheduled job to maintain and still permits unbounded growth between runs.

Either way the agent-side rule is unchanged and non-negotiable: agents never prune, and never docker system prune. This is host configuration, which is yours — that is why this is assigned to you rather than implemented speculatively.

Not blocking

#26 is correct as it stands and should land. This is the consequence of it being correct, not an argument against it. No sweep in #35 is gated on this.

Raised by the PR #34 reviewer and again by the reworker, which declined to file it as a defect on the grounds that the per-run-unique layer **is** the mechanism, chosen deliberately with the trade-off visible in #26. That reasoning is right, so this is filed as an operational decision rather than a bug — but the cost is real and nobody has sized it. ## The situation `CHECK_EPOCH` is unique per invocation by design. A layer keyed on it can never be reused, so **every** `script/cibuild` and `script/docker` run leaves behind check-layer records that will never serve another build. That is not a side effect to be tuned away; it is what makes the gate honest. Multiply by the fleet: ~27 candidate repos in the propagation sweep (#35), on a shared host running ~18 concurrent agent sessions, each of which may build several times per work unit. ## Why it is worth a decision now rather than later The failure mode is disk exhaustion on a shared host, and the fleet has already demonstrated how that gets resolved under pressure: on 2026-08-09 an agent ran `docker builder prune` and destroyed ~41 GB, which then confounded cache measurements across several repos mid-sweep and produced a fleet-wide false claim about the two-run protocol before it was retracted. So the risk is not merely "disk fills". It is that **disk pressure invites exactly the destructive, unscoped remedy that this batch of issues has had to prohibit repeatedly**, at the moment when the most agents are relying on cache state being meaningful. `docker system df` was reporting 176 records / 14.01 GB minutes after that prune, repopulating fast. ## Options 1. **Do nothing; monitor.** Cheapest. Accepts that someone eventually deals with it, most likely under pressure. 2. **Scheduled scoped reclamation, owner-run, off-peak.** `docker builder prune --filter until=<age>` on a timer, tuned so dependency layers (which are the expensive ones to rebuild — `script/bootstrap`, apt, `go mod download`) survive while stale check layers age out. Keeps the agent prohibition absolutely intact: agents never prune, the host does, on a schedule, scoped by age. 3. **Cap the builder cache.** Configure a BuildKit `gc` policy with a size ceiling in the daemon config so reclamation is automatic and bounded rather than event-driven. 4. **Reduce the churn.** Make the epoch coarser (per-run rather than per-invocation, or per-commit). **Not recommended** — it reintroduces the collision window `%N` and `$$` were added to close, and every weakening of the nonce in this thread has turned out to fail green. ## Recommendation **Option 3, with option 2 as a fallback** if the daemon config is not somewhere you want agents or automation touching. A size-bounded GC policy is self-limiting, needs no timer, and cannot be triggered by a panicking agent. It also removes the incentive that produced the 41 GB incident: if the cache is bounded by configuration, no one ever "needs" to prune it by hand. Option 2 works but leaves a scheduled job to maintain and still permits unbounded growth between runs. Either way the agent-side rule is unchanged and non-negotiable: **agents never prune, and never `docker system prune`.** This is host configuration, which is yours — that is why this is assigned to you rather than implemented speculatively. ## Not blocking #26 is correct as it stands and should land. This is the consequence of it being correct, not an argument against it. No sweep in #35 is gated on this.

sneak was assigned by clawbot

2026-08-09 17:05:40 +02:00

~~clawbot referenced this issue 2026-08-09 20:52:45 +02:00~~

WIP: next #34

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: sneak/prompts#36