GOPATH is /home/builder/go, so the build stage's RUN chown -R builder:builder /src /home/builder walks the entire Go module cache the dependency download just populated, not just the sources. And the layer sits directly below COPY . ., so any source change invalidates it — it is paid on every branch build and every non-no-op CI run, not only on a cold cache.
Timings, all on this shared host under varying concurrent load, so treat the absolute numbers as noisy:
build
total
chown
unchanged tree (layer CACHED)
2m13s
—
one source file changed
2m17s
76.9s
one source file changed
4m29s
203.7s
cold (--no-cache-filter=builder)
5m14s
210.1s
main, cold, for comparison
5m03s
209.4s
The chown alone ranged 77s to 210s across those runs, and that spread accounts for essentially the whole spread in the totals — it is the single largest layer in every build where it is not cached, and it is what takes a loaded-host cold build past the five-minute policy ceiling.
The intent of the chown is right and must be preserved: the build never runs as root, and the unprivileged user needs to own the sources and the Go caches. The cost is incidental to how that ownership is established. Options worth weighing: COPY --chown=builder:builder on the source copies, creating the cache directories owned by builder up front and running the dependency download as that user, or moving GOMODCACHE out from under the chowned tree.
Definition of done
The chown is no longer re-walking the module cache on a source change. Report the timing of a build after a one-file change and of a cold build (docker build --no-cache-filter=builder — never docker builder prune), both comfortably under the ceiling.
The build still never runs as root: make check executes as builder, and the suite run in the image as --user 0:0 still FAILS TestScanHardlinkRunFailsTogether. Disable the Go test cache when checking this — as root the shared cache returns ok ... (cached) and proves nothing.
Builds on an unchanged tree still serve the dependency and bootstrap layers CACHED while the CHECK_EPOCH gate layers execute.
Measured while verifying #42, and predates it.
`GOPATH` is `/home/builder/go`, so the build stage's `RUN chown -R builder:builder /src /home/builder` walks the entire Go module cache the dependency download just populated, not just the sources. And the layer sits directly below `COPY . .`, so **any** source change invalidates it — it is paid on every branch build and every non-no-op CI run, not only on a cold cache.
Timings, all on this shared host under varying concurrent load, so treat the absolute numbers as noisy:
| build | total | `chown` |
| --- | --- | --- |
| unchanged tree (layer `CACHED`) | 2m13s | — |
| one source file changed | 2m17s | 76.9s |
| one source file changed | 4m29s | 203.7s |
| cold (`--no-cache-filter=builder`) | 5m14s | 210.1s |
| `main`, cold, for comparison | 5m03s | 209.4s |
The chown alone ranged 77s to 210s across those runs, and that spread accounts for essentially the whole spread in the totals — it is the single largest layer in every build where it is not cached, and it is what takes a loaded-host cold build past the five-minute policy ceiling.
The intent of the chown is right and must be preserved: the build never runs as root, and the unprivileged user needs to own the sources and the Go caches. The cost is incidental to how that ownership is established. Options worth weighing: `COPY --chown=builder:builder` on the source copies, creating the cache directories owned by `builder` up front and running the dependency download as that user, or moving `GOMODCACHE` out from under the chowned tree.
## Definition of done
1. The chown is no longer re-walking the module cache on a source change. Report the timing of a build after a one-file change and of a cold build (`docker build --no-cache-filter=builder` — never `docker builder prune`), both comfortably under the ceiling.
2. The build still never runs as root: `make check` executes as `builder`, and the suite run in the image as `--user 0:0` still FAILS `TestScanHardlinkRunFailsTogether`. Disable the Go test cache when checking this — as root the shared cache returns `ok ... (cached)` and proves nothing.
3. Builds on an unchanged tree still serve the dependency and bootstrap layers `CACHED` while the `CHECK_EPOCH` gate layers execute.
clawbot
added this to the 1.0.0 milestone 2026-08-09 16:42:31 +02:00
clawbot
changed title from Cold Docker builds exceed the five-minute ceiling: `chown -R` over the module cache costs 210s to `chown -R` over the module cache costs ~205s and re-runs on every source change2026-08-09 16:48:26 +02:00
clawbot
changed title from `chown -R` over the module cache costs ~205s and re-runs on every source change to `chown -R` over the module cache is the largest build layer and re-runs on every source change2026-08-09 16:51:36 +02:00
Fresh measurements from the containerised-lint work
(#47), independently reproduced
by its reviewer: cold-cache make docker is 5m37s, still over the REPO_POLICIES.md five-minute ceiling; warm is 1m16s and CI 1m5s.
Relevant here because that change removed one whole lint run from the
image build (the build stage no longer lints a second time), and the
cold time barely moved — consistent with this issue's diagnosis that chown -R builder:builder /src /home/builder is what dominates, not
the gates.
Fresh measurements from the containerised-lint work
(https://git.eeqj.de/sneak/sfdupes/pulls/47), independently reproduced
by its reviewer: cold-cache `make docker` is 5m37s, still over the
`REPO_POLICIES.md` five-minute ceiling; warm is 1m16s and CI 1m5s.
Relevant here because that change removed one whole lint run from the
image build (the build stage no longer lints a second time), and the
cold time barely moved — consistent with this issue's diagnosis that
`chown -R builder:builder /src /home/builder` is what dominates, not
the gates.
This now keeps next from going green. script/cibuild on a fresh clone of next (705c8729) has not finished in four tries since 00:40 UTC on 4 October, with and without the build cache. Each stalled in RUN chown -R builder:builder /src /home/builder, with no output for 12 to 38 minutes before I stopped it. The last green head, 01ff3bb5, now stalls the same way at the same step, though that step took 82 s there on 3 October. So the cause is this step on the shared host as it is now, not the latest commit. Another session's sfdupes build hung at the same step meanwhile, so the repo's own gate runs are blocked too.
main (c2804d08) has no such step and is not affected.
Top priority: this comes before all other sfdupes work. It meets the critical bar (a red next) and is labelled. To reproduce: clone next fresh and run script/cibuild. Done when that completes and passes, in addition to the definition of done above.
Model: opus-5-5
This now keeps `next` from going green. `script/cibuild` on a fresh clone of `next` (`705c8729`) has not finished in four tries since 00:40 UTC on 4 October, with and without the build cache. Each stalled in `RUN chown -R builder:builder /src /home/builder`, with no output for 12 to 38 minutes before I stopped it. The last green head, `01ff3bb5`, now stalls the same way at the same step, though that step took 82 s there on 3 October. So the cause is this step on the shared host as it is now, not the latest commit. Another session's sfdupes build hung at the same step meanwhile, so the repo's own gate runs are blocked too.
`main` (`c2804d08`) has no such step and is not affected.
Top priority: this comes before all other sfdupes work. It meets the `critical` bar (a red `next`) and is labelled. To reproduce: clone `next` fresh and run `script/cibuild`. Done when that completes and passes, in addition to the definition of done above.
Model: opus-5-5
Plan (implementer's brief). Top priority: this keeps next from going green.
Make ownership cheap instead of walking the caches after the fact. Two plain ways, pick one and say which in the PR: (a) point GOMODCACHE at a directory outside /home/builder that root fills during script/bootstrap and builder only reads (Go reads, never writes, a fully downloaded module cache; GOCACHE stays in builder's home), or (b) create the cache directories owned by builder up front and run the download as builder. Either way copy the sources with COPY --chown=builder:builder . . so nothing re-walks /src, and any remaining chown touches only directories that are small and fixed.
Keep everything the definition of done protects: never build or test as root, the lint-stage ordering edge (COPY --from=lint), the CHECK_EPOCH gate layers executing on an unchanged tree while the dependency and bootstrap layers stay CACHED, and the version stamping on next.
Item 2's root check: run the suite in the image as --user 0:0 with the test cache disabled and confirm TestScanHardlinkRunFailsTogether still fails; remove that container.
Done also means script/cibuild on a fresh clone of the branch completes and passes on this host as it is now. Never run docker builder prune.
Shared host: wrap every Docker build in timeout (about 10 minutes) inside the shared lock (flock -w 3600 /srv/code/tmp/docker-gate.lock timeout -s INT 600 ...) so a stall never holds the lock; a timeout is a result to investigate, not something to retry blindly.
PR body: the one-file-change and cold build times, one line each, as the definition of done asks.
Model: opus-5-5
Plan (implementer's brief). Top priority: this keeps `next` from going green.
- Make ownership cheap instead of walking the caches after the fact. Two plain ways, pick one and say which in the PR: (a) point `GOMODCACHE` at a directory outside `/home/builder` that root fills during `script/bootstrap` and `builder` only reads (Go reads, never writes, a fully downloaded module cache; `GOCACHE` stays in `builder`'s home), or (b) create the cache directories owned by `builder` up front and run the download as `builder`. Either way copy the sources with `COPY --chown=builder:builder . .` so nothing re-walks `/src`, and any remaining `chown` touches only directories that are small and fixed.
- Keep everything the definition of done protects: never build or test as root, the lint-stage ordering edge (`COPY --from=lint`), the `CHECK_EPOCH` gate layers executing on an unchanged tree while the dependency and bootstrap layers stay `CACHED`, and the version stamping on `next`.
- Item 2's root check: run the suite in the image as `--user 0:0` with the test cache disabled and confirm `TestScanHardlinkRunFailsTogether` still fails; remove that container.
- Done also means `script/cibuild` on a fresh clone of the branch completes and passes on this host as it is now. Never run `docker builder prune`.
- Shared host: wrap every Docker build in `timeout` (about 10 minutes) inside the shared lock (`flock -w 3600 /srv/code/tmp/docker-gate.lock timeout -s INT 600 ...`) so a stall never holds the lock; a timeout is a result to investigate, not something to retry blindly.
- PR body: the one-file-change and cold build times, one line each, as the definition of done asks.
Model: opus-5-5
clawbot
self-assigned this 2026-10-04 04:04:53 +02:00
Correction: next was not red. 705c8729 passed script/cibuild on a fresh clone at 02:11 UTC once the host's load dropped, with the chown step at 96 s. A full run of the older head 01ff3bb5 also passed, with that step at 662 s. My earlier runs were stopped before they could finish. The critical label is removed. The issue's own definition of done above stands: on this host the step takes from 80 s to over 10 minutes, depending on load.
Model: opus-5-5
Correction: `next` was not red. `705c8729` passed `script/cibuild` on a fresh clone at 02:11 UTC once the host's load dropped, with the `chown` step at 96 s. A full run of the older head `01ff3bb5` also passed, with that step at 662 s. My earlier runs were stopped before they could finish. The `critical` label is removed. The issue's own definition of done above stands: on this host the step takes from 80 s to over 10 minutes, depending on load.
Model: opus-5-5
clawbot
removed the critical label 2026-10-04 04:12:09 +02:00
Implemented in #77: the Go module cache moves to /go/pkg/mod and stays root's, the sources are copied with --chown, and the one remaining chown covers only the /src directory and builder's home, in a layer cached with bootstrap.
Model: opus-5-5
Implemented in https://git.eeqj.de/sneak/sfdupes/pulls/77: the Go module cache moves to `/go/pkg/mod` and stays root's, the sources are copied with `--chown`, and the one remaining `chown` covers only the `/src` directory and `builder`'s home, in a layer cached with bootstrap.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Measured while verifying #42, and predates it.
GOPATHis/home/builder/go, so the build stage'sRUN chown -R builder:builder /src /home/builderwalks the entire Go module cache the dependency download just populated, not just the sources. And the layer sits directly belowCOPY . ., so any source change invalidates it — it is paid on every branch build and every non-no-op CI run, not only on a cold cache.Timings, all on this shared host under varying concurrent load, so treat the absolute numbers as noisy:
chownCACHED)--no-cache-filter=builder)main, cold, for comparisonThe chown alone ranged 77s to 210s across those runs, and that spread accounts for essentially the whole spread in the totals — it is the single largest layer in every build where it is not cached, and it is what takes a loaded-host cold build past the five-minute policy ceiling.
The intent of the chown is right and must be preserved: the build never runs as root, and the unprivileged user needs to own the sources and the Go caches. The cost is incidental to how that ownership is established. Options worth weighing:
COPY --chown=builder:builderon the source copies, creating the cache directories owned bybuilderup front and running the dependency download as that user, or movingGOMODCACHEout from under the chowned tree.Definition of done
docker build --no-cache-filter=builder— neverdocker builder prune), both comfortably under the ceiling.make checkexecutes asbuilder, and the suite run in the image as--user 0:0still FAILSTestScanHardlinkRunFailsTogether. Disable the Go test cache when checking this — as root the shared cache returnsok ... (cached)and proves nothing.CACHEDwhile theCHECK_EPOCHgate layers execute.Cold Docker builds exceed the five-minute ceiling: `chown -R` over the module cache costs 210sto `chown -R` over the module cache costs ~205s and re-runs on every source change`chown -R` over the module cache costs ~205s and re-runs on every source changeto `chown -R` over the module cache is the largest build layer and re-runs on every source changeFresh measurements from the containerised-lint work
(#47), independently reproduced
by its reviewer: cold-cache
make dockeris 5m37s, still over theREPO_POLICIES.mdfive-minute ceiling; warm is 1m16s and CI 1m5s.Relevant here because that change removed one whole lint run from the
image build (the build stage no longer lints a second time), and the
cold time barely moved — consistent with this issue's diagnosis that
chown -R builder:builder /src /home/builderis what dominates, notthe gates.
This now keeps
nextfrom going green.script/cibuildon a fresh clone ofnext(705c8729) has not finished in four tries since 00:40 UTC on 4 October, with and without the build cache. Each stalled inRUN chown -R builder:builder /src /home/builder, with no output for 12 to 38 minutes before I stopped it. The last green head,01ff3bb5, now stalls the same way at the same step, though that step took 82 s there on 3 October. So the cause is this step on the shared host as it is now, not the latest commit. Another session's sfdupes build hung at the same step meanwhile, so the repo's own gate runs are blocked too.main(c2804d08) has no such step and is not affected.Top priority: this comes before all other sfdupes work. It meets the
criticalbar (a rednext) and is labelled. To reproduce: clonenextfresh and runscript/cibuild. Done when that completes and passes, in addition to the definition of done above.Model: opus-5-5
Plan (implementer's brief). Top priority: this keeps
nextfrom going green.GOMODCACHEat a directory outside/home/builderthat root fills duringscript/bootstrapandbuilderonly reads (Go reads, never writes, a fully downloaded module cache;GOCACHEstays inbuilder's home), or (b) create the cache directories owned bybuilderup front and run the download asbuilder. Either way copy the sources withCOPY --chown=builder:builder . .so nothing re-walks/src, and any remainingchowntouches only directories that are small and fixed.COPY --from=lint), theCHECK_EPOCHgate layers executing on an unchanged tree while the dependency and bootstrap layers stayCACHED, and the version stamping onnext.--user 0:0with the test cache disabled and confirmTestScanHardlinkRunFailsTogetherstill fails; remove that container.script/cibuildon a fresh clone of the branch completes and passes on this host as it is now. Never rundocker builder prune.timeout(about 10 minutes) inside the shared lock (flock -w 3600 /srv/code/tmp/docker-gate.lock timeout -s INT 600 ...) so a stall never holds the lock; a timeout is a result to investigate, not something to retry blindly.Model: opus-5-5
Correction:
nextwas not red.705c8729passedscript/cibuildon a fresh clone at 02:11 UTC once the host's load dropped, with thechownstep at 96 s. A full run of the older head01ff3bb5also passed, with that step at 662 s. My earlier runs were stopped before they could finish. Thecriticallabel is removed. The issue's own definition of done above stands: on this host the step takes from 80 s to over 10 minutes, depending on load.Model: opus-5-5
Implemented in #77: the Go module cache moves to
/go/pkg/modand stays root's, the sources are copied with--chown, and the one remainingchowncovers only the/srcdirectory andbuilder's home, in a layer cached with bootstrap.Model: opus-5-5