chown -R over the module cache is the largest build layer and re-runs on every source change #43

Open
opened 2026-08-09 16:42:31 +02:00 by clawbot · 1 comment
Collaborator

Measured while verifying #42, and predates it.

GOPATH is /home/builder/go, so the build stage's RUN chown -R builder:builder /src /home/builder walks the entire Go module cache the dependency download just populated, not just the sources. And the layer sits directly below COPY . ., so any source change invalidates it — it is paid on every branch build and every non-no-op CI run, not only on a cold cache.

Timings, all on this shared host under varying concurrent load, so treat the absolute numbers as noisy:

build total chown
unchanged tree (layer CACHED) 2m13s
one source file changed 2m17s 76.9s
one source file changed 4m29s 203.7s
cold (--no-cache-filter=builder) 5m14s 210.1s
main, cold, for comparison 5m03s 209.4s

The chown alone ranged 77s to 210s across those runs, and that spread accounts for essentially the whole spread in the totals — it is the single largest layer in every build where it is not cached, and it is what takes a loaded-host cold build past the five-minute policy ceiling.

The intent of the chown is right and must be preserved: the build never runs as root, and the unprivileged user needs to own the sources and the Go caches. The cost is incidental to how that ownership is established. Options worth weighing: COPY --chown=builder:builder on the source copies, creating the cache directories owned by builder up front and running the dependency download as that user, or moving GOMODCACHE out from under the chowned tree.

Definition of done

  1. The chown is no longer re-walking the module cache on a source change. Report the timing of a build after a one-file change and of a cold build (docker build --no-cache-filter=builder — never docker builder prune), both comfortably under the ceiling.
  2. The build still never runs as root: make check executes as builder, and the suite run in the image as --user 0:0 still FAILS TestScanHardlinkRunFailsTogether. Disable the Go test cache when checking this — as root the shared cache returns ok ... (cached) and proves nothing.
  3. Builds on an unchanged tree still serve the dependency and bootstrap layers CACHED while the CHECK_EPOCH gate layers execute.
Measured while verifying #42, and predates it. `GOPATH` is `/home/builder/go`, so the build stage's `RUN chown -R builder:builder /src /home/builder` walks the entire Go module cache the dependency download just populated, not just the sources. And the layer sits directly below `COPY . .`, so **any** source change invalidates it — it is paid on every branch build and every non-no-op CI run, not only on a cold cache. Timings, all on this shared host under varying concurrent load, so treat the absolute numbers as noisy: | build | total | `chown` | | --- | --- | --- | | unchanged tree (layer `CACHED`) | 2m13s | — | | one source file changed | 2m17s | 76.9s | | one source file changed | 4m29s | 203.7s | | cold (`--no-cache-filter=builder`) | 5m14s | 210.1s | | `main`, cold, for comparison | 5m03s | 209.4s | The chown alone ranged 77s to 210s across those runs, and that spread accounts for essentially the whole spread in the totals — it is the single largest layer in every build where it is not cached, and it is what takes a loaded-host cold build past the five-minute policy ceiling. The intent of the chown is right and must be preserved: the build never runs as root, and the unprivileged user needs to own the sources and the Go caches. The cost is incidental to how that ownership is established. Options worth weighing: `COPY --chown=builder:builder` on the source copies, creating the cache directories owned by `builder` up front and running the dependency download as that user, or moving `GOMODCACHE` out from under the chowned tree. ## Definition of done 1. The chown is no longer re-walking the module cache on a source change. Report the timing of a build after a one-file change and of a cold build (`docker build --no-cache-filter=builder` — never `docker builder prune`), both comfortably under the ceiling. 2. The build still never runs as root: `make check` executes as `builder`, and the suite run in the image as `--user 0:0` still FAILS `TestScanHardlinkRunFailsTogether`. Disable the Go test cache when checking this — as root the shared cache returns `ok ... (cached)` and proves nothing. 3. Builds on an unchanged tree still serve the dependency and bootstrap layers `CACHED` while the `CHECK_EPOCH` gate layers execute.
clawbot added this to the 1.0.0 milestone 2026-08-09 16:42:31 +02:00
clawbot changed title from Cold Docker builds exceed the five-minute ceiling: `chown -R` over the module cache costs 210s to `chown -R` over the module cache costs ~205s and re-runs on every source change 2026-08-09 16:48:26 +02:00
clawbot changed title from `chown -R` over the module cache costs ~205s and re-runs on every source change to `chown -R` over the module cache is the largest build layer and re-runs on every source change 2026-08-09 16:51:36 +02:00
Author
Collaborator

Fresh measurements from the containerised-lint work
(#47), independently reproduced
by its reviewer: cold-cache make docker is 5m37s, still over the
REPO_POLICIES.md five-minute ceiling; warm is 1m16s and CI 1m5s.

Relevant here because that change removed one whole lint run from the
image build (the build stage no longer lints a second time), and the
cold time barely moved — consistent with this issue's diagnosis that
chown -R builder:builder /src /home/builder is what dominates, not
the gates.

Fresh measurements from the containerised-lint work (https://git.eeqj.de/sneak/sfdupes/pulls/47), independently reproduced by its reviewer: cold-cache `make docker` is 5m37s, still over the `REPO_POLICIES.md` five-minute ceiling; warm is 1m16s and CI 1m5s. Relevant here because that change removed one whole lint run from the image build (the build stage no longer lints a second time), and the cold time barely moved — consistent with this issue's diagnosis that `chown -R builder:builder /src /home/builder` is what dominates, not the gates.
Sign in to join this conversation.