From 15aab336ad33d43f74bbe42570ea55905563af9a Mon Sep 17 00:00:00 2001 From: sneak Date: Mon, 16 Nov 2026 21:17:42 +0000 Subject: [PATCH] Check builds use Docker's normal cache (closes #177) Drop the CACHEBUST build argument from script/lint, script/test, script/docker, script/cibuild and every Dockerfile stage. REPO_POLICIES now says check builds use Docker's normal cache: an unchanged tree reuses each step's stored result, which Docker keeps only for steps that succeeded. The rules on placing ARG CACHEBUST and re-vendoring it in one commit are gone; the rule that a planted defect proves a new or changed check stays. The README, the Go style guide and both checklists restated the cache-busting rules and are updated to match. The existing-repo checklist asks that no script pass --no-cache or CACHEBUST. Model: opus-5-5 --- Dockerfile | 14 +-- README.md | 34 ++----- prompts/CODE_STYLEGUIDE_GO.md | 8 +- prompts/EXISTING_REPO_CHECKLIST.md | 62 +++++-------- prompts/NEW_REPO_CHECKLIST.md | 59 ++++-------- prompts/REPO_POLICIES.md | 138 +++++++++-------------------- script/cibuild | 10 +-- script/docker | 10 +-- script/lint | 13 +-- script/test | 13 +-- 10 files changed, 101 insertions(+), 260 deletions(-) diff --git a/Dockerfile b/Dockerfile index e490670..531152d 100644 --- a/Dockerfile +++ b/Dockerfile @@ -2,18 +2,14 @@ # lint` or `script/lint`, which are themselves a docker build and would # recurse into a daemon that does not exist in a build step. # -# In every stage, ARG CACHEBUST comes right after the step that installs -# system packages. script/ passes a new value on every build. That value -# reruns every RUN step below the ARG line, and the install step keeps -# its layer. node and yarn come from the distribution, which is trusted, -# so they need no version and no hash. That yarn only starts the release -# file in .yarn/releases/, so every stage copies that file and -# .yarnrc.yml before it runs script/bootstrap. +# node and yarn come from the distribution, which is trusted, so they +# need no version and no hash. That yarn only starts the release file in +# .yarn/releases/, so every stage copies that file and .yarnrc.yml +# before it runs script/bootstrap. # # alpine 3.24.3, 2026-11-04 FROM alpine@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a AS lint RUN apk add --no-cache git make nodejs-current yarn -ARG CACHEBUST WORKDIR /app @@ -31,7 +27,6 @@ RUN yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always # alpine 3.24.3, 2026-11-04 FROM alpine@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a AS test RUN apk add --no-cache git make nodejs-current yarn -ARG CACHEBUST WORKDIR /app @@ -53,7 +48,6 @@ RUN timeout 90 test/update-policy.sh # alpine 3.24.3, 2026-11-04 FROM alpine@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a RUN apk add --no-cache git make nodejs-current yarn -ARG CACHEBUST WORKDIR /app diff --git a/README.md b/README.md index 37493a3..b1b5934 100644 --- a/README.md +++ b/README.md @@ -116,12 +116,10 @@ alpine. We provide: `script/bootstrap`, then `script/install-precommit` - `script/projectname` — output the project name (our own extension); used by `script/docker` for the image tag -- `script/test` — - `docker build --build-arg CACHEBUST="$(date +%s)-$$" --target test --output type=cacheonly .`, +- `script/test` — `docker build --target test --output type=cacheonly .`, building the `test` phase of the `Dockerfile`, which runs `test/update-policy.sh`, without writing an image -- `script/lint` — - `docker build --build-arg CACHEBUST="$(date +%s)-$$" --target lint --output type=cacheonly .`, +- `script/lint` — `docker build --target lint --output type=cacheonly .`, building the `lint` phase, which runs prettier over the markdown files, without writing an image - `script/fmt` — format all markdown files with prettier (writes; native, not in @@ -129,14 +127,13 @@ alpine. We provide: - `script/fmt-check` — check formatting (read-only; native) - `script/check` — run all checks: `test`, `lint`, `fmt-check` (our own extension); builds no image of its own -- `script/docker` — - `docker build --build-arg CACHEBUST="$(date +%s)-$$" --build-arg VERSION="$version" -t prompts .`, +- `script/docker` — `docker build --build-arg VERSION="$version" -t prompts .`, the tag coming from `script/projectname` (byte-identical across repos) - `script/cibuild` — cd to the repo root, run `script/bootstrap`, run `script/check`, compute `version` from `git describe`, then - `docker build --build-arg CACHEBUST="$(date +%s)-$$" --build-arg VERSION="$version" -t prompts .` - (what CI runs; it bootstraps because CI checks out and runs this alone while - `script/fmt-check` is native) + `docker build --build-arg VERSION="$version" -t prompts .` (what CI runs; it + bootstraps because CI checks out and runs this alone while `script/fmt-check` + is native) - `script/precommit` — run by the git pre-commit hook (our own extension); calls `script/check` - `script/install-precommit` — installs the git pre-commit hook (our own @@ -147,22 +144,9 @@ alpine. We provide: `make hooks` installs the pre-commit hook that runs `script/precommit`. -In every stage of a `Dockerfile`, every step that lints, tests or compiles comes -after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps: - -- the steps that install system packages -- the steps that download Go's modules -- the step that runs `script/bootstrap` to install a Python `test` phase's - packages from `Pipfile.lock` - -A stage with none of these steps declares `ARG CACHEBUST` right after `FROM`. -The `script/` builds pass a new `CACHEBUST` value on every build, which reruns -every `RUN` step after that line and leaves the layers above it and the cache -mounts in place; `--no-cache` would build those layers again and empty the -mounts. This repo's own `Dockerfile` installs git, make, node and yarn right -after `FROM` in every stage, above `ARG CACHEBUST`. `script/bootstrap` installs -the node packages below that line, because check builds keep system packages but -not node packages. +The builds use Docker's normal cache: a step runs again only when its inputs +change. This repo's own `Dockerfile` installs git, make, node and yarn right +after `FROM` in every stage, so a change to `script/` keeps that layer. ## Rationale diff --git a/prompts/CODE_STYLEGUIDE_GO.md b/prompts/CODE_STYLEGUIDE_GO.md index 7cadb9c..b15c2e4 100644 --- a/prompts/CODE_STYLEGUIDE_GO.md +++ b/prompts/CODE_STYLEGUIDE_GO.md @@ -113,13 +113,7 @@ last_modified: 2026-11-16 1. Write a `Dockerfile` for every repo, even if it only runs the tests and linting. It carries the lint and test phases, and the final stage depends on both, so a build makes sure the code is in an able-to-be-compiled state, - linted, and its tests run. Go through `script/cibuild` or `script/docker` - rather than a bare `docker build .`: they pass a new `CACHEBUST` value on - every build, which reruns every step that lints, tests or compiles and keeps - the system packages, Go's modules and build cache, and golangci-lint's - cache; `--no-cache` would throw all of them away. Without a new value an - unchanged tree serves the gate layers from cache and the build reports a - green it never ran. + linted, and its tests run. 1. Every repo must have a `Makefile`. See [Repository Policies](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/REPO_POLICIES.md) diff --git a/prompts/EXISTING_REPO_CHECKLIST.md b/prompts/EXISTING_REPO_CHECKLIST.md index 8c3849d..feb8a5a 100644 --- a/prompts/EXISTING_REPO_CHECKLIST.md +++ b/prompts/EXISTING_REPO_CHECKLIST.md @@ -1,6 +1,6 @@ --- title: Existing Repo Checklist -last_modified: 2026-11-09 +last_modified: 2026-11-16 --- Use this checklist when beginning work in a repo that may not yet conform to our @@ -49,23 +49,11 @@ with your task. `--target` names it, so a lost `COPY --from=` edge leaves `docker build .` passing while the gate never runs. Confirm by planting a violation, not by reading the file. -- [ ] In every stage, every step that lints, tests or compiles comes after - `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps: - - the steps that install system packages - - the steps that download Go's modules - - the step that runs `script/bootstrap` to install a Python `test` phase's - packages from `Pipfile.lock` -- [ ] A stage with none of these steps declares `ARG CACHEBUST` right after - `FROM`. A step that also installs anything else comes after - `ARG CACHEBUST` too. A `script/bootstrap` that also installs node packages - is one. The `script/` builds pass a new `CACHEBUST` value on every build, - which reruns every `RUN` step after that line and leaves the layers above - it and the cache mounts in place; `--no-cache` would build those layers - again and empty the mounts. A stage without the `ARG` line is served from - cache on an unchanged tree, so the scripts and the `ARG` lines are - re-vendored in one commit. +- [ ] No `docker build` in `script/` passes `--no-cache` or a cache-busting + build argument such as `CACHEBUST`, and no `Dockerfile` stage declares + `ARG CACHEBUST`. - [ ] Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`, - `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile + `RUN go mod download`) before `COPY . .`; the steps that compile (`golangci-lint run`, `go test`, `go build`) mount Go's build cache (`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run` also mounts golangci-lint's @@ -74,7 +62,7 @@ with your task. that timed out gets no rerun, as in the canonical Go `Dockerfile` in `REPO_POLICIES.md` - [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does - it above `ARG CACHEBUST`, with `COPY script/ script/`, + it before `COPY . .`, with `COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When `script/bootstrap` runs in that step, it installs nothing beyond these: - system packages @@ -91,9 +79,8 @@ with your task. a package with no wheel for the image stops the install. In that step it installs no Python, because a step above it already did. `Pipfile` names no `python_version`, because `--deploy` stops when the distribution's Python - differs from it. The tests run after `ARG CACHEBUST`, and every other stage - installs its Python packages after it, as in the Python `test` phase example - in `REPO_POLICIES.md` + differs from it. The tests run after `COPY . .`, as in the Python `test` + phase example in `REPO_POLICIES.md` - [ ] The gate phases invoke their tools directly, never through `make lint` or `script/test` — those are themselves a `docker build` and would recurse @@ -223,23 +210,21 @@ with your task. - [ ] README has an **Entrypoints** section documenting the `script/` entrypoints and linking the standard - [ ] `script/lint` and `script/test` each run - `docker build --build-arg CACHEBUST="$(date +%s)-$$" --target --output type=cacheonly .`, - which builds their phase by name and writes no image, and no host - invocation anywhere in the repo can produce a lint verdict — grep for the - linter's own name across `script/`, the `Makefile` and CI config, not just - `script/lint`. A second path is likeliest here: a `make lint-fast`, an - older host-versus-container branch, or a CI step calling the binary - directly. `script/fmt` and `script/fmt-check` are expected hits and stay - on the host. + `docker build --target --output type=cacheonly .`, which builds + their phase by name and writes no image, and no host invocation anywhere + in the repo can produce a lint verdict — grep for the linter's own name + across `script/`, the `Makefile` and CI config, not just `script/lint`. A + second path is likeliest here: a `make lint-fast`, an older + host-versus-container branch, or a CI step calling the binary directly. + `script/fmt` and `script/fmt-check` are expected hits and stay on the + host. - [ ] No `docker build` in `script/` leaves a dangling image behind: `script/lint` and `script/test` write no image, and `script/docker` and `script/cibuild` tag theirs. A build that writes an untagged image leaves one behind on every run, on every host and CI runner. -- [ ] `script/cibuild` runs `script/bootstrap` before `script/check`, and builds - the image with a new `CACHEBUST` value on every build, like every - `docker build` in `script/`. Without the bootstrap the CI run dies in - `script/fmt-check`, which runs the formatter on the host and finds nothing - installed. +- [ ] `script/cibuild` runs `script/bootstrap` before `script/check`. Without + the bootstrap the CI run dies in `script/fmt-check`, which runs the + formatter on the host and finds nothing installed. - [ ] `script/bootstrap` installs node, yarn and the node packages only when `yarn.lock` is in the tree. node and yarn come from the distribution when they are missing. It installs the node packages with @@ -327,14 +312,7 @@ with your task. - [ ] `script/cibuild` succeeds in a fresh clone on a host carrying what CI has. That host is the runner image `docker.gitea.com/runner-images:ubuntu-latest`, which has docker, git and - node 24. Take any yarn off `PATH` first. The run must show that the checks - executed. A sub-second build means nothing ran. So does `CACHED` on a step - that lints or tests, one of the `RUN` steps below `ARG CACHEBUST`. Above - that line, these steps are expected to show `CACHED`: - - the steps that install system packages - - the steps that download Go's modules - - the step that runs `script/bootstrap` to install a Python `test` phase's - packages from `Pipfile.lock` + node 24. Take any yarn off `PATH` first. - [ ] A planted lint violation fails both `make lint` and a plain `docker build .`; revert it afterwards - [ ] Commit and merge fixes before starting your actual task diff --git a/prompts/NEW_REPO_CHECKLIST.md b/prompts/NEW_REPO_CHECKLIST.md index ff769b1..2eb3d48 100644 --- a/prompts/NEW_REPO_CHECKLIST.md +++ b/prompts/NEW_REPO_CHECKLIST.md @@ -1,6 +1,6 @@ --- title: New Repo Checklist -last_modified: 2026-11-09 +last_modified: 2026-11-16 --- Use this checklist when creating a new repository from scratch. Follow the steps @@ -118,23 +118,8 @@ Template files can be fetched from: stage carries a `COPY --from=` of a harmless file from each so the image cannot be built unless both passed. Keep the final stage last: a stage nothing depends on is built only when `--target` names it. - - In every stage, every step that lints, tests or compiles comes after - `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps: - - the steps that install system packages - - the steps that download Go's modules - - the step that runs `script/bootstrap` to install a Python `test` - phase's packages from `Pipfile.lock` - - A stage with none of these steps declares `ARG CACHEBUST` right after - `FROM`. A step that also installs anything else comes after - `ARG CACHEBUST` too. A `script/bootstrap` that also installs node packages - is one. The `script/` builds pass a new `CACHEBUST` value on every build, - which reruns every `RUN` step after that line and leaves the layers above - it and the cache mounts in place; `--no-cache` would build those layers - again and empty the mounts. A stage without the `ARG` line is served from - cache on an unchanged tree, so the scripts and every stage's - `ARG CACHEBUST` line are added in the same commit. - Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`, - `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile + `RUN go mod download`) before `COPY . .`; the steps that compile (`golangci-lint run`, `go test`, `go build`) mount Go's build cache (`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run` also mounts golangci-lint's @@ -143,7 +128,7 @@ Template files can be fetched from: that timed out gets no rerun, as in the canonical Go `Dockerfile` in `REPO_POLICIES.md` - Python: a `test` phase that installs its packages from `Pipfile.lock` does - it above `ARG CACHEBUST`, with `COPY script/ script/`, + it before `COPY . .`, with `COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When `script/bootstrap` runs in that step, it installs nothing beyond these: - system packages @@ -161,8 +146,7 @@ Template files can be fetched from: for the image stops the install. In that step it installs no Python, because a step above it already did. `Pipfile` names no `python_version`, because `--deploy` stops when the distribution's - Python differs from it. The tests run after `ARG CACHEBUST`, and every - other stage installs its Python packages after it, as in the Python + Python differs from it. The tests run after `COPY . .`, as in the Python `test` phase example in `REPO_POLICIES.md` - Server: the final stage builds and runs the application @@ -227,13 +211,12 @@ are thin shims calling them. Model scripts: - [ ] `script/setup` / `make setup` — readies a fresh clone: runs `bootstrap`, then `install-precommit`, plus repo-specific init - [ ] `script/test` / `make test` — - `docker build --build-arg CACHEBUST="$(date +%s)-$$" --target test --output type=cacheonly .`, - which writes no image; the phase runs real tests, not a no-op (90-second - timeout, 60-second hard cap on wall time) + `docker build --target test --output type=cacheonly .`, which writes no + image; the phase runs real tests, not a no-op (90-second timeout, + 60-second hard cap on wall time) - [ ] `script/lint` / `make lint` — - `docker build --build-arg CACHEBUST="$(date +%s)-$$" --target lint --output type=cacheonly .`, - which writes no image. No lint verdict may come from a host invocation of - the linter. + `docker build --target lint --output type=cacheonly .`, which writes no + image. No lint verdict may come from a host invocation of the linter. - [ ] `script/fmt` / `make fmt` — formats code (writes; native, never in a container). Go code gets `gofmt`, then `golines -m 77`. - [ ] `script/fmt-check` / `make fmt-check` — checks formatting (read-only; @@ -245,16 +228,15 @@ are thin shims calling them. Model scripts: - [ ] `script/projectname` — outputs the project name (used by `script/docker` for the image tag) - [ ] `script/docker` / `make docker` — builds Docker image, tagged via - `script/projectname` (byte-identical across repos); a new `CACHEBUST` - value on every build, plus the version as a build arg + `script/projectname` (byte-identical across repos), with the version as a + build arg - [ ] `script/cibuild` — cd to repo root, run `script/bootstrap`, run `script/check`, then - `docker build --build-arg CACHEBUST="$(date +%s)-$$" --build-arg VERSION="$version" -t "$tag" .` - (what CI runs), with `$version` from - `git describe --tags --always --dirty` (`unknown` if empty) and `$tag` - from `script/projectname`, each assigned on its own line before the build. - The bootstrap is required: CI checks out and runs this alone, and - `script/fmt-check` runs the formatter on the host. + `docker build --build-arg VERSION="$version" -t "$tag" .` (what CI runs), + with `$version` from `git describe --tags --always --dirty` (`unknown` if + empty) and `$tag` from `script/projectname`, each assigned on its own line + before the build. The bootstrap is required: CI checks out and runs this + alone, and `script/fmt-check` runs the formatter on the host. - [ ] `script/bootstrap`, `script/fmt` and `script/fmt-check` run yarn by whichever name is present: `yarn`, or `yarnpkg` on Debian and Ubuntu. With neither name on `PATH`, `script/fmt` and `script/fmt-check` print one line @@ -279,14 +261,7 @@ are thin shims calling them. Model scripts: - [ ] `script/cibuild` succeeds in a fresh clone on a host carrying what CI has. That host is the runner image `docker.gitea.com/runner-images:ubuntu-latest`, which has docker, git and - node 24. Take any yarn off `PATH` first. The run must show that the checks - executed. A sub-second build means nothing ran. So does `CACHED` on a step - that lints or tests, one of the `RUN` steps below `ARG CACHEBUST`. Above - that line, these steps are expected to show `CACHED`: - - the steps that install system packages - - the steps that download Go's modules - - the step that runs `script/bootstrap` to install a Python `test` phase's - packages from `Pipfile.lock` + node 24. Take any yarn off `PATH` first. - [ ] Plant a lint violation and confirm both `make lint` and a plain `docker build .` fail on it; revert. A plain build that passes proves the final stage is missing its `COPY --from=` edge to the gate phases. diff --git a/prompts/REPO_POLICIES.md b/prompts/REPO_POLICIES.md index 528bcfa..b5e93cf 100644 --- a/prompts/REPO_POLICIES.md +++ b/prompts/REPO_POLICIES.md @@ -1,6 +1,6 @@ --- title: Repository Policies -last_modified: 2026-11-09 +last_modified: 2026-11-16 --- This document covers repository structure, tooling, and workflow standards. Code @@ -254,10 +254,8 @@ style conventions are in separate documents: phase and nothing else: ```sh - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --target lint --output type=cacheonly . - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --target test --output type=cacheonly . + docker build --target lint --output type=cacheonly . + docker build --target test --output type=cacheonly . ``` **A stage that is not the last one in the file is built only when the final @@ -275,7 +273,8 @@ style conventions are in separate documents: CI runner. `script/cibuild` and `script/docker` build the image that ships and tag it, so each build replaces the previous image. Each script assigns the tag on its own line before the build, so `set -e` stops it where - `script/projectname` fails. + `script/projectname` fails. `script/check` runs the two gate phases and + `script/fmt-check`, and builds no image of its own. Inside a phase the tool is invoked directly: `golangci-lint`, `go test`, `eslint`, `prettier`. The tool is never invoked through `make lint` or @@ -298,48 +297,16 @@ style conventions are in separate documents: - the settings - the golangci-lint version -- **Any build that runs checks is given a new `CACHEBUST` value.** Docker - invalidates a `COPY` layer only when the copied content changes. On an - unchanged tree the check `RUN` is therefore served from cache, nothing - executes, and the build still exits 0. Every `docker build` in `script/` - therefore passes `--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the - script's process id. So no two builds share a value, not even the three in one - `script/cibuild`. Four scripts do this, and there is no fifth: - - `script/lint` - - `script/test` - - `script/cibuild` - - `script/docker` - - `script/check` runs the two gate phases and `script/fmt-check`, and builds - no image of its own. In every stage of the `Dockerfile`, every step that - lints, tests or compiles comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes - right after these steps: - - the steps that install system packages - - the steps that download Go's modules - - the step that runs `script/bootstrap` to install a Python `test` phase's - packages from `Pipfile.lock` - - A stage with none of these steps declares `ARG CACHEBUST` right after - `FROM`. A step that also installs anything else comes after `ARG CACHEBUST` - too. A `script/bootstrap` that also installs node packages is one. A new - value reruns every `RUN` step after that line; `COPY` steps are compared by - content and need nothing. Docker keeps the layers above the line until one - of these changes: +- **Check builds use Docker's normal cache.** On an unchanged tree Docker reuses + each step's stored result, and it stores a result only for a step that + succeeded. A step runs again when one of these changes: - the base image - - their own lines - - the files they copy + - its own line or a line above it + - the files copied before it - A stage without the `ARG` line is served from cache on an unchanged tree, so - a repository re-vendors the scripts and adds the `ARG` lines in one commit. - The scripts do not pass `--no-cache`. That flag reruns every step too, but - it also builds the layers above the line again and empties every cache - mount. Go's build cache and golangci-lint's would then start empty every - time; the canonical Go `Dockerfile` below mounts them so that they survive - between check builds. A bare `docker build .` is not evidence that anything - ran: a sub-second build reporting success is a cache hit, not a result. - Never invalidate by pruning: `docker builder prune` and friends destroy a - build cache shared with every other build on the host. When a check is added - or changed, prove it works: + Nothing needs to run again on an unchanged tree. Never prune the build cache + (`docker builder prune` and the like): it is shared with every other build + on the host. When a check is added or changed, prove it works: - plant a defect it must catch - watch the run fail on it - revert the defect @@ -354,12 +321,8 @@ style conventions are in separate documents: `Dockerfile`: ```dockerfile - # In every stage ARG CACHEBUST comes right after the steps that install - # system packages and download Go's modules, or right after FROM in a - # stage with neither, and every step that lints, tests or compiles - # comes after it. script/ passes a new value on every build, which - # reruns every RUN step after it. Docker keeps the layers above it - # until the base image, their own lines or go.mod and go.sum change. + # The steps that install system packages and download Go's modules + # come before COPY . ., so a change to the code keeps their layers. # The steps that compile mount Go's build cache, and golangci-lint run # also mounts golangci-lint's; both survive from build to build. @@ -369,7 +332,6 @@ style conventions are in separate documents: WORKDIR /src COPY go.mod go.sum ./ RUN go mod download - ARG CACHEBUST COPY . . RUN --mount=type=cache,target=/root/.cache/go-build \ --mount=type=cache,target=/root/.cache/golangci-lint \ @@ -385,7 +347,6 @@ style conventions are in separate documents: WORKDIR /src COPY go.mod go.sum ./ RUN go mod download - ARG CACHEBUST COPY . . RUN --mount=type=cache,target=/root/.cache/go-build \ status=0; \ @@ -410,7 +371,6 @@ style conventions are in separate documents: WORKDIR /src COPY go.mod go.sum ./ RUN go mod download - ARG CACHEBUST COPY --from=lint /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null # A tar-stream context keeps the sender's file owners, which git refuses. @@ -436,7 +396,6 @@ style conventions are in separate documents: # Runtime stage, and the last one FROM alpine@sha256:... - ARG CACHEBUST COPY --from=builder /app /usr/local/bin/app ENTRYPOINT ["app"] ``` @@ -452,15 +411,15 @@ style conventions are in separate documents: two copies. A plain `docker build .` builds the last stage's chain and nothing else. - Check builds keep these, and none of them can hand back a stale result: - - The layers above `ARG CACHEBUST`: the system packages and Go's - modules. Docker installs the packages again when the base image or the - install line changes. Docker downloads the modules again when `go.mod` - or `go.sum` changes. Go checks every module against `go.sum` as it - downloads it, and no later build can change a kept layer. The modules - are kept in a layer rather than a cache mount. A cache mount is shared - by every build on the host, and Go builds from whatever files the - mount holds without checking them again (only `go mod verify` does). A - file changed there would go unnoticed. + - The layers of the system packages and Go's modules. Docker installs + the packages again when the base image or the install line changes. + Docker downloads the modules again when `go.mod` or `go.sum` changes. + Go checks every module against `go.sum` as it downloads it, and no + later build can change a kept layer. The modules are kept in a layer + rather than a cache mount. A cache mount is shared by every build on + the host, and Go builds from whatever files the mount holds without + checking them again (only `go mod verify` does). A file changed there + would go unnoticed. - Go's build cache (`/root/.cache/go-build`, where Go keeps it for root), mounted on the steps that compile: `golangci-lint run`, both `go test` runs and `go build`. Go reuses a compiled package only when @@ -487,8 +446,8 @@ style conventions are in separate documents: (`libvips-dev`, where alpine says `vips-dev`), and delete the package lists in the same `RUN`, so the layer does not keep them. The test phase, on the Debian Go image, installs the libraries its tests need the same - way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so - Docker keeps it until the base image or the install line changes: + way. Either install goes right after `FROM`, so Docker keeps it until the + base image or the install line changes: ```dockerfile FROM golangci/golangci-lint@sha256:... AS lint @@ -498,7 +457,6 @@ style conventions are in separate documents: WORKDIR /src COPY go.mod go.sum ./ RUN go mod download - ARG CACHEBUST ``` - `.dockerignore` lets `.git` into the build context. It keeps out every git @@ -539,21 +497,19 @@ style conventions are in separate documents: already do. - **A Python `test` phase that installs its packages from `Pipfile.lock` does it - above `ARG CACHEBUST`, so check builds keep them.** The steps are: + before `COPY . .`, so check builds keep them.** The steps are: - `COPY script/ script/` - `COPY Pipfile Pipfile.lock ./` - `RUN script/bootstrap` - `ARG CACHEBUST` comes right after them. Docker keeps what that `RUN` - installed until one of these changes: + Docker keeps what that `RUN` installed until one of these changes: - the base image - `script/` - `Pipfile` - `Pipfile.lock` - a line up to and including the `RUN` - The tests run after `ARG CACHEBUST`, on every check build. In every other - stage, a step that installs Python packages comes after `ARG CACHEBUST`. + The tests run after `COPY . .`. - When `script/bootstrap` runs in that step, it installs nothing beyond these: - system packages @@ -561,8 +517,7 @@ style conventions are in separate documents: - the packages from `Pipfile.lock` `script/bootstrap` installs no Python there: the tests use the - distribution's Python, which a step above it installs. If it installs - anything else there, the step goes after `ARG CACHEBUST`. + distribution's Python, which a step above it installs. - Debian's Python refuses a system-wide `pip install`, so pipenv goes in a virtual environment at `$HOME/.pipenv`. `script/bootstrap` creates it with @@ -644,7 +599,6 @@ style conventions are in separate documents: COPY script/ script/ COPY Pipfile Pipfile.lock ./ RUN script/bootstrap - ARG CACHEBUST COPY . . RUN timeout 90 sh -c 'pipenv run pytest || { \ echo "--- Rerunning with -v for details ---"; \ @@ -663,20 +617,17 @@ style conventions are in separate documents: A new push then cancels the older run on the same branch, queued or running, and no other: runs for replaced commits do not hold up the shared runner. `script/cibuild` bootstraps, runs the gate phases, and then builds the image, - so a successful run means every check passed. A bare `docker build .` does not - carry the same guarantee, because its gate phases may come from the cache. The - image build gets a `CACHEBUST` value of its own and so runs the gate phases a - second time. That is the price of the rule above, and it is worth paying. The - image that ships is built from a run of its own gates rather than from a cache - entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the - shared runner after 20 minutes. That allows for the three Docker builds - described above, each held to the 5-minute Docker build limit below, plus the - bootstrap. Those builds are the test phase, the lint phase, then the image. A - separate workflow limited to `main` by a `branches` list under `on: push` - cannot be checked by review. To try a change to it, add the feature branch to - that list and push, then remove the branch from the list again before merging. - Keep any job in it that publishes behind `if: github.ref_name == 'main'`, so - the run from the feature branch publishes nothing. + so a successful run means every check passed. The image build reuses the gate + phases that the check has just built. The `check` job sets + `timeout-minutes: 20`, so a hung build frees the shared runner after 20 + minutes. That allows for the three Docker builds described above, each held to + the 5-minute Docker build limit below, plus the bootstrap. Those builds are + the test phase, the lint phase, then the image. A separate workflow limited to + `main` by a `branches` list under `on: push` cannot be checked by review. To + try a change to it, add the feature branch to that list and push, then remove + the branch from the list again before merging. Keep any job in it that + publishes behind `if: github.ref_name == 'main'`, so the run from the feature + branch publishes nothing. - Use platform-standard formatters: - `black` for Python @@ -787,8 +738,7 @@ style conventions are in separate documents: running them. The directory lasts from one run to the next on a developer's machine, where this target runs. The directory also lasts in the `test` phase of the `Dockerfile`, which mounts it to keep compiled packages between - check builds, so the phase's `go test` runs carry `-count=1` too. A new - `CACHEBUST` value (above) is what makes that step run on an unchanged tree. + check builds, so the phase's `go test` runs carry `-count=1` too. Python example: @@ -891,9 +841,7 @@ style conventions are in separate documents: version="$(git describe --tags --always --dirty 2>/dev/null || true)" [ -n "$version" ] || version="unknown" tag="$(script/projectname)" - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --build-arg VERSION="$version" \ - -t "$tag" . + docker build --build-arg VERSION="$version" -t "$tag" . ``` `--always` makes an untagged repo yield an abbreviated commit hash rather diff --git a/script/cibuild b/script/cibuild index 2656771..89a0c24 100755 --- a/script/cibuild +++ b/script/cibuild @@ -2,12 +2,6 @@ # script/cibuild: run the CI build. It bootstraps first: a CI runner # checks out and runs this and nothing else, and script/fmt-check runs # the formatter on the host, which a pristine checkout cannot do. -# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST -# line, since a cached gate phase is a check that did not run; unlike -# --no-cache, it keeps the layers above that line and the cache mounts. -# This relies on every stage of the Dockerfile declaring ARG CACHEBUST -# before every step that lints, tests or compiles; a stage without that -# line is served from cache on an unchanged tree, with no warning. set -eu SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" @@ -25,9 +19,7 @@ main() { version="$(git describe --tags --always --dirty 2>/dev/null || true)" [ -n "$version" ] || version="unknown" tag="$("$SCRIPT_DIR/projectname")" - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --build-arg VERSION="$version" \ - -t "$tag" . + docker build --build-arg VERSION="$version" -t "$tag" . } main "$@" diff --git a/script/docker b/script/docker index cc00d9c..56bc3e0 100755 --- a/script/docker +++ b/script/docker @@ -1,12 +1,6 @@ #!/bin/sh # script/docker: build the Docker image tagged with the project name. # Identical in all repos; the tag comes from script/projectname. -# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST -# line, since a cached gate phase is a check that did not run; unlike -# --no-cache, it keeps the layers above that line and the cache mounts. -# This relies on every stage of the Dockerfile declaring ARG CACHEBUST -# before every step that lints, tests or compiles; a stage without that -# line is served from cache on an unchanged tree, with no warning. set -eu SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" @@ -22,9 +16,7 @@ main() { version="$(git describe --tags --always --dirty 2>/dev/null || true)" [ -n "$version" ] || version="unknown" tag="$("$SCRIPT_DIR/projectname")" - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --build-arg VERSION="$version" \ - -t "$tag" . + docker build --build-arg VERSION="$version" -t "$tag" . } main "$@" diff --git a/script/lint b/script/lint index 697fe82..6ae01b5 100755 --- a/script/lint +++ b/script/lint @@ -5,14 +5,7 @@ # make its answer untrustworthy. # # --target builds the lint phase and the stages it depends on, nothing -# else. A new CACHEBUST value reruns every RUN step after -# the ARG CACHEBUST line, since a cached lint layer is a lint that did -# not run; unlike --no-cache, it keeps the layers above that line and -# the cache mounts. This relies on every stage of the Dockerfile -# declaring ARG CACHEBUST before every step that lints, tests or -# compiles; a stage without that line is served from cache on an -# unchanged tree, with no warning. -# --output type=cacheonly writes no image, since nothing uses one. +# else. --output type=cacheonly writes no image, since nothing uses one. set -eu SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" @@ -20,9 +13,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)" main() { cd "$ROOT" - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --target lint \ - --output type=cacheonly . + docker build --target lint --output type=cacheonly . } main "$@" diff --git a/script/test b/script/test index bf78b35..9240fd0 100755 --- a/script/test +++ b/script/test @@ -2,13 +2,8 @@ # script/test: run the test suite. Testing is a phase of the Dockerfile # and this builds that phase, on the same terms as script/lint: # --target so that only the test phase and the stages it depends on are -# built, and a new CACHEBUST value because a cached test layer is a test -# that did not run; unlike --no-cache, it keeps the layers above the ARG -# CACHEBUST line and the cache mounts. This relies on every stage of the -# Dockerfile declaring ARG CACHEBUST before every step that lints, tests -# or compiles; a stage without that line is served from cache on an -# unchanged tree, with no warning. --output type=cacheonly writes no -# image, since nothing uses one. +# built, and --output type=cacheonly so that no image is written, since +# nothing uses one. set -eu SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" @@ -16,9 +11,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)" main() { cd "$ROOT" - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --target test \ - --output type=cacheonly . + docker build --target test --output type=cacheonly . } main "$@"