diff --git a/README.md b/README.md index 91c74e4..7479b61 100644 --- a/README.md +++ b/README.md @@ -144,17 +144,22 @@ alpine. We provide: `make hooks` installs the pre-commit hook that runs `script/precommit`. -In every stage of a `Dockerfile`, `ARG CACHEBUST` comes right after the steps -that install system packages, download Go's modules or run `script/bootstrap` to -install a Python `test` phase's packages from `Pipfile.lock`, or right after -`FROM` in a stage with none of these, and every step that lints, tests or -compiles comes after it. The `script/` builds pass a new `CACHEBUST` value on -every build, which reruns every `RUN` step after that line and leaves the layers -above it and the cache mounts in place; `--no-cache` would build those layers -again and empty the mounts. This repo's own `Dockerfile` declares -`ARG CACHEBUST` right after `FROM` in every stage: each stage installs system -packages and node packages together in one `script/bootstrap` step, and check -builds may keep system packages but not node packages. +In every stage of a `Dockerfile`, every step that lints, tests or compiles comes +after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps: + +- the steps that install system packages +- the steps that download Go's modules +- the step that runs `script/bootstrap` to install a Python `test` phase's + packages from `Pipfile.lock` + +A stage with none of these steps declares `ARG CACHEBUST` right after `FROM`. +The `script/` builds pass a new `CACHEBUST` value on every build, which reruns +every `RUN` step after that line and leaves the layers above it and the cache +mounts in place; `--no-cache` would build those layers again and empty the +mounts. This repo's own `Dockerfile` declares `ARG CACHEBUST` right after `FROM` +in every stage: each stage installs system packages and node packages together +in one `script/bootstrap` step, and check builds may keep system packages but +not node packages. ## Rationale diff --git a/TODO.md b/TODO.md index 530645f..4022afa 100644 --- a/TODO.md +++ b/TODO.md @@ -27,17 +27,16 @@ fmt-check, and commit. comes before `ARG CACHEBUST`: `COPY script/ script/`, `COPY Pipfile Pipfile.lock ./`, then `RUN script/bootstrap`. In that step `script/bootstrap` installs nothing beyond system packages, pipenv and its - dependencies by hash, and the packages from the lockfile, and no Python, since - the tests use the base image's. `--deploy` installs only what the lockfile - names, checks every package against its hashes, and fails when the lockfile is - out of date with `Pipfile`. A kept layer holds nothing a new install would - refuse, except that a package built from source and the system packages may - differ from a new install. The packages are kept as a layer, not a cache - mount, for the reason Go's modules are. The tests still run after - `ARG CACHEBUST`, and every other stage still installs its Python packages - after it. `REPO_POLICIES.md` gains a short Python `test` phase example, and - both checklists and the README state the rule the same way. rfscan picks this - up on its next re-vendor. + dependencies by hash, and the packages from the lockfile. It installs no + Python, because the tests use the base image's. `--deploy` installs only what + the lockfile names, checks every package against its hashes, and fails when + the lockfile is out of date with `Pipfile`. A kept layer holds nothing a new + install would refuse. A package built from source and the system packages may + differ in a new install. As with Go's modules, the packages are kept in a + layer, not a cache mount. The tests still run after `ARG CACHEBUST`, and every + other stage still installs its Python packages after it. `REPO_POLICIES.md` + gains a short Python `test` phase example, and both checklists and the README + state the rule the same way. rfscan picks this up on its next re-vendor. - 2026-10-21: The canonical `.golangci.yml` now accepts a local `replace` of `github.com/grafana/sobek` in a repository's root `go.mod` (issue 134), for homoicon, which keeps a local copy of sobek on sneak's decision in diff --git a/prompts/EXISTING_REPO_CHECKLIST.md b/prompts/EXISTING_REPO_CHECKLIST.md index 029c780..0cf3e40 100644 --- a/prompts/EXISTING_REPO_CHECKLIST.md +++ b/prompts/EXISTING_REPO_CHECKLIST.md @@ -49,18 +49,21 @@ with your task. `--target` names it, so a lost `COPY --from=` edge leaves `docker build .` passing while the gate never runs. Confirm by planting a violation, not by reading the file. -- [ ] In every stage `ARG CACHEBUST` comes right after the steps that install - system packages, download Go's modules or run `script/bootstrap` to - install a Python `test` phase's packages from `Pipfile.lock`, or right - after `FROM` in a stage with none of these, and every step that lints, - tests or compiles comes after it. A step that installs anything else as - well, such as a `script/bootstrap` that also installs node packages, comes - after it too. The `script/` builds pass a new `CACHEBUST` value on every - build, which reruns every `RUN` step after that line and leaves the layers - above it and the cache mounts in place; `--no-cache` would build those - layers again and empty the mounts. A stage without the `ARG` line is - served from cache on an unchanged tree, so the scripts and the `ARG` lines - are re-vendored in one commit. +- [ ] In every stage, every step that lints, tests or compiles comes after + `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps: + - the steps that install system packages + - the steps that download Go's modules + - the step that runs `script/bootstrap` to install a Python `test` phase's + packages from `Pipfile.lock` +- [ ] A stage with none of these steps declares `ARG CACHEBUST` right after + `FROM`. A step that installs anything else as well, such as a + `script/bootstrap` that also installs node packages, comes after it too. + The `script/` builds pass a new `CACHEBUST` value on every build, which + reruns every `RUN` step after that line and leaves the layers above it and + the cache mounts in place; `--no-cache` would build those layers again and + empty the mounts. A stage without the `ARG` line is served from cache on + an unchanged tree, so the scripts and the `ARG` lines are re-vendored in + one commit. - [ ] Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`, `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile (`golangci-lint run`, `go test`, `go build`) mount Go's build cache @@ -71,15 +74,16 @@ with your task. canonical Go `Dockerfile` in `REPO_POLICIES.md` - [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does it above `ARG CACHEBUST`, with `COPY script/ script/`, - `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. - `script/bootstrap`, when it runs in that step, installs nothing beyond - system packages, pipenv and its dependencies by hash - (`pip install --require-hashes`), and the packages from the lockfile - (`pipenv install --deploy`, with `--dev` when the tests need the - development packages). It installs no Python there, because the tests use - the base image's. The tests run after `ARG CACHEBUST`, and every other - stage installs its Python packages after it, as in the Python `test` phase - example in `REPO_POLICIES.md` + `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When + `script/bootstrap` runs in that step, it installs nothing beyond system + packages, pipenv and its dependencies, and the packages from + `Pipfile.lock`. It installs pipenv and its dependencies by hash, with + `pip install --require-hashes`. It installs the packages from + `Pipfile.lock` with `pipenv install --deploy`, plus `--dev` when the tests + need the development packages. It installs no Python there, because the + tests use the base image's. The tests run after `ARG CACHEBUST`, and every + other stage installs its Python packages after it, as in the Python `test` + phase example in `REPO_POLICIES.md` - [ ] The gate phases invoke their tools directly, never through `make lint` or `script/test` — those are themselves a `docker build` and would recurse inside a build step @@ -245,9 +249,11 @@ with your task. git and node 24, with no yarn on `PATH`), and demonstrably executed the checks — a sub-second build, or `CACHED` on a step that lints or tests, one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above - that line, the steps that install system packages, download Go's modules - or run `script/bootstrap` to install a Python `test` phase's packages from - `Pipfile.lock` are expected to show `CACHED`. + that line, these steps are expected to show `CACHED`: + - the steps that install system packages + - the steps that download Go's modules + - the step that runs `script/bootstrap` to install a Python `test` phase's + packages from `Pipfile.lock` - [ ] A planted lint violation fails both `make lint` and a plain `docker build .`; revert it afterwards - [ ] Commit and merge fixes before starting your actual task diff --git a/prompts/NEW_REPO_CHECKLIST.md b/prompts/NEW_REPO_CHECKLIST.md index 1ec3ec2..0b4f15a 100644 --- a/prompts/NEW_REPO_CHECKLIST.md +++ b/prompts/NEW_REPO_CHECKLIST.md @@ -108,18 +108,21 @@ Template files can be fetched from: stage carries a `COPY --from=` of a harmless file from each so the image cannot be built unless both passed. Keep the final stage last: a stage nothing depends on is built only when `--target` names it. - - In every stage `ARG CACHEBUST` comes right after the steps that install - system packages, download Go's modules or run `script/bootstrap` to - install a Python `test` phase's packages from `Pipfile.lock`, or right - after `FROM` in a stage with none of these, and every step that lints, - tests or compiles comes after it. A step that installs anything else as - well, such as a `script/bootstrap` that also installs node packages, comes - after it too. The `script/` builds pass a new `CACHEBUST` value on every - build, which reruns every `RUN` step after that line and leaves the layers - above it and the cache mounts in place; `--no-cache` would build those - layers again and empty the mounts. A stage without the `ARG` line is - served from cache on an unchanged tree, so the scripts and every stage's - `ARG CACHEBUST` line are added in the same commit. + - In every stage, every step that lints, tests or compiles comes after + `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps: + - the steps that install system packages + - the steps that download Go's modules + - the step that runs `script/bootstrap` to install a Python `test` + phase's packages from `Pipfile.lock` + - A stage with none of these steps declares `ARG CACHEBUST` right after + `FROM`. A step that installs anything else as well, such as a + `script/bootstrap` that also installs node packages, comes after it too. + The `script/` builds pass a new `CACHEBUST` value on every build, which + reruns every `RUN` step after that line and leaves the layers above it and + the cache mounts in place; `--no-cache` would build those layers again and + empty the mounts. A stage without the `ARG` line is served from cache on + an unchanged tree, so the scripts and every stage's `ARG CACHEBUST` line + are added in the same commit. - Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`, `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile (`golangci-lint run`, `go test`, `go build`) mount Go's build cache @@ -130,15 +133,16 @@ Template files can be fetched from: canonical Go `Dockerfile` in `REPO_POLICIES.md` - Python: a `test` phase that installs its packages from `Pipfile.lock` does it above `ARG CACHEBUST`, with `COPY script/ script/`, - `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. - `script/bootstrap`, when it runs in that step, installs nothing beyond - system packages, pipenv and its dependencies by hash - (`pip install --require-hashes`), and the packages from the lockfile - (`pipenv install --deploy`, with `--dev` when the tests need the - development packages). It installs no Python there, because the tests use - the base image's. The tests run after `ARG CACHEBUST`, and every other - stage installs its Python packages after it, as in the Python `test` phase - example in `REPO_POLICIES.md` + `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When + `script/bootstrap` runs in that step, it installs nothing beyond system + packages, pipenv and its dependencies, and the packages from + `Pipfile.lock`. It installs pipenv and its dependencies by hash, with + `pip install --require-hashes`. It installs the packages from + `Pipfile.lock` with `pipenv install --deploy`, plus `--dev` when the tests + need the development packages. It installs no Python there, because the + tests use the base image's. The tests run after `ARG CACHEBUST`, and every + other stage installs its Python packages after it, as in the Python `test` + phase example in `REPO_POLICIES.md` - Server: the final stage builds and runs the application - Non-server: the final stage brings up the dev environment - Image pinned by sha256 hash with version/date comment @@ -228,9 +232,11 @@ are thin shims calling them. Model scripts: git and node 24, with no yarn on `PATH`), and demonstrably executed the checks — a sub-second build, or `CACHED` on a step that lints or tests, one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above - that line, the steps that install system packages, download Go's modules - or run `script/bootstrap` to install a Python `test` phase's packages from - `Pipfile.lock` are expected to show `CACHED`. + that line, these steps are expected to show `CACHED`: + - the steps that install system packages + - the steps that download Go's modules + - the step that runs `script/bootstrap` to install a Python `test` phase's + packages from `Pipfile.lock` - [ ] Plant a lint violation and confirm both `make lint` and a plain `docker build .` fail on it; revert. A plain build that passes proves the final stage is missing its `COPY --from=` edge to the gate phases. diff --git a/prompts/REPO_POLICIES.md b/prompts/REPO_POLICIES.md index eadb8f3..0cd41b7 100644 --- a/prompts/REPO_POLICIES.md +++ b/prompts/REPO_POLICIES.md @@ -174,28 +174,32 @@ style conventions are in separate documents: `script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and `script/docker` are the four, and there is no fifth — `script/check` runs the two gate phases and `script/fmt-check`, and builds no image of its own. In - every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps - that install system packages, download Go's modules or run `script/bootstrap` - to install a Python `test` phase's packages from `Pipfile.lock`, or right - after `FROM` in a stage with none of these, and every step that lints, tests - or compiles comes after it. A step that installs anything else as well, such - as a `script/bootstrap` that also installs node packages, comes after it too. - A new value reruns every `RUN` step after that line; `COPY` steps are compared - by content and need nothing. Docker keeps the layers above the line until the - base image, their own lines or the files they copy change. A stage without the - `ARG` line is served from cache on an unchanged tree, so a repository - re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do - not pass `--no-cache`: it reruns every step too, but it also builds the layers - above the line again and empties every cache mount, so Go's build cache and - golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they - survive between check builds, would start empty every time. A bare - `docker build .` is not evidence that anything ran: a sub-second build - reporting success is a cache hit, not a result. Never invalidate by pruning — - `docker builder prune` and friends destroy a build cache shared with every - other build on the host. When a check is added or changed, prove it works by - planting a defect it must catch and watching the run fail on it, then revert - the defect. A green run alone shows neither that the check ran nor that it - covers what it should. + every stage of the `Dockerfile`, every step that lints, tests or compiles + comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps: + - the steps that install system packages + - the steps that download Go's modules + - the step that runs `script/bootstrap` to install a Python `test` phase's + packages from `Pipfile.lock` + + A stage with none of these steps declares `ARG CACHEBUST` right after + `FROM`. A step that installs anything else as well, such as a + `script/bootstrap` that also installs node packages, comes after it too. A + new value reruns every `RUN` step after that line; `COPY` steps are compared + by content and need nothing. Docker keeps the layers above the line until + the base image, their own lines or the files they copy change. A stage + without the `ARG` line is served from cache on an unchanged tree, so a + repository re-vendors the scripts and adds the `ARG` lines in one commit. + The scripts do not pass `--no-cache`: it reruns every step too, but it also + builds the layers above the line again and empties every cache mount, so + Go's build cache and golangci-lint's, which the canonical Go `Dockerfile` + below mounts so that they survive between check builds, would start empty + every time. A bare `docker build .` is not evidence that anything ran: a + sub-second build reporting success is a cache hit, not a result. Never + invalidate by pruning — `docker builder prune` and friends destroy a build + cache shared with every other build on the host. When a check is added or + changed, prove it works by planting a defect it must catch and watching the + run fail on it, then revert the defect. A green run alone shows neither that + the check ran nor that it covers what it should. - **The gate phases are separate stages, and the build stage depends on both.** The lint phase is based on the `golangci/golangci-lint` image (pinned by @@ -380,8 +384,8 @@ style conventions are in separate documents: `Pipfile.lock`. It installs no Python there: the tests use the base image's Python, which the image's hash pins. If it installs anything else there, the step goes after `ARG CACHEBUST`. - - It installs pipenv and its dependencies by hash, as the rule that every - external reference is pinned by hash requires. It does so with + - Every external reference must be pinned by hash, so `script/bootstrap` + installs pipenv and its dependencies by hash. It does so with `pip install --require-hashes -r` and a requirements file in `script/` that lists each of them with `==` and its `--hash`. pip refuses to install a dependency the file does not list. @@ -394,14 +398,14 @@ style conventions are in separate documents: their hashes. The packages `virtualenv` puts into the virtualenv that pipenv creates come from wheels the `virtualenv` package ships, which its hash covers. Two things in the layer are not fixed by a hash. A package - with no ready-built wheel for the image is built from source, with build - tools pip downloads without a hash check, so a kept layer's build of it - may differ from a new one. And system packages keep the versions the - package manager gave when the layer was built; a new install may get newer - ones. - - The packages are kept in a layer rather than a cache mount for the reason - Go's modules are: a cache mount is shared by every build on the host, and - nothing checks its files again. + with no ready-built wheel for the image is built from source. pip + downloads the build tools without a hash check, so a kept layer's build of + such a package may differ from a new one. And system packages keep the + versions the package manager gave when the layer was built; a new install + may get newer ones. + - As with Go's modules, the packages are kept in a layer, not a cache mount. + A cache mount is shared by every build on the host, and nothing checks its + files again. In this example, `script/bootstrap` finds `make` and `git` already installed, installs pipenv and its dependencies by hash, and runs