From a3ec8461afb086c5bea6baa1c2e4c8b2f4f40cc4 Mon Sep 17 00:00:00 2001 From: sneak Date: Wed, 21 Oct 2026 15:48:09 +0000 Subject: [PATCH] Pin pipenv by hash and keep one form of the Python lockfile install (closes #133) Rework of https://git.eeqj.de/sneak/prompts/pulls/135 on the review at https://git.eeqj.de/sneak/prompts/pulls/135#issuecomment-153985. A Python test phase now has one form: COPY script/, COPY Pipfile Pipfile.lock, RUN script/bootstrap, then ARG CACHEBUST, and every list of what goes above the line names that step. In that step script/bootstrap installs only system packages, pipenv and its dependencies by hash with pip --require-hashes, and the lockfile's packages, and no Python. The claim about a kept layer covers only what is pinned by hash and says its system packages may be older. The bold rule is conditional, the script/cibuild items expect the step to show CACHED, and the example says why it installs make and git. Model: opus-5-5 --- README.md | 20 +++--- TODO.md | 23 ++++--- prompts/EXISTING_REPO_CHECKLIST.md | 46 +++++++------ prompts/NEW_REPO_CHECKLIST.md | 47 +++++++------ prompts/REPO_POLICIES.md | 106 +++++++++++++++++------------ 5 files changed, 137 insertions(+), 105 deletions(-) diff --git a/README.md b/README.md index f188c5f..91c74e4 100644 --- a/README.md +++ b/README.md @@ -145,16 +145,16 @@ alpine. We provide: `make hooks` installs the pre-commit hook that runs `script/precommit`. In every stage of a `Dockerfile`, `ARG CACHEBUST` comes right after the steps -that install system packages, download Go's modules or install a `test` phase's -Python packages from `Pipfile.lock`, or right after `FROM` in a stage with none -of these, and every step that lints, tests or compiles comes after it. The -`script/` builds pass a new `CACHEBUST` value on every build, which reruns every -`RUN` step after that line and leaves the layers above it and the cache mounts -in place; `--no-cache` would build those layers again and empty the mounts. This -repo's own `Dockerfile` declares `ARG CACHEBUST` right after `FROM` in every -stage: each stage installs system packages and node packages together in one -`script/bootstrap` step, and check builds may keep system packages but not node -packages. +that install system packages, download Go's modules or run `script/bootstrap` to +install a Python `test` phase's packages from `Pipfile.lock`, or right after +`FROM` in a stage with none of these, and every step that lints, tests or +compiles comes after it. The `script/` builds pass a new `CACHEBUST` value on +every build, which reruns every `RUN` step after that line and leaves the layers +above it and the cache mounts in place; `--no-cache` would build those layers +again and empty the mounts. This repo's own `Dockerfile` declares +`ARG CACHEBUST` right after `FROM` in every stage: each stage installs system +packages and node packages together in one `script/bootstrap` step, and check +builds may keep system packages but not node packages. ## Rationale diff --git a/TODO.md b/TODO.md index f6621ec..5bfcaf3 100644 --- a/TODO.md +++ b/TODO.md @@ -24,16 +24,19 @@ fmt-check, and commit. - 2026-10-21: Check builds may now also keep the Python packages a `test` phase installs from `Pipfile.lock` (issue 133), on sneak's ruling in https://git.eeqj.de/sneak/rfscan/issues/80#issuecomment-153437. The install - comes before `ARG CACHEBUST`: `COPY Pipfile Pipfile.lock ./`, then one `RUN` - that runs `pipenv install --deploy`, or a `script/bootstrap` that installs - nothing beyond system packages, the pinned pipenv and the packages from the - lockfile. `--deploy` installs only what the lockfile names, checks every - package against its hashes, and fails when the lockfile is out of date with - `Pipfile`. The packages are kept as a layer, not a cache mount, for the reason - Go's modules are. The tests still run after `ARG CACHEBUST`, and every other - stage still installs its Python packages after it. `REPO_POLICIES.md` gains a - short Python `test` phase example, and both checklists and the README state - the rule the same way. rfscan picks this up on its next re-vendor. + comes before `ARG CACHEBUST`: `COPY script/ script/`, + `COPY Pipfile Pipfile.lock ./`, then `RUN script/bootstrap`. In that step + `script/bootstrap` installs nothing beyond system packages, pipenv and its + dependencies by hash, and the packages from the lockfile, and no Python, since + the tests use the base image's. `--deploy` installs only what the lockfile + names, checks every package against its hashes, and fails when the lockfile is + out of date with `Pipfile`. A kept layer holds the same hashed packages a new + install would, but its system packages may be older. The packages are kept as + a layer, not a cache mount, for the reason Go's modules are. The tests still + run after `ARG CACHEBUST`, and every other stage still installs its Python + packages after it. `REPO_POLICIES.md` gains a short Python `test` phase + example, and both checklists and the README state the rule the same way. + rfscan picks this up on its next re-vendor. - 2026-10-21: The canonical `.golangci.yml` now accepts a local `replace` of `github.com/grafana/sobek` in a repository's root `go.mod` (issue 134), for homoicon, which keeps a local copy of sobek on sneak's decision in diff --git a/prompts/EXISTING_REPO_CHECKLIST.md b/prompts/EXISTING_REPO_CHECKLIST.md index 5678f8f..029c780 100644 --- a/prompts/EXISTING_REPO_CHECKLIST.md +++ b/prompts/EXISTING_REPO_CHECKLIST.md @@ -50,16 +50,17 @@ with your task. passing while the gate never runs. Confirm by planting a violation, not by reading the file. - [ ] In every stage `ARG CACHEBUST` comes right after the steps that install - system packages, download Go's modules or install a `test` phase's Python - packages from `Pipfile.lock`, or right after `FROM` in a stage with none - of these, and every step that lints, tests or compiles comes after it. A - step that installs anything else as well, such as a `script/bootstrap` - that also installs node packages, comes after it too. The `script/` builds - pass a new `CACHEBUST` value on every build, which reruns every `RUN` step - after that line and leaves the layers above it and the cache mounts in - place; `--no-cache` would build those layers again and empty the mounts. A - stage without the `ARG` line is served from cache on an unchanged tree, so - the scripts and the `ARG` lines are re-vendored in one commit. + system packages, download Go's modules or run `script/bootstrap` to + install a Python `test` phase's packages from `Pipfile.lock`, or right + after `FROM` in a stage with none of these, and every step that lints, + tests or compiles comes after it. A step that installs anything else as + well, such as a `script/bootstrap` that also installs node packages, comes + after it too. The `script/` builds pass a new `CACHEBUST` value on every + build, which reruns every `RUN` step after that line and leaves the layers + above it and the cache mounts in place; `--no-cache` would build those + layers again and empty the mounts. A stage without the `ARG` line is + served from cache on an unchanged tree, so the scripts and the `ARG` lines + are re-vendored in one commit. - [ ] Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`, `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile (`golangci-lint run`, `go test`, `go build`) mount Go's build cache @@ -69,14 +70,16 @@ with your task. mounts anything else; both `go test` runs carry `-count=1`, as in the canonical Go `Dockerfile` in `REPO_POLICIES.md` - [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does - it above `ARG CACHEBUST`, with `COPY Pipfile Pipfile.lock ./` and then one - `RUN` that runs `pipenv install --deploy` (with `--dev` when the tests - need the development packages), or that runs `script/bootstrap` after - `COPY script/ script/`; that `script/bootstrap` installs nothing beyond - system packages, the pinned pipenv and the packages from the lockfile. The - tests run after `ARG CACHEBUST`, and every other stage installs its Python - packages after it, as in the Python `test` phase example in - `REPO_POLICIES.md` + it above `ARG CACHEBUST`, with `COPY script/ script/`, + `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. + `script/bootstrap`, when it runs in that step, installs nothing beyond + system packages, pipenv and its dependencies by hash + (`pip install --require-hashes`), and the packages from the lockfile + (`pipenv install --deploy`, with `--dev` when the tests need the + development packages). It installs no Python there, because the tests use + the base image's. The tests run after `ARG CACHEBUST`, and every other + stage installs its Python packages after it, as in the Python `test` phase + example in `REPO_POLICIES.md` - [ ] The gate phases invoke their tools directly, never through `make lint` or `script/test` — those are themselves a `docker build` and would recurse inside a build step @@ -241,9 +244,10 @@ with your task. (the runner image `docker.gitea.com/runner-images:ubuntu-latest`: docker, git and node 24, with no yarn on `PATH`), and demonstrably executed the checks — a sub-second build, or `CACHED` on a step that lints or tests, - one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. The steps - that install system packages and download Go's modules above that line are - expected to show `CACHED`. + one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above + that line, the steps that install system packages, download Go's modules + or run `script/bootstrap` to install a Python `test` phase's packages from + `Pipfile.lock` are expected to show `CACHED`. - [ ] A planted lint violation fails both `make lint` and a plain `docker build .`; revert it afterwards - [ ] Commit and merge fixes before starting your actual task diff --git a/prompts/NEW_REPO_CHECKLIST.md b/prompts/NEW_REPO_CHECKLIST.md index ed83ad2..1ec3ec2 100644 --- a/prompts/NEW_REPO_CHECKLIST.md +++ b/prompts/NEW_REPO_CHECKLIST.md @@ -109,17 +109,17 @@ Template files can be fetched from: cannot be built unless both passed. Keep the final stage last: a stage nothing depends on is built only when `--target` names it. - In every stage `ARG CACHEBUST` comes right after the steps that install - system packages, download Go's modules or install a `test` phase's Python - packages from `Pipfile.lock`, or right after `FROM` in a stage with none - of these, and every step that lints, tests or compiles comes after it. A - step that installs anything else as well, such as a `script/bootstrap` - that also installs node packages, comes after it too. The `script/` builds - pass a new `CACHEBUST` value on every build, which reruns every `RUN` step - after that line and leaves the layers above it and the cache mounts in - place; `--no-cache` would build those layers again and empty the mounts. A - stage without the `ARG` line is served from cache on an unchanged tree, so - the scripts and every stage's `ARG CACHEBUST` line are added in the same - commit. + system packages, download Go's modules or run `script/bootstrap` to + install a Python `test` phase's packages from `Pipfile.lock`, or right + after `FROM` in a stage with none of these, and every step that lints, + tests or compiles comes after it. A step that installs anything else as + well, such as a `script/bootstrap` that also installs node packages, comes + after it too. The `script/` builds pass a new `CACHEBUST` value on every + build, which reruns every `RUN` step after that line and leaves the layers + above it and the cache mounts in place; `--no-cache` would build those + layers again and empty the mounts. A stage without the `ARG` line is + served from cache on an unchanged tree, so the scripts and every stage's + `ARG CACHEBUST` line are added in the same commit. - Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`, `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile (`golangci-lint run`, `go test`, `go build`) mount Go's build cache @@ -129,14 +129,16 @@ Template files can be fetched from: mounts anything else; both `go test` runs carry `-count=1`, as in the canonical Go `Dockerfile` in `REPO_POLICIES.md` - Python: a `test` phase that installs its packages from `Pipfile.lock` does - it above `ARG CACHEBUST`, with `COPY Pipfile Pipfile.lock ./` and then one - `RUN` that runs `pipenv install --deploy` (with `--dev` when the tests - need the development packages), or that runs `script/bootstrap` after - `COPY script/ script/`; that `script/bootstrap` installs nothing beyond - system packages, the pinned pipenv and the packages from the lockfile. The - tests run after `ARG CACHEBUST`, and every other stage installs its Python - packages after it, as in the Python `test` phase example in - `REPO_POLICIES.md` + it above `ARG CACHEBUST`, with `COPY script/ script/`, + `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. + `script/bootstrap`, when it runs in that step, installs nothing beyond + system packages, pipenv and its dependencies by hash + (`pip install --require-hashes`), and the packages from the lockfile + (`pipenv install --deploy`, with `--dev` when the tests need the + development packages). It installs no Python there, because the tests use + the base image's. The tests run after `ARG CACHEBUST`, and every other + stage installs its Python packages after it, as in the Python `test` phase + example in `REPO_POLICIES.md` - Server: the final stage builds and runs the application - Non-server: the final stage brings up the dev environment - Image pinned by sha256 hash with version/date comment @@ -225,9 +227,10 @@ are thin shims calling them. Model scripts: (the runner image `docker.gitea.com/runner-images:ubuntu-latest`: docker, git and node 24, with no yarn on `PATH`), and demonstrably executed the checks — a sub-second build, or `CACHED` on a step that lints or tests, - one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. The steps - that install system packages and download Go's modules above that line are - expected to show `CACHED`. + one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above + that line, the steps that install system packages, download Go's modules + or run `script/bootstrap` to install a Python `test` phase's packages from + `Pipfile.lock` are expected to show `CACHED`. - [ ] Plant a lint violation and confirm both `make lint` and a plain `docker build .` fail on it; revert. A plain build that passes proves the final stage is missing its `COPY --from=` edge to the gate phases. diff --git a/prompts/REPO_POLICIES.md b/prompts/REPO_POLICIES.md index 7ff50a9..b4e2a5e 100644 --- a/prompts/REPO_POLICIES.md +++ b/prompts/REPO_POLICIES.md @@ -175,26 +175,27 @@ style conventions are in separate documents: `script/docker` are the four, and there is no fifth — `script/check` runs the two gate phases and `script/fmt-check`, and builds no image of its own. In every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps - that install system packages, download Go's modules or install a `test` - phase's Python packages from `Pipfile.lock`, or right after `FROM` in a stage - with none of these, and every step that lints, tests or compiles comes after - it. A step that installs anything else as well, such as a `script/bootstrap` - that also installs node packages, comes after it too. A new value reruns every - `RUN` step after that line; `COPY` steps are compared by content and need - nothing. Docker keeps the layers above the line until the base image, their - own lines or the files they copy change. A stage without the `ARG` line is - served from cache on an unchanged tree, so a repository re-vendors the scripts - and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`: - it reruns every step too, but it also builds the layers above the line again - and empties every cache mount, so Go's build cache and golangci-lint's, which - the canonical Go `Dockerfile` below mounts so that they survive between check - builds, would start empty every time. A bare `docker build .` is not evidence - that anything ran: a sub-second build reporting success is a cache hit, not a - result. Never invalidate by pruning — `docker builder prune` and friends - destroy a build cache shared with every other build on the host. When a check - is added or changed, prove it works by planting a defect it must catch and - watching the run fail on it, then revert the defect. A green run alone shows - neither that the check ran nor that it covers what it should. + that install system packages, download Go's modules or run `script/bootstrap` + to install a Python `test` phase's packages from `Pipfile.lock`, or right + after `FROM` in a stage with none of these, and every step that lints, tests + or compiles comes after it. A step that installs anything else as well, such + as a `script/bootstrap` that also installs node packages, comes after it too. + A new value reruns every `RUN` step after that line; `COPY` steps are compared + by content and need nothing. Docker keeps the layers above the line until the + base image, their own lines or the files they copy change. A stage without the + `ARG` line is served from cache on an unchanged tree, so a repository + re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do + not pass `--no-cache`: it reruns every step too, but it also builds the layers + above the line again and empties every cache mount, so Go's build cache and + golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they + survive between check builds, would start empty every time. A bare + `docker build .` is not evidence that anything ran: a sub-second build + reporting success is a cache hit, not a result. Never invalidate by pruning — + `docker builder prune` and friends destroy a build cache shared with every + other build on the host. When a check is added or changed, prove it works by + planting a defect it must catch and watching the run fail on it, then revert + the defect. A green run alone shows neither that the check ran nor that it + covers what it should. - **The gate phases are separate stages, and the build stage depends on both.** The lint phase is based on the `golangci/golangci-lint` image (pinned by @@ -309,8 +310,9 @@ style conventions are in separate documents: a package's findings only when the package's files and those of every package it imports, `go.mod`, the linters and their settings in `.golangci.yml`, and the golangci-lint version are all unchanged. - - In a Python `test` phase, the packages installed from `Pipfile.lock`, - as described for that phase below. + - In a Python `test` phase, the layer of the step that runs + `script/bootstrap` to install the packages from `Pipfile.lock`, as + described for that phase below. - If the project uses `//go:embed` directives that reference build artifacts (e.g. a web frontend compiled in a separate stage), the lint phase must create placeholder files so the embed directives resolve. Example: @@ -365,31 +367,51 @@ style conventions are in separate documents: `--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already do. -- **A Python `test` phase installs its packages from `Pipfile.lock` above - `ARG CACHEBUST`, so check builds keep them.** The install has steps of its - own: `COPY Pipfile Pipfile.lock ./`, then one `RUN` that runs - `pipenv install --deploy`, with `--dev` when the tests need the development - packages. Docker keeps what that `RUN` installed until the base image, - `Pipfile`, `Pipfile.lock` or a line up to and including the `RUN` changes. If - the `RUN` runs `script/bootstrap` instead, as in the example below, - `COPY script/ script/` comes before it, so Docker installs again when - `script/` changes too. That `script/bootstrap` may install nothing beyond - system packages, the pinned pipenv and the packages from the lockfile; one - that installs anything else comes after `ARG CACHEBUST`. A kept layer holds - the same packages a new install would: `--deploy` installs only what - `Pipfile.lock` names, checks every package against the lockfile's hashes, and - fails when the lockfile is out of date with `Pipfile`. The packages are kept - in a layer rather than a cache mount for the reason Go's modules are: a cache - mount is shared by every build on the host, and nothing checks its files - again. The tests run after `ARG CACHEBUST`, on every check build. In every - other stage, a step that installs Python packages comes after `ARG CACHEBUST`. - An example, with a `script/bootstrap` that installs the pinned pipenv and then - runs `pipenv install --dev --deploy`: +- **A Python `test` phase that installs its packages from `Pipfile.lock` does it + above `ARG CACHEBUST`, so check builds keep them.** The steps are + `COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and + `RUN script/bootstrap`, with `ARG CACHEBUST` right after them. Docker keeps + what that `RUN` installed until the base image, `script/`, `Pipfile`, + `Pipfile.lock` or a line up to and including the `RUN` changes. The tests run + after `ARG CACHEBUST`, on every check build. In every other stage, a step that + installs Python packages comes after `ARG CACHEBUST`. + - When `script/bootstrap` runs in that step, it installs nothing beyond + system packages, pipenv and its dependencies, and the packages from + `Pipfile.lock`. It installs no Python there: the tests use the base + image's Python, which the image's hash pins. If it installs anything else + there, the step goes after `ARG CACHEBUST`. + - It installs pipenv and its dependencies by hash, as the rule that every + external reference is pinned by hash requires. It does so with + `pip install --require-hashes -r` and a requirements file in `script/` + that lists each of them with `==` and its `--hash`. pip refuses to install + a dependency the file does not list. + - It then runs `pipenv install --deploy`, with `--dev` when the tests need + the development packages. `--deploy` installs only what `Pipfile.lock` + names, checks every package against the lockfile's hashes, and fails when + the lockfile is out of date with `Pipfile`. + - A kept layer holds the same pipenv, the same dependencies of pipenv and + the same lockfile packages that a new install would, because each was + checked against its hash. The pip and setuptools inside the virtualenv + that pipenv creates are the same too: the `virtualenv` package, one of + pipenv's dependencies, copies them from wheels it ships, and its hash + covers those wheels. Any system packages the step installed are not pinned + by hash: the layer keeps the versions the package manager gave when the + layer was built, and a new install may get newer ones. + - The packages are kept in a layer rather than a cache mount for the reason + Go's modules are: a cache mount is shared by every build on the host, and + nothing checks its files again. + + In this example, `script/bootstrap` finds `make` and `git` already + installed, installs pipenv and its dependencies by hash, and runs + `pipenv install --dev --deploy`: ```dockerfile # Test phase. The 90-second timeout covers both pytest runs. # python:3.x-slim, YYYY-MM-DD FROM python@sha256:... AS test + # make and git are base tools that script/bootstrap installs when they + # are missing. Installed here, they stay in their own layer when + # script/ changes. RUN apt-get update \ && apt-get install -y --no-install-recommends make git \ && rm -rf /var/lib/apt/lists/*