Pin pipenv by hash and keep one form of the Python lockfile install (closes #133)

Rework of #135 on the review at
#135 (comment). A Python
test phase now has one form: COPY script/, COPY Pipfile Pipfile.lock,
RUN script/bootstrap, then ARG CACHEBUST, and every list of what goes
above the line names that step. In that step script/bootstrap installs
only system packages, pipenv and its dependencies by hash with pip
--require-hashes, and the lockfile's packages, and no Python. The claim
about a kept layer covers only what is pinned by hash and says its
system packages may be older. The bold rule is conditional, the
script/cibuild items expect the step to show CACHED, and the example
says why it installs make and git.

Model: opus-5-5

README.md

@@ -145,16 +145,16 @@ alpine. We provide:
 `make hooks` installs the pre-commit hook that runs `script/precommit`.
 
 In every stage of a `Dockerfile`, `ARG CACHEBUST` comes right after the steps
-that install system packages, download Go's modules or install a `test` phase's
-Python packages from `Pipfile.lock`, or right after `FROM` in a stage with none
-of these, and every step that lints, tests or compiles comes after it. The
-`script/` builds pass a new `CACHEBUST` value on every build, which reruns every
-`RUN` step after that line and leaves the layers above it and the cache mounts
-in place; `--no-cache` would build those layers again and empty the mounts. This
-repo's own `Dockerfile` declares `ARG CACHEBUST` right after `FROM` in every
-stage: each stage installs system packages and node packages together in one
-`script/bootstrap` step, and check builds may keep system packages but not node
-packages.
+that install system packages, download Go's modules or run `script/bootstrap` to
+install a Python `test` phase's packages from `Pipfile.lock`, or right after
+`FROM` in a stage with none of these, and every step that lints, tests or
+compiles comes after it. The `script/` builds pass a new `CACHEBUST` value on
+every build, which reruns every `RUN` step after that line and leaves the layers
+above it and the cache mounts in place; `--no-cache` would build those layers
+again and empty the mounts. This repo's own `Dockerfile` declares
+`ARG CACHEBUST` right after `FROM` in every stage: each stage installs system
+packages and node packages together in one `script/bootstrap` step, and check
+builds may keep system packages but not node packages.
 
 ## Rationale
 

TODO.md

@@ -24,16 +24,19 @@ fmt-check, and commit.
 - 2026-10-21: Check builds may now also keep the Python packages a `test` phase
   installs from `Pipfile.lock` (issue 133), on sneak's ruling in
   https://git.eeqj.de/sneak/rfscan/issues/80#issuecomment-153437. The install
-  comes before `ARG CACHEBUST`: `COPY Pipfile Pipfile.lock ./`, then one `RUN`
-  that runs `pipenv install --deploy`, or a `script/bootstrap` that installs
-  nothing beyond system packages, the pinned pipenv and the packages from the
-  lockfile. `--deploy` installs only what the lockfile names, checks every
-  package against its hashes, and fails when the lockfile is out of date with
-  `Pipfile`. The packages are kept as a layer, not a cache mount, for the reason
-  Go's modules are. The tests still run after `ARG CACHEBUST`, and every other
-  stage still installs its Python packages after it. `REPO_POLICIES.md` gains a
-  short Python `test` phase example, and both checklists and the README state
-  the rule the same way. rfscan picks this up on its next re-vendor.
+  comes before `ARG CACHEBUST`: `COPY script/ script/`,
+  `COPY Pipfile Pipfile.lock ./`, then `RUN script/bootstrap`. In that step
+  `script/bootstrap` installs nothing beyond system packages, pipenv and its
+  dependencies by hash, and the packages from the lockfile, and no Python, since
+  the tests use the base image's. `--deploy` installs only what the lockfile
+  names, checks every package against its hashes, and fails when the lockfile is
+  out of date with `Pipfile`. A kept layer holds the same hashed packages a new
+  install would, but its system packages may be older. The packages are kept as
+  a layer, not a cache mount, for the reason Go's modules are. The tests still
+  run after `ARG CACHEBUST`, and every other stage still installs its Python
+  packages after it. `REPO_POLICIES.md` gains a short Python `test` phase
+  example, and both checklists and the README state the rule the same way.
+  rfscan picks this up on its next re-vendor.
 - 2026-10-21: The canonical `.golangci.yml` now accepts a local `replace` of
   `github.com/grafana/sobek` in a repository's root `go.mod` (issue 134), for
   homoicon, which keeps a local copy of sobek on sneak's decision in

prompts/EXISTING_REPO_CHECKLIST.md

@@ -50,16 +50,17 @@ with your task.
       passing while the gate never runs. Confirm by planting a violation, not by
       reading the file.
 - [ ] In every stage `ARG CACHEBUST` comes right after the steps that install
-      system packages, download Go's modules or install a `test` phase's Python
-      packages from `Pipfile.lock`, or right after `FROM` in a stage with none
-      of these, and every step that lints, tests or compiles comes after it. A
-      step that installs anything else as well, such as a `script/bootstrap`
-      that also installs node packages, comes after it too. The `script/` builds
-      pass a new `CACHEBUST` value on every build, which reruns every `RUN` step
-      after that line and leaves the layers above it and the cache mounts in
-      place; `--no-cache` would build those layers again and empty the mounts. A
-      stage without the `ARG` line is served from cache on an unchanged tree, so
-      the scripts and the `ARG` lines are re-vendored in one commit.
+      system packages, download Go's modules or run `script/bootstrap` to
+      install a Python `test` phase's packages from `Pipfile.lock`, or right
+      after `FROM` in a stage with none of these, and every step that lints,
+      tests or compiles comes after it. A step that installs anything else as
+      well, such as a `script/bootstrap` that also installs node packages, comes
+      after it too. The `script/` builds pass a new `CACHEBUST` value on every
+      build, which reruns every `RUN` step after that line and leaves the layers
+      above it and the cache mounts in place; `--no-cache` would build those
+      layers again and empty the mounts. A stage without the `ARG` line is
+      served from cache on an unchanged tree, so the scripts and the `ARG` lines
+      are re-vendored in one commit.
 - [ ] Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`,
       `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile
       (`golangci-lint run`, `go test`, `go build`) mount Go's build cache
@@ -69,14 +70,16 @@ with your task.
       mounts anything else; both `go test` runs carry `-count=1`, as in the
       canonical Go `Dockerfile` in `REPO_POLICIES.md`
 - [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does
-      it above `ARG CACHEBUST`, with `COPY Pipfile Pipfile.lock ./` and then one
-      `RUN` that runs `pipenv install --deploy` (with `--dev` when the tests
-      need the development packages), or that runs `script/bootstrap` after
-      `COPY script/ script/`; that `script/bootstrap` installs nothing beyond
-      system packages, the pinned pipenv and the packages from the lockfile. The
-      tests run after `ARG CACHEBUST`, and every other stage installs its Python
-      packages after it, as in the Python `test` phase example in
-      `REPO_POLICIES.md`
+      it above `ARG CACHEBUST`, with `COPY script/ script/`,
+      `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`.
+      `script/bootstrap`, when it runs in that step, installs nothing beyond
+      system packages, pipenv and its dependencies by hash
+      (`pip install --require-hashes`), and the packages from the lockfile
+      (`pipenv install --deploy`, with `--dev` when the tests need the
+      development packages). It installs no Python there, because the tests use
+      the base image's. The tests run after `ARG CACHEBUST`, and every other
+      stage installs its Python packages after it, as in the Python `test` phase
+      example in `REPO_POLICIES.md`
 - [ ] The gate phases invoke their tools directly, never through `make lint` or
       `script/test` — those are themselves a `docker build` and would recurse
       inside a build step
@@ -241,9 +244,10 @@ with your task.
       (the runner image `docker.gitea.com/runner-images:ubuntu-latest`: docker,
       git and node 24, with no yarn on `PATH`), and demonstrably executed the
       checks — a sub-second build, or `CACHED` on a step that lints or tests,
-      one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. The steps
-      that install system packages and download Go's modules above that line are
-      expected to show `CACHED`.
+      one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above
+      that line, the steps that install system packages, download Go's modules
+      or run `script/bootstrap` to install a Python `test` phase's packages from
+      `Pipfile.lock` are expected to show `CACHED`.
 - [ ] A planted lint violation fails both `make lint` and a plain
       `docker build .`; revert it afterwards
 - [ ] Commit and merge fixes before starting your actual task

prompts/NEW_REPO_CHECKLIST.md

@@ -109,17 +109,17 @@ Template files can be fetched from:
       cannot be built unless both passed. Keep the final stage last: a stage
       nothing depends on is built only when `--target` names it.
     - In every stage `ARG CACHEBUST` comes right after the steps that install
-      system packages, download Go's modules or install a `test` phase's Python
-      packages from `Pipfile.lock`, or right after `FROM` in a stage with none
-      of these, and every step that lints, tests or compiles comes after it. A
-      step that installs anything else as well, such as a `script/bootstrap`
-      that also installs node packages, comes after it too. The `script/` builds
-      pass a new `CACHEBUST` value on every build, which reruns every `RUN` step
-      after that line and leaves the layers above it and the cache mounts in
-      place; `--no-cache` would build those layers again and empty the mounts. A
-      stage without the `ARG` line is served from cache on an unchanged tree, so
-      the scripts and every stage's `ARG CACHEBUST` line are added in the same
-      commit.
+      system packages, download Go's modules or run `script/bootstrap` to
+      install a Python `test` phase's packages from `Pipfile.lock`, or right
+      after `FROM` in a stage with none of these, and every step that lints,
+      tests or compiles comes after it. A step that installs anything else as
+      well, such as a `script/bootstrap` that also installs node packages, comes
+      after it too. The `script/` builds pass a new `CACHEBUST` value on every
+      build, which reruns every `RUN` step after that line and leaves the layers
+      above it and the cache mounts in place; `--no-cache` would build those
+      layers again and empty the mounts. A stage without the `ARG` line is
+      served from cache on an unchanged tree, so the scripts and every stage's
+      `ARG CACHEBUST` line are added in the same commit.
     - Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`,
       `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile
       (`golangci-lint run`, `go test`, `go build`) mount Go's build cache
@@ -129,14 +129,16 @@ Template files can be fetched from:
       mounts anything else; both `go test` runs carry `-count=1`, as in the
       canonical Go `Dockerfile` in `REPO_POLICIES.md`
     - Python: a `test` phase that installs its packages from `Pipfile.lock` does
-      it above `ARG CACHEBUST`, with `COPY Pipfile Pipfile.lock ./` and then one
-      `RUN` that runs `pipenv install --deploy` (with `--dev` when the tests
-      need the development packages), or that runs `script/bootstrap` after
-      `COPY script/ script/`; that `script/bootstrap` installs nothing beyond
-      system packages, the pinned pipenv and the packages from the lockfile. The
-      tests run after `ARG CACHEBUST`, and every other stage installs its Python
-      packages after it, as in the Python `test` phase example in
-      `REPO_POLICIES.md`
+      it above `ARG CACHEBUST`, with `COPY script/ script/`,
+      `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`.
+      `script/bootstrap`, when it runs in that step, installs nothing beyond
+      system packages, pipenv and its dependencies by hash
+      (`pip install --require-hashes`), and the packages from the lockfile
+      (`pipenv install --deploy`, with `--dev` when the tests need the
+      development packages). It installs no Python there, because the tests use
+      the base image's. The tests run after `ARG CACHEBUST`, and every other
+      stage installs its Python packages after it, as in the Python `test` phase
+      example in `REPO_POLICIES.md`
     - Server: the final stage builds and runs the application
     - Non-server: the final stage brings up the dev environment
     - Image pinned by sha256 hash with version/date comment
@@ -225,9 +227,10 @@ are thin shims calling them. Model scripts:
       (the runner image `docker.gitea.com/runner-images:ubuntu-latest`: docker,
       git and node 24, with no yarn on `PATH`), and demonstrably executed the
       checks — a sub-second build, or `CACHED` on a step that lints or tests,
-      one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. The steps
-      that install system packages and download Go's modules above that line are
-      expected to show `CACHED`.
+      one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above
+      that line, the steps that install system packages, download Go's modules
+      or run `script/bootstrap` to install a Python `test` phase's packages from
+      `Pipfile.lock` are expected to show `CACHED`.
 - [ ] Plant a lint violation and confirm both `make lint` and a plain
       `docker build .` fail on it; revert. A plain build that passes proves the
       final stage is missing its `COPY --from=` edge to the gate phases.

prompts/REPO_POLICIES.md

@@ -175,26 +175,27 @@ style conventions are in separate documents:
   `script/docker` are the four, and there is no fifth — `script/check` runs the
   two gate phases and `script/fmt-check`, and builds no image of its own. In
   every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
-  that install system packages, download Go's modules or install a `test`
-  phase's Python packages from `Pipfile.lock`, or right after `FROM` in a stage
-  with none of these, and every step that lints, tests or compiles comes after
-  it. A step that installs anything else as well, such as a `script/bootstrap`
-  that also installs node packages, comes after it too. A new value reruns every
-  `RUN` step after that line; `COPY` steps are compared by content and need
-  nothing. Docker keeps the layers above the line until the base image, their
-  own lines or the files they copy change. A stage without the `ARG` line is
-  served from cache on an unchanged tree, so a repository re-vendors the scripts
-  and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
-  it reruns every step too, but it also builds the layers above the line again
-  and empties every cache mount, so Go's build cache and golangci-lint's, which
-  the canonical Go `Dockerfile` below mounts so that they survive between check
-  builds, would start empty every time. A bare `docker build .` is not evidence
-  that anything ran: a sub-second build reporting success is a cache hit, not a
-  result. Never invalidate by pruning — `docker builder prune` and friends
-  destroy a build cache shared with every other build on the host. When a check
-  is added or changed, prove it works by planting a defect it must catch and
-  watching the run fail on it, then revert the defect. A green run alone shows
-  neither that the check ran nor that it covers what it should.
+  that install system packages, download Go's modules or run `script/bootstrap`
+  to install a Python `test` phase's packages from `Pipfile.lock`, or right
+  after `FROM` in a stage with none of these, and every step that lints, tests
+  or compiles comes after it. A step that installs anything else as well, such
+  as a `script/bootstrap` that also installs node packages, comes after it too.
+  A new value reruns every `RUN` step after that line; `COPY` steps are compared
+  by content and need nothing. Docker keeps the layers above the line until the
+  base image, their own lines or the files they copy change. A stage without the
+  `ARG` line is served from cache on an unchanged tree, so a repository
+  re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
+  not pass `--no-cache`: it reruns every step too, but it also builds the layers
+  above the line again and empties every cache mount, so Go's build cache and
+  golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
+  survive between check builds, would start empty every time. A bare
+  `docker build .` is not evidence that anything ran: a sub-second build
+  reporting success is a cache hit, not a result. Never invalidate by pruning —
+  `docker builder prune` and friends destroy a build cache shared with every
+  other build on the host. When a check is added or changed, prove it works by
+  planting a defect it must catch and watching the run fail on it, then revert
+  the defect. A green run alone shows neither that the check ran nor that it
+  covers what it should.
 
 - **The gate phases are separate stages, and the build stage depends on both.**
   The lint phase is based on the `golangci/golangci-lint` image (pinned by
@@ -309,8 +310,9 @@ style conventions are in separate documents:
           a package's findings only when the package's files and those of every
           package it imports, `go.mod`, the linters and their settings in
           `.golangci.yml`, and the golangci-lint version are all unchanged.
-        - In a Python `test` phase, the packages installed from `Pipfile.lock`,
-          as described for that phase below.
+        - In a Python `test` phase, the layer of the step that runs
+          `script/bootstrap` to install the packages from `Pipfile.lock`, as
+          described for that phase below.
     - If the project uses `//go:embed` directives that reference build artifacts
       (e.g. a web frontend compiled in a separate stage), the lint phase must
       create placeholder files so the embed directives resolve. Example:
@@ -365,31 +367,51 @@ style conventions are in separate documents:
       `--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
       do.
 
-- **A Python `test` phase installs its packages from `Pipfile.lock` above
-  `ARG CACHEBUST`, so check builds keep them.** The install has steps of its
-  own: `COPY Pipfile Pipfile.lock ./`, then one `RUN` that runs
-  `pipenv install --deploy`, with `--dev` when the tests need the development
-  packages. Docker keeps what that `RUN` installed until the base image,
-  `Pipfile`, `Pipfile.lock` or a line up to and including the `RUN` changes. If
-  the `RUN` runs `script/bootstrap` instead, as in the example below,
-  `COPY script/ script/` comes before it, so Docker installs again when
-  `script/` changes too. That `script/bootstrap` may install nothing beyond
-  system packages, the pinned pipenv and the packages from the lockfile; one
-  that installs anything else comes after `ARG CACHEBUST`. A kept layer holds
-  the same packages a new install would: `--deploy` installs only what
-  `Pipfile.lock` names, checks every package against the lockfile's hashes, and
-  fails when the lockfile is out of date with `Pipfile`. The packages are kept
-  in a layer rather than a cache mount for the reason Go's modules are: a cache
-  mount is shared by every build on the host, and nothing checks its files
-  again. The tests run after `ARG CACHEBUST`, on every check build. In every
-  other stage, a step that installs Python packages comes after `ARG CACHEBUST`.
-  An example, with a `script/bootstrap` that installs the pinned pipenv and then
-  runs `pipenv install --dev --deploy`:
+- **A Python `test` phase that installs its packages from `Pipfile.lock` does it
+  above `ARG CACHEBUST`, so check builds keep them.** The steps are
+  `COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and
+  `RUN script/bootstrap`, with `ARG CACHEBUST` right after them. Docker keeps
+  what that `RUN` installed until the base image, `script/`, `Pipfile`,
+  `Pipfile.lock` or a line up to and including the `RUN` changes. The tests run
+  after `ARG CACHEBUST`, on every check build. In every other stage, a step that
+  installs Python packages comes after `ARG CACHEBUST`.
+    - When `script/bootstrap` runs in that step, it installs nothing beyond
+      system packages, pipenv and its dependencies, and the packages from
+      `Pipfile.lock`. It installs no Python there: the tests use the base
+      image's Python, which the image's hash pins. If it installs anything else
+      there, the step goes after `ARG CACHEBUST`.
+    - It installs pipenv and its dependencies by hash, as the rule that every
+      external reference is pinned by hash requires. It does so with
+      `pip install --require-hashes -r` and a requirements file in `script/`
+      that lists each of them with `==` and its `--hash`. pip refuses to install
+      a dependency the file does not list.
+    - It then runs `pipenv install --deploy`, with `--dev` when the tests need
+      the development packages. `--deploy` installs only what `Pipfile.lock`
+      names, checks every package against the lockfile's hashes, and fails when
+      the lockfile is out of date with `Pipfile`.
+    - A kept layer holds the same pipenv, the same dependencies of pipenv and
+      the same lockfile packages that a new install would, because each was
+      checked against its hash. The pip and setuptools inside the virtualenv
+      that pipenv creates are the same too: the `virtualenv` package, one of
+      pipenv's dependencies, copies them from wheels it ships, and its hash
+      covers those wheels. Any system packages the step installed are not pinned
+      by hash: the layer keeps the versions the package manager gave when the
+      layer was built, and a new install may get newer ones.
+    - The packages are kept in a layer rather than a cache mount for the reason
+      Go's modules are: a cache mount is shared by every build on the host, and
+      nothing checks its files again.
+
+    In this example, `script/bootstrap` finds `make` and `git` already
+    installed, installs pipenv and its dependencies by hash, and runs
+    `pipenv install --dev --deploy`:
 
     ```dockerfile
     # Test phase. The 90-second timeout covers both pytest runs.
     # python:3.x-slim, YYYY-MM-DD
     FROM python@sha256:... AS test
+    # make and git are base tools that script/bootstrap installs when they
+    # are missing. Installed here, they stay in their own layer when
+    # script/ changes.
     RUN apt-get update \
         && apt-get install -y --no-install-recommends make git \
         && rm -rf /var/lib/apt/lists/*