From 83f148f3b208b7f76579f6732d3028841bb76451 Mon Sep 17 00:00:00 2001 From: sneak Date: Wed, 21 Oct 2026 15:11:57 +0000 Subject: [PATCH] Keep a Python test phase's lockfile install in check builds (closes #133) On sneak's ruling in https://git.eeqj.de/sneak/rfscan/issues/80#issuecomment-153437, a Python test phase installs its packages from Pipfile.lock above ARG CACHEBUST, so Docker keeps them until the base image, Pipfile, Pipfile.lock or a line up to the install changes. The step is COPY Pipfile Pipfile.lock ./ and one RUN of pipenv install --deploy, or a script/bootstrap that installs nothing beyond system packages, the pinned pipenv and the lockfile's packages. --deploy installs only what the lockfile names and checks its hashes. The tests, and every other stage's Python installs, stay after the line. REPO_POLICIES.md gains a short example; both checklists and the README state the rule the same way. Model: opus-5-5 --- README.md | 19 +++---- TODO.md | 13 +++++ prompts/EXISTING_REPO_CHECKLIST.md | 31 +++++++---- prompts/NEW_REPO_CHECKLIST.md | 32 ++++++++---- prompts/REPO_POLICIES.md | 83 ++++++++++++++++++++++-------- 5 files changed, 126 insertions(+), 52 deletions(-) diff --git a/README.md b/README.md index fce9748..f188c5f 100644 --- a/README.md +++ b/README.md @@ -145,15 +145,16 @@ alpine. We provide: `make hooks` installs the pre-commit hook that runs `script/precommit`. In every stage of a `Dockerfile`, `ARG CACHEBUST` comes right after the steps -that install system packages and download Go's modules, or right after `FROM` in -a stage with neither, and every step that lints, tests or compiles comes after -it. The `script/` builds pass a new `CACHEBUST` value on every build, which -reruns every `RUN` step after that line and leaves the layers above it and the -cache mounts in place; `--no-cache` would build those layers again and empty the -mounts. This repo's own `Dockerfile` declares `ARG CACHEBUST` right after `FROM` -in every stage: each stage installs system packages and node packages together -in one `script/bootstrap` step, and check builds may keep system packages but -not node packages. +that install system packages, download Go's modules or install a `test` phase's +Python packages from `Pipfile.lock`, or right after `FROM` in a stage with none +of these, and every step that lints, tests or compiles comes after it. The +`script/` builds pass a new `CACHEBUST` value on every build, which reruns every +`RUN` step after that line and leaves the layers above it and the cache mounts +in place; `--no-cache` would build those layers again and empty the mounts. This +repo's own `Dockerfile` declares `ARG CACHEBUST` right after `FROM` in every +stage: each stage installs system packages and node packages together in one +`script/bootstrap` step, and check builds may keep system packages but not node +packages. ## Rationale diff --git a/TODO.md b/TODO.md index 8c927fd..f6621ec 100644 --- a/TODO.md +++ b/TODO.md @@ -21,6 +21,19 @@ fmt-check, and commit. # Completed Steps +- 2026-10-21: Check builds may now also keep the Python packages a `test` phase + installs from `Pipfile.lock` (issue 133), on sneak's ruling in + https://git.eeqj.de/sneak/rfscan/issues/80#issuecomment-153437. The install + comes before `ARG CACHEBUST`: `COPY Pipfile Pipfile.lock ./`, then one `RUN` + that runs `pipenv install --deploy`, or a `script/bootstrap` that installs + nothing beyond system packages, the pinned pipenv and the packages from the + lockfile. `--deploy` installs only what the lockfile names, checks every + package against its hashes, and fails when the lockfile is out of date with + `Pipfile`. The packages are kept as a layer, not a cache mount, for the reason + Go's modules are. The tests still run after `ARG CACHEBUST`, and every other + stage still installs its Python packages after it. `REPO_POLICIES.md` gains a + short Python `test` phase example, and both checklists and the README state + the rule the same way. rfscan picks this up on its next re-vendor. - 2026-10-21: The canonical `.golangci.yml` now accepts a local `replace` of `github.com/grafana/sobek` in a repository's root `go.mod` (issue 134), for homoicon, which keeps a local copy of sobek on sneak's decision in diff --git a/prompts/EXISTING_REPO_CHECKLIST.md b/prompts/EXISTING_REPO_CHECKLIST.md index 4f1358e..5678f8f 100644 --- a/prompts/EXISTING_REPO_CHECKLIST.md +++ b/prompts/EXISTING_REPO_CHECKLIST.md @@ -1,6 +1,6 @@ --- title: Existing Repo Checklist -last_modified: 2026-10-18 +last_modified: 2026-10-21 --- Use this checklist when beginning work in a repo that may not yet conform to our @@ -50,16 +50,16 @@ with your task. passing while the gate never runs. Confirm by planting a violation, not by reading the file. - [ ] In every stage `ARG CACHEBUST` comes right after the steps that install - system packages and download Go's modules, or right after `FROM` in a - stage with neither, and every step that lints, tests or compiles comes - after it. A step that installs anything else as well, such as a - `script/bootstrap` that also installs node packages, comes after it too. - The `script/` builds pass a new `CACHEBUST` value on every build, which - reruns every `RUN` step after that line and leaves the layers above it and - the cache mounts in place; `--no-cache` would build those layers again and - empty the mounts. A stage without the `ARG` line is served from cache on - an unchanged tree, so the scripts and the `ARG` lines are re-vendored in - one commit. + system packages, download Go's modules or install a `test` phase's Python + packages from `Pipfile.lock`, or right after `FROM` in a stage with none + of these, and every step that lints, tests or compiles comes after it. A + step that installs anything else as well, such as a `script/bootstrap` + that also installs node packages, comes after it too. The `script/` builds + pass a new `CACHEBUST` value on every build, which reruns every `RUN` step + after that line and leaves the layers above it and the cache mounts in + place; `--no-cache` would build those layers again and empty the mounts. A + stage without the `ARG` line is served from cache on an unchanged tree, so + the scripts and the `ARG` lines are re-vendored in one commit. - [ ] Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`, `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile (`golangci-lint run`, `go test`, `go build`) mount Go's build cache @@ -68,6 +68,15 @@ with your task. (`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step mounts anything else; both `go test` runs carry `-count=1`, as in the canonical Go `Dockerfile` in `REPO_POLICIES.md` +- [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does + it above `ARG CACHEBUST`, with `COPY Pipfile Pipfile.lock ./` and then one + `RUN` that runs `pipenv install --deploy` (with `--dev` when the tests + need the development packages), or that runs `script/bootstrap` after + `COPY script/ script/`; that `script/bootstrap` installs nothing beyond + system packages, the pinned pipenv and the packages from the lockfile. The + tests run after `ARG CACHEBUST`, and every other stage installs its Python + packages after it, as in the Python `test` phase example in + `REPO_POLICIES.md` - [ ] The gate phases invoke their tools directly, never through `make lint` or `script/test` — those are themselves a `docker build` and would recurse inside a build step diff --git a/prompts/NEW_REPO_CHECKLIST.md b/prompts/NEW_REPO_CHECKLIST.md index 31f4afc..ed83ad2 100644 --- a/prompts/NEW_REPO_CHECKLIST.md +++ b/prompts/NEW_REPO_CHECKLIST.md @@ -1,6 +1,6 @@ --- title: New Repo Checklist -last_modified: 2026-10-18 +last_modified: 2026-10-21 --- Use this checklist when creating a new repository from scratch. Follow the steps @@ -109,16 +109,17 @@ Template files can be fetched from: cannot be built unless both passed. Keep the final stage last: a stage nothing depends on is built only when `--target` names it. - In every stage `ARG CACHEBUST` comes right after the steps that install - system packages and download Go's modules, or right after `FROM` in a - stage with neither, and every step that lints, tests or compiles comes - after it. A step that installs anything else as well, such as a - `script/bootstrap` that also installs node packages, comes after it too. - The `script/` builds pass a new `CACHEBUST` value on every build, which - reruns every `RUN` step after that line and leaves the layers above it and - the cache mounts in place; `--no-cache` would build those layers again and - empty the mounts. A stage without the `ARG` line is served from cache on - an unchanged tree, so the scripts and every stage's `ARG CACHEBUST` line - are added in the same commit. + system packages, download Go's modules or install a `test` phase's Python + packages from `Pipfile.lock`, or right after `FROM` in a stage with none + of these, and every step that lints, tests or compiles comes after it. A + step that installs anything else as well, such as a `script/bootstrap` + that also installs node packages, comes after it too. The `script/` builds + pass a new `CACHEBUST` value on every build, which reruns every `RUN` step + after that line and leaves the layers above it and the cache mounts in + place; `--no-cache` would build those layers again and empty the mounts. A + stage without the `ARG` line is served from cache on an unchanged tree, so + the scripts and every stage's `ARG CACHEBUST` line are added in the same + commit. - Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`, `RUN go mod download`) above `ARG CACHEBUST`; the steps that compile (`golangci-lint run`, `go test`, `go build`) mount Go's build cache @@ -127,6 +128,15 @@ Template files can be fetched from: (`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step mounts anything else; both `go test` runs carry `-count=1`, as in the canonical Go `Dockerfile` in `REPO_POLICIES.md` + - Python: a `test` phase that installs its packages from `Pipfile.lock` does + it above `ARG CACHEBUST`, with `COPY Pipfile Pipfile.lock ./` and then one + `RUN` that runs `pipenv install --deploy` (with `--dev` when the tests + need the development packages), or that runs `script/bootstrap` after + `COPY script/ script/`; that `script/bootstrap` installs nothing beyond + system packages, the pinned pipenv and the packages from the lockfile. The + tests run after `ARG CACHEBUST`, and every other stage installs its Python + packages after it, as in the Python `test` phase example in + `REPO_POLICIES.md` - Server: the final stage builds and runs the application - Non-server: the final stage brings up the dev environment - Image pinned by sha256 hash with version/date comment diff --git a/prompts/REPO_POLICIES.md b/prompts/REPO_POLICIES.md index 1067698..7ff50a9 100644 --- a/prompts/REPO_POLICIES.md +++ b/prompts/REPO_POLICIES.md @@ -1,6 +1,6 @@ --- title: Repository Policies -last_modified: 2026-10-18 +last_modified: 2026-10-21 --- This document covers repository structure, tooling, and workflow standards. Code @@ -175,26 +175,26 @@ style conventions are in separate documents: `script/docker` are the four, and there is no fifth — `script/check` runs the two gate phases and `script/fmt-check`, and builds no image of its own. In every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps - that install system packages and download Go's modules, or right after `FROM` - in a stage with neither, and every step that lints, tests or compiles comes - after it. A step that installs anything else as well, such as a - `script/bootstrap` that also installs node packages, comes after it too. A new - value reruns every `RUN` step after that line; `COPY` steps are compared by - content and need nothing. Docker keeps the layers above the line until the - base image, their own lines or the files they copy change. A stage without the - `ARG` line is served from cache on an unchanged tree, so a repository - re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do - not pass `--no-cache`: it reruns every step too, but it also builds the layers - above the line again and empties every cache mount, so Go's build cache and - golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they - survive between check builds, would start empty every time. A bare - `docker build .` is not evidence that anything ran: a sub-second build - reporting success is a cache hit, not a result. Never invalidate by pruning — - `docker builder prune` and friends destroy a build cache shared with every - other build on the host. When a check is added or changed, prove it works by - planting a defect it must catch and watching the run fail on it, then revert - the defect. A green run alone shows neither that the check ran nor that it - covers what it should. + that install system packages, download Go's modules or install a `test` + phase's Python packages from `Pipfile.lock`, or right after `FROM` in a stage + with none of these, and every step that lints, tests or compiles comes after + it. A step that installs anything else as well, such as a `script/bootstrap` + that also installs node packages, comes after it too. A new value reruns every + `RUN` step after that line; `COPY` steps are compared by content and need + nothing. Docker keeps the layers above the line until the base image, their + own lines or the files they copy change. A stage without the `ARG` line is + served from cache on an unchanged tree, so a repository re-vendors the scripts + and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`: + it reruns every step too, but it also builds the layers above the line again + and empties every cache mount, so Go's build cache and golangci-lint's, which + the canonical Go `Dockerfile` below mounts so that they survive between check + builds, would start empty every time. A bare `docker build .` is not evidence + that anything ran: a sub-second build reporting success is a cache hit, not a + result. Never invalidate by pruning — `docker builder prune` and friends + destroy a build cache shared with every other build on the host. When a check + is added or changed, prove it works by planting a defect it must catch and + watching the run fail on it, then revert the defect. A green run alone shows + neither that the check ran nor that it covers what it should. - **The gate phases are separate stages, and the build stage depends on both.** The lint phase is based on the `golangci/golangci-lint` image (pinned by @@ -309,6 +309,8 @@ style conventions are in separate documents: a package's findings only when the package's files and those of every package it imports, `go.mod`, the linters and their settings in `.golangci.yml`, and the golangci-lint version are all unchanged. + - In a Python `test` phase, the packages installed from `Pipfile.lock`, + as described for that phase below. - If the project uses `//go:embed` directives that reference build artifacts (e.g. a web frontend compiled in a separate stage), the lint phase must create placeholder files so the embed directives resolve. Example: @@ -363,6 +365,45 @@ style conventions are in separate documents: `--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already do. +- **A Python `test` phase installs its packages from `Pipfile.lock` above + `ARG CACHEBUST`, so check builds keep them.** The install has steps of its + own: `COPY Pipfile Pipfile.lock ./`, then one `RUN` that runs + `pipenv install --deploy`, with `--dev` when the tests need the development + packages. Docker keeps what that `RUN` installed until the base image, + `Pipfile`, `Pipfile.lock` or a line up to and including the `RUN` changes. If + the `RUN` runs `script/bootstrap` instead, as in the example below, + `COPY script/ script/` comes before it, so Docker installs again when + `script/` changes too. That `script/bootstrap` may install nothing beyond + system packages, the pinned pipenv and the packages from the lockfile; one + that installs anything else comes after `ARG CACHEBUST`. A kept layer holds + the same packages a new install would: `--deploy` installs only what + `Pipfile.lock` names, checks every package against the lockfile's hashes, and + fails when the lockfile is out of date with `Pipfile`. The packages are kept + in a layer rather than a cache mount for the reason Go's modules are: a cache + mount is shared by every build on the host, and nothing checks its files + again. The tests run after `ARG CACHEBUST`, on every check build. In every + other stage, a step that installs Python packages comes after `ARG CACHEBUST`. + An example, with a `script/bootstrap` that installs the pinned pipenv and then + runs `pipenv install --dev --deploy`: + + ```dockerfile + # Test phase. The 90-second timeout covers both pytest runs. + # python:3.x-slim, YYYY-MM-DD + FROM python@sha256:... AS test + RUN apt-get update \ + && apt-get install -y --no-install-recommends make git \ + && rm -rf /var/lib/apt/lists/* + WORKDIR /app + COPY script/ script/ + COPY Pipfile Pipfile.lock ./ + RUN script/bootstrap + ARG CACHEBUST + COPY . . + RUN timeout 90 sh -c 'pipenv run pytest || { \ + echo "--- Rerunning with -v for details ---"; \ + pipenv run pytest -v; exit 1; }' + ``` + - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that runs `script/cibuild` on push, and checks out the repo as its only other step, with `persist-credentials: false`: `script/cibuild` needs no token, and