Keep a Python test phase's lockfile install in check builds #135
No files matched your search
@@ -144,17 +144,22 @@ alpine. We provide:
|
|||||||
|
|
||||||
`make hooks` installs the pre-commit hook that runs `script/precommit`.
|
`make hooks` installs the pre-commit hook that runs `script/precommit`.
|
||||||
|
|
||||||
In every stage of a `Dockerfile`, `ARG CACHEBUST` comes right after the steps
|
In every stage of a `Dockerfile`, every step that lints, tests or compiles comes
|
||||||
that install system packages, download Go's modules or run `script/bootstrap` to
|
after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps:
|
||||||
install a Python `test` phase's packages from `Pipfile.lock`, or right after
|
|
||||||
`FROM` in a stage with none of these, and every step that lints, tests or
|
- the steps that install system packages
|
||||||
compiles comes after it. The `script/` builds pass a new `CACHEBUST` value on
|
- the steps that download Go's modules
|
||||||
every build, which reruns every `RUN` step after that line and leaves the layers
|
- the step that runs `script/bootstrap` to install a Python `test` phase's
|
||||||
above it and the cache mounts in place; `--no-cache` would build those layers
|
packages from `Pipfile.lock`
|
||||||
again and empty the mounts. This repo's own `Dockerfile` declares
|
|
||||||
`ARG CACHEBUST` right after `FROM` in every stage: each stage installs system
|
A stage with none of these steps declares `ARG CACHEBUST` right after `FROM`.
|
||||||
packages and node packages together in one `script/bootstrap` step, and check
|
The `script/` builds pass a new `CACHEBUST` value on every build, which reruns
|
||||||
builds may keep system packages but not node packages.
|
every `RUN` step after that line and leaves the layers above it and the cache
|
||||||
|
mounts in place; `--no-cache` would build those layers again and empty the
|
||||||
|
mounts. This repo's own `Dockerfile` declares `ARG CACHEBUST` right after `FROM`
|
||||||
|
in every stage: each stage installs system packages and node packages together
|
||||||
|
in one `script/bootstrap` step, and check builds may keep system packages but
|
||||||
|
not node packages.
|
||||||
|
|
||||||
## Rationale
|
## Rationale
|
||||||
|
|
||||||
|
|||||||
@@ -27,17 +27,16 @@ fmt-check, and commit.
|
|||||||
comes before `ARG CACHEBUST`: `COPY script/ script/`,
|
comes before `ARG CACHEBUST`: `COPY script/ script/`,
|
||||||
`COPY Pipfile Pipfile.lock ./`, then `RUN script/bootstrap`. In that step
|
`COPY Pipfile Pipfile.lock ./`, then `RUN script/bootstrap`. In that step
|
||||||
`script/bootstrap` installs nothing beyond system packages, pipenv and its
|
`script/bootstrap` installs nothing beyond system packages, pipenv and its
|
||||||
dependencies by hash, and the packages from the lockfile, and no Python, since
|
dependencies by hash, and the packages from the lockfile. It installs no
|
||||||
the tests use the base image's. `--deploy` installs only what the lockfile
|
Python, because the tests use the base image's. `--deploy` installs only what
|
||||||
names, checks every package against its hashes, and fails when the lockfile is
|
the lockfile names, checks every package against its hashes, and fails when
|
||||||
out of date with `Pipfile`. A kept layer holds nothing a new install would
|
the lockfile is out of date with `Pipfile`. A kept layer holds nothing a new
|
||||||
refuse, except that a package built from source and the system packages may
|
install would refuse. A package built from source and the system packages may
|
||||||
differ from a new install. The packages are kept as a layer, not a cache
|
differ in a new install. As with Go's modules, the packages are kept in a
|
||||||
mount, for the reason Go's modules are. The tests still run after
|
layer, not a cache mount. The tests still run after `ARG CACHEBUST`, and every
|
||||||
`ARG CACHEBUST`, and every other stage still installs its Python packages
|
other stage still installs its Python packages after it. `REPO_POLICIES.md`
|
||||||
after it. `REPO_POLICIES.md` gains a short Python `test` phase example, and
|
gains a short Python `test` phase example, and both checklists and the README
|
||||||
both checklists and the README state the rule the same way. rfscan picks this
|
state the rule the same way. rfscan picks this up on its next re-vendor.
|
||||||
up on its next re-vendor.
|
|
||||||
- 2026-10-21: The canonical `.golangci.yml` now accepts a local `replace` of
|
- 2026-10-21: The canonical `.golangci.yml` now accepts a local `replace` of
|
||||||
`github.com/grafana/sobek` in a repository's root `go.mod` (issue 134), for
|
`github.com/grafana/sobek` in a repository's root `go.mod` (issue 134), for
|
||||||
homoicon, which keeps a local copy of sobek on sneak's decision in
|
homoicon, which keeps a local copy of sobek on sneak's decision in
|
||||||
|
|||||||
@@ -49,18 +49,21 @@ with your task.
|
|||||||
`--target` names it, so a lost `COPY --from=` edge leaves `docker build .`
|
`--target` names it, so a lost `COPY --from=` edge leaves `docker build .`
|
||||||
passing while the gate never runs. Confirm by planting a violation, not by
|
passing while the gate never runs. Confirm by planting a violation, not by
|
||||||
reading the file.
|
reading the file.
|
||||||
- [ ] In every stage `ARG CACHEBUST` comes right after the steps that install
|
- [ ] In every stage, every step that lints, tests or compiles comes after
|
||||||
system packages, download Go's modules or run `script/bootstrap` to
|
`ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps:
|
||||||
install a Python `test` phase's packages from `Pipfile.lock`, or right
|
- the steps that install system packages
|
||||||
after `FROM` in a stage with none of these, and every step that lints,
|
- the steps that download Go's modules
|
||||||
tests or compiles comes after it. A step that installs anything else as
|
- the step that runs `script/bootstrap` to install a Python `test` phase's
|
||||||
well, such as a `script/bootstrap` that also installs node packages, comes
|
packages from `Pipfile.lock`
|
||||||
after it too. The `script/` builds pass a new `CACHEBUST` value on every
|
- [ ] A stage with none of these steps declares `ARG CACHEBUST` right after
|
||||||
build, which reruns every `RUN` step after that line and leaves the layers
|
`FROM`. A step that installs anything else as well, such as a
|
||||||
above it and the cache mounts in place; `--no-cache` would build those
|
`script/bootstrap` that also installs node packages, comes after it too.
|
||||||
layers again and empty the mounts. A stage without the `ARG` line is
|
The `script/` builds pass a new `CACHEBUST` value on every build, which
|
||||||
served from cache on an unchanged tree, so the scripts and the `ARG` lines
|
reruns every `RUN` step after that line and leaves the layers above it and
|
||||||
are re-vendored in one commit.
|
the cache mounts in place; `--no-cache` would build those layers again and
|
||||||
|
empty the mounts. A stage without the `ARG` line is served from cache on
|
||||||
|
an unchanged tree, so the scripts and the `ARG` lines are re-vendored in
|
||||||
|
one commit.
|
||||||
- [ ] Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`,
|
- [ ] Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`,
|
||||||
`RUN go mod download`) above `ARG CACHEBUST`; the steps that compile
|
`RUN go mod download`) above `ARG CACHEBUST`; the steps that compile
|
||||||
(`golangci-lint run`, `go test`, `go build`) mount Go's build cache
|
(`golangci-lint run`, `go test`, `go build`) mount Go's build cache
|
||||||
@@ -71,15 +74,16 @@ with your task.
|
|||||||
canonical Go `Dockerfile` in `REPO_POLICIES.md`
|
canonical Go `Dockerfile` in `REPO_POLICIES.md`
|
||||||
- [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does
|
- [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does
|
||||||
it above `ARG CACHEBUST`, with `COPY script/ script/`,
|
it above `ARG CACHEBUST`, with `COPY script/ script/`,
|
||||||
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`.
|
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When
|
||||||
`script/bootstrap`, when it runs in that step, installs nothing beyond
|
`script/bootstrap` runs in that step, it installs nothing beyond system
|
||||||
system packages, pipenv and its dependencies by hash
|
packages, pipenv and its dependencies, and the packages from
|
||||||
(`pip install --require-hashes`), and the packages from the lockfile
|
`Pipfile.lock`. It installs pipenv and its dependencies by hash, with
|
||||||
(`pipenv install --deploy`, with `--dev` when the tests need the
|
`pip install --require-hashes`. It installs the packages from
|
||||||
development packages). It installs no Python there, because the tests use
|
`Pipfile.lock` with `pipenv install --deploy`, plus `--dev` when the tests
|
||||||
the base image's. The tests run after `ARG CACHEBUST`, and every other
|
need the development packages. It installs no Python there, because the
|
||||||
stage installs its Python packages after it, as in the Python `test` phase
|
tests use the base image's. The tests run after `ARG CACHEBUST`, and every
|
||||||
example in `REPO_POLICIES.md`
|
other stage installs its Python packages after it, as in the Python `test`
|
||||||
|
phase example in `REPO_POLICIES.md`
|
||||||
- [ ] The gate phases invoke their tools directly, never through `make lint` or
|
- [ ] The gate phases invoke their tools directly, never through `make lint` or
|
||||||
`script/test` — those are themselves a `docker build` and would recurse
|
`script/test` — those are themselves a `docker build` and would recurse
|
||||||
inside a build step
|
inside a build step
|
||||||
@@ -245,9 +249,11 @@ with your task.
|
|||||||
git and node 24, with no yarn on `PATH`), and demonstrably executed the
|
git and node 24, with no yarn on `PATH`), and demonstrably executed the
|
||||||
checks — a sub-second build, or `CACHED` on a step that lints or tests,
|
checks — a sub-second build, or `CACHED` on a step that lints or tests,
|
||||||
one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above
|
one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above
|
||||||
that line, the steps that install system packages, download Go's modules
|
that line, these steps are expected to show `CACHED`:
|
||||||
or run `script/bootstrap` to install a Python `test` phase's packages from
|
- the steps that install system packages
|
||||||
`Pipfile.lock` are expected to show `CACHED`.
|
- the steps that download Go's modules
|
||||||
|
- the step that runs `script/bootstrap` to install a Python `test` phase's
|
||||||
|
packages from `Pipfile.lock`
|
||||||
- [ ] A planted lint violation fails both `make lint` and a plain
|
- [ ] A planted lint violation fails both `make lint` and a plain
|
||||||
`docker build .`; revert it afterwards
|
`docker build .`; revert it afterwards
|
||||||
- [ ] Commit and merge fixes before starting your actual task
|
- [ ] Commit and merge fixes before starting your actual task
|
||||||
@@ -108,18 +108,21 @@ Template files can be fetched from:
|
|||||||
stage carries a `COPY --from=` of a harmless file from each so the image
|
stage carries a `COPY --from=` of a harmless file from each so the image
|
||||||
cannot be built unless both passed. Keep the final stage last: a stage
|
cannot be built unless both passed. Keep the final stage last: a stage
|
||||||
nothing depends on is built only when `--target` names it.
|
nothing depends on is built only when `--target` names it.
|
||||||
- In every stage `ARG CACHEBUST` comes right after the steps that install
|
- In every stage, every step that lints, tests or compiles comes after
|
||||||
system packages, download Go's modules or run `script/bootstrap` to
|
`ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps:
|
||||||
install a Python `test` phase's packages from `Pipfile.lock`, or right
|
- the steps that install system packages
|
||||||
after `FROM` in a stage with none of these, and every step that lints,
|
- the steps that download Go's modules
|
||||||
tests or compiles comes after it. A step that installs anything else as
|
- the step that runs `script/bootstrap` to install a Python `test`
|
||||||
well, such as a `script/bootstrap` that also installs node packages, comes
|
phase's packages from `Pipfile.lock`
|
||||||
after it too. The `script/` builds pass a new `CACHEBUST` value on every
|
- A stage with none of these steps declares `ARG CACHEBUST` right after
|
||||||
build, which reruns every `RUN` step after that line and leaves the layers
|
`FROM`. A step that installs anything else as well, such as a
|
||||||
above it and the cache mounts in place; `--no-cache` would build those
|
`script/bootstrap` that also installs node packages, comes after it too.
|
||||||
layers again and empty the mounts. A stage without the `ARG` line is
|
The `script/` builds pass a new `CACHEBUST` value on every build, which
|
||||||
served from cache on an unchanged tree, so the scripts and every stage's
|
reruns every `RUN` step after that line and leaves the layers above it and
|
||||||
`ARG CACHEBUST` line are added in the same commit.
|
the cache mounts in place; `--no-cache` would build those layers again and
|
||||||
|
empty the mounts. A stage without the `ARG` line is served from cache on
|
||||||
|
an unchanged tree, so the scripts and every stage's `ARG CACHEBUST` line
|
||||||
|
are added in the same commit.
|
||||||
- Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`,
|
- Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`,
|
||||||
`RUN go mod download`) above `ARG CACHEBUST`; the steps that compile
|
`RUN go mod download`) above `ARG CACHEBUST`; the steps that compile
|
||||||
(`golangci-lint run`, `go test`, `go build`) mount Go's build cache
|
(`golangci-lint run`, `go test`, `go build`) mount Go's build cache
|
||||||
@@ -130,15 +133,16 @@ Template files can be fetched from:
|
|||||||
canonical Go `Dockerfile` in `REPO_POLICIES.md`
|
canonical Go `Dockerfile` in `REPO_POLICIES.md`
|
||||||
- Python: a `test` phase that installs its packages from `Pipfile.lock` does
|
- Python: a `test` phase that installs its packages from `Pipfile.lock` does
|
||||||
it above `ARG CACHEBUST`, with `COPY script/ script/`,
|
it above `ARG CACHEBUST`, with `COPY script/ script/`,
|
||||||
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`.
|
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`. When
|
||||||
`script/bootstrap`, when it runs in that step, installs nothing beyond
|
`script/bootstrap` runs in that step, it installs nothing beyond system
|
||||||
system packages, pipenv and its dependencies by hash
|
packages, pipenv and its dependencies, and the packages from
|
||||||
(`pip install --require-hashes`), and the packages from the lockfile
|
`Pipfile.lock`. It installs pipenv and its dependencies by hash, with
|
||||||
(`pipenv install --deploy`, with `--dev` when the tests need the
|
`pip install --require-hashes`. It installs the packages from
|
||||||
development packages). It installs no Python there, because the tests use
|
`Pipfile.lock` with `pipenv install --deploy`, plus `--dev` when the tests
|
||||||
the base image's. The tests run after `ARG CACHEBUST`, and every other
|
need the development packages. It installs no Python there, because the
|
||||||
stage installs its Python packages after it, as in the Python `test` phase
|
tests use the base image's. The tests run after `ARG CACHEBUST`, and every
|
||||||
example in `REPO_POLICIES.md`
|
other stage installs its Python packages after it, as in the Python `test`
|
||||||
|
phase example in `REPO_POLICIES.md`
|
||||||
- Server: the final stage builds and runs the application
|
- Server: the final stage builds and runs the application
|
||||||
- Non-server: the final stage brings up the dev environment
|
- Non-server: the final stage brings up the dev environment
|
||||||
- Image pinned by sha256 hash with version/date comment
|
- Image pinned by sha256 hash with version/date comment
|
||||||
@@ -228,9 +232,11 @@ are thin shims calling them. Model scripts:
|
|||||||
git and node 24, with no yarn on `PATH`), and demonstrably executed the
|
git and node 24, with no yarn on `PATH`), and demonstrably executed the
|
||||||
checks — a sub-second build, or `CACHED` on a step that lints or tests,
|
checks — a sub-second build, or `CACHED` on a step that lints or tests,
|
||||||
one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above
|
one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above
|
||||||
that line, the steps that install system packages, download Go's modules
|
that line, these steps are expected to show `CACHED`:
|
||||||
or run `script/bootstrap` to install a Python `test` phase's packages from
|
- the steps that install system packages
|
||||||
`Pipfile.lock` are expected to show `CACHED`.
|
- the steps that download Go's modules
|
||||||
|
- the step that runs `script/bootstrap` to install a Python `test` phase's
|
||||||
|
packages from `Pipfile.lock`
|
||||||
- [ ] Plant a lint violation and confirm both `make lint` and a plain
|
- [ ] Plant a lint violation and confirm both `make lint` and a plain
|
||||||
`docker build .` fail on it; revert. A plain build that passes proves the
|
`docker build .` fail on it; revert. A plain build that passes proves the
|
||||||
final stage is missing its `COPY --from=` edge to the gate phases.
|
final stage is missing its `COPY --from=` edge to the gate phases.
|
||||||
|
|||||||
+36
-32
@@ -174,28 +174,32 @@ style conventions are in separate documents:
|
|||||||
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
|
||||||
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
`script/docker` are the four, and there is no fifth — `script/check` runs the
|
||||||
two gate phases and `script/fmt-check`, and builds no image of its own. In
|
two gate phases and `script/fmt-check`, and builds no image of its own. In
|
||||||
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
|
every stage of the `Dockerfile`, every step that lints, tests or compiles
|
||||||
that install system packages, download Go's modules or run `script/bootstrap`
|
comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps:
|
||||||
to install a Python `test` phase's packages from `Pipfile.lock`, or right
|
- the steps that install system packages
|
||||||
after `FROM` in a stage with none of these, and every step that lints, tests
|
- the steps that download Go's modules
|
||||||
or compiles comes after it. A step that installs anything else as well, such
|
- the step that runs `script/bootstrap` to install a Python `test` phase's
|
||||||
as a `script/bootstrap` that also installs node packages, comes after it too.
|
packages from `Pipfile.lock`
|
||||||
A new value reruns every `RUN` step after that line; `COPY` steps are compared
|
|
||||||
by content and need nothing. Docker keeps the layers above the line until the
|
A stage with none of these steps declares `ARG CACHEBUST` right after
|
||||||
base image, their own lines or the files they copy change. A stage without the
|
`FROM`. A step that installs anything else as well, such as a
|
||||||
`ARG` line is served from cache on an unchanged tree, so a repository
|
`script/bootstrap` that also installs node packages, comes after it too. A
|
||||||
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
|
new value reruns every `RUN` step after that line; `COPY` steps are compared
|
||||||
not pass `--no-cache`: it reruns every step too, but it also builds the layers
|
by content and need nothing. Docker keeps the layers above the line until
|
||||||
above the line again and empties every cache mount, so Go's build cache and
|
the base image, their own lines or the files they copy change. A stage
|
||||||
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
|
without the `ARG` line is served from cache on an unchanged tree, so a
|
||||||
survive between check builds, would start empty every time. A bare
|
repository re-vendors the scripts and adds the `ARG` lines in one commit.
|
||||||
`docker build .` is not evidence that anything ran: a sub-second build
|
The scripts do not pass `--no-cache`: it reruns every step too, but it also
|
||||||
reporting success is a cache hit, not a result. Never invalidate by pruning —
|
builds the layers above the line again and empties every cache mount, so
|
||||||
`docker builder prune` and friends destroy a build cache shared with every
|
Go's build cache and golangci-lint's, which the canonical Go `Dockerfile`
|
||||||
other build on the host. When a check is added or changed, prove it works by
|
below mounts so that they survive between check builds, would start empty
|
||||||
planting a defect it must catch and watching the run fail on it, then revert
|
every time. A bare `docker build .` is not evidence that anything ran: a
|
||||||
the defect. A green run alone shows neither that the check ran nor that it
|
sub-second build reporting success is a cache hit, not a result. Never
|
||||||
covers what it should.
|
invalidate by pruning — `docker builder prune` and friends destroy a build
|
||||||
|
cache shared with every other build on the host. When a check is added or
|
||||||
|
changed, prove it works by planting a defect it must catch and watching the
|
||||||
|
run fail on it, then revert the defect. A green run alone shows neither that
|
||||||
|
the check ran nor that it covers what it should.
|
||||||
|
|
||||||
- **The gate phases are separate stages, and the build stage depends on both.**
|
- **The gate phases are separate stages, and the build stage depends on both.**
|
||||||
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
The lint phase is based on the `golangci/golangci-lint` image (pinned by
|
||||||
@@ -380,8 +384,8 @@ style conventions are in separate documents:
|
|||||||
`Pipfile.lock`. It installs no Python there: the tests use the base
|
`Pipfile.lock`. It installs no Python there: the tests use the base
|
||||||
image's Python, which the image's hash pins. If it installs anything else
|
image's Python, which the image's hash pins. If it installs anything else
|
||||||
there, the step goes after `ARG CACHEBUST`.
|
there, the step goes after `ARG CACHEBUST`.
|
||||||
- It installs pipenv and its dependencies by hash, as the rule that every
|
- Every external reference must be pinned by hash, so `script/bootstrap`
|
||||||
external reference is pinned by hash requires. It does so with
|
installs pipenv and its dependencies by hash. It does so with
|
||||||
`pip install --require-hashes -r` and a requirements file in `script/`
|
`pip install --require-hashes -r` and a requirements file in `script/`
|
||||||
that lists each of them with `==` and its `--hash`. pip refuses to install
|
that lists each of them with `==` and its `--hash`. pip refuses to install
|
||||||
a dependency the file does not list.
|
a dependency the file does not list.
|
||||||
@@ -394,14 +398,14 @@ style conventions are in separate documents:
|
|||||||
their hashes. The packages `virtualenv` puts into the virtualenv that
|
their hashes. The packages `virtualenv` puts into the virtualenv that
|
||||||
pipenv creates come from wheels the `virtualenv` package ships, which its
|
pipenv creates come from wheels the `virtualenv` package ships, which its
|
||||||
hash covers. Two things in the layer are not fixed by a hash. A package
|
hash covers. Two things in the layer are not fixed by a hash. A package
|
||||||
with no ready-built wheel for the image is built from source, with build
|
with no ready-built wheel for the image is built from source. pip
|
||||||
tools pip downloads without a hash check, so a kept layer's build of it
|
downloads the build tools without a hash check, so a kept layer's build of
|
||||||
may differ from a new one. And system packages keep the versions the
|
such a package may differ from a new one. And system packages keep the
|
||||||
package manager gave when the layer was built; a new install may get newer
|
versions the package manager gave when the layer was built; a new install
|
||||||
ones.
|
may get newer ones.
|
||||||
- The packages are kept in a layer rather than a cache mount for the reason
|
- As with Go's modules, the packages are kept in a layer, not a cache mount.
|
||||||
Go's modules are: a cache mount is shared by every build on the host, and
|
A cache mount is shared by every build on the host, and nothing checks its
|
||||||
nothing checks its files again.
|
files again.
|
||||||
|
|
||||||
In this example, `script/bootstrap` finds `make` and `git` already
|
In this example, `script/bootstrap` finds `make` and `git` already
|
||||||
installed, installs pipenv and its dependencies by hash, and runs
|
installed, installs pipenv and its dependencies by hash, and runs
|
||||||
|
|||||||
Reference in new issue
Block a user