Author SHA1 Message Date
sneak 22b2f19a4c Install the pinned node when the installed one is another major version (closes #118)
check / check (push) Canceled after 0s
script/bootstrap used whatever node was installed, so on the CI runner
image, which ships node 24, the pinned yarn 1.22.22 printed the
deprecation warning DEP0169 on every bootstrap. It now uses the
installed node only when its major version is the pinned one, and
otherwise installs the pinned node with nvm, as it already did when node
was missing, with yarn and the packages under it. It prints the node
version it uses. Only the major is compared because the Dockerfile
stages use a node 22 alpine image that nvm cannot replace. script/fmt
and script/fmt-check carry the same test to pick their yarn: the yarn on
PATH, or yarn under the pinned node loaded through nvm.

Model: opus-5-5
2026-10-08 01:34:57 +00:00
12 changed files with 103 additions and 403 deletions
-11
View File
@@ -95,17 +95,6 @@ linters:
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
exclusions:
rules:
# homoicon keeps a local copy of sobek, its JavaScript engine, as sneak
# decided in
# https://git.eeqj.de/sneak/homoicon/issues/1759#issuecomment-153261.
# This is an exclusion because gomoddirectives does not check its
# replace-allow-list for a replace that points at a local path.
- path: ^go\.mod$
linters:
- gomoddirectives
text: '^local replacement are not allowed: github\.com/grafana/sobek$'
issues:
max-issues-per-linter: 0
-6
View File
@@ -2,12 +2,8 @@
# lint` or `script/lint`, which are themselves a docker build and would
# recurse into a daemon that does not exist in a build step.
#
# Every stage declares ARG CACHEBUST first: script/ passes a new value on
# every build, which reruns every RUN step after it.
#
# node 22-alpine, 2026-02-22
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS lint
ARG CACHEBUST
WORKDIR /app
@@ -23,7 +19,6 @@ RUN yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always
#
# node 22-alpine, 2026-02-22
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS test
ARG CACHEBUST
WORKDIR /app
@@ -43,7 +38,6 @@ RUN echo "No tests defined."
#
# node 22-alpine, 2026-02-22
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34
ARG CACHEBUST
WORKDIR /app
+10 -22
View File
@@ -117,25 +117,25 @@ alpine. We provide:
- `script/projectname` — output the project name (our own extension); used by
`script/docker` for the image tag
- `script/test` —
`docker build --build-arg CACHEBUST="$(date +%s)-$$" --target test --output type=cacheonly .`,
building the `test` phase of the `Dockerfile` without writing an image (no
tests defined here)
`docker build --no-cache --target test --output type=cacheonly .`, building
the `test` phase of the `Dockerfile` without writing an image (no tests
defined here)
- `script/lint` —
`docker build --build-arg CACHEBUST="$(date +%s)-$$" --target lint --output type=cacheonly .`,
building the `lint` phase, which runs prettier over the markdown files,
without writing an image
`docker build --no-cache --target lint --output type=cacheonly .`, building
the `lint` phase, which runs prettier over the markdown files, without writing
an image
- `script/fmt` — format all markdown files with prettier (writes; native, not in
a container)
- `script/fmt-check` — check formatting (read-only; native)
- `script/check` — run all checks: `test`, `lint`, `fmt-check` (our own
extension); builds no image of its own
- `script/docker` —
`docker build --build-arg CACHEBUST="$(date +%s)-$$" --build-arg VERSION="$version" -t prompts .`,
the tag coming from `script/projectname` (byte-identical across repos)
`docker build --no-cache --build-arg VERSION="$version" -t prompts .`, the tag
coming from `script/projectname` (byte-identical across repos)
- `script/cibuild` — cd to the repo root, run `script/bootstrap`, run
`script/check`, compute `version` from `git describe`, then
`docker build --build-arg CACHEBUST="$(date +%s)-$$" --build-arg VERSION="$version" -t prompts .`
(what CI runs; it bootstraps because CI checks out and runs this alone while
`docker build --no-cache --build-arg VERSION="$version" -t prompts .` (what CI
runs; it bootstraps because CI checks out and runs this alone while
`script/fmt-check` is native)
- `script/precommit` — run by the git pre-commit hook (our own extension); calls
`script/check`
@@ -144,18 +144,6 @@ alpine. We provide:
`make hooks` installs the pre-commit hook that runs `script/precommit`.
In every stage of a `Dockerfile`, `ARG CACHEBUST` comes right after the steps
that install system packages, download Go's modules or run `script/bootstrap` to
install a Python `test` phase's packages from `Pipfile.lock`, or right after
`FROM` in a stage with none of these, and every step that lints, tests or
compiles comes after it. The `script/` builds pass a new `CACHEBUST` value on
every build, which reruns every `RUN` step after that line and leaves the layers
above it and the cache mounts in place; `--no-cache` would build those layers
again and empty the mounts. This repo's own `Dockerfile` declares
`ARG CACHEBUST` right after `FROM` in every stage: each stage installs system
packages and node packages together in one `script/bootstrap` step, and check
builds may keep system packages but not node packages.
## Rationale
LLM prompts, especially development policies, benefit from version control and a
-50
View File
@@ -21,56 +21,6 @@ fmt-check, and commit.
# Completed Steps
- 2026-10-21: Check builds may now also keep the Python packages a `test` phase
installs from `Pipfile.lock` (issue 133), on sneak's ruling in
https://git.eeqj.de/sneak/rfscan/issues/80#issuecomment-153437. The install
comes before `ARG CACHEBUST`: `COPY script/ script/`,
`COPY Pipfile Pipfile.lock ./`, then `RUN script/bootstrap`. In that step
`script/bootstrap` installs nothing beyond system packages, pipenv and its
dependencies by hash, and the packages from the lockfile, and no Python, since
the tests use the base image's. `--deploy` installs only what the lockfile
names, checks every package against its hashes, and fails when the lockfile is
out of date with `Pipfile`. A kept layer holds nothing a new install would
refuse, except that a package built from source and the system packages may
differ from a new install. The packages are kept as a layer, not a cache
mount, for the reason Go's modules are. The tests still run after
`ARG CACHEBUST`, and every other stage still installs its Python packages
after it. `REPO_POLICIES.md` gains a short Python `test` phase example, and
both checklists and the README state the rule the same way. rfscan picks this
up on its next re-vendor.
- 2026-10-21: The canonical `.golangci.yml` now accepts a local `replace` of
`github.com/grafana/sobek` in a repository's root `go.mod` (issue 134), for
homoicon, which keeps a local copy of sobek on sneak's decision in
https://git.eeqj.de/sneak/homoicon/issues/1759#issuecomment-153261. Every
other local `replace` is still refused. It is an exclusion rather than a
`replace-allow-list` entry because gomoddirectives does not check that list
for a `replace` that points at a local path. homoicon picks this up on its
next re-vendor.
- 2026-10-18: Check builds may now also keep Go's downloaded modules,
golangci-lint's cache and the layers that install system packages (issue 130),
on sneak's ruling there. In each stage of the canonical Go `Dockerfile`, the
system-package install and the module download (`COPY go.mod go.sum ./`,
`RUN go mod download`) come before `ARG CACHEBUST`, so Docker keeps them until
the base image, the install line, `go.mod` or `go.sum` changes. The modules
are kept as a layer, not a cache mount, because Go does not check the files in
a cache mount again. `golangci-lint run` also mounts golangci-lint's cache,
`/root/.cache/golangci-lint`. Every step that lints, tests or compiles still
comes after `ARG CACHEBUST` and runs on every check build, and both `go test`
runs keep `-count=1`. `REPO_POLICIES.md`, both checklists, the Go style guide,
the README and the script comments state the rule the same way. This repo's
own `Dockerfile` is unchanged: its `script/bootstrap` step also installs node
packages, which the ruling does not cover. Repositories pick this up on their
next re-vendor.
- 2026-10-15: Check builds now keep Go's build cache between runs (issue 113),
on sneak's ruling there that Go's compile cache may survive. `script/lint`,
`script/test`, `script/cibuild` and `script/docker` pass a new `CACHEBUST`
build argument on every build in place of `--no-cache`, which also emptied the
cache mounts. Every `Dockerfile` stage declares `ARG CACHEBUST` first, so
every `RUN` step still runs on every check build. The canonical Go
`Dockerfile` mounts Go's build cache on the steps that compile and runs both
`go test` runs with `-count=1`, so no test result is reused. A stage without
the `ARG` line is served from cache, so repositories re-vendor the scripts and
the `ARG` lines in one commit. Not yet tried on the shared runner.
- 2026-10-08: The canonical `script/bootstrap` now uses the installed node only
when its major version is the pinned one (issue 118). Otherwise, as when node
is missing, it installs the pinned node under nvm and installs yarn and the
+3 -6
View File
@@ -1,6 +1,6 @@
---
title: Code Styleguide — Go
last_modified: 2026-10-18
last_modified: 2026-10-06
---
1. Try to hard wrap long lines at 77 characters or less.
@@ -111,11 +111,8 @@ last_modified: 2026-10-18
linting. It carries the lint and test phases, and the final stage depends on
both, so a build makes sure the code is in an able-to-be-compiled state,
linted, and its tests run. Go through `script/cibuild` or `script/docker`
rather than a bare `docker build .`: they pass a new `CACHEBUST` value on
every build, which reruns every step that lints, tests or compiles and keeps
the system packages, Go's modules and build cache, and golangci-lint's
cache; `--no-cache` would throw all of them away. Without a new value an
unchanged tree serves the gate layers from cache and the build reports a
rather than a bare `docker build .`: they pass `--no-cache`, without which
an unchanged tree serves the gate layers from cache and the build reports a
green it never ran.
1. Every repo must have a `Makefile`. See
+5 -40
View File
@@ -1,6 +1,6 @@
---
title: Existing Repo Checklist
last_modified: 2026-10-21
last_modified: 2026-10-08
---
Use this checklist when beginning work in a repo that may not yet conform to our
@@ -49,37 +49,6 @@ with your task.
`--target` names it, so a lost `COPY --from=` edge leaves `docker build .`
passing while the gate never runs. Confirm by planting a violation, not by
reading the file.
- [ ] In every stage `ARG CACHEBUST` comes right after the steps that install
system packages, download Go's modules or run `script/bootstrap` to
install a Python `test` phase's packages from `Pipfile.lock`, or right
after `FROM` in a stage with none of these, and every step that lints,
tests or compiles comes after it. A step that installs anything else as
well, such as a `script/bootstrap` that also installs node packages, comes
after it too. The `script/` builds pass a new `CACHEBUST` value on every
build, which reruns every `RUN` step after that line and leaves the layers
above it and the cache mounts in place; `--no-cache` would build those
layers again and empty the mounts. A stage without the `ARG` line is
served from cache on an unchanged tree, so the scripts and the `ARG` lines
are re-vendored in one commit.
- [ ] Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`,
`RUN go mod download`) above `ARG CACHEBUST`; the steps that compile
(`golangci-lint run`, `go test`, `go build`) mount Go's build cache
(`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run`
also mounts golangci-lint's
(`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step
mounts anything else; both `go test` runs carry `-count=1`, as in the
canonical Go `Dockerfile` in `REPO_POLICIES.md`
- [ ] Python: a `test` phase that installs its packages from `Pipfile.lock` does
it above `ARG CACHEBUST`, with `COPY script/ script/`,
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`.
`script/bootstrap`, when it runs in that step, installs nothing beyond
system packages, pipenv and its dependencies by hash
(`pip install --require-hashes`), and the packages from the lockfile
(`pipenv install --deploy`, with `--dev` when the tests need the
development packages). It installs no Python there, because the tests use
the base image's. The tests run after `ARG CACHEBUST`, and every other
stage installs its Python packages after it, as in the Python `test` phase
example in `REPO_POLICIES.md`
- [ ] The gate phases invoke their tools directly, never through `make lint` or
`script/test` — those are themselves a `docker build` and would recurse
inside a build step
@@ -164,7 +133,7 @@ with your task.
- [ ] README has an **Entrypoints** section documenting the `script/`
entrypoints and linking the standard
- [ ] `script/lint` and `script/test` each run
`docker build --build-arg CACHEBUST="$(date +%s)-$$" --target <phase> --output type=cacheonly .`,
`docker build --no-cache --target <phase> --output type=cacheonly .`,
which builds their phase by name and writes no image, and no host
invocation anywhere in the repo can produce a lint verdict — grep for the
linter's own name across `script/`, the `Makefile` and CI config, not just
@@ -177,8 +146,7 @@ with your task.
`script/cibuild` tag theirs. A build that writes an untagged image leaves
one behind on every run, on every host and CI runner.
- [ ] `script/cibuild` runs `script/bootstrap` before `script/check`, and builds
the image with a new `CACHEBUST` value on every build, like every
`docker build` in `script/`. Without the bootstrap the CI run dies in
the image with `--no-cache`. Without the bootstrap the CI run dies in
`script/fmt-check`, which runs the formatter on the host and finds nothing
installed.
- [ ] `script/fmt` and `script/fmt-check` pick their `yarn` with the same test
@@ -243,11 +211,8 @@ with your task.
- [ ] `script/cibuild` succeeds in a fresh clone on a host carrying what CI has
(the runner image `docker.gitea.com/runner-images:ubuntu-latest`: docker,
git and node 24, with no yarn on `PATH`), and demonstrably executed the
checks — a sub-second build, or `CACHED` on a step that lints or tests,
one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above
that line, the steps that install system packages, download Go's modules
or run `script/bootstrap` to install a Python `test` phase's packages from
`Pipfile.lock` are expected to show `CACHED`.
checks — a sub-second build, or `CACHED` on a gate layer, means nothing
ran
- [ ] A planted lint violation fails both `make lint` and a plain
`docker build .`; revert it afterwards
- [ ] Commit and merge fixes before starting your actual task
+16 -50
View File
@@ -1,6 +1,6 @@
---
title: New Repo Checklist
last_modified: 2026-10-21
last_modified: 2026-10-08
---
Use this checklist when creating a new repository from scratch. Follow the steps
@@ -108,37 +108,6 @@ Template files can be fetched from:
stage carries a `COPY --from=` of a harmless file from each so the image
cannot be built unless both passed. Keep the final stage last: a stage
nothing depends on is built only when `--target` names it.
- In every stage `ARG CACHEBUST` comes right after the steps that install
system packages, download Go's modules or run `script/bootstrap` to
install a Python `test` phase's packages from `Pipfile.lock`, or right
after `FROM` in a stage with none of these, and every step that lints,
tests or compiles comes after it. A step that installs anything else as
well, such as a `script/bootstrap` that also installs node packages, comes
after it too. The `script/` builds pass a new `CACHEBUST` value on every
build, which reruns every `RUN` step after that line and leaves the layers
above it and the cache mounts in place; `--no-cache` would build those
layers again and empty the mounts. A stage without the `ARG` line is
served from cache on an unchanged tree, so the scripts and every stage's
`ARG CACHEBUST` line are added in the same commit.
- Go: each Go stage downloads the modules (`COPY go.mod go.sum ./`,
`RUN go mod download`) above `ARG CACHEBUST`; the steps that compile
(`golangci-lint run`, `go test`, `go build`) mount Go's build cache
(`--mount=type=cache,target=/root/.cache/go-build`), `golangci-lint run`
also mounts golangci-lint's
(`--mount=type=cache,target=/root/.cache/golangci-lint`), and no step
mounts anything else; both `go test` runs carry `-count=1`, as in the
canonical Go `Dockerfile` in `REPO_POLICIES.md`
- Python: a `test` phase that installs its packages from `Pipfile.lock` does
it above `ARG CACHEBUST`, with `COPY script/ script/`,
`COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`.
`script/bootstrap`, when it runs in that step, installs nothing beyond
system packages, pipenv and its dependencies by hash
(`pip install --require-hashes`), and the packages from the lockfile
(`pipenv install --deploy`, with `--dev` when the tests need the
development packages). It installs no Python there, because the tests use
the base image's. The tests run after `ARG CACHEBUST`, and every other
stage installs its Python packages after it, as in the Python `test` phase
example in `REPO_POLICIES.md`
- Server: the final stage builds and runs the application
- Non-server: the final stage brings up the dev environment
- Image pinned by sha256 hash with version/date comment
@@ -176,13 +145,13 @@ are thin shims calling them. Model scripts:
- [ ] `script/setup` / `make setup` — readies a fresh clone: runs `bootstrap`,
then `install-precommit`, plus repo-specific init
- [ ] `script/test` / `make test` —
`docker build --build-arg CACHEBUST="$(date +%s)-$$" --target test --output type=cacheonly .`,
which writes no image; the phase runs real tests, not a no-op (90-second
`docker build --no-cache --target test --output type=cacheonly .`, which
writes no image; the phase runs real tests, not a no-op (90-second
timeout, 60-second hard cap on wall time)
- [ ] `script/lint` / `make lint` —
`docker build --build-arg CACHEBUST="$(date +%s)-$$" --target lint --output type=cacheonly .`,
which writes no image. No lint verdict may come from a host invocation of
the linter.
`docker build --no-cache --target lint --output type=cacheonly .`, which
writes no image. No lint verdict may come from a host invocation of the
linter.
- [ ] `script/fmt` / `make fmt` — formats code (writes; native, never in a
container)
- [ ] `script/fmt-check` / `make fmt-check` — checks formatting (read-only;
@@ -192,16 +161,16 @@ are thin shims calling them. Model scripts:
- [ ] `script/projectname` — outputs the project name (used by `script/docker`
for the image tag)
- [ ] `script/docker` / `make docker` — builds Docker image, tagged via
`script/projectname` (byte-identical across repos); a new `CACHEBUST`
value on every build, plus the version as a build arg
`script/projectname` (byte-identical across repos); `--no-cache`, plus the
version as a build arg
- [ ] `script/cibuild` — cd to repo root, run `script/bootstrap`, run
`script/check`, then
`docker build --build-arg CACHEBUST="$(date +%s)-$$" --build-arg VERSION="$version" -t "$tag" .`
(what CI runs), with `$version` from
`git describe --tags --always --dirty` (`unknown` if empty) and `$tag`
from `script/projectname`, each assigned on its own line before the build.
The bootstrap is required: CI checks out and runs this alone, and
`script/fmt-check` runs the formatter on the host.
`docker build --no-cache --build-arg VERSION="$version" -t "$tag" .` (what
CI runs), with `$version` from `git describe --tags --always --dirty`
(`unknown` if empty) and `$tag` from `script/projectname`, each assigned
on its own line before the build. The bootstrap is required: CI checks out
and runs this alone, and `script/fmt-check` runs the formatter on the
host.
- [ ] `script/fmt` and `script/fmt-check` pick their `yarn` with the same test
as `script/bootstrap`: the `yarn` on `PATH` when the node on `PATH` has
the pinned major version, and otherwise yarn under the pinned node, with
@@ -226,11 +195,8 @@ are thin shims calling them. Model scripts:
- [ ] `script/cibuild` succeeds in a fresh clone on a host carrying what CI has
(the runner image `docker.gitea.com/runner-images:ubuntu-latest`: docker,
git and node 24, with no yarn on `PATH`), and demonstrably executed the
checks — a sub-second build, or `CACHED` on a step that lints or tests,
one of the `RUN` steps below `ARG CACHEBUST`, means nothing ran. Above
that line, the steps that install system packages, download Go's modules
or run `script/bootstrap` to install a Python `test` phase's packages from
`Pipfile.lock` are expected to show `CACHED`.
checks — a sub-second build, or `CACHED` on a gate layer, means nothing
ran
- [ ] Plant a lint violation and confirm both `make lint` and a plain
`docker build .` fail on it; revert. A plain build that passes proves the
final stage is missing its `COPY --from=` edge to the gate phases.
+55 -187
View File
@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-10-21
last_modified: 2026-10-08
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -123,10 +123,8 @@ style conventions are in separate documents:
and nothing else:
```sh
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--target lint --output type=cacheonly .
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
--target test --output type=cacheonly .
docker build --no-cache --target lint --output type=cacheonly .
docker build --no-cache --target test --output type=cacheonly .
```
**A stage that is not the last one in the file is built only when the final
@@ -139,12 +137,12 @@ style conventions are in separate documents:
**The gate builds write no image.** With `--output type=cacheonly` the phase
runs and a failing step fails the build, but the result is not exported.
Nothing uses those images, and writing one out is slow: a Go test phase's
image holds the toolchain. A build given neither `--output` nor `-t` writes
an untagged image and leaves it dangling, on every developer host and every
CI runner. `script/cibuild` and `script/docker` build the image that ships
and tag it, so each build replaces the previous image; each assigns the tag
on its own line before the build, so `set -e` stops it where
`script/projectname` fails.
image holds the toolchain and every compiled package. A build given neither
`--output` nor `-t` writes an untagged image and leaves it dangling, on
every developer host and every CI runner. `script/cibuild` and
`script/docker` build the image that ships and tag it, so each build
replaces the previous image; each assigns the tag on its own line before the
build, so `set -e` stops it where `script/projectname` fails.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
@@ -159,43 +157,22 @@ style conventions are in separate documents:
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own `TMPDIR`, so the lock is
out of reach. The lint phase does keep golangci-lint's cache between check
builds (below), but it lints every checkout at `/src`, and a finding is
taken from that cache only for the same files, settings and golangci-lint
version.
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is given a new `CACHEBUST` value.** Docker
invalidates a `COPY` layer only when the copied content changes, so on an
unchanged tree the check `RUN` is served from cache, nothing executes, and the
build still exits 0. Every `docker build` in `script/` therefore passes
`--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process
id, so no two builds share a value, not even the three in one
`script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
`script/docker` are the four, and there is no fifth — `script/check` runs the
two gate phases and `script/fmt-check`, and builds no image of its own. In
every stage of the `Dockerfile`, `ARG CACHEBUST` comes right after the steps
that install system packages, download Go's modules or run `script/bootstrap`
to install a Python `test` phase's packages from `Pipfile.lock`, or right
after `FROM` in a stage with none of these, and every step that lints, tests
or compiles comes after it. A step that installs anything else as well, such
as a `script/bootstrap` that also installs node packages, comes after it too.
A new value reruns every `RUN` step after that line; `COPY` steps are compared
by content and need nothing. Docker keeps the layers above the line until the
base image, their own lines or the files they copy change. A stage without the
`ARG` line is served from cache on an unchanged tree, so a repository
re-vendors the scripts and adds the `ARG` lines in one commit. The scripts do
not pass `--no-cache`: it reruns every step too, but it also builds the layers
above the line again and empties every cache mount, so Go's build cache and
golangci-lint's, which the canonical Go `Dockerfile` below mounts so that they
survive between check builds, would start empty every time. A bare
`docker build .` is not evidence that anything ran: a sub-second build
reporting success is a cache hit, not a result. Never invalidate by pruning —
`docker builder prune` and friends destroy a build cache shared with every
other build on the host. When a check is added or changed, prove it works by
planting a defect it must catch and watching the run fail on it, then revert
the defect. A green run alone shows neither that the check ran nor that it
covers what it should.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
@@ -204,56 +181,40 @@ style conventions are in separate documents:
`Dockerfile`:
```dockerfile
# In every stage ARG CACHEBUST comes right after the steps that install
# system packages and download Go's modules, or right after FROM in a
# stage with neither, and every step that lints, tests or compiles
# comes after it. script/ passes a new value on every build, which
# reruns every RUN step after it. Docker keeps the layers above it
# until the base image, their own lines or go.mod and go.sum change.
# The steps that compile mount Go's build cache, and golangci-lint run
# also mounts golangci-lint's; both survive from build to build.
# Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \
--mount=type=cache,target=/root/.cache/golangci-lint \
golangci-lint run --config .golangci.yml ./...
RUN golangci-lint run --config .golangci.yml ./...
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not. -count=1 because the mounted
# cache also holds test results, which go test would otherwise replay.
# image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
COPY . .
RUN --mount=type=cache,target=/root/.cache/go-build \
go test -count=1 -timeout 90s -race -cover ./... || \
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so nothing after them can
# run unless lint and test passed.
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
COPY . .
# The VERSION build arg when one is given, otherwise
@@ -261,8 +222,7 @@ style conventions are in separate documents:
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN --mount=type=cache,target=/root/.cache/go-build \
VERSION="${VERSION:-$(git describe --tags --always)}"; \
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
@@ -275,7 +235,6 @@ style conventions are in separate documents:
# Runtime stage, and the last one
FROM alpine@sha256:...
ARG CACHEBUST
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
@@ -290,29 +249,6 @@ style conventions are in separate documents:
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- Check builds keep these, and none of them can hand back a stale result:
- The layers above `ARG CACHEBUST`: the system packages and Go's
modules. Docker installs the packages again when the base image or the
install line changes, and downloads the modules again when `go.mod` or
`go.sum` changes. Go checks every module against `go.sum` as it
downloads it, and no later build can change a kept layer. The modules
are kept in a layer rather than a cache mount because a cache mount is
shared by every build on the host, and Go builds from whatever files
it holds without checking them again (only `go mod verify` does), so a
file changed there would go unnoticed.
- Go's build cache (`/root/.cache/go-build`, where Go keeps it for
root), mounted on the steps that compile: `golangci-lint run`, both
`go test` runs and `go build`. Go reuses a compiled package only when
all its inputs are unchanged. The cache also holds test results, which
is why both `go test` runs carry `-count=1`.
- golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned
image keeps it), mounted on `golangci-lint run`. golangci-lint reuses
a package's findings only when the package's files and those of every
package it imports, `go.mod`, the linters and their settings in
`.golangci.yml`, and the golangci-lint version are all unchanged.
- In a Python `test` phase, the layer of the step that runs
`script/bootstrap` to install the packages from `Pipfile.lock`, as
described for that phase below.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example:
@@ -321,20 +257,12 @@ style conventions are in separate documents:
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them. The test phase,
on the Debian Go image, installs the libraries its tests need the same
way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so
Docker keeps it until the base image or the install line changes:
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
FROM golangci/golangci-lint@sha256:... AS lint
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
ARG CACHEBUST
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
@@ -367,67 +295,6 @@ style conventions are in separate documents:
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
do.
- **A Python `test` phase that installs its packages from `Pipfile.lock` does it
above `ARG CACHEBUST`, so check builds keep them.** The steps are
`COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and
`RUN script/bootstrap`, with `ARG CACHEBUST` right after them. Docker keeps
what that `RUN` installed until the base image, `script/`, `Pipfile`,
`Pipfile.lock` or a line up to and including the `RUN` changes. The tests run
after `ARG CACHEBUST`, on every check build. In every other stage, a step that
installs Python packages comes after `ARG CACHEBUST`.
- When `script/bootstrap` runs in that step, it installs nothing beyond
system packages, pipenv and its dependencies, and the packages from
`Pipfile.lock`. It installs no Python there: the tests use the base
image's Python, which the image's hash pins. If it installs anything else
there, the step goes after `ARG CACHEBUST`.
- It installs pipenv and its dependencies by hash, as the rule that every
external reference is pinned by hash requires. It does so with
`pip install --require-hashes -r` and a requirements file in `script/`
that lists each of them with `==` and its `--hash`. pip refuses to install
a dependency the file does not list.
- It then runs `pipenv install --deploy`, with `--dev` when the tests need
the development packages. `--deploy` installs only what `Pipfile.lock`
names, checks every package against the lockfile's hashes, and fails when
the lockfile is out of date with `Pipfile`.
- A kept layer holds nothing a new install would refuse: pip checked pipenv
and its dependencies, and pipenv checked the lockfile's packages, against
their hashes. The packages `virtualenv` puts into the virtualenv that
pipenv creates come from wheels the `virtualenv` package ships, which its
hash covers. Two things in the layer are not fixed by a hash. A package
with no ready-built wheel for the image is built from source, with build
tools pip downloads without a hash check, so a kept layer's build of it
may differ from a new one. And system packages keep the versions the
package manager gave when the layer was built; a new install may get newer
ones.
- The packages are kept in a layer rather than a cache mount for the reason
Go's modules are: a cache mount is shared by every build on the host, and
nothing checks its files again.
In this example, `script/bootstrap` finds `make` and `git` already
installed, installs pipenv and its dependencies by hash, and runs
`pipenv install --dev --deploy`:
```dockerfile
# Test phase. The 90-second timeout covers both pytest runs.
# python:3.x-slim, YYYY-MM-DD
FROM python@sha256:... AS test
# make and git are base tools that script/bootstrap installs when they
# are missing. Installed here, they stay in their own layer when
# script/ changes.
RUN apt-get update \
&& apt-get install -y --no-install-recommends make git \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY script/ script/
COPY Pipfile Pipfile.lock ./
RUN script/bootstrap
ARG CACHEBUST
COPY . .
RUN timeout 90 sh -c 'pipenv run pytest || { \
echo "--- Rerunning with -v for details ---"; \
pipenv run pytest -v; exit 1; }'
```
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step,
with `persist-credentials: false`: `script/cibuild` needs no token, and
@@ -442,18 +309,18 @@ style conventions are in separate documents:
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
so a successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build gets a `CACHEBUST` value of its own and so runs the gate phases a
second time. That is the price of the rule above, and it is worth paying: the
image that ships is built from a run of its own gates rather than from a cache
entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the
shared runner after 20 minutes. That allows for the three Docker builds
described above (the test phase, the lint phase, then the image), each held to
the 5-minute Docker build limit below, plus the bootstrap. A separate workflow
limited to `main` by a `branches` list under `on: push` cannot be checked by
review: to try a change to it, add the feature branch to that list and push,
then remove the branch from the list again before merging. Keep any job in it
that publishes behind `if: github.ref_name == 'main'`, so the run from the
feature branch publishes nothing.
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. The `check` job
sets `timeout-minutes: 20`, so a hung build frees the shared runner after 20
minutes. That allows for the three Docker builds described above (the test
phase, the lint phase, then the image), each held to the 5-minute Docker build
limit below, plus the bootstrap. A separate workflow limited to `main` by a
`branches` list under `on: push` cannot be checked by review: to try a change
to it, add the feature branch to that list and push, then remove the branch
from the list again before merging. Keep any job in it that publishes behind
`if: github.ref_name == 'main'`, so the run from the feature branch publishes
nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -513,11 +380,12 @@ style conventions are in separate documents:
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. The directory lasts from one run to the next on a
developer's machine, where this target runs, and in the `test` phase of the
`Dockerfile`, which mounts it to keep compiled packages between check
builds, so the phase's `go test` runs carry `-count=1` too. A new
`CACHEBUST` value (above) is what makes that step run on an unchanged tree.
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example:
@@ -604,7 +472,7 @@ style conventions are in separate documents:
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
tag="$(script/projectname)"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$tag" .
```
+4 -7
View File
@@ -2,12 +2,9 @@
# script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do.
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
# line, since a cached gate phase is a check that did not run; unlike
# --no-cache, it keeps the layers above that line and the cache mounts.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# before every step that lints, tests or compiles; a stage without that
# line is served from cache on an unchanged tree, with no warning.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -25,7 +22,7 @@ main() {
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$tag" .
}
+3 -7
View File
@@ -1,12 +1,8 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# A new CACHEBUST value reruns every RUN step after the ARG CACHEBUST
# line, since a cached gate phase is a check that did not run; unlike
# --no-cache, it keeps the layers above that line and the cache mounts.
# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
# before every step that lints, tests or compiles; a stage without that
# line is served from cache on an unchanged tree, with no warning.
# --no-cache because the gate phases the final stage depends on are RUN
# steps, and a cached one is a check that did not run.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -22,7 +18,7 @@ main() {
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$tag" .
}
+4 -9
View File
@@ -5,14 +5,9 @@
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. A new CACHEBUST value reruns every RUN step after
# the ARG CACHEBUST line, since a cached lint layer is a lint that did
# not run; unlike --no-cache, it keeps the layers above that line and
# the cache mounts. This relies on every stage of the Dockerfile
# declaring ARG CACHEBUST before every step that lints, tests or
# compiles; a stage without that line is served from cache on an
# unchanged tree, with no warning.
# --output type=cacheonly writes no image, since nothing uses one.
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. --output type=cacheonly writes no image, since
# nothing uses one.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -20,7 +15,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
docker build --no-cache \
--target lint \
--output type=cacheonly .
}
+3 -8
View File
@@ -2,13 +2,8 @@
# script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, and a new CACHEBUST value because a cached test layer is a test
# that did not run; unlike --no-cache, it keeps the layers above the ARG
# CACHEBUST line and the cache mounts. This relies on every stage of the
# Dockerfile declaring ARG CACHEBUST before every step that lints, tests
# or compiles; a stage without that line is served from cache on an
# unchanged tree, with no warning. --output type=cacheonly writes no
# image, since nothing uses one.
# named, and --no-cache because a cached test layer is a test that did
# not run. --output type=cacheonly writes no image, since nothing uses one.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -16,7 +11,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --build-arg CACHEBUST="$(date +%s)-$$" \
docker build --no-cache \
--target test \
--output type=cacheonly .
}