diff --git a/TODO.md b/TODO.md index 83c8480..60db56a 100644 --- a/TODO.md +++ b/TODO.md @@ -21,6 +21,13 @@ fmt-check, and commit. # Completed Steps +- 2026-11-06: Compacted `prompts/REPO_POLICIES.md` from 1,145 lines and 8,427 + words to 1,066 lines and 7,812 words + (https://git.eeqj.de/sneak/prompts/issues/158). Every rule keeps its meaning. + The rules now sit under headings, the pinning rule first. Each rule that was + stated in several places is stated once. The Makefile copies of the Go and + Python test commands are gone, because the `Dockerfile` examples carry the + same commands. Both checklists name the `.golangci.yml` section. - 2026-11-06: The pinning rule in `prompts/REPO_POLICIES.md` now opens with its scope: what we download and install ourselves in scripts and Dockerfiles (https://git.eeqj.de/sneak/prompts/issues/154). Distribution packages are diff --git a/prompts/EXISTING_REPO_CHECKLIST.md b/prompts/EXISTING_REPO_CHECKLIST.md index 8fe3f42..dfd0d03 100644 --- a/prompts/EXISTING_REPO_CHECKLIST.md +++ b/prompts/EXISTING_REPO_CHECKLIST.md @@ -163,7 +163,7 @@ with your task. - [ ] Go: `go.mod`, `go.sum`, `.golangci.yml` (fetch from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and, in the same commit, set the lint phase digest to the one named in the - `.golangci.yml` paragraph of `REPO_POLICIES.md`) + `.golangci.yml` section of `REPO_POLICIES.md`) - [ ] JS: `package.json`, `yarn.lock`, `.yarnrc.yml`, `.yarn/releases/`, `.gitattributes`, `.prettierrc`, `.prettierignore` (fetch the last two from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.prettierrc` diff --git a/prompts/NEW_REPO_CHECKLIST.md b/prompts/NEW_REPO_CHECKLIST.md index 0380f38..544eeb3 100644 --- a/prompts/NEW_REPO_CHECKLIST.md +++ b/prompts/NEW_REPO_CHECKLIST.md @@ -178,7 +178,7 @@ Template files can be fetched from: - [ ] Go: `go mod init sneak.berlin/go/`, `.golangci.yml` (fetch from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and, in the same commit, set the lint phase digest to the one named in the - `.golangci.yml` paragraph of `REPO_POLICIES.md`) + `.golangci.yml` section of `REPO_POLICIES.md`) - [ ] JS, or any repository that runs prettier through yarn: make the first `yarn.lock` with the steps below. `script/bootstrap` installs no yarn until `yarn.lock` exists, so run them with any distribution's node and diff --git a/prompts/REPO_POLICIES.md b/prompts/REPO_POLICIES.md index e44995d..207d20c 100644 --- a/prompts/REPO_POLICIES.md +++ b/prompts/REPO_POLICIES.md @@ -14,881 +14,64 @@ style conventions are in separate documents: - [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md) - [API Policies](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/API_POLICIES.md) ---- - -- Cross-project documentation (such as this file) must include - `last_modified: YYYY-MM-DD` in the YAML front matter so it can be kept in sync - with the authoritative source as policies evolve. - -- **External references must be pinned by cryptographic hash.** This is the - single most important rule in this document. It covers what we download and - install ourselves in scripts and Dockerfiles: - - Docker base images - - Go modules - - Python packages from PyPI - - npm packages - - GitHub Actions - - release archives fetched with `curl` or `wget` - - Packages from the distribution's own package manager are outside this rule. - They need no version and no hash, because the distribution is trusted. In a - `Dockerfile`, it is the pinned base image's distribution. On a host, it is - the host's. The host's package manager stands for its distribution, and that - includes nix and brew. Updating the package index and upgrading installed - packages are fine too. This command, for example, is allowed: - - ```sh - apt update && apt upgrade -y && apt install -y build-essential - ``` - - A package source that a `Dockerfile` adds itself is not the distribution's - own, so its packages are pinned like any other download. - - A version tag such as `@v4`, `@latest` or `:3.21` is server-mutable, and - therefore a remote code execution vulnerability. Reference an external - dependency by its content hash: - - Docker: `@sha256:...` - - Go: the module hash in `go.sum` - - Python: the package hash in `Pipfile.lock` - - npm: the `checksum` in `yarn.lock` - - GitHub Actions: `@` - - This also means never `curl | bash` to install a tool such as rustup. - Instead, download a specific release archive from GitHub, verify its hash, - and only then install. The hash is hardcoded in the Dockerfile or script. - Unverified install scripts are arbitrary remote code execution. Double-check - every external reference in every file before committing. - - These come from the distribution wherever possible: - - node, meaning the distribution's newest Node package, such as - `nodejs-current` on Alpine 3.24 - - yarn - - npm - - python - - pip and venv, the installer tools bundled with python - - If the distribution's version is too old, use a newer distribution, never a - download. That rules out all of these: - - nvm - - pyenv - - corepack - - `npm install -g yarn` - - yarn's release file is third-party code, and git pins it. `yarn set version` - writes it to `.yarn/releases/` once, given an exact version. - - These are not trusted, so everything from them stays pinned by hash: - - Docker Hub - - PyPI - - the npm registry - - `go get` - -- Every repo with software must have a root `Makefile` with these targets: - - `make bootstrap` - - `make setup` - - `make test` - - `make lint` - - `make fmt` (writes) - - `make fmt-check` (read-only) - - `make check` (runs `test`, `lint`, `fmt-check`) - - `make docker` - - `make hooks` (installs pre-commit hook) - - A model Makefile is at - `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`. - -- Repos follow the - [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) - pattern. The implementation of each Makefile target lives in an executable - script in `script/`, and the Makefile targets are thin shims that call them: - - `script/bootstrap` - - `script/setup` - - `script/test` - - `script/lint` - - `script/fmt` - - `script/fmt-check` - - `script/check` - - `script/docker` - - The scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they - run in minimal containers; alpine images, for example, have no bash. Locate - the repo root with `$(cd "$(dirname "$0")/.." && pwd -P)`, and `cd` there - before acting. - - From the standard's canonical set we use `bootstrap`, `setup`, `test`, and - `cibuild`. `setup` makes the repo ready for development after a fresh clone: - it runs `bootstrap`, then `install-precommit`, plus any repo-specific - initialization. - - `script/bootstrap` installs all dependencies idempotently and assumes - nothing is present. Base tools, node and yarn come from nix, apt, brew, or - apk, detected in that order; apt runs noninteractive. Each one is installed - only when it is missing. A tool from outside the distribution is never - installed as "latest" or "lts"; it is always an exact version. - - `script/bootstrap` installs node, yarn and the node packages only when - `yarn.lock` is in the tree. With no `yarn.lock`, it installs no node and no - yarn. It installs the node packages with `yarn install --immutable`. That - install fails when a package's `checksum` is wrong, or when `package.json` - is out of step with `yarn.lock`. - - Each repository commits the yarn that runs. `yarnPath` in `.yarnrc.yml` - points at the release file in `.yarn/releases/`. The distribution's yarn - only starts that file, and that file does the work. Alpine packages yarn 1, - and Debian 13 packages yarn 4.1. The same committed yarn does the work on - both. Corepack never runs, and no yarn is downloaded. - - `.yarnrc.yml` holds these settings: - - `yarnPath` - - `nodeLinker: node-modules` - - `enableTelemetry: false`, because yarn otherwise sends usage reports from - any host that is not CI - - By default, yarn runs no dependency's install script. A dependency that must - run one is named in `package.json`, as - `"dependenciesMeta": { "": { "built": true } }`. - - `.gitattributes` carries `.yarn/releases/** binary`, so diffs and merges - treat the release file as one opaque file. - - yarn 1 ignores `--immutable` and exits 0, so on a tree with no `.yarnrc.yml` - the install is not frozen. A repository therefore re-vendors - `script/bootstrap` and converts its `yarn.lock` in one commit. - - Debian and Ubuntu name the yarn command `yarnpkg`. `script/bootstrap`, - `script/fmt` and `script/fmt-check` each carry the same small function that - runs whichever name is present. With neither name on `PATH`, `script/fmt` - and `script/fmt-check` print one line saying to run `script/bootstrap`, and - exit 1. - - A `Dockerfile` stage that needs yarn may start from any distribution that - packages node and yarn. It copies all of these before it runs - `script/bootstrap`: - - `package.json` - - `yarn.lock` - - `.yarnrc.yml` - - `.yarn/releases/` - - With only `package.json` and `yarn.lock`, the install fails. Alpine's yarn 1 - then says to enable corepack; never do that, and copy the missing files - instead. The Python and Go stages below need no yarn and copy no - `yarn.lock`, so `script/bootstrap` installs no node and no yarn there. - - `script/cibuild` runs the CI build, and the Gitea workflow calls it. The - script changes to the repo root, runs `script/bootstrap`, runs - `script/check`, and builds the image with the version. **`script/cibuild` - runs `script/bootstrap` first**, because the workflow checks out the repo - and runs nothing else, while `script/fmt-check` runs the formatter on the - host. On a pristine checkout with nothing installed, the run dies there, - after the containerised gates have passed. - - Four further scripts are our own extensions to the standard: - - `script/check` runs `script/test`, `script/lint` and `script/fmt-check`. - - `script/precommit` is what the git pre-commit hook runs, and it calls - `script/check`. - - `script/install-precommit` installs the git pre-commit hook. The - `make hooks` target shims to it. - - `script/projectname` (literally that filename) simply outputs the - project's name. - - Scripts that need the name call `script/projectname`, so those scripts stay - byte-identical across all repos. `script/docker`, for example, assembles its - image tag from it. Repo-type-specific pre-commit extras belong in - `script/precommit`, not in the hook itself; `go mod tidy` verification in Go - repos is one. Model scripts are at - `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/`. The - README must document the provided scripts in an **Entrypoints** section (see - the README requirements below). - -- Run the repo's own entrypoints, never the underlying tools directly. The - `script/` scripts are the single source of truth for how each operation runs. - Calling them directly is fine, and so is calling the Makefile targets. The - Makefile targets must still exist, as the thin shims described above. - - An agent whose work is confined to one package may run `go test` directly on - that package's suite, or on individual tests in it, while iterating. This is - the only exception to this rule. `go build`, `go vet` and the linter still - go through the entrypoints. A run while iterating is not a gate, so the full - `make check` is still required before every push. - -- The Makefile is authoritative documentation for how the repo is used. Beyond - the required targets above, it should have targets for every common operation: - - running a local development server (`make run`, `make dev`) - - re-initializing or migrating the database (`make db-reset`, - `make migrate`) - - building artifacts (`make build`) - - generating code - - seeding data - - anything else a developer would do regularly - - If someone checks out the repo and types `make`, they should see every - meaningful operation available. A new contributor should be able to - understand the entire development workflow by reading the Makefile. - -- Every repo should have a `Dockerfile`, and it carries the repo's gates: a - `lint` phase and a `test` phase. The final stage depends on both, so the image - cannot be built unless they pass. For non-server repos the final stage brings - up a development environment; for server repos it is the runtime image. The - gate phases and the build stage start from their pinned base images and - install what those images lack in either of these ways: - - inline, as the canonical Go `Dockerfile` below does for `git` - - by running `script/bootstrap`, as the `prompts` repo's own `Dockerfile` - does for its yarn packages - - The development environment stage installs development prerequisites by - running `script/bootstrap`. Only system packages may be installed inline - above it, so that check builds keep them; the `prompts` repo's own - `Dockerfile` does this. A stage that runs `script/bootstrap` COPYs `script/` - and the dependency manifests it needs before running it. These are examples: - - `package.json`, `yarn.lock`, `.yarnrc.yml` and `.yarn/releases/` - - `go.mod` and `go.sum` - -- **The lint and test gates run in Docker, as phases of the `Dockerfile`.** - There is no separate lint file. `script/lint` and `script/test` each build one - phase and nothing else: - - ```sh - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --target lint --output type=cacheonly . - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --target test --output type=cacheonly . - ``` - - **A stage that is not the last one in the file is built only when the final - stage's chain depends on it, or when `--target` names it.** That is why the - two gates are always invoked by name here, and why the final stage carries a - `COPY --from=` of a harmless file from each of them. Without that - dependency, a plain `docker build .` builds the last stage alone and exits 0 - having linted and tested nothing. - - **The gate builds write no image.** With `--output type=cacheonly` the phase - runs and a failing step fails the build, but the result is not exported. - Nothing uses those images, and writing one out is slow: a Go test phase's - image holds the toolchain. A build given neither `--output` nor `-t` writes - an untagged image and leaves it dangling, on every developer host and every - CI runner. `script/cibuild` and `script/docker` build the image that ships - and tag it, so each build replaces the previous image. Each script assigns - the tag on its own line before the build, so `set -e` stops it where - `script/projectname` fails. - - Inside a phase the tool is invoked directly: `golangci-lint`, `go test`, - `eslint`, `prettier`. The tool is never invoked through `make lint` or - `script/test`. Those are themselves a `docker build`, and would recurse into - a daemon that does not exist in a build step. Formatting is the exception - and stays on the host: `script/fmt` writes the working tree, and - `script/fmt-check` is its read-only twin. - - **No lint verdict may come from a host invocation of the linter.** On a - shared host golangci-lint reads a result cache keyed on file content rather - than location. A second checkout of the same content is therefore served the - first one's findings. A host-global lock in `$TMPDIR` also makes concurrent - runs exit non-zero with `parallel golangci-lint is running`, a status a - caller cannot tell from real findings. Both have produced wrong verdicts in - this org, in both directions. A container has its own `TMPDIR`, so the lock - is out of reach. The lint phase does keep golangci-lint's cache between - check builds (below), but it lints every checkout at `/src`. A finding is - taken from that cache only when all of these are the same: - - the files - - the settings - - the golangci-lint version - -- **Any build that runs checks is given a new `CACHEBUST` value.** Docker - invalidates a `COPY` layer only when the copied content changes. On an - unchanged tree the check `RUN` is therefore served from cache, nothing - executes, and the build still exits 0. Every `docker build` in `script/` - therefore passes `--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the - script's process id. So no two builds share a value, not even the three in one - `script/cibuild`. Four scripts do this, and there is no fifth: - - `script/lint` - - `script/test` - - `script/cibuild` - - `script/docker` - - `script/check` runs the two gate phases and `script/fmt-check`, and builds - no image of its own. In every stage of the `Dockerfile`, every step that - lints, tests or compiles comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes - right after these steps: - - the steps that install system packages - - the steps that download Go's modules - - the step that runs `script/bootstrap` to install a Python `test` phase's - packages from `Pipfile.lock` - - A stage with none of these steps declares `ARG CACHEBUST` right after - `FROM`. A step that also installs anything else comes after `ARG CACHEBUST` - too. A `script/bootstrap` that also installs node packages is one. A new - value reruns every `RUN` step after that line; `COPY` steps are compared by - content and need nothing. Docker keeps the layers above the line until one - of these changes: - - the base image - - their own lines - - the files they copy - - A stage without the `ARG` line is served from cache on an unchanged tree, so - a repository re-vendors the scripts and adds the `ARG` lines in one commit. - The scripts do not pass `--no-cache`. That flag reruns every step too, but - it also builds the layers above the line again and empties every cache - mount. Go's build cache and golangci-lint's would then start empty every - time; the canonical Go `Dockerfile` below mounts them so that they survive - between check builds. A bare `docker build .` is not evidence that anything - ran: a sub-second build reporting success is a cache hit, not a result. - Never invalidate by pruning: `docker builder prune` and friends destroy a - build cache shared with every other build on the host. When a check is added - or changed, prove it works: - - plant a defect it must catch - - watch the run fail on it - - revert the defect - - A green run alone shows neither that the check ran nor that it covers what - it should. - -- **The gate phases are separate stages, and the build stage depends on both.** - The lint phase is based on the `golangci/golangci-lint` image (pinned by - hash), so lint failures surface in seconds rather than after a full compile. - The test phase is based on the Debian Go image. The canonical Go repo - `Dockerfile`: - - ```dockerfile - # In every stage ARG CACHEBUST comes right after the steps that install - # system packages and download Go's modules, or right after FROM in a - # stage with neither, and every step that lints, tests or compiles - # comes after it. script/ passes a new value on every build, which - # reruns every RUN step after it. Docker keeps the layers above it - # until the base image, their own lines or go.mod and go.sum change. - # The steps that compile mount Go's build cache, and golangci-lint run - # also mounts golangci-lint's; both survive from build to build. - - # Lint phase - # golangci/golangci-lint:v2.x.x, YYYY-MM-DD - FROM golangci/golangci-lint@sha256:... AS lint - WORKDIR /src - COPY go.mod go.sum ./ - RUN go mod download - ARG CACHEBUST - COPY . . - RUN --mount=type=cache,target=/root/.cache/go-build \ - --mount=type=cache,target=/root/.cache/golangci-lint \ - golangci-lint run --config .golangci.yml ./... - - # Test phase. -race needs cgo and so a C compiler, which the Debian Go - # image ships and the alpine one does not. -count=1 because the mounted - # cache also holds test results, which go test would otherwise replay. - # A first run that timed out gets no rerun, because a rerun would only - # wait for the same hang. - # golang:1.x, YYYY-MM-DD - FROM golang@sha256:... AS test - WORKDIR /src - COPY go.mod go.sum ./ - RUN go mod download - ARG CACHEBUST - COPY . . - RUN --mount=type=cache,target=/root/.cache/go-build \ - status=0; \ - output=$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$?; \ - printf '%s\n' "$output"; \ - test "$status" -eq 0 && exit 0; \ - case "$output" in \ - *"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \ - esac; \ - echo "--- Rerunning with -v for details ---"; \ - go test -count=1 -timeout 90s -race -v ./...; \ - exit 1 - - # Build stage. Nothing is wanted from either phase above; the copies - # are what make BuildKit build them first, so nothing after them can - # run unless lint and test passed. - # golang:1.x-alpine, YYYY-MM-DD - FROM golang@sha256:... AS builder - # git comes from the base image's own distribution package index, which - # is trusted, so it needs no version and no hash. - RUN apk add --no-cache git - WORKDIR /src - COPY go.mod go.sum ./ - RUN go mod download - ARG CACHEBUST - COPY --from=lint /src/go.sum /dev/null - COPY --from=test /src/go.sum /dev/null - # A tar-stream context keeps the sender's file owners, which git refuses. - RUN git config --system --add safe.directory /src - COPY . . - - # The VERSION build arg when one is given, otherwise - # `git describe --tags --always` on the .git in the build context. With - # .git present, a version that is still empty, dev or unknown fails the - # build: git is missing or could not read the checkout. - ARG VERSION - RUN --mount=type=cache,target=/root/.cache/go-build \ - VERSION="${VERSION:-$(git describe --tags --always)}"; \ - if [ -e .git ]; then \ - case "$VERSION" in ""|dev|unknown) \ - echo "version is '$VERSION' although .git is present" >&2; \ - exit 1 ;; \ - esac; \ - fi; \ - CGO_ENABLED=0 go build -trimpath \ - -ldflags="-s -w -X main.Version=${VERSION}" \ - -o /app ./cmd/app/ - - # Runtime stage, and the last one - FROM alpine@sha256:... - ARG CACHEBUST - COPY --from=builder /app /usr/local/bin/app - ENTRYPOINT ["app"] - ``` - - Key points: - - The lint phase uses the `golangci/golangci-lint` image directly, so - nothing needs installing: it has both Go and the linter. - - `COPY --from= /src/go.sum /dev/null` is a no-op copy whose only - purpose is the dependency on that phase. BuildKit runs stages in parallel - by default, and a stage nothing depends on is not built at all. Without - these two lines a red gate would not fail the build. - - Keep the runtime stage last. If you add a stage after it, give it the same - two copies. A plain `docker build .` builds the last stage's chain and - nothing else. - - Check builds keep these, and none of them can hand back a stale result: - - The layers above `ARG CACHEBUST`: the system packages and Go's - modules. Docker installs the packages again when the base image or the - install line changes. Docker downloads the modules again when `go.mod` - or `go.sum` changes. Go checks every module against `go.sum` as it - downloads it, and no later build can change a kept layer. The modules - are kept in a layer rather than a cache mount. A cache mount is shared - by every build on the host, and Go builds from whatever files the - mount holds without checking them again (only `go mod verify` does). A - file changed there would go unnoticed. - - Go's build cache (`/root/.cache/go-build`, where Go keeps it for - root), mounted on the steps that compile: `golangci-lint run`, both - `go test` runs and `go build`. Go reuses a compiled package only when - all its inputs are unchanged. The cache also holds test results, which - is why both `go test` runs carry `-count=1`. - - golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned - image keeps it), mounted on `golangci-lint run`. golangci-lint reuses - a package's findings only when all of these are unchanged: - - the package's files and those of every package it imports - - `go.mod` - - the linters and their settings in `.golangci.yml` - - the golangci-lint version - - In a Python `test` phase, the layer of the step that runs - `script/bootstrap` to install the packages from `Pipfile.lock`, as - described for that phase below. - - If the project uses `//go:embed` directives that reference build - artifacts, the lint phase must create placeholder files so the embed - directives resolve. A web frontend compiled in a separate stage is such an - artifact. Example: - `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. - - If the project requires CGO or system libraries for linting, install them - in the lint phase. The `golangci/golangci-lint` image is Debian-based and - has no `apk`. So install with `apt-get` under the Debian package name - (`libvips-dev`, where alpine says `vips-dev`), and delete the package - lists in the same `RUN`, so the layer does not keep them. The test phase, - on the Debian Go image, installs the libraries its tests need the same - way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so - Docker keeps it until the base image or the install line changes: - - ```dockerfile - FROM golangci/golangci-lint@sha256:... AS lint - RUN apt-get update \ - && apt-get install -y --no-install-recommends libvips-dev \ - && rm -rf /var/lib/apt/lists/* - WORKDIR /src - COPY go.mod go.sum ./ - RUN go mod download - ARG CACHEBUST - ``` - - - `.dockerignore` lets `.git` into the build context. It keeps out every git - `config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): - - the repository's own - - each submodule's under `.git/modules/` - - that of a submodule keeping its own `.git` directory - - `git describe` does not need them, and each can hold a credential: a - password in a remote URL, or the token the CI checkout step stores - there. Some submodule names have a `config` segment, as in `config`, - `deploy/config` or `config/lib`. Such a submodule loses its whole git - directory to `**/.git/modules/**/config`, and Go's version stamping then - fails the build. Give it a name without that segment - (`git submodule add --name`). The stage that compiles has `git`: the - Debian Go image has it, and an alpine one needs - `apk add --no-cache git`. That stage takes the version from the - `VERSION` build argument when one is given, otherwise from - `git describe --tags --always`. That gives: - - the tag, on a tagged commit - - the tag, the number of commits since it and the short commit - (`v1.2.3-4-gabc1234`), on a later commit - - the short commit, when no tag is reachable - - The stage that compiles also marks its working directory safe for git - (`git config --system --add safe.directory /src`). A context sent as a - tar stream keeps the sender's file owners, and git refuses a checkout - owned by another user, so the version would come out empty. - `ARG VERSION` has no default. The build fails if the context carries - `.git` and the version still comes out empty, `dev` or `unknown`. A - plain `docker build .` with no build arguments must succeed; a - Dockerfile that refuses an empty build argument drops that refusal and - keeps the argument. A checkout whose `.git` is a file is the exception: - a linked worktree, or a repository checked out as a submodule. That file - points to a git directory outside the build context, so the build cannot - read the version and a plain `docker build .` fails. Pass the version - with `--build-arg VERSION=...`, as `script/docker` and `script/cibuild` - already do. - -- **A Python `test` phase that installs its packages from `Pipfile.lock` does it - above `ARG CACHEBUST`, so check builds keep them.** The steps are: - - `COPY script/ script/` - - `COPY Pipfile Pipfile.lock ./` - - `RUN script/bootstrap` - - `ARG CACHEBUST` comes right after them. Docker keeps what that `RUN` - installed until one of these changes: - - the base image - - `script/` - - `Pipfile` - - `Pipfile.lock` - - a line up to and including the `RUN` - - The tests run after `ARG CACHEBUST`, on every check build. In every other - stage, a step that installs Python packages comes after `ARG CACHEBUST`. - - When `script/bootstrap` runs in that step, it installs nothing beyond - these: - - system packages - - pipenv and its dependencies - - the packages from `Pipfile.lock` - - `script/bootstrap` installs no Python there: the tests use the - distribution's Python, which a step above it installs. If it installs - anything else there, the step goes after `ARG CACHEBUST`. - - - Debian's Python refuses a system-wide `pip install`, so pipenv goes in a - virtual environment at `$HOME/.pipenv`. `script/bootstrap` creates it with - `python3 -m venv` when it is missing, then uses that environment's `pip` - and `pipenv`. On a host, it first installs the distribution's python and - venv packages when they are missing. One script then works on a host and - in a `Dockerfile`. `Pipfile` names no `python_version`, because - `pipenv install --deploy` stops when the distribution's Python differs - from it. - - PyPI is not trusted, so `script/bootstrap` installs pipenv and its - dependencies by hash. It does so with `pip install --require-hashes -r` - and a requirements file in `script/` that lists each of them with `==` and - its `--hash`. pip refuses to install a dependency the file does not list. - - `script/bootstrap` then runs `pipenv install --deploy`, with `--dev` when - the tests need the development packages. `--deploy` does all of these: - - installs only what `Pipfile.lock` names - - checks every package against the lockfile's hashes - - fails when the lockfile is out of date with `Pipfile` - - `script/bootstrap` sets `PIP_ONLY_BINARY=:all:` for both installs, and - pipenv passes it on to pip. A package with no ready-built wheel for the - image then stops the install. Without the setting, pip builds that package - from source and downloads the build tools without a hash check. - - A kept layer holds nothing a new install would refuse: pip checked the - hashes of pipenv and its dependencies, and pipenv checked the hashes of - the lockfile's packages. pipenv creates its virtual environment with the - `virtualenv` package. The packages that `virtualenv` adds come from wheels - inside that package, so the hash pip checked for it covers them. System - packages, Python among them, are not fixed by a hash. They keep the - versions the package manager gave when the layer was built; a new install - may get newer ones. - - As with Go's modules, the packages are kept in a layer, not a cache mount. - A cache mount is shared by every build on the host, and nothing checks its - files again. - - A Python repository's `script/bootstrap` adds these lines in `main`, above - the last `echo`. The `ensurepip` test fails when python is missing, and also - when Debian's venv package is missing: - - ```sh - if [ ! -x "$HOME/.pipenv/bin/pip" ]; then - if ! python3 -m ensurepip --version >/dev/null 2>&1; then - pkg_install python3 python3-venv python3 python3 - fi - python3 -m venv "$HOME/.pipenv" - fi - export PIP_ONLY_BINARY=:all: - "$HOME/.pipenv/bin/pip" install --require-hashes -r script/requirements.txt - "$HOME/.pipenv/bin/pipenv" install --dev --deploy - ``` - - Make `script/requirements.txt` once, with - `echo pipenv | pip-compile --generate-hashes --allow-unsafe -o script/requirements.txt -`. - `pip-compile` comes from Alpine's `py3-pip-tools` package; Debian and Ubuntu - do not package `pip-tools`. Make the first `Pipfile.lock` with - `$HOME/.pipenv/bin/pipenv lock`, after the `pip install` line has installed - pipenv. - - In the example below, `script/bootstrap` finds `make`, `git`, python and - venv already installed. The example copies no `yarn.lock`, so - `script/bootstrap` installs no node and no yarn. - - ```dockerfile - # Test phase. Both pytest runs share one 90-second timeout, so a hang - # uses it once and gets no rerun. - # debian:trixie-slim, YYYY-MM-DD - FROM debian@sha256:... AS test - # Python, pip and venv come from the distribution, which is trusted, so - # they need no version and no hash. make and git are base tools that - # script/bootstrap installs when they are missing. Installed here, they - # stay in their own layer when script/ changes. - RUN apt-get update \ - && apt-get install -y --no-install-recommends ca-certificates make git \ - python3 python3-venv \ - && rm -rf /var/lib/apt/lists/* - # script/bootstrap installs pipenv in a virtual environment at - # $HOME/.pipenv, and the tests below run that pipenv. - ENV PATH=/root/.pipenv/bin:$PATH - WORKDIR /app - COPY script/ script/ - COPY Pipfile Pipfile.lock ./ - RUN script/bootstrap - ARG CACHEBUST - COPY . . - RUN timeout 90 sh -c 'pipenv run pytest || { \ - echo "--- Rerunning with -v for details ---"; \ - pipenv run pytest -v; exit 1; }' - ``` - -- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - runs `script/cibuild` on push. The workflow's only other step checks out the - repo, with `persist-credentials: false`. `script/cibuild` needs no token, and - without that setting the checkout leaves the job's token in `.git/config` for - every later step. The checkout step also sets `fetch-depth: 0`, which fetches - the tags `git describe` needs. By default it clones shallow with no tags, and - a tagged repository's CI build would stamp a bare short commit id. The - workflow's `concurrency` block groups runs by workflow and branch - (`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`. - A new push then cancels the older run on the same branch, queued or running, - and no other: runs for replaced commits do not hold up the shared runner. - `script/cibuild` bootstraps, runs the gate phases, and then builds the image, - so a successful run means every check passed. A bare `docker build .` does not - carry the same guarantee, because its gate phases may come from the cache. The - image build gets a `CACHEBUST` value of its own and so runs the gate phases a - second time. That is the price of the rule above, and it is worth paying. The - image that ships is built from a run of its own gates rather than from a cache - entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the - shared runner after 20 minutes. That allows for the three Docker builds - described above, each held to the 5-minute Docker build limit below, plus the - bootstrap. Those builds are the test phase, the lint phase, then the image. A - separate workflow limited to `main` by a `branches` list under `on: push` - cannot be checked by review. To try a change to it, add the feature branch to - that list and push, then remove the branch from the list again before merging. - Keep any job in it that publishes behind `if: github.ref_name == 'main'`, so - the run from the feature branch publishes nothing. - -- Use platform-standard formatters: - - `black` for Python - - `prettier` for JS/CSS/Markdown/HTML - - `go fmt` for Go - - Always use default configuration, with two exceptions: - - four-space indents (except Go) - - `proseWrap: always` for Markdown (hard-wrap at 80 columns) - - Documentation and writing repos (Markdown, HTML, CSS) should also have - `.prettierrc` and `.prettierignore`. - -- Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local - testing is not possible in the repo, `script/precommit` may skip `script/test` - and run only `script/lint` and `script/fmt-check`. `script/install-precommit` - installs the hook; the Makefile must provide a `make hooks` target that shims - to it. - -- All repos with software must have tests that run via the platform-standard - test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful - tests exist yet, add the most minimal test possible: importing the module - under test to verify it compiles/parses, for example. There is no excuse for - `make test` to be a no-op. - -- `make test` must complete in under 60 seconds. That is the hard cap, and a - suite that exceeds it fails. Under 20 seconds is the target. A suite between - 20 and 60 seconds is still green, but the overage must be filed as an - improvement bug against that repo. Add a 90-second timeout to the test - invocation (`go test -timeout 90s`). The backstop deliberately sits above the - hard cap so that it catches a genuinely hung test rather than a merely slow - one. A hung package uses the backstop once, because a first run that timed out - gets no rerun. `-timeout` applies to each package, so several hung packages - can each use it. - -- **The test command should use the conditional verbose rerun pattern.** Run - tests without `-v` (verbose) first. If tests fail, automatically rerun with - `-v` to show full output. This keeps CI logs and `docker build` output clean - on success, with just package/suite summaries. On failure it provides full - diagnostic detail: every test case, every assertion. A first run that timed - out gets no rerun, because the rerun would only wait for the same hang. The - command lives in the `test` phase of the `Dockerfile`, since `script/test` - builds that phase. The Makefile form below is the same pattern for any - repo-local invocation: - - ```makefile - test: - @ || \ - { echo "--- Rerunning with -v for details ---"; \ - ; exit 1; } - ``` - - Go example: - - ```makefile - test: - @status=0; \ - output=$$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$$?; \ - printf '%s\n' "$$output"; \ - test "$$status" -eq 0 && exit 0; \ - case "$$output" in \ - *"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \ - esac; \ - echo "--- Rerunning with -v for details ---"; \ - go test -count=1 -timeout 90s -race -v ./...; \ - exit 1 - ``` - - The first run's output is kept so that the recipe can look in it for what Go - prints for a hang: - - `panic: test timed out` means the test's own timer fired. - - `*** Test killed` means `go test` stopped a test that ignored that timer, - about a minute later. - - Go prints the goroutine stacks with either line, as long as the test can - still respond. A rerun would only wait for the same hang, so the recipe - exits 1. A failure with neither line in its output reruns with `-v`. The - kept output prints when the first run ends, not while it runs. - - `-count=1` is required on both invocations. It defeats Go's test _result_ - cache, so neither run can report a stored pass in place of running the - tests. It leaves the build cache alone, so it costs the runtime of the suite - and no recompilation. - - That cache is Go's own, separate from Docker's layer cache. Go stores a - passing result in its cache directory (`GOCACHE`). When the same tests run - again on unchanged code, it prints that result, marked `(cached)`, without - running them. The directory lasts from one run to the next on a developer's - machine, where this target runs. The directory also lasts in the `test` - phase of the `Dockerfile`, which mounts it to keep compiled packages between - check builds, so the phase's `go test` runs carry `-count=1` too. A new - `CACHEBUST` value (above) is what makes that step run on an unchanged tree. - - Python example: - - ```makefile - test: - @timeout 90 sh -c 'pipenv run pytest || { \ - echo "--- Rerunning with -v for details ---"; \ - pipenv run pytest -v; exit 1; }' - ``` - - Both pytest runs share that one 90-second timeout, as in the `Dockerfile` - example above, so a hang uses it once and gets no rerun. Ctrl-C does not - stop a running `make test`, which goes on until the tests end or the 90 - seconds run out. On a Mac without `timeout`, the target fails until it is - installed. On a host, the target needs `$HOME/.pipenv/bin` on `PATH`. - - The `exit 1` ensures the target always fails after a rerun. The first run - already proved the tests are broken, so the build must not pass even if a - flaky test happens to succeed on the second attempt. The rerun exists solely - for diagnostic output. - -- Docker builds must complete in under 5 minutes. A hung package uses the - 90-second test backstop once, because a first run that timed out gets no - rerun. - -- `make check` must not modify any files in the repo. Tests may use temporary - directories. - -- `main` must always pass `make check`, no exceptions. +# Pin external references by hash + +**External references must be pinned by cryptographic hash.** This is the single +most important rule in this document. It covers what we download and install +ourselves in scripts and Dockerfiles. Reference each by its content hash: + +- Docker base images: `@sha256:...` +- Go modules: the module hash in `go.sum` +- Python packages from PyPI: the package hash in `Pipfile.lock` +- npm packages: the `checksum` in `yarn.lock` +- GitHub Actions: `@` +- release archives fetched with `curl` or `wget`: a hash hardcoded in the + Dockerfile or script + +Packages from the distribution's own package manager are outside this rule. They +need no version and no hash, because the distribution is trusted. In a +`Dockerfile`, it is the pinned base image's distribution. On a host, it is the +host's. The host's package manager stands for its distribution, and that +includes nix and brew. Updating the package index and upgrading installed +packages are fine too. This command, for example, is allowed: + +```sh +apt update && apt upgrade -y && apt install -y build-essential +``` + +A package source that a `Dockerfile` adds itself is not the distribution's own, +so its packages are pinned like any other download. + +- A version tag such as `@v4`, `@latest` or `:3.21` is server-mutable, and + therefore a remote code execution vulnerability. +- An unverified install script is arbitrary remote code execution. Never + `curl | bash` to install a tool such as rustup. Instead, download a specific + release archive from GitHub, verify its hash, and only then install. +- Docker Hub, PyPI, the npm registry and `go get` are not trusted, so everything + from them stays pinned by hash. +- Above each image or package pinned by hash, add a comment with the version and + date (YYYY-MM-DD). +- Double-check every external reference in every file before committing. + +node, yarn, npm, python, pip and venv come from the distribution wherever +possible: + +- node means the distribution's newest Node package, such as `nodejs-current` on + Alpine 3.24. +- pip and venv are the installer tools bundled with python. +- If the distribution's version is too old, use a newer distribution, never a + download. That rules out nvm, pyenv, corepack and `npm install -g yarn`. +- yarn's release file is third-party code, and git pins it. `yarn set version` + writes it to `.yarn/releases/` once, given an exact version. + +# Git and secrets - Never commit secrets. `.env` files, credentials, API keys, and private keys must be in `.gitignore`. No exceptions. - -- `.gitignore` should be comprehensive from the start: - - OS files (`.DS_Store`) - - editor files (`.swp`, `*~`) - - in-repo agent scratch directories (`.claude/`) - - `node_modules/` - - `.yarn/*`, with `!.yarn/releases` so that git keeps yarn's release file - - the repo's own build outputs - - Fetch the standard `.gitignore` from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting - up a new repo. A repo's `.gitignore` is the standard file followed by the - repo's own entries, such as its binaries. A re-vendor replaces the standard - part and keeps those entries. These patterns are written to `.gitignore`'s - own semantics, in which an unanchored pattern already matches at every - depth. They are not a `.dockerignore` and must not be transplanted into one - unmodified. - -- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns - across unmodified leaves secrets in the build context.** Docker matches with - `moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension. So - `*` does not cross `/`, and a pattern without a leading `**/` is anchored at - the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key` - therefore excludes only the copies at the repository root. `config/.env` and - `certs/server.key` still reach the context and can land in an image layer. - Such a `.dockerignore` is more dangerous than a short file with no secret - patterns at all, because it reads as solved and stops anyone looking. Give - every depth-independent pattern the `**/` prefix, and leave only genuinely - root-anchored entries unprefixed. Those are `.claude`, `.yarn/*`, - `!.yarn/releases`, and the repo's own host-built binary. The binary is written - `/myapp` and never `**/myapp`, which would also match `cmd/myapp/` and delete - the package directory from the context. Matching is case-sensitive, and an - all-caps copy of each pattern still misses `Server.Key`. So secret names use - character ranges: `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`, and likewise for - `.envrc` and the extensionless SSH keys. Where such a pattern also catches - something the build needs, re-include it with a negation - (`!docs/example.env`). Deleting the pattern reopens the exposure for every - other file it covers. Fetch the standard `.dockerignore` from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend - it with the repo's own artifacts. - -- **In-repo agent scratch belongs in both files, written to each file's own - semantics.** `.claude/` holds one worktree per in-flight agent, an entire - additional checkout of the repo. Under `COPY . .` the build context therefore - inflates by a multiple of the repo, and another session's unreviewed work can - be copied into an image layer. In `.gitignore` the entry is `.claude/`, - unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/` - prefix, because the prefixed form would also delete any nested directory of - that name from the build. Anchoring carries a known gap. The canonical - `.dockerignore` states the gap in its own comment, since consuming repos - receive the file and not the tracker. The gap is that the directory is created - in the agent's working directory. A repo running agents in subdirectories - therefore still ships `services/api/.claude/` and must add its own anchored - entry there. - -- **A plain `docker build .` of a clone stamps the version that - `git describe --tags --always` gives.** The version is derived from the `.git` - in the build context, as the canonical `Dockerfile` above shows. Without its - failure check, a missing `git` or an unreadable checkout would leave - `-X main.Version=` empty, and the build would still exit 0. `script/docker` - and `script/cibuild` pass the version they compute on the host; it takes - precedence. They do this byte-identically across repos: - - ```sh - # The version and the tag each get their own line: a failing command - # substitution inside an argument does not trip `set -e`, so the inline - # form degrades to an empty constant. - version="$(git describe --tags --always --dirty 2>/dev/null || true)" - [ -n "$version" ] || version="unknown" - tag="$(script/projectname)" - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --build-arg VERSION="$version" \ - -t "$tag" . - ``` - - `--always` makes an untagged repo yield an abbreviated commit hash rather - than failing. The `[ -n "$version" ]` line is the single place the fallback - is applied. That line is a live check that fires on a build from an export - with no `.git`, and on a repository with no commits yet. Do not fold it into - the substitution as `|| echo unknown`, which makes the guard unreachable. - The Dockerfile's side is `ARG VERSION` in the stage that compiles, declared - there because `ARG` is stage-scoped. Passing `VERSION` to a repo whose - Dockerfile declares no such `ARG` is ignored and costs nothing, which is why - the scripts stay byte-identical. One consequence for CI: the standard - checkout action clones shallow and fetches no tags, so the canonical - `.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step. - -- **Verify `.dockerignore` by enumerating the image, not by reading the - patterns.** Plant files at the root _and_ at least two directories deep. Build - a probe image that does `COPY . .`, and list what actually landed - (`docker run --rm --entrypoint find IMAGE /app`). The `transferring context` - size is not a substitute: a nested secret is a few bytes, and BuildKit - transfers only the delta from the previous build. - +- Make all changes on a feature branch. You can do whatever you want on a + feature branch. +- Never force-push to `main`. +- Never use `git add -A` or `git add .`. Always stage files explicitly by name. - **No build artifacts in version control.** Code-derived data must never be committed to the repository if it can be avoided. Code-derived data includes: - compiled bundles @@ -897,249 +80,987 @@ style conventions are in separate documents: The build process (e.g. Dockerfile, Makefile) should generate these at build time. Notable exception: Go protobuf generated files (`.pb.go`) ARE - committed because repos need to work with `go get`, which downloads code but - does not execute code generation. + committed, because repos need to work with `go get`, which downloads code + but does not execute code generation. -- Never use `git add -A` or `git add .`. Always stage files explicitly by name. +# Entrypoints -- Never force-push to `main`. +Repos follow the +[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) +pattern. Every repo with software must have a root `Makefile` with the targets +below. Each target is a thin shim that calls an executable script in `script/`, +where its implementation lives: -- Make all changes on a feature branch. You can do whatever you want on a - feature branch. +- `make bootstrap`: `script/bootstrap` installs all dependencies. +- `make setup`: `script/setup` readies a fresh clone for development. It runs + `bootstrap`, then `install-precommit`, plus any repo-specific initialization. +- `make test`: `script/test` +- `make lint`: `script/lint` +- `make fmt`: `script/fmt` (writes) +- `make fmt-check`: `script/fmt-check` (read-only) +- `make check`: `script/check` runs `script/test`, `script/lint` and + `script/fmt-check`. +- `make docker`: `script/docker` +- `make hooks`: `script/install-precommit` installs the git pre-commit hook. -- `.golangci.yml` is standardized. The vendored copy in a consuming repo must - _NEVER_ be modified by an agent. Fetch it from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it - byte-identical, so that no repo can quietly loosen its own linting. Linter - configuration changes are made to the canonical copy in the `prompts` repo and - reach consuming repos by re-vendoring. An agent may open a PR against - canonical, which only the user merges. One list is exempt from byte-identity, - because it cannot be written once for every repo: the `deny` list of the - `test-support` depguard rule. There a repo names its own test-support packages - by full import path. A repo adds entries there and changes nothing else, and a - re-vendor carries its entries forward. The canonical golangci-lint version is - v2.15.0 (released 2026-10-16). The version is pinned as the digest of the lint - phase's base image, +Three more scripts have no required target: + +- `script/precommit` is what the pre-commit hook runs. It calls `script/check`. + Repo-type-specific extras belong in it, not in the hook itself; `go mod tidy` + verification in Go repos is one. If local testing is not possible in the repo, + it may skip `script/test` and run only `script/lint` and `script/fmt-check`. +- `script/cibuild` runs the CI build, and the Gitea workflow calls it. +- `script/projectname` (literally that filename) outputs the project's name. + Scripts that need the name call it, so they stay byte-identical across all + repos. `script/docker`, for example, assembles its image tag from it. + +From the standard's canonical set we use `bootstrap`, `setup`, `test`, and +`cibuild`. `check`, `precommit`, `install-precommit` and `projectname` are our +own extensions to the standard. + +The scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run +in minimal containers; alpine images, for example, have no bash. Locate the repo +root with `$(cd "$(dirname "$0")/.." && pwd -P)`, and `cd` there before acting. +A model Makefile is at +`https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`. Model scripts are +at `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/`. + +Run the repo's own entrypoints, never the underlying tools directly. The +`script/` scripts are the single source of truth for how each operation runs. +Calling them directly is fine, and so is calling the Makefile targets. The +Makefile targets must still exist, as the thin shims described above. + +An agent whose work is confined to one package may run `go test` directly on +that package's suite, or on individual tests in it, while iterating. This is the +only exception to this rule. `go build`, `go vet` and the linter still go +through the entrypoints. A run while iterating is not a gate, so the full +`make check` is still required before every push. + +The Makefile is authoritative documentation for how the repo is used. Beyond the +required targets above, it should have targets for every common operation: + +- running a local development server (`make run`, `make dev`) +- re-initializing or migrating the database (`make db-reset`, `make migrate`) +- building artifacts (`make build`) +- generating code +- seeding data +- anything else a developer would do regularly + +Typing `make` in a checkout should show every meaningful operation. A new +contributor should be able to understand the entire development workflow by +reading the Makefile. + +# Tests and checks + +- All repos with software must have tests that run via the platform-standard + test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). There is no + excuse for `make test` to be a no-op. If no meaningful tests exist yet, add + the most minimal one: importing the module under test to verify it + compiles/parses, for example. +- `make test` must complete in under 60 seconds. That is the hard cap, and a + suite that exceeds it fails. Under 20 seconds is the target. A suite between + 20 and 60 seconds is still green, but the overage must be filed as an + improvement bug against that repo. +- Add a 90-second timeout to the test invocation (`go test -timeout 90s`). This + backstop sits above the hard cap on purpose: it catches a hung test, not a + merely slow one. `-timeout` applies to each package, so several hung packages + can each use it. +- Docker builds must complete in under 5 minutes. +- `make check` must not modify any files in the repo. Tests may use temporary + directories. +- `main` must always pass `make check`, no exceptions. + +**The test command should use the conditional verbose rerun pattern.** Run tests +without `-v` (verbose) first. If they fail, rerun automatically with `-v`. +Success then leaves CI logs and `docker build` output clean, with just +package/suite summaries. Failure shows every test case and every assertion. + +- The rerun always ends in `exit 1`. The first run already proved the tests are + broken, so the build must not pass even if a flaky test succeeds on the second + attempt. The rerun exists solely for diagnostic output. +- A first run that timed out gets no rerun, which would only wait for the same + hang. A hung package therefore uses the 90-second backstop once. + +The command lives in the `test` phase of the `Dockerfile`, since `script/test` +builds that phase. The canonical Go `Dockerfile` and the Python `test` phase +example below carry it. The Makefile form below is the same pattern for any +repo-local invocation. In a Makefile, each `$` of those two commands is written +`$$`. + +```makefile +test: + @ || \ + { echo "--- Rerunning with -v for details ---"; \ + ; exit 1; } +``` + +# `script/bootstrap` + +`script/bootstrap` installs all dependencies idempotently and assumes nothing is +present. + +- A tool from the distribution has no pin, so it is installed only when it is + missing. make, git, node and yarn are such tools. They come from nix, apt, + brew, or apk, detected in that order; apt runs noninteractive. +- A tool from outside the distribution is never installed as "latest" or "lts"; + it is always an exact version. + +## node and yarn + +Use `yarn`, not `npm`. + +- `script/bootstrap` installs node, yarn and the node packages only when + `yarn.lock` is in the tree. A `Dockerfile` stage that needs no yarn copies no + `yarn.lock` before it runs `script/bootstrap`, so `script/bootstrap` installs + no node and no yarn there. The Go and Python stages below are such stages. +- It installs the node packages with `yarn install --immutable`, which fails + when a package's `checksum` is wrong, or when `package.json` is out of step + with `yarn.lock`. +- Each repository commits the yarn that runs. `yarnPath` in `.yarnrc.yml` points + at the release file in `.yarn/releases/`. The distribution's yarn only starts + that file, and that file does the work. So Alpine's yarn 1 and Debian 13's + yarn 4.1 run the same committed yarn. Corepack never runs, and no yarn is + downloaded. +- `.yarnrc.yml` holds `yarnPath`, `nodeLinker: node-modules` and + `enableTelemetry: false`. Without the last, yarn sends usage reports from any + host that is not CI. +- By default, yarn runs no dependency's install script. A dependency that must + run one is named in `package.json`, as + `"dependenciesMeta": { "": { "built": true } }`. +- `.gitattributes` carries `.yarn/releases/** binary`, so diffs and merges treat + the release file as one opaque file. +- yarn 1 ignores `--immutable` and exits 0, so on a tree with no `.yarnrc.yml` + the install is not frozen. A repository therefore re-vendors + `script/bootstrap` and converts its `yarn.lock` in one commit. +- Debian and Ubuntu name the yarn command `yarnpkg`. `script/bootstrap`, + `script/fmt` and `script/fmt-check` each carry the same small function that + runs whichever name is present. With neither name on `PATH`, `script/fmt` and + `script/fmt-check` print one line saying to run `script/bootstrap`, and + exit 1. +- A `Dockerfile` stage that needs yarn may start from any distribution that + packages node and yarn. Before it runs `script/bootstrap`, it copies + `package.json`, `yarn.lock`, `.yarnrc.yml` and `.yarn/releases/`. With only + the first two, the install fails, and Alpine's yarn 1 says to enable corepack. + Never do that; copy the missing files instead. + +## Pinned tools + +**`script/bootstrap` installs a pinned tool by comparing versions, never by +testing presence.** An `if ! command -v ; then install; fi` guard tests +`PATH` only. On an already-provisioned machine the pin is then inert, and a +version bump is a silent no-op. The Dockerfile's clean image still gets the +pinned version, so a local `make check` and `make docker` can disagree about +what the tool even is. The canonical form: + +- compares the installed version against the pin over the **whole** version + token. A parser that stops at the first `-` reports `2.12.2` for a host + running `2.12.2-rc1` and skips the install. +- treats `--version` output as a mismatch when it is any of these: + - absent + - non-zero + - empty + - unrecognised + + So a failure causes a redundant install, never a skipped one. + +- after installing, re-resolves the binary the way callers do: `hash -r`, then + `PATH`, not the directory the installer wrote to. A shadowing binary can hide + an install that reported success. In that case it fails and names the resolved + path. +- is actually called, and prints the version on both success paths. A function + defined and never invoked has the same exit status and the same empty output + as one that worked. +- stays POSIX sh: no arrays, no `[[`, no `grep -P`. + +A Go tool a repo needs on the host is installed with `go install` pinned to a +commit hash (`go install @`). It is never tracked as a +`go.mod` tool dependency or through a `tools.go` file, either of which pulls the +tool's own dependencies into the repo's `go.mod` and `go.sum`. + +# Dockerfile + +Every repo should have a `Dockerfile`, and it carries the repo's gates: a `lint` +phase and a `test` phase. The final stage depends on both, so the image cannot +be built unless they pass. For non-server repos the final stage brings up a +development environment; for server repos it is the runtime image. + +**The lint and test gates run in Docker, as phases of the `Dockerfile`.** There +is no separate lint file. `script/lint` and `script/test` each build one phase +and nothing else: + +```sh +docker build --build-arg CACHEBUST="$(date +%s)-$$" \ + --target lint --output type=cacheonly . +docker build --build-arg CACHEBUST="$(date +%s)-$$" \ + --target test --output type=cacheonly . +``` + +- **A stage that is not the last one in the file is built only when the final + stage's chain depends on it, or when `--target` names it.** That is why the + two gates are always invoked by name here, and why the final stage carries a + `COPY --from=` of a harmless file from each of them. Without those copies, a + plain `docker build .` builds the last stage's chain alone and exits 0 having + linted and tested nothing. Keep the stage that ships last: the runtime image, + or the development environment in a non-server repo. If you add a stage after + it, give it the same two copies. +- **The gate builds write no image.** With `--output type=cacheonly` the phase + runs and a failing step fails the build, but the result is not exported. + Nothing uses those images, and writing one out is slow. A build given neither + `--output` nor `-t` leaves an untagged image dangling on every developer host + and CI runner. `script/cibuild` and `script/docker` tag the image that ships, + so each build replaces the previous image. +- **Inside a phase the tool is invoked directly:** `golangci-lint`, `go test`, + `eslint`, `prettier`. It is never invoked through `make lint` or + `script/test`, which are themselves a `docker build` and would recurse into a + daemon that does not exist in a build step. +- **Formatting is the exception and stays on the host.** `script/fmt` writes the + working tree, and `script/fmt-check` is its read-only twin. +- **No lint verdict may come from a host invocation of the linter.** On a shared + host golangci-lint reads a result cache keyed on file content rather than + location. A second checkout of the same content is therefore served the first + one's findings. A host-global lock in `$TMPDIR` also makes concurrent runs + exit non-zero with `parallel golangci-lint is running`, which a caller cannot + tell from real findings. A container has its own `TMPDIR`, out of the lock's + reach. The lint phase does keep golangci-lint's cache between check builds, + but it lints every checkout at `/src`. It reuses a finding only on the + conditions listed under the canonical Go `Dockerfile` below. + +What each stage installs: + +- The gate phases and the build stage start from their pinned base images. They + install what those images lack in either of these ways: + - inline, as the canonical Go `Dockerfile` below does for `git` + - by running `script/bootstrap`, as the `prompts` repo's own `Dockerfile` + does for its yarn packages +- The development environment stage installs development prerequisites by + running `script/bootstrap`. Only system packages may be installed inline above + it, so that check builds keep them; the `prompts` repo's own `Dockerfile` does + this. +- A stage that runs `script/bootstrap` first COPYs `script/` and the dependency + manifests it needs: `go.mod` and `go.sum`, for example, or the four yarn files + named above. + +## `CACHEBUST` + +**Any build that runs checks is given a new `CACHEBUST` value.** On an unchanged +tree Docker otherwise serves the check `RUN` from cache: nothing executes, and +the build still exits 0. + +Every `docker build` in `script/` therefore passes +`--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process id. +So no two builds share a value, not even the three in one `script/cibuild`. Four +scripts do this, and there is no fifth: `script/lint`, `script/test`, +`script/cibuild` and `script/docker`. `script/check` builds no image of its own. + +In every stage of the `Dockerfile`, every step that lints, tests or compiles +comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps: + +- the steps that install system packages +- the steps that download Go's modules +- the step that runs `script/bootstrap` to install a Python `test` phase's + packages from `Pipfile.lock` + +A stage with none of these steps declares `ARG CACHEBUST` right after `FROM`. A +step that also installs anything else comes after `ARG CACHEBUST` too. A +`script/bootstrap` that also installs node packages is one. + +- A new value reruns every `RUN` step after that line; `COPY` steps are compared + by content and need nothing. Docker keeps the layers above the line until one + of these changes: + - the base image + - their own lines + - the files they copy + +- A stage without the `ARG` line is served from cache on an unchanged tree, so a + repository re-vendors the scripts and adds the `ARG` lines in one commit. +- The scripts do not pass `--no-cache`. That flag reruns every step too, but it + also rebuilds the layers above the line and empties every cache mount. The + canonical Go `Dockerfile` below mounts Go's build cache and golangci-lint's so + that they survive between check builds. +- A bare `docker build .` is not evidence that anything ran: a sub-second build + reporting success is a cache hit, not a result. +- Never invalidate by pruning: `docker builder prune` and friends destroy a + build cache shared with every other build on the host. +- When a check is added or changed, prove it works: plant a defect it must + catch, watch the run fail on it, then revert the defect. A green run alone + shows neither that the check ran nor that it covers what it should. + +## Canonical Go `Dockerfile` + +**The gate phases are separate stages, and the build stage depends on both.** +The lint phase is based on the `golangci/golangci-lint` image (pinned by hash), +so lint failures surface in seconds rather than after a full compile. That image +has both Go and the linter, so nothing needs installing. The test phase is based +on the Debian Go image. + +```dockerfile +# In every stage ARG CACHEBUST comes right after the steps that install +# system packages and download Go's modules, or right after FROM in a +# stage with neither, and every step that lints, tests or compiles +# comes after it. script/ passes a new value on every build, which +# reruns every RUN step after it. Docker keeps the layers above it +# until the base image, their own lines or go.mod and go.sum change. +# The steps that compile mount Go's build cache, and golangci-lint run +# also mounts golangci-lint's; both survive from build to build. + +# Lint phase +# golangci/golangci-lint:v2.x.x, YYYY-MM-DD +FROM golangci/golangci-lint@sha256:... AS lint +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +ARG CACHEBUST +COPY . . +RUN --mount=type=cache,target=/root/.cache/go-build \ + --mount=type=cache,target=/root/.cache/golangci-lint \ + golangci-lint run --config .golangci.yml ./... + +# Test phase. -race needs cgo and so a C compiler, which the Debian Go +# image ships and the alpine one does not. -count=1 because the mounted +# cache also holds test results, which go test would otherwise replay. +# A first run that timed out gets no rerun, because a rerun would only +# wait for the same hang. +# golang:1.x, YYYY-MM-DD +FROM golang@sha256:... AS test +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +ARG CACHEBUST +COPY . . +RUN --mount=type=cache,target=/root/.cache/go-build \ + status=0; \ + output=$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$?; \ + printf '%s\n' "$output"; \ + test "$status" -eq 0 && exit 0; \ + case "$output" in \ + *"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \ + esac; \ + echo "--- Rerunning with -v for details ---"; \ + go test -count=1 -timeout 90s -race -v ./...; \ + exit 1 + +# Build stage. Nothing is wanted from either phase above; the copies +# are what make BuildKit build them first, so nothing after them can +# run unless lint and test passed. +# golang:1.x-alpine, YYYY-MM-DD +FROM golang@sha256:... AS builder +# git comes from the base image's own distribution package index, which +# is trusted, so it needs no version and no hash. +RUN apk add --no-cache git +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +ARG CACHEBUST +COPY --from=lint /src/go.sum /dev/null +COPY --from=test /src/go.sum /dev/null +# A tar-stream context keeps the sender's file owners, which git refuses. +RUN git config --system --add safe.directory /src +COPY . . + +# The VERSION build arg when one is given, otherwise +# `git describe --tags --always` on the .git in the build context. With +# .git present, a version that is still empty, dev or unknown fails the +# build: git is missing or could not read the checkout. +ARG VERSION +RUN --mount=type=cache,target=/root/.cache/go-build \ + VERSION="${VERSION:-$(git describe --tags --always)}"; \ + if [ -e .git ]; then \ + case "$VERSION" in ""|dev|unknown) \ + echo "version is '$VERSION' although .git is present" >&2; \ + exit 1 ;; \ + esac; \ + fi; \ + CGO_ENABLED=0 go build -trimpath \ + -ldflags="-s -w -X main.Version=${VERSION}" \ + -o /app ./cmd/app/ + +# Runtime stage, and the last one +FROM alpine@sha256:... +ARG CACHEBUST +COPY --from=builder /app /usr/local/bin/app +ENTRYPOINT ["app"] +``` + +Key points: + +- Check builds keep these, and none of them can hand back a stale result: + - The layers above `ARG CACHEBUST`: the system packages and Go's modules. Go + checks every module against `go.sum` as it downloads it, and no later + build can change a kept layer. + - Go's build cache (`/root/.cache/go-build`, where Go keeps it for root), + mounted on the steps that compile: `golangci-lint run`, both `go test` + runs and `go build`. Go reuses a compiled package only when all its inputs + are unchanged. + - golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned + image keeps it), mounted on `golangci-lint run`. golangci-lint reuses a + package's findings only when all of these are unchanged: + - the package's files and those of every package it imports + - `go.mod` + - the linters and their settings in `.golangci.yml` + - the golangci-lint version +- The modules are kept in a layer rather than a cache mount. A cache mount is + shared by every build on the host, and Go builds from whatever files the mount + holds without checking them again (only `go mod verify` does). A file changed + there would go unnoticed. +- The test command keeps the first run's output, to look in it for what Go + prints for a hang. `panic: test timed out` means the test's own timer fired. + `*** Test killed` means `go test` stopped a test that ignored that timer, + about a minute later. Go prints the goroutine stacks with either line, as long + as the test can still respond. On either line the command exits 1 without a + rerun. A failure with neither line reruns with `-v`. The kept output prints + when the first run ends, not while it runs. +- `-count=1` is required on both `go test` invocations. Go stores a passing + result in its cache directory (`GOCACHE`). When the same tests run again on + unchanged code, it prints that result, marked `(cached)`, without running + them. That directory lasts between runs on a developer's machine, and in the + `test` phase, which mounts it to keep compiled packages. +- `-count=1` defeats only that test _result_ cache. It leaves the build cache + alone, so it costs the runtime of the suite and no recompilation. The cache is + Go's own, separate from Docker's layer cache, which `CACHEBUST` handles. +- If the project uses `//go:embed` directives that reference build artifacts, + the lint phase must create placeholder files so the embed directives resolve. + A web frontend compiled in a separate stage is such an artifact. Example: + `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. +- If the project requires CGO or system libraries for linting, install them in + the lint phase. The `golangci/golangci-lint` image is Debian-based and has no + `apk`. So install with `apt-get` under the Debian package name (`libvips-dev`, + where alpine says `vips-dev`), and delete the package lists in the same `RUN`, + so the layer does not keep them. The test phase, on the Debian Go image, + installs the libraries its tests need the same way. Either install goes right + after `FROM`, above `ARG CACHEBUST`: + + ```dockerfile + FROM golangci/golangci-lint@sha256:... AS lint + RUN apt-get update \ + && apt-get install -y --no-install-recommends libvips-dev \ + && rm -rf /var/lib/apt/lists/* + WORKDIR /src + COPY go.mod go.sum ./ + RUN go mod download + ARG CACHEBUST + ``` + +## Python `test` phase + +**A Python `test` phase that installs its packages from `Pipfile.lock` does it +above `ARG CACHEBUST`, so check builds keep them.** The steps are +`COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and +`RUN script/bootstrap`, with `ARG CACHEBUST` right after them. Docker keeps what +that `RUN` installed until one of these changes: + +- the base image +- `script/` +- `Pipfile` +- `Pipfile.lock` +- a line up to and including the `RUN` + +The tests run after `ARG CACHEBUST`, on every check build. In every other stage, +a step that installs Python packages comes after `ARG CACHEBUST`. + +In that step, `script/bootstrap`: + +- installs nothing beyond these: + - system packages + - pipenv and its dependencies + - the packages from `Pipfile.lock` + + If it installs anything else there, the step goes after `ARG CACHEBUST`. + +- installs no Python: the tests use the distribution's Python, which a step + above it installs. + +A Python repository's `script/bootstrap`: + +- puts pipenv in a virtual environment at `$HOME/.pipenv`, because Debian's + Python refuses a system-wide `pip install`. It creates the environment with + `python3 -m venv` when it is missing, then uses that environment's `pip` and + `pipenv`. On a host, it first installs the distribution's python and venv + packages when they are missing. One script then works on a host and in a + `Dockerfile`. +- installs pipenv and its dependencies by hash, because PyPI is not trusted. It + runs `pip install --require-hashes -r` on a requirements file in `script/` + that lists each of them with `==` and its `--hash`. pip refuses to install a + dependency the file does not list. +- then runs `pipenv install --deploy`, with `--dev` when the tests need the + development packages. `--deploy` does all of these: + - installs only what `Pipfile.lock` names + - checks every package against the lockfile's hashes + - fails when the lockfile is out of date with `Pipfile` + +- sets `PIP_ONLY_BINARY=:all:` for both installs, and pipenv passes it on to + pip. A package with no ready-built wheel for the image then stops the install. + Without the setting, pip builds that package from source and downloads the + build tools without a hash check. + +`Pipfile` names no `python_version`, because `pipenv install --deploy` stops +when the distribution's Python differs from it. + +A kept layer holds nothing a new install would refuse. pip checked the hashes of +pipenv and its dependencies, and pipenv checked the hashes of the lockfile's +packages. pipenv creates its virtual environment with the `virtualenv` package. +The packages that `virtualenv` adds come from wheels inside it, so the hash pip +checked for it covers them. System packages, Python among them, are not fixed by +a hash. They keep the versions the package manager gave when the layer was +built; a new install may get newer ones. + +As with Go's modules, the packages are kept in a layer, not a cache mount, which +every build on the host shares and nothing checks again. + +A Python repository's `script/bootstrap` adds these lines in `main`, above the +last `echo`. The `ensurepip` test fails when python is missing, and also when +Debian's venv package is missing: + +```sh +if [ ! -x "$HOME/.pipenv/bin/pip" ]; then + if ! python3 -m ensurepip --version >/dev/null 2>&1; then + pkg_install python3 python3-venv python3 python3 + fi + python3 -m venv "$HOME/.pipenv" +fi +export PIP_ONLY_BINARY=:all: +"$HOME/.pipenv/bin/pip" install --require-hashes -r script/requirements.txt +"$HOME/.pipenv/bin/pipenv" install --dev --deploy +``` + +Make `script/requirements.txt` once, with +`echo pipenv | pip-compile --generate-hashes --allow-unsafe -o script/requirements.txt -`. +`pip-compile` comes from Alpine's `py3-pip-tools` package; Debian and Ubuntu do +not package `pip-tools`. Make the first `Pipfile.lock` with +`$HOME/.pipenv/bin/pipenv lock`, after the `pip install` line has installed +pipenv. + +In the example below, `script/bootstrap` finds `make`, `git`, python and venv +already installed: + +```dockerfile +# Test phase. Both pytest runs share one 90-second timeout, so a hang +# uses it once and gets no rerun. +# debian:trixie-slim, YYYY-MM-DD +FROM debian@sha256:... AS test +# Python, pip and venv come from the distribution, which is trusted, so +# they need no version and no hash. make and git are base tools that +# script/bootstrap installs when they are missing. Installed here, they +# stay in their own layer when script/ changes. +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates make git \ + python3 python3-venv \ + && rm -rf /var/lib/apt/lists/* +# script/bootstrap installs pipenv in a virtual environment at +# $HOME/.pipenv, and the tests below run that pipenv. +ENV PATH=/root/.pipenv/bin:$PATH +WORKDIR /app +COPY script/ script/ +COPY Pipfile Pipfile.lock ./ +RUN script/bootstrap +ARG CACHEBUST +COPY . . +RUN timeout 90 sh -c 'pipenv run pytest || { \ + echo "--- Rerunning with -v for details ---"; \ + pipenv run pytest -v; exit 1; }' +``` + +Run from a `make test` target on a host, that test command: + +- does not stop on Ctrl-C. It goes on until the tests end or the 90 seconds run + out. +- fails on a Mac without `timeout`, until it is installed. +- needs `$HOME/.pipenv/bin` on `PATH`. + +## Version + +**A plain `docker build .` of a clone stamps the version that +`git describe --tags --always` gives.** That gives: + +- the tag, on a tagged commit +- the tag, the number of commits since it and the short commit + (`v1.2.3-4-gabc1234`), on a later commit +- the short commit, when no tag is reachable + +The stage that compiles derives it from the `.git` in the build context, as the +canonical Go `Dockerfile` above shows: + +- It takes the version from the `VERSION` build argument when one is given, + otherwise from `git describe`. `ARG VERSION` is declared in that stage, + because `ARG` is stage-scoped. It has no default. +- It has `git`: the Debian Go image has it, and an alpine one needs + `apk add --no-cache git`. +- It marks its working directory safe for git + (`git config --system --add safe.directory /src`). A context sent as a tar + stream keeps the sender's file owners, and git refuses a checkout owned by + another user, so the version would come out empty. +- The build fails if the context carries `.git` and the version still comes out + empty, `dev` or `unknown`. Without that check, a missing `git` or an + unreadable checkout would leave `-X main.Version=` empty, and the build would + still exit 0. +- A plain `docker build .` with no build arguments must succeed; a Dockerfile + that refuses an empty build argument drops that refusal and keeps the + argument. +- A checkout whose `.git` is a file is the exception: a linked worktree, or a + repository checked out as a submodule. That file points to a git directory + outside the build context, so the build cannot read the version and a plain + `docker build .` fails. Pass the version with `--build-arg VERSION=...`, as + `script/docker` and `script/cibuild` already do. + +`.dockerignore` lets `.git` into the build context. It keeps out every git +`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): + +- the repository's own +- each submodule's under `.git/modules/` +- that of a submodule keeping its own `.git` directory + +`git describe` does not need them, and each can hold a credential: a password in +a remote URL, or the token the CI checkout step stores there. Some submodule +names have a `config` segment, as in `config`, `deploy/config` or `config/lib`. +Such a submodule loses its whole git directory to `**/.git/modules/**/config`, +and Go's version stamping then fails the build. Give it a name without that +segment (`git submodule add --name`). + +`script/docker` and `script/cibuild` pass the version they compute on the host; +it takes precedence. They do this byte-identically across repos: + +```sh +# The version and the tag each get their own line: a failing command +# substitution inside an argument does not trip `set -e`, so the inline +# form degrades to an empty constant. +version="$(git describe --tags --always --dirty 2>/dev/null || true)" +[ -n "$version" ] || version="unknown" +tag="$(script/projectname)" +docker build --build-arg CACHEBUST="$(date +%s)-$$" \ + --build-arg VERSION="$version" \ + -t "$tag" . +``` + +- `--always` makes an untagged repo yield an abbreviated commit hash rather than + failing. +- The `[ -n "$version" ]` line is the single place the fallback is applied. It + is a live check that fires on a build from an export with no `.git`, and on a + repository with no commits yet. Do not fold it into the substitution as + `|| echo unknown`, which makes the guard unreachable. +- Passing `VERSION` to a repo whose Dockerfile declares no such `ARG` is ignored + and costs nothing, which is why the scripts stay byte-identical. + +# CI + +Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that runs +`script/cibuild` on push. The workflow's only other step checks out the repo. +The workflow sets these: + +- `persist-credentials: false` on the checkout step. `script/cibuild` needs no + token, and without that setting the checkout leaves the job's token in + `.git/config` for every later step. +- `fetch-depth: 0` on the checkout step, which fetches the tags `git describe` + needs. The default is a shallow clone with no tags, and a tagged repository's + CI build would then stamp a bare short commit id. +- a `concurrency` block that groups runs by workflow and branch + (`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`. + A new push then cancels the older run on the same branch, queued or running, + and no other: runs for replaced commits do not hold up the shared runner. +- `timeout-minutes: 20` on the `check` job, so a hung build frees the shared + runner after 20 minutes. That allows for the bootstrap and three Docker + builds, each held to the 5-minute limit. + +`script/cibuild` changes to the repo root, runs `script/bootstrap`, runs +`script/check`, and builds the image with the version: + +- **`script/cibuild` runs `script/bootstrap` first**, because the workflow + checks out the repo and runs nothing else, while `script/fmt-check` runs the + formatter on the host. Without it, a pristine checkout dies there, after the + containerised gates have passed. +- Its three Docker builds are the test phase, the lint phase, then the image. So + a successful run means every check passed, which a bare `docker build .` does + not show. +- The image build gets a `CACHEBUST` value of its own and so runs the gate + phases a second time. That price is worth paying: the image that ships is + built from a run of its own gates rather than from a cache entry. + +A separate workflow limited to `main` by a `branches` list under `on: push` +cannot be checked by review. To try a change to it, add the feature branch to +that list and push, then remove the branch from the list again before merging. +Keep any job in it that publishes behind `if: github.ref_name == 'main'`, so the +run from the feature branch publishes nothing. + +# `.gitignore` and `.dockerignore` + +`.gitignore` should be comprehensive from the start: + +- OS files (`.DS_Store`) +- editor files (`.swp`, `*~`) +- in-repo agent scratch directories (`.claude/`) +- `node_modules/` +- `.yarn/*`, with `!.yarn/releases` so that git keeps yarn's release file +- the repo's own build outputs + +Fetch the standard `.gitignore` from +`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up a +new repo. A repo's `.gitignore` is the standard file followed by the repo's own +entries, such as its binaries. A re-vendor replaces the standard part and keeps +those entries. + +**`.dockerignore` does not use `.gitignore` semantics, and copying patterns +across unmodified leaves secrets in the build context.** The standard +`.gitignore` patterns are written to `.gitignore`'s own semantics, in which an +unanchored pattern already matches at every depth. They must not be transplanted +into a `.dockerignore` unmodified. Docker matches with `moby/patternmatcher`: +`filepath.Match` semantics plus a `**` extension. So `*` does not cross `/`, and +a pattern without a leading `**/` is anchored at the build-context root. + +- A `.dockerignore` listing `.env`, `*.pem` and `*.key` therefore excludes only + the copies at the repository root. `config/.env` and `certs/server.key` still + reach the context and can land in an image layer. Such a `.dockerignore` is + more dangerous than a short file with no secret patterns at all, because it + reads as solved and stops anyone looking. +- Give every depth-independent pattern the `**/` prefix, and leave only + genuinely root-anchored entries unprefixed. Those are `.claude`, `.yarn/*`, + `!.yarn/releases`, and the repo's own host-built binary. The binary is written + `/myapp` and never `**/myapp`, which would also match `cmd/myapp/` and delete + the package directory from the context. +- Matching is case-sensitive, and an all-caps copy of each pattern still misses + `Server.Key`. So secret names use character ranges: `**/*.[kK][eE][yY]`, + `**/*.[pP][eE][mM]`, and likewise for `.envrc` and the extensionless SSH keys. +- Where such a pattern also catches something the build needs, re-include it + with a negation (`!docs/example.env`). Deleting the pattern reopens the + exposure for every other file it covers. +- Fetch the standard `.dockerignore` from + `https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend + it with the repo's own artifacts. + +**In-repo agent scratch belongs in both files, written to each file's own +semantics.** `.claude/` holds one worktree per in-flight agent, an entire +additional checkout of the repo. Under `COPY . .` the build context therefore +inflates by a multiple of the repo, and another session's unreviewed work can be +copied into an image layer. + +- In `.gitignore` the entry is `.claude/`, unanchored. +- In `.dockerignore` it is `.claude`, anchored and with **no** `**/` prefix, + because the prefixed form would also delete any nested directory of that name + from the build. +- Anchoring carries a known gap, which the canonical `.dockerignore` states in + its own comment. The directory is created in the agent's working directory. A + repo running agents in subdirectories therefore still ships + `services/api/.claude/` and must add its own anchored entry there. + +**Verify `.dockerignore` by enumerating the image, not by reading the +patterns.** Plant files at the root _and_ at least two directories deep. Build a +probe image that does `COPY . .`, and list what actually landed +(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context` +size is not a substitute: a nested secret is a few bytes, and BuildKit transfers +only the delta from the previous build. + +# `.golangci.yml` + +`.golangci.yml` is standardized. The vendored copy in a consuming repo must +_NEVER_ be modified by an agent. Fetch it from +`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it +byte-identical, so that no repo can quietly loosen its own linting. + +- Linter configuration changes are made to the canonical copy in the `prompts` + repo and reach consuming repos by re-vendoring. An agent may open a PR against + canonical, which only the user merges. +- One list is exempt from byte-identity, because it cannot be written once for + every repo: the `deny` list of the `test-support` depguard rule. There a repo + names its own test-support packages by full import path. A repo adds entries + there and changes nothing else, and a re-vendor carries its entries forward. +- The canonical golangci-lint version is v2.15.0 (released 2026-10-16). The + version is pinned as the digest of the lint phase's base image, `golangci/golangci-lint@sha256:fbfcfd12a6f8d7f1b97b3362ca80fc2a244d46863148c73cb88e7793af96c481`, - which reports `2.15.0 built with go1.27.0 from 25c5369a`. A module's `go` - directive must not name a newer Go minor version than the one golangci-lint - was built with, or golangci-lint refuses to lint it. This release lints - `go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo - installs golangci-lint on the host. A repo sets the lint phase digest to the - one named here and re-vendors `.golangci.yml` in the same commit, whichever of - the two prompted the change. The canonical copy can name linters that an older - golangci-lint rejects. A newer golangci-lint can add linters that - `default: all` switches on until the canonical copy disables them. + which reports `2.15.0 built with go1.27.0 from 25c5369a`. That digest is the + only pin, since no repo installs golangci-lint on the host. +- A module's `go` directive must not name a newer Go minor version than the one + golangci-lint was built with, or golangci-lint refuses to lint it. This + release lints `go 1.27.1` but not `go 1.28`. +- A repo sets the lint phase digest to the one named here and re-vendors + `.golangci.yml` in the same commit, whichever of the two prompted the change. + The canonical copy can name linters that an older golangci-lint rejects. A + newer golangci-lint can add linters that `default: all` switches on until the + canonical copy disables them. -- **`script/bootstrap` installs a pinned tool by comparing versions, never by - testing presence.** An `if ! command -v ; then install; fi` guard tests - `PATH` only. On an already-provisioned machine the pin is then inert, and a - version bump is a silent no-op. The Dockerfile, installing into a clean image, - gets the pinned version, so a local `make check` and `make docker` can - disagree about what the tool even is. The canonical form: - - compares the installed version against the pin over the **whole** version - token. A parser that stops at the first `-` reports `2.12.2` for a host - running `2.12.2-rc1` and skips the install. - - treats `--version` output as a mismatch when it is any of these: - - absent - - non-zero - - empty - - unrecognised +# Formatting - So a failure causes a redundant install, never a skipped one. +Use platform-standard formatters: `black` for Python, `prettier` for +JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration, with +two exceptions: - - after installing, re-resolves the binary the way callers do, with - `hash -r` and then through `PATH`, not through the directory the installer - wrote to. A shadowing binary can hide an install, which then succeeds - while no caller sees any change. In that case it fails and names the - resolved path. - - is actually called, and prints the version on both success paths. A - function defined and never invoked has the same exit status and the same - empty output as one that worked. +- four-space indents (except Go) +- `proseWrap: always` for Markdown (hard-wrap at 80 columns) - A tool from the distribution has no pin, so `script/bootstrap` installs it - when it is missing. make, git, node and yarn are such tools. +Documentation and writing repos (Markdown, HTML, CSS) should also have +`.prettierrc` and `.prettierignore`. - Keep it POSIX sh: no arrays, no `[[`, no `grep -P`. +All repos should have an `.editorconfig` enforcing the project's indentation +settings. It is the standard file from +`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, followed by +the repo's own sections, such as one for another language it uses. The standard +file sets tabs for `Makefile` and Go files. A re-vendor replaces the standard +part and keeps those sections. - A Go tool a repo needs on the host is installed with `go install` pinned to - a commit hash (`go install @`). It is never tracked as - a `go.mod` tool dependency or through a `tools.go` file, either of which - pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`. +# Web services -- When pinning images or packages by hash, add a comment above the reference - with the version and date (YYYY-MM-DD). +Dockerized web services listen on port 8080 by default, overridable with `PORT`. -- Use `yarn`, not `npm`. +**HTTP/web services must be hardened for production internet exposure before +tagging 1.0.** This means full compliance with security best practices +including, without limitation, all of the following: + +- **Security headers** on every response: + - `Strict-Transport-Security` (HSTS) with `max-age` of at least one year and + `includeSubDomains`. + - `Content-Security-Policy` (CSP) with a restrictive default policy + (`default-src 'self'` as a baseline, tightened per-resource as needed). + Never use `unsafe-inline` or `unsafe-eval` unless unavoidable, and + document the reason. + - `X-Frame-Options: DENY` (or `SAMEORIGIN` if framing is required). Prefer + the `frame-ancestors` CSP directive as the primary control. + - `X-Content-Type-Options: nosniff`. + - `Referrer-Policy: strict-origin-when-cross-origin` (or stricter). + - `Permissions-Policy` restricting access to browser features the + application does not use (camera, microphone, geolocation, etc.). +- **Request and response limits:** + - Maximum request body size enforced on all endpoints (e.g. Go + `http.MaxBytesReader`). Choose a sane default per-route; never accept + unbounded input. + - Maximum response body size where applicable (e.g. paginated APIs). + - `ReadTimeout` and `ReadHeaderTimeout` on the `http.Server` to defend + against slowloris attacks. + - `WriteTimeout` and `IdleTimeout` on the `http.Server`. + - Per-handler execution time limits via `context.WithTimeout` or chi/stdlib + `middleware.Timeout`. +- **Authentication and session security:** + - Rate limiting on password-based authentication endpoints. API keys are + high-entropy and not susceptible to brute force, so they are exempt. + - CSRF tokens on all state-mutating HTML forms. API endpoints authenticated + via `Authorization` header (Bearer token, API key) are exempt because the + browser does not attach these automatically. + - Passwords stored using bcrypt, scrypt, or argon2 — never plain-text, MD5, + or SHA. + - Session cookies set with `HttpOnly`, `Secure`, and `SameSite=Lax` (or + `Strict`) attributes. +- **Reverse proxy awareness:** True client IP detection when behind a reverse + proxy (`X-Forwarded-For`, `X-Real-IP`). The application must accept forwarded + headers only from a configured set of trusted proxy addresses, never + unconditionally. +- **CORS:** Authenticated endpoints must restrict `Access-Control-Allow-Origin` + to an explicit allowlist of known origins. Wildcard (`*`) is acceptable only + for public, unauthenticated read-only APIs. +- **Error handling:** Internal errors must never leak implementation details to + the client. Those details include: + - stack traces + - SQL queries + - file paths + + Return generic error messages in production; detailed errors only when + `DEBUG` is enabled. + +- **TLS:** Services never terminate TLS directly. They are always deployed + behind a TLS-terminating reverse proxy. The service itself listens on plain + HTTP. However, HSTS headers and `Secure` cookie flags must still be set by the + application so that the browser enforces HTTPS end-to-end. + +Apply defense-in-depth: if a standard security hardening measure exists for HTTP +services and is not listed here, it is still expected. When in doubt, harden. + +# Database migrations + +Database migrations live in `internal/db/migrations/` and must be embedded in +the binary. + +- `000_migration.sql` — contains ONLY the creation of the migrations tracking + table itself. Nothing else. +- `001_schema.sql` — the full application schema. +- **Pre-1.0.0:** never add additional migration files (002, 003, etc.). There is + no installed base to migrate. Edit `001_schema.sql` directly. +- **Post-1.0.0:** add new numbered migration files for each schema change. Never + edit existing migrations after release. + +# README + +`README.md` is the primary documentation. Required sections: + +- **Description**: First line must include: + - the project name + - purpose + - category (web server, SPA, CLI tool, etc.) + - license + - author + + Example: "µPaaS is an MIT-licensed Go web application by @sneak that + receives git-frontend webhooks and deploys applications via Docker in + realtime." + +- **Getting Started**: Copy-pasteable install/usage code block. +- **Entrypoints**: Opens by stating that the repo adheres to the + [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) + standard (with that link), then documents each provided `script/` entrypoint + and its purpose. +- **Rationale**: Why does this exist? +- **Design**: How is the program structured? +- **TODO**: Update meticulously, even between commits. When planning, put the + todo list in the README so a new agent can pick up where the last one left + off. +- **License**: MIT, GPL, or WTFPL. Ask the user for new projects. Include a + `LICENSE` file in the repo root. +- **Author**: [@sneak](https://sneak.berlin). + +# Repository layout + +Avoid putting files in the repo root unless necessary. Root should contain only +project-level config files: + +- `README.md` +- `AGENTS.md` +- `Makefile` +- `Dockerfile` +- `LICENSE` +- `.gitignore` +- `.editorconfig` +- `REPO_POLICIES.md` +- language-specific config + +Everything else goes in a subdirectory. Canonical subdirectory names: + +- `bin/` — executable scripts and tools +- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose + body is a single call into `internal/` or `pkg/`, no project logic in `cmd/` +- `configs/` — configuration templates and examples +- `deploy/` — deployment manifests (k8s, compose, terraform) +- `docs/` — documentation and markdown (README.md stays in root) +- `internal/` — Go internal packages +- `internal/db/migrations/` — database migrations +- `pkg/` — Go library packages +- `share/` — systemd units, data files +- `static/` — static assets (images, fonts, etc.) +- `web/` — web frontend source + +Guidance for coding agents lives in one `AGENTS.md` at the repository root. It +is never committed under a file or directory named after one agent tool, such as +`CLAUDE.md` or `.claude/`, and never split into separate memory files. + +New repos must contain at minimum: + +- `README.md`, `.git`, `.gitignore`, `.editorconfig` +- `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo) +- `Makefile` +- `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`, `lint`, + `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, + `install-precommit`) +- `Dockerfile`, `.dockerignore` +- `.gitea/workflows/check.yml` +- Go: `go.mod`, `go.sum`, `.golangci.yml` +- JS: `package.json`, `yarn.lock`, `.yarnrc.yml`, `.yarn/releases/`, + `.gitattributes`, `.prettierrc`, `.prettierignore` +- Python: `pyproject.toml` + +When setting up a new repo, files from the `prompts` repo may be used as +templates. Fetch them from +`https://git.eeqj.de/sneak/prompts/raw/branch/main/`. First commit of a +new repo should contain only `README.md`. + +# Other conventions - Write all dates as YYYY-MM-DD (ISO 8601). - -- Simple projects should be configured with environment variables. - -- Dockerized web services listen on port 8080 by default, overridable with - `PORT`. - -- **HTTP/web services must be hardened for production internet exposure before - tagging 1.0.** This means full compliance with security best practices - including, without limitation, all of the following: - - **Security headers** on every response: - - `Strict-Transport-Security` (HSTS) with `max-age` of at least one year - and `includeSubDomains`. - - `Content-Security-Policy` (CSP) with a restrictive default policy - (`default-src 'self'` as a baseline, tightened per-resource as - needed). Never use `unsafe-inline` or `unsafe-eval` unless - unavoidable, and document the reason. - - `X-Frame-Options: DENY` (or `SAMEORIGIN` if framing is required). - Prefer the `frame-ancestors` CSP directive as the primary control. - - `X-Content-Type-Options: nosniff`. - - `Referrer-Policy: strict-origin-when-cross-origin` (or stricter). - - `Permissions-Policy` restricting access to browser features the - application does not use (camera, microphone, geolocation, etc.). - - **Request and response limits:** - - Maximum request body size enforced on all endpoints (e.g. Go - `http.MaxBytesReader`). Choose a sane default per-route; never accept - unbounded input. - - Maximum response body size where applicable (e.g. paginated APIs). - - `ReadTimeout` and `ReadHeaderTimeout` on the `http.Server` to defend - against slowloris attacks. - - `WriteTimeout` on the `http.Server`. - - `IdleTimeout` on the `http.Server`. - - Per-handler execution time limits via `context.WithTimeout` or - chi/stdlib `middleware.Timeout`. - - **Authentication and session security:** - - Rate limiting on password-based authentication endpoints. API keys are - high-entropy and not susceptible to brute force, so they are exempt. - - CSRF tokens on all state-mutating HTML forms. API endpoints - authenticated via `Authorization` header (Bearer token, API key) are - exempt because the browser does not attach these automatically. - - Passwords stored using bcrypt, scrypt, or argon2 — never plain-text, - MD5, or SHA. - - Session cookies set with `HttpOnly`, `Secure`, and `SameSite=Lax` (or - `Strict`) attributes. - - **Reverse proxy awareness:** - - True client IP detection when behind a reverse proxy - (`X-Forwarded-For`, `X-Real-IP`). The application must accept - forwarded headers only from a configured set of trusted proxy - addresses. Never trust `X-Forwarded-For` unconditionally. - - **CORS:** - - Authenticated endpoints must restrict `Access-Control-Allow-Origin` to - an explicit allowlist of known origins. Wildcard (`*`) is acceptable - only for public, unauthenticated read-only APIs. - - **Error handling:** - - Internal errors must never leak implementation details to the client. - Those details include: - - stack traces - - SQL queries - - file paths - - Return generic error messages in production; detailed errors only - when `DEBUG` is enabled. - - - **TLS:** - - Services never terminate TLS directly. They are always deployed behind - a TLS-terminating reverse proxy. The service itself listens on plain - HTTP. However, HSTS headers and `Secure` cookie flags must still be - set by the application so that the browser enforces HTTPS end-to-end. - - This list is non-exhaustive. Apply defense-in-depth: if a standard security - hardening measure exists for HTTP services and is not listed here, it is - still expected. When in doubt, harden. - -- `README.md` is the primary documentation. Required sections: - - **Description**: First line must include: - - the project name - - purpose - - category (web server, SPA, CLI tool, etc.) - - license - - author - - Example: "µPaaS is an MIT-licensed Go web application by @sneak that - receives git-frontend webhooks and deploys applications via Docker in - realtime." - - - **Getting Started**: Copy-pasteable install/usage code block. - - **Entrypoints**: Opens by stating that the repo adheres to the - [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) - standard (with that link), then documents each provided `script/` - entrypoint and its purpose. - - **Rationale**: Why does this exist? - - **Design**: How is the program structured? - - **TODO**: Update meticulously, even between commits. When planning, put - the todo list in the README so a new agent can pick up where the last one - left off. - - **License**: MIT, GPL, or WTFPL. Ask the user for new projects. Include a - `LICENSE` file in the repo root and a License section in the README. - - **Author**: [@sneak](https://sneak.berlin). - -- First commit of a new repo should contain only `README.md`. - +- Cross-project documentation (such as this file) must include + `last_modified: YYYY-MM-DD` in the YAML front matter so it can be kept in sync + with the authoritative source as policies evolve. +- Use SemVer. - Go module root: `sneak.berlin/go/`. Always run `go mod tidy` before committing. - -- Use SemVer. - -- Database migrations live in `internal/db/migrations/` and must be embedded in - the binary. - - `000_migration.sql` — contains ONLY the creation of the migrations - tracking table itself. Nothing else. - - `001_schema.sql` — the full application schema. - - **Pre-1.0.0:** never add additional migration files (002, 003, etc.). - There is no installed base to migrate. Edit `001_schema.sql` directly. - - **Post-1.0.0:** add new numbered migration files for each schema change. - Never edit existing migrations after release. - -- All repos should have an `.editorconfig` enforcing the project's indentation - settings. It is the standard file from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, followed by - the repo's own sections, such as one for another language it uses. The - standard file sets tabs for `Makefile` and Go files. A re-vendor replaces the - standard part and keeps those sections. - -- Avoid putting files in the repo root unless necessary. Root should contain - only project-level config files: - - `README.md` - - `AGENTS.md` - - `Makefile` - - `Dockerfile` - - `LICENSE` - - `.gitignore` - - `.editorconfig` - - `REPO_POLICIES.md` - - language-specific config - - Everything else goes in a subdirectory. Canonical subdirectory names: - - `bin/` — executable scripts and tools - - `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose - body is a single call into `internal/` or `pkg/`, no project logic in - `cmd/` - - `configs/` — configuration templates and examples - - `deploy/` — deployment manifests (k8s, compose, terraform) - - `docs/` — documentation and markdown (README.md stays in root) - - `internal/` — Go internal packages - - `internal/db/migrations/` — database migrations - - `pkg/` — Go library packages - - `share/` — systemd units, data files - - `static/` — static assets (images, fonts, etc.) - - `web/` — web frontend source - -- When setting up a new repo, files from the `prompts` repo may be used as - templates. Fetch them from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/`. - -- New repos must contain at minimum: - - `README.md`, `.git`, `.gitignore`, `.editorconfig` - - `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo) - - `Makefile` - - `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`, - `lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, - `install-precommit`) - - `Dockerfile`, `.dockerignore` - - `.gitea/workflows/check.yml` - - Go: `go.mod`, `go.sum`, `.golangci.yml` - - JS: `package.json`, `yarn.lock`, `.yarnrc.yml`, `.yarn/releases/`, - `.gitattributes`, `.prettierrc`, `.prettierignore` - - Python: `pyproject.toml` - -- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It - is never committed under a file or directory named after one agent tool, such - as `CLAUDE.md` or `.claude/`, and never split into separate memory files. +- Simple projects should be configured with environment variables.