From 6dcacf673f3e17fe70835d9dfaa3e4b047a424dc Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Fri, 6 Nov 2026 00:57:30 +0000 Subject: [PATCH 1/3] Compact REPO_POLICIES.md without losing rules (closes #158) The rules now sit under headings, the pinning rule first. Each rule that was stated in several places is stated once. From 1,145 lines and 8,427 words to 1,012 lines and 7,709 words. Code blocks are unchanged, except that the Makefile copies of the Go and Python test commands are gone: the Dockerfile examples carry the same commands. Both checklists name the `.golangci.yml` section. Model: fable-5-1 --- TODO.md | 7 + prompts/EXISTING_REPO_CHECKLIST.md | 2 +- prompts/NEW_REPO_CHECKLIST.md | 2 +- prompts/REPO_POLICIES.md | 2067 +++++++++++++--------------- 4 files changed, 976 insertions(+), 1102 deletions(-) diff --git a/TODO.md b/TODO.md index 83c8480..acb2991 100644 --- a/TODO.md +++ b/TODO.md @@ -21,6 +21,13 @@ fmt-check, and commit. # Completed Steps +- 2026-11-06: Compacted `prompts/REPO_POLICIES.md` from 1,145 lines and 8,427 + words to 1,012 lines and 7,709 words + (https://git.eeqj.de/sneak/prompts/issues/158). Every rule keeps its meaning. + The rules now sit under headings, the pinning rule first. Each rule that was + stated in several places is stated once. The Makefile copies of the Go and + Python test commands are gone, because the `Dockerfile` examples carry the + same commands. Both checklists name the `.golangci.yml` section. - 2026-11-06: The pinning rule in `prompts/REPO_POLICIES.md` now opens with its scope: what we download and install ourselves in scripts and Dockerfiles (https://git.eeqj.de/sneak/prompts/issues/154). Distribution packages are diff --git a/prompts/EXISTING_REPO_CHECKLIST.md b/prompts/EXISTING_REPO_CHECKLIST.md index 8fe3f42..dfd0d03 100644 --- a/prompts/EXISTING_REPO_CHECKLIST.md +++ b/prompts/EXISTING_REPO_CHECKLIST.md @@ -163,7 +163,7 @@ with your task. - [ ] Go: `go.mod`, `go.sum`, `.golangci.yml` (fetch from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and, in the same commit, set the lint phase digest to the one named in the - `.golangci.yml` paragraph of `REPO_POLICIES.md`) + `.golangci.yml` section of `REPO_POLICIES.md`) - [ ] JS: `package.json`, `yarn.lock`, `.yarnrc.yml`, `.yarn/releases/`, `.gitattributes`, `.prettierrc`, `.prettierignore` (fetch the last two from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.prettierrc` diff --git a/prompts/NEW_REPO_CHECKLIST.md b/prompts/NEW_REPO_CHECKLIST.md index 0380f38..544eeb3 100644 --- a/prompts/NEW_REPO_CHECKLIST.md +++ b/prompts/NEW_REPO_CHECKLIST.md @@ -178,7 +178,7 @@ Template files can be fetched from: - [ ] Go: `go mod init sneak.berlin/go/`, `.golangci.yml` (fetch from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and, in the same commit, set the lint phase digest to the one named in the - `.golangci.yml` paragraph of `REPO_POLICIES.md`) + `.golangci.yml` section of `REPO_POLICIES.md`) - [ ] JS, or any repository that runs prettier through yarn: make the first `yarn.lock` with the steps below. `script/bootstrap` installs no yarn until `yarn.lock` exists, so run them with any distribution's node and diff --git a/prompts/REPO_POLICIES.md b/prompts/REPO_POLICIES.md index e44995d..f2f5ad7 100644 --- a/prompts/REPO_POLICIES.md +++ b/prompts/REPO_POLICIES.md @@ -14,1132 +14,999 @@ style conventions are in separate documents: - [Go HTTP Server Conventions](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/GO_HTTP_SERVER_CONVENTIONS.md) - [API Policies](https://git.eeqj.de/sneak/prompts/raw/branch/main/prompts/API_POLICIES.md) ---- - -- Cross-project documentation (such as this file) must include - `last_modified: YYYY-MM-DD` in the YAML front matter so it can be kept in sync - with the authoritative source as policies evolve. - -- **External references must be pinned by cryptographic hash.** This is the - single most important rule in this document. It covers what we download and - install ourselves in scripts and Dockerfiles: - - Docker base images - - Go modules - - Python packages from PyPI - - npm packages - - GitHub Actions - - release archives fetched with `curl` or `wget` - - Packages from the distribution's own package manager are outside this rule. - They need no version and no hash, because the distribution is trusted. In a - `Dockerfile`, it is the pinned base image's distribution. On a host, it is - the host's. The host's package manager stands for its distribution, and that - includes nix and brew. Updating the package index and upgrading installed - packages are fine too. This command, for example, is allowed: - - ```sh - apt update && apt upgrade -y && apt install -y build-essential - ``` - - A package source that a `Dockerfile` adds itself is not the distribution's - own, so its packages are pinned like any other download. - - A version tag such as `@v4`, `@latest` or `:3.21` is server-mutable, and - therefore a remote code execution vulnerability. Reference an external - dependency by its content hash: - - Docker: `@sha256:...` - - Go: the module hash in `go.sum` - - Python: the package hash in `Pipfile.lock` - - npm: the `checksum` in `yarn.lock` - - GitHub Actions: `@` - - This also means never `curl | bash` to install a tool such as rustup. - Instead, download a specific release archive from GitHub, verify its hash, - and only then install. The hash is hardcoded in the Dockerfile or script. - Unverified install scripts are arbitrary remote code execution. Double-check - every external reference in every file before committing. - - These come from the distribution wherever possible: - - node, meaning the distribution's newest Node package, such as - `nodejs-current` on Alpine 3.24 - - yarn - - npm - - python - - pip and venv, the installer tools bundled with python - - If the distribution's version is too old, use a newer distribution, never a - download. That rules out all of these: - - nvm - - pyenv - - corepack - - `npm install -g yarn` - - yarn's release file is third-party code, and git pins it. `yarn set version` - writes it to `.yarn/releases/` once, given an exact version. - - These are not trusted, so everything from them stays pinned by hash: - - Docker Hub - - PyPI - - the npm registry - - `go get` - -- Every repo with software must have a root `Makefile` with these targets: - - `make bootstrap` - - `make setup` - - `make test` - - `make lint` - - `make fmt` (writes) - - `make fmt-check` (read-only) - - `make check` (runs `test`, `lint`, `fmt-check`) - - `make docker` - - `make hooks` (installs pre-commit hook) - - A model Makefile is at - `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`. - -- Repos follow the - [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) - pattern. The implementation of each Makefile target lives in an executable - script in `script/`, and the Makefile targets are thin shims that call them: - - `script/bootstrap` - - `script/setup` - - `script/test` - - `script/lint` - - `script/fmt` - - `script/fmt-check` - - `script/check` - - `script/docker` - - The scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they - run in minimal containers; alpine images, for example, have no bash. Locate - the repo root with `$(cd "$(dirname "$0")/.." && pwd -P)`, and `cd` there - before acting. - - From the standard's canonical set we use `bootstrap`, `setup`, `test`, and - `cibuild`. `setup` makes the repo ready for development after a fresh clone: - it runs `bootstrap`, then `install-precommit`, plus any repo-specific - initialization. - - `script/bootstrap` installs all dependencies idempotently and assumes - nothing is present. Base tools, node and yarn come from nix, apt, brew, or - apk, detected in that order; apt runs noninteractive. Each one is installed - only when it is missing. A tool from outside the distribution is never - installed as "latest" or "lts"; it is always an exact version. - - `script/bootstrap` installs node, yarn and the node packages only when - `yarn.lock` is in the tree. With no `yarn.lock`, it installs no node and no - yarn. It installs the node packages with `yarn install --immutable`. That - install fails when a package's `checksum` is wrong, or when `package.json` - is out of step with `yarn.lock`. - - Each repository commits the yarn that runs. `yarnPath` in `.yarnrc.yml` - points at the release file in `.yarn/releases/`. The distribution's yarn - only starts that file, and that file does the work. Alpine packages yarn 1, - and Debian 13 packages yarn 4.1. The same committed yarn does the work on - both. Corepack never runs, and no yarn is downloaded. - - `.yarnrc.yml` holds these settings: - - `yarnPath` - - `nodeLinker: node-modules` - - `enableTelemetry: false`, because yarn otherwise sends usage reports from - any host that is not CI - - By default, yarn runs no dependency's install script. A dependency that must - run one is named in `package.json`, as - `"dependenciesMeta": { "": { "built": true } }`. - - `.gitattributes` carries `.yarn/releases/** binary`, so diffs and merges - treat the release file as one opaque file. - - yarn 1 ignores `--immutable` and exits 0, so on a tree with no `.yarnrc.yml` - the install is not frozen. A repository therefore re-vendors - `script/bootstrap` and converts its `yarn.lock` in one commit. - - Debian and Ubuntu name the yarn command `yarnpkg`. `script/bootstrap`, - `script/fmt` and `script/fmt-check` each carry the same small function that - runs whichever name is present. With neither name on `PATH`, `script/fmt` - and `script/fmt-check` print one line saying to run `script/bootstrap`, and - exit 1. - - A `Dockerfile` stage that needs yarn may start from any distribution that - packages node and yarn. It copies all of these before it runs - `script/bootstrap`: - - `package.json` - - `yarn.lock` - - `.yarnrc.yml` - - `.yarn/releases/` - - With only `package.json` and `yarn.lock`, the install fails. Alpine's yarn 1 - then says to enable corepack; never do that, and copy the missing files - instead. The Python and Go stages below need no yarn and copy no - `yarn.lock`, so `script/bootstrap` installs no node and no yarn there. - - `script/cibuild` runs the CI build, and the Gitea workflow calls it. The - script changes to the repo root, runs `script/bootstrap`, runs - `script/check`, and builds the image with the version. **`script/cibuild` - runs `script/bootstrap` first**, because the workflow checks out the repo - and runs nothing else, while `script/fmt-check` runs the formatter on the - host. On a pristine checkout with nothing installed, the run dies there, - after the containerised gates have passed. - - Four further scripts are our own extensions to the standard: - - `script/check` runs `script/test`, `script/lint` and `script/fmt-check`. - - `script/precommit` is what the git pre-commit hook runs, and it calls - `script/check`. - - `script/install-precommit` installs the git pre-commit hook. The - `make hooks` target shims to it. - - `script/projectname` (literally that filename) simply outputs the - project's name. - - Scripts that need the name call `script/projectname`, so those scripts stay - byte-identical across all repos. `script/docker`, for example, assembles its - image tag from it. Repo-type-specific pre-commit extras belong in - `script/precommit`, not in the hook itself; `go mod tidy` verification in Go - repos is one. Model scripts are at - `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/`. The - README must document the provided scripts in an **Entrypoints** section (see - the README requirements below). - -- Run the repo's own entrypoints, never the underlying tools directly. The - `script/` scripts are the single source of truth for how each operation runs. - Calling them directly is fine, and so is calling the Makefile targets. The - Makefile targets must still exist, as the thin shims described above. - - An agent whose work is confined to one package may run `go test` directly on - that package's suite, or on individual tests in it, while iterating. This is - the only exception to this rule. `go build`, `go vet` and the linter still - go through the entrypoints. A run while iterating is not a gate, so the full - `make check` is still required before every push. - -- The Makefile is authoritative documentation for how the repo is used. Beyond - the required targets above, it should have targets for every common operation: - - running a local development server (`make run`, `make dev`) - - re-initializing or migrating the database (`make db-reset`, - `make migrate`) - - building artifacts (`make build`) - - generating code - - seeding data - - anything else a developer would do regularly - - If someone checks out the repo and types `make`, they should see every - meaningful operation available. A new contributor should be able to - understand the entire development workflow by reading the Makefile. - -- Every repo should have a `Dockerfile`, and it carries the repo's gates: a - `lint` phase and a `test` phase. The final stage depends on both, so the image - cannot be built unless they pass. For non-server repos the final stage brings - up a development environment; for server repos it is the runtime image. The - gate phases and the build stage start from their pinned base images and - install what those images lack in either of these ways: - - inline, as the canonical Go `Dockerfile` below does for `git` - - by running `script/bootstrap`, as the `prompts` repo's own `Dockerfile` - does for its yarn packages - - The development environment stage installs development prerequisites by - running `script/bootstrap`. Only system packages may be installed inline - above it, so that check builds keep them; the `prompts` repo's own - `Dockerfile` does this. A stage that runs `script/bootstrap` COPYs `script/` - and the dependency manifests it needs before running it. These are examples: - - `package.json`, `yarn.lock`, `.yarnrc.yml` and `.yarn/releases/` - - `go.mod` and `go.sum` - -- **The lint and test gates run in Docker, as phases of the `Dockerfile`.** - There is no separate lint file. `script/lint` and `script/test` each build one - phase and nothing else: - - ```sh - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --target lint --output type=cacheonly . - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --target test --output type=cacheonly . - ``` - - **A stage that is not the last one in the file is built only when the final - stage's chain depends on it, or when `--target` names it.** That is why the - two gates are always invoked by name here, and why the final stage carries a - `COPY --from=` of a harmless file from each of them. Without that - dependency, a plain `docker build .` builds the last stage alone and exits 0 - having linted and tested nothing. - - **The gate builds write no image.** With `--output type=cacheonly` the phase - runs and a failing step fails the build, but the result is not exported. - Nothing uses those images, and writing one out is slow: a Go test phase's - image holds the toolchain. A build given neither `--output` nor `-t` writes - an untagged image and leaves it dangling, on every developer host and every - CI runner. `script/cibuild` and `script/docker` build the image that ships - and tag it, so each build replaces the previous image. Each script assigns - the tag on its own line before the build, so `set -e` stops it where - `script/projectname` fails. - - Inside a phase the tool is invoked directly: `golangci-lint`, `go test`, - `eslint`, `prettier`. The tool is never invoked through `make lint` or - `script/test`. Those are themselves a `docker build`, and would recurse into - a daemon that does not exist in a build step. Formatting is the exception - and stays on the host: `script/fmt` writes the working tree, and - `script/fmt-check` is its read-only twin. - - **No lint verdict may come from a host invocation of the linter.** On a - shared host golangci-lint reads a result cache keyed on file content rather - than location. A second checkout of the same content is therefore served the - first one's findings. A host-global lock in `$TMPDIR` also makes concurrent - runs exit non-zero with `parallel golangci-lint is running`, a status a - caller cannot tell from real findings. Both have produced wrong verdicts in - this org, in both directions. A container has its own `TMPDIR`, so the lock - is out of reach. The lint phase does keep golangci-lint's cache between - check builds (below), but it lints every checkout at `/src`. A finding is - taken from that cache only when all of these are the same: - - the files - - the settings - - the golangci-lint version - -- **Any build that runs checks is given a new `CACHEBUST` value.** Docker - invalidates a `COPY` layer only when the copied content changes. On an - unchanged tree the check `RUN` is therefore served from cache, nothing - executes, and the build still exits 0. Every `docker build` in `script/` - therefore passes `--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the - script's process id. So no two builds share a value, not even the three in one - `script/cibuild`. Four scripts do this, and there is no fifth: - - `script/lint` - - `script/test` - - `script/cibuild` - - `script/docker` - - `script/check` runs the two gate phases and `script/fmt-check`, and builds - no image of its own. In every stage of the `Dockerfile`, every step that - lints, tests or compiles comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes - right after these steps: - - the steps that install system packages - - the steps that download Go's modules - - the step that runs `script/bootstrap` to install a Python `test` phase's - packages from `Pipfile.lock` - - A stage with none of these steps declares `ARG CACHEBUST` right after - `FROM`. A step that also installs anything else comes after `ARG CACHEBUST` - too. A `script/bootstrap` that also installs node packages is one. A new - value reruns every `RUN` step after that line; `COPY` steps are compared by - content and need nothing. Docker keeps the layers above the line until one - of these changes: - - the base image - - their own lines - - the files they copy - - A stage without the `ARG` line is served from cache on an unchanged tree, so - a repository re-vendors the scripts and adds the `ARG` lines in one commit. - The scripts do not pass `--no-cache`. That flag reruns every step too, but - it also builds the layers above the line again and empties every cache - mount. Go's build cache and golangci-lint's would then start empty every - time; the canonical Go `Dockerfile` below mounts them so that they survive - between check builds. A bare `docker build .` is not evidence that anything - ran: a sub-second build reporting success is a cache hit, not a result. - Never invalidate by pruning: `docker builder prune` and friends destroy a - build cache shared with every other build on the host. When a check is added - or changed, prove it works: - - plant a defect it must catch - - watch the run fail on it - - revert the defect - - A green run alone shows neither that the check ran nor that it covers what - it should. - -- **The gate phases are separate stages, and the build stage depends on both.** - The lint phase is based on the `golangci/golangci-lint` image (pinned by - hash), so lint failures surface in seconds rather than after a full compile. - The test phase is based on the Debian Go image. The canonical Go repo - `Dockerfile`: - - ```dockerfile - # In every stage ARG CACHEBUST comes right after the steps that install - # system packages and download Go's modules, or right after FROM in a - # stage with neither, and every step that lints, tests or compiles - # comes after it. script/ passes a new value on every build, which - # reruns every RUN step after it. Docker keeps the layers above it - # until the base image, their own lines or go.mod and go.sum change. - # The steps that compile mount Go's build cache, and golangci-lint run - # also mounts golangci-lint's; both survive from build to build. - - # Lint phase - # golangci/golangci-lint:v2.x.x, YYYY-MM-DD - FROM golangci/golangci-lint@sha256:... AS lint - WORKDIR /src - COPY go.mod go.sum ./ - RUN go mod download - ARG CACHEBUST - COPY . . - RUN --mount=type=cache,target=/root/.cache/go-build \ - --mount=type=cache,target=/root/.cache/golangci-lint \ - golangci-lint run --config .golangci.yml ./... - - # Test phase. -race needs cgo and so a C compiler, which the Debian Go - # image ships and the alpine one does not. -count=1 because the mounted - # cache also holds test results, which go test would otherwise replay. - # A first run that timed out gets no rerun, because a rerun would only - # wait for the same hang. - # golang:1.x, YYYY-MM-DD - FROM golang@sha256:... AS test - WORKDIR /src - COPY go.mod go.sum ./ - RUN go mod download - ARG CACHEBUST - COPY . . - RUN --mount=type=cache,target=/root/.cache/go-build \ - status=0; \ - output=$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$?; \ - printf '%s\n' "$output"; \ - test "$status" -eq 0 && exit 0; \ - case "$output" in \ - *"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \ - esac; \ - echo "--- Rerunning with -v for details ---"; \ - go test -count=1 -timeout 90s -race -v ./...; \ - exit 1 - - # Build stage. Nothing is wanted from either phase above; the copies - # are what make BuildKit build them first, so nothing after them can - # run unless lint and test passed. - # golang:1.x-alpine, YYYY-MM-DD - FROM golang@sha256:... AS builder - # git comes from the base image's own distribution package index, which - # is trusted, so it needs no version and no hash. - RUN apk add --no-cache git - WORKDIR /src - COPY go.mod go.sum ./ - RUN go mod download - ARG CACHEBUST - COPY --from=lint /src/go.sum /dev/null - COPY --from=test /src/go.sum /dev/null - # A tar-stream context keeps the sender's file owners, which git refuses. - RUN git config --system --add safe.directory /src - COPY . . - - # The VERSION build arg when one is given, otherwise - # `git describe --tags --always` on the .git in the build context. With - # .git present, a version that is still empty, dev or unknown fails the - # build: git is missing or could not read the checkout. - ARG VERSION - RUN --mount=type=cache,target=/root/.cache/go-build \ - VERSION="${VERSION:-$(git describe --tags --always)}"; \ - if [ -e .git ]; then \ - case "$VERSION" in ""|dev|unknown) \ - echo "version is '$VERSION' although .git is present" >&2; \ - exit 1 ;; \ - esac; \ - fi; \ - CGO_ENABLED=0 go build -trimpath \ - -ldflags="-s -w -X main.Version=${VERSION}" \ - -o /app ./cmd/app/ - - # Runtime stage, and the last one - FROM alpine@sha256:... - ARG CACHEBUST - COPY --from=builder /app /usr/local/bin/app - ENTRYPOINT ["app"] - ``` - - Key points: - - The lint phase uses the `golangci/golangci-lint` image directly, so - nothing needs installing: it has both Go and the linter. - - `COPY --from= /src/go.sum /dev/null` is a no-op copy whose only - purpose is the dependency on that phase. BuildKit runs stages in parallel - by default, and a stage nothing depends on is not built at all. Without - these two lines a red gate would not fail the build. - - Keep the runtime stage last. If you add a stage after it, give it the same - two copies. A plain `docker build .` builds the last stage's chain and - nothing else. - - Check builds keep these, and none of them can hand back a stale result: - - The layers above `ARG CACHEBUST`: the system packages and Go's - modules. Docker installs the packages again when the base image or the - install line changes. Docker downloads the modules again when `go.mod` - or `go.sum` changes. Go checks every module against `go.sum` as it - downloads it, and no later build can change a kept layer. The modules - are kept in a layer rather than a cache mount. A cache mount is shared - by every build on the host, and Go builds from whatever files the - mount holds without checking them again (only `go mod verify` does). A - file changed there would go unnoticed. - - Go's build cache (`/root/.cache/go-build`, where Go keeps it for - root), mounted on the steps that compile: `golangci-lint run`, both - `go test` runs and `go build`. Go reuses a compiled package only when - all its inputs are unchanged. The cache also holds test results, which - is why both `go test` runs carry `-count=1`. - - golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned - image keeps it), mounted on `golangci-lint run`. golangci-lint reuses - a package's findings only when all of these are unchanged: - - the package's files and those of every package it imports - - `go.mod` - - the linters and their settings in `.golangci.yml` - - the golangci-lint version - - In a Python `test` phase, the layer of the step that runs - `script/bootstrap` to install the packages from `Pipfile.lock`, as - described for that phase below. - - If the project uses `//go:embed` directives that reference build - artifacts, the lint phase must create placeholder files so the embed - directives resolve. A web frontend compiled in a separate stage is such an - artifact. Example: - `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. - - If the project requires CGO or system libraries for linting, install them - in the lint phase. The `golangci/golangci-lint` image is Debian-based and - has no `apk`. So install with `apt-get` under the Debian package name - (`libvips-dev`, where alpine says `vips-dev`), and delete the package - lists in the same `RUN`, so the layer does not keep them. The test phase, - on the Debian Go image, installs the libraries its tests need the same - way. Either install goes right after `FROM`, above `ARG CACHEBUST`, so - Docker keeps it until the base image or the install line changes: - - ```dockerfile - FROM golangci/golangci-lint@sha256:... AS lint - RUN apt-get update \ - && apt-get install -y --no-install-recommends libvips-dev \ - && rm -rf /var/lib/apt/lists/* - WORKDIR /src - COPY go.mod go.sum ./ - RUN go mod download - ARG CACHEBUST - ``` - - - `.dockerignore` lets `.git` into the build context. It keeps out every git - `config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): - - the repository's own - - each submodule's under `.git/modules/` - - that of a submodule keeping its own `.git` directory - - `git describe` does not need them, and each can hold a credential: a - password in a remote URL, or the token the CI checkout step stores - there. Some submodule names have a `config` segment, as in `config`, - `deploy/config` or `config/lib`. Such a submodule loses its whole git - directory to `**/.git/modules/**/config`, and Go's version stamping then - fails the build. Give it a name without that segment - (`git submodule add --name`). The stage that compiles has `git`: the - Debian Go image has it, and an alpine one needs - `apk add --no-cache git`. That stage takes the version from the - `VERSION` build argument when one is given, otherwise from - `git describe --tags --always`. That gives: - - the tag, on a tagged commit - - the tag, the number of commits since it and the short commit - (`v1.2.3-4-gabc1234`), on a later commit - - the short commit, when no tag is reachable - - The stage that compiles also marks its working directory safe for git - (`git config --system --add safe.directory /src`). A context sent as a - tar stream keeps the sender's file owners, and git refuses a checkout - owned by another user, so the version would come out empty. - `ARG VERSION` has no default. The build fails if the context carries - `.git` and the version still comes out empty, `dev` or `unknown`. A - plain `docker build .` with no build arguments must succeed; a - Dockerfile that refuses an empty build argument drops that refusal and - keeps the argument. A checkout whose `.git` is a file is the exception: - a linked worktree, or a repository checked out as a submodule. That file - points to a git directory outside the build context, so the build cannot - read the version and a plain `docker build .` fails. Pass the version - with `--build-arg VERSION=...`, as `script/docker` and `script/cibuild` - already do. - -- **A Python `test` phase that installs its packages from `Pipfile.lock` does it - above `ARG CACHEBUST`, so check builds keep them.** The steps are: - - `COPY script/ script/` - - `COPY Pipfile Pipfile.lock ./` - - `RUN script/bootstrap` - - `ARG CACHEBUST` comes right after them. Docker keeps what that `RUN` - installed until one of these changes: - - the base image - - `script/` - - `Pipfile` - - `Pipfile.lock` - - a line up to and including the `RUN` - - The tests run after `ARG CACHEBUST`, on every check build. In every other - stage, a step that installs Python packages comes after `ARG CACHEBUST`. - - When `script/bootstrap` runs in that step, it installs nothing beyond - these: - - system packages - - pipenv and its dependencies - - the packages from `Pipfile.lock` - - `script/bootstrap` installs no Python there: the tests use the - distribution's Python, which a step above it installs. If it installs - anything else there, the step goes after `ARG CACHEBUST`. - - - Debian's Python refuses a system-wide `pip install`, so pipenv goes in a - virtual environment at `$HOME/.pipenv`. `script/bootstrap` creates it with - `python3 -m venv` when it is missing, then uses that environment's `pip` - and `pipenv`. On a host, it first installs the distribution's python and - venv packages when they are missing. One script then works on a host and - in a `Dockerfile`. `Pipfile` names no `python_version`, because - `pipenv install --deploy` stops when the distribution's Python differs - from it. - - PyPI is not trusted, so `script/bootstrap` installs pipenv and its - dependencies by hash. It does so with `pip install --require-hashes -r` - and a requirements file in `script/` that lists each of them with `==` and - its `--hash`. pip refuses to install a dependency the file does not list. - - `script/bootstrap` then runs `pipenv install --deploy`, with `--dev` when - the tests need the development packages. `--deploy` does all of these: - - installs only what `Pipfile.lock` names - - checks every package against the lockfile's hashes - - fails when the lockfile is out of date with `Pipfile` - - `script/bootstrap` sets `PIP_ONLY_BINARY=:all:` for both installs, and - pipenv passes it on to pip. A package with no ready-built wheel for the - image then stops the install. Without the setting, pip builds that package - from source and downloads the build tools without a hash check. - - A kept layer holds nothing a new install would refuse: pip checked the - hashes of pipenv and its dependencies, and pipenv checked the hashes of - the lockfile's packages. pipenv creates its virtual environment with the - `virtualenv` package. The packages that `virtualenv` adds come from wheels - inside that package, so the hash pip checked for it covers them. System - packages, Python among them, are not fixed by a hash. They keep the - versions the package manager gave when the layer was built; a new install - may get newer ones. - - As with Go's modules, the packages are kept in a layer, not a cache mount. - A cache mount is shared by every build on the host, and nothing checks its - files again. - - A Python repository's `script/bootstrap` adds these lines in `main`, above - the last `echo`. The `ensurepip` test fails when python is missing, and also - when Debian's venv package is missing: - - ```sh - if [ ! -x "$HOME/.pipenv/bin/pip" ]; then - if ! python3 -m ensurepip --version >/dev/null 2>&1; then - pkg_install python3 python3-venv python3 python3 - fi - python3 -m venv "$HOME/.pipenv" - fi - export PIP_ONLY_BINARY=:all: - "$HOME/.pipenv/bin/pip" install --require-hashes -r script/requirements.txt - "$HOME/.pipenv/bin/pipenv" install --dev --deploy - ``` - - Make `script/requirements.txt` once, with - `echo pipenv | pip-compile --generate-hashes --allow-unsafe -o script/requirements.txt -`. - `pip-compile` comes from Alpine's `py3-pip-tools` package; Debian and Ubuntu - do not package `pip-tools`. Make the first `Pipfile.lock` with - `$HOME/.pipenv/bin/pipenv lock`, after the `pip install` line has installed - pipenv. - - In the example below, `script/bootstrap` finds `make`, `git`, python and - venv already installed. The example copies no `yarn.lock`, so - `script/bootstrap` installs no node and no yarn. - - ```dockerfile - # Test phase. Both pytest runs share one 90-second timeout, so a hang - # uses it once and gets no rerun. - # debian:trixie-slim, YYYY-MM-DD - FROM debian@sha256:... AS test - # Python, pip and venv come from the distribution, which is trusted, so - # they need no version and no hash. make and git are base tools that - # script/bootstrap installs when they are missing. Installed here, they - # stay in their own layer when script/ changes. - RUN apt-get update \ - && apt-get install -y --no-install-recommends ca-certificates make git \ - python3 python3-venv \ - && rm -rf /var/lib/apt/lists/* - # script/bootstrap installs pipenv in a virtual environment at - # $HOME/.pipenv, and the tests below run that pipenv. - ENV PATH=/root/.pipenv/bin:$PATH - WORKDIR /app - COPY script/ script/ - COPY Pipfile Pipfile.lock ./ - RUN script/bootstrap - ARG CACHEBUST - COPY . . - RUN timeout 90 sh -c 'pipenv run pytest || { \ - echo "--- Rerunning with -v for details ---"; \ - pipenv run pytest -v; exit 1; }' - ``` - -- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - runs `script/cibuild` on push. The workflow's only other step checks out the - repo, with `persist-credentials: false`. `script/cibuild` needs no token, and - without that setting the checkout leaves the job's token in `.git/config` for - every later step. The checkout step also sets `fetch-depth: 0`, which fetches - the tags `git describe` needs. By default it clones shallow with no tags, and - a tagged repository's CI build would stamp a bare short commit id. The - workflow's `concurrency` block groups runs by workflow and branch - (`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`. - A new push then cancels the older run on the same branch, queued or running, - and no other: runs for replaced commits do not hold up the shared runner. - `script/cibuild` bootstraps, runs the gate phases, and then builds the image, - so a successful run means every check passed. A bare `docker build .` does not - carry the same guarantee, because its gate phases may come from the cache. The - image build gets a `CACHEBUST` value of its own and so runs the gate phases a - second time. That is the price of the rule above, and it is worth paying. The - image that ships is built from a run of its own gates rather than from a cache - entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the - shared runner after 20 minutes. That allows for the three Docker builds - described above, each held to the 5-minute Docker build limit below, plus the - bootstrap. Those builds are the test phase, the lint phase, then the image. A - separate workflow limited to `main` by a `branches` list under `on: push` - cannot be checked by review. To try a change to it, add the feature branch to - that list and push, then remove the branch from the list again before merging. - Keep any job in it that publishes behind `if: github.ref_name == 'main'`, so - the run from the feature branch publishes nothing. - -- Use platform-standard formatters: - - `black` for Python - - `prettier` for JS/CSS/Markdown/HTML - - `go fmt` for Go - - Always use default configuration, with two exceptions: - - four-space indents (except Go) - - `proseWrap: always` for Markdown (hard-wrap at 80 columns) - - Documentation and writing repos (Markdown, HTML, CSS) should also have - `.prettierrc` and `.prettierignore`. - -- Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local - testing is not possible in the repo, `script/precommit` may skip `script/test` - and run only `script/lint` and `script/fmt-check`. `script/install-precommit` - installs the hook; the Makefile must provide a `make hooks` target that shims - to it. - -- All repos with software must have tests that run via the platform-standard - test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful - tests exist yet, add the most minimal test possible: importing the module - under test to verify it compiles/parses, for example. There is no excuse for - `make test` to be a no-op. - -- `make test` must complete in under 60 seconds. That is the hard cap, and a - suite that exceeds it fails. Under 20 seconds is the target. A suite between - 20 and 60 seconds is still green, but the overage must be filed as an - improvement bug against that repo. Add a 90-second timeout to the test - invocation (`go test -timeout 90s`). The backstop deliberately sits above the - hard cap so that it catches a genuinely hung test rather than a merely slow - one. A hung package uses the backstop once, because a first run that timed out - gets no rerun. `-timeout` applies to each package, so several hung packages - can each use it. - -- **The test command should use the conditional verbose rerun pattern.** Run - tests without `-v` (verbose) first. If tests fail, automatically rerun with - `-v` to show full output. This keeps CI logs and `docker build` output clean - on success, with just package/suite summaries. On failure it provides full - diagnostic detail: every test case, every assertion. A first run that timed - out gets no rerun, because the rerun would only wait for the same hang. The - command lives in the `test` phase of the `Dockerfile`, since `script/test` - builds that phase. The Makefile form below is the same pattern for any - repo-local invocation: - - ```makefile - test: - @ || \ - { echo "--- Rerunning with -v for details ---"; \ - ; exit 1; } - ``` - - Go example: - - ```makefile - test: - @status=0; \ - output=$$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$$?; \ - printf '%s\n' "$$output"; \ - test "$$status" -eq 0 && exit 0; \ - case "$$output" in \ - *"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \ - esac; \ - echo "--- Rerunning with -v for details ---"; \ - go test -count=1 -timeout 90s -race -v ./...; \ - exit 1 - ``` - - The first run's output is kept so that the recipe can look in it for what Go - prints for a hang: - - `panic: test timed out` means the test's own timer fired. - - `*** Test killed` means `go test` stopped a test that ignored that timer, - about a minute later. - - Go prints the goroutine stacks with either line, as long as the test can - still respond. A rerun would only wait for the same hang, so the recipe - exits 1. A failure with neither line in its output reruns with `-v`. The - kept output prints when the first run ends, not while it runs. - - `-count=1` is required on both invocations. It defeats Go's test _result_ - cache, so neither run can report a stored pass in place of running the - tests. It leaves the build cache alone, so it costs the runtime of the suite - and no recompilation. - - That cache is Go's own, separate from Docker's layer cache. Go stores a - passing result in its cache directory (`GOCACHE`). When the same tests run - again on unchanged code, it prints that result, marked `(cached)`, without - running them. The directory lasts from one run to the next on a developer's - machine, where this target runs. The directory also lasts in the `test` - phase of the `Dockerfile`, which mounts it to keep compiled packages between - check builds, so the phase's `go test` runs carry `-count=1` too. A new - `CACHEBUST` value (above) is what makes that step run on an unchanged tree. - - Python example: - - ```makefile - test: - @timeout 90 sh -c 'pipenv run pytest || { \ - echo "--- Rerunning with -v for details ---"; \ - pipenv run pytest -v; exit 1; }' - ``` - - Both pytest runs share that one 90-second timeout, as in the `Dockerfile` - example above, so a hang uses it once and gets no rerun. Ctrl-C does not - stop a running `make test`, which goes on until the tests end or the 90 - seconds run out. On a Mac without `timeout`, the target fails until it is - installed. On a host, the target needs `$HOME/.pipenv/bin` on `PATH`. - - The `exit 1` ensures the target always fails after a rerun. The first run - already proved the tests are broken, so the build must not pass even if a - flaky test happens to succeed on the second attempt. The rerun exists solely - for diagnostic output. - -- Docker builds must complete in under 5 minutes. A hung package uses the - 90-second test backstop once, because a first run that timed out gets no - rerun. - -- `make check` must not modify any files in the repo. Tests may use temporary - directories. - -- `main` must always pass `make check`, no exceptions. +# Pin external references by hash + +**External references must be pinned by cryptographic hash.** This is the single +most important rule in this document. It covers what we download and install +ourselves in scripts and Dockerfiles. Reference each by its content hash: + +- Docker base images: `@sha256:...` +- Go modules: the module hash in `go.sum` +- Python packages from PyPI: the package hash in `Pipfile.lock` +- npm packages: the `checksum` in `yarn.lock` +- GitHub Actions: `@` +- release archives fetched with `curl` or `wget`: a hash hardcoded in the + Dockerfile or script + +Packages from the distribution's own package manager are outside this rule. They +need no version and no hash, because the distribution is trusted. In a +`Dockerfile`, it is the pinned base image's distribution. On a host, it is the +host's. The host's package manager stands for its distribution, and that +includes nix and brew. Updating the package index and upgrading installed +packages are fine too. This command, for example, is allowed: + +```sh +apt update && apt upgrade -y && apt install -y build-essential +``` + +A package source that a `Dockerfile` adds itself is not the distribution's own, +so its packages are pinned like any other download. + +- A version tag such as `@v4`, `@latest` or `:3.21` is server-mutable, and + therefore a remote code execution vulnerability. +- An unverified install script is arbitrary remote code execution. Never + `curl | bash` to install a tool such as rustup. Instead, download a specific + release archive from GitHub, verify its hash, and only then install. +- Docker Hub, PyPI, the npm registry and `go get` are not trusted, so everything + from them stays pinned by hash. +- Above each image or package pinned by hash, add a comment with the version and + date (YYYY-MM-DD). +- Double-check every external reference in every file before committing. + +node, yarn, npm, python, pip and venv come from the distribution wherever +possible: + +- node means the distribution's newest Node package, such as `nodejs-current` on + Alpine 3.24. +- pip and venv are the installer tools bundled with python. +- If the distribution's version is too old, use a newer distribution, never a + download. That rules out nvm, pyenv, corepack and `npm install -g yarn`. +- yarn's release file is third-party code, and git pins it. `yarn set version` + writes it to `.yarn/releases/` once, given an exact version. + +# Git and secrets - Never commit secrets. `.env` files, credentials, API keys, and private keys must be in `.gitignore`. No exceptions. +- Make all changes on a feature branch. You can do whatever you want on a + feature branch. +- Never force-push to `main`. +- Never use `git add -A` or `git add .`. Always stage files explicitly by name. +- **No build artifacts in version control.** Code-derived data must never be + committed if it can be avoided: compiled bundles, minified output, generated + assets. The build process (e.g. Dockerfile, Makefile) should generate these at + build time. Notable exception: Go protobuf generated files (`.pb.go`) ARE + committed, because repos need to work with `go get`, which downloads code but + does not execute code generation. -- `.gitignore` should be comprehensive from the start: - - OS files (`.DS_Store`) - - editor files (`.swp`, `*~`) - - in-repo agent scratch directories (`.claude/`) - - `node_modules/` - - `.yarn/*`, with `!.yarn/releases` so that git keeps yarn's release file - - the repo's own build outputs +# Entrypoints - Fetch the standard `.gitignore` from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting - up a new repo. A repo's `.gitignore` is the standard file followed by the - repo's own entries, such as its binaries. A re-vendor replaces the standard - part and keeps those entries. These patterns are written to `.gitignore`'s - own semantics, in which an unanchored pattern already matches at every - depth. They are not a `.dockerignore` and must not be transplanted into one - unmodified. +Repos follow the +[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) +pattern. Every repo with software must have a root `Makefile` with the targets +below. Each target is a thin shim that calls an executable script in `script/`, +where its implementation lives: -- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns - across unmodified leaves secrets in the build context.** Docker matches with - `moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension. So - `*` does not cross `/`, and a pattern without a leading `**/` is anchored at - the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key` - therefore excludes only the copies at the repository root. `config/.env` and - `certs/server.key` still reach the context and can land in an image layer. - Such a `.dockerignore` is more dangerous than a short file with no secret - patterns at all, because it reads as solved and stops anyone looking. Give - every depth-independent pattern the `**/` prefix, and leave only genuinely - root-anchored entries unprefixed. Those are `.claude`, `.yarn/*`, +- `make bootstrap`: `script/bootstrap` installs all dependencies. +- `make setup`: `script/setup` readies a fresh clone for development. It runs + `bootstrap`, then `install-precommit`, plus any repo-specific initialization. +- `make test`: `script/test` +- `make lint`: `script/lint` +- `make fmt`: `script/fmt` (writes) +- `make fmt-check`: `script/fmt-check` (read-only) +- `make check`: `script/check` runs `script/test`, `script/lint` and + `script/fmt-check`. +- `make docker`: `script/docker` +- `make hooks`: `script/install-precommit` installs the git pre-commit hook. + +Three more scripts have no target: + +- `script/precommit` is what the pre-commit hook runs. It calls `script/check`. + Repo-type-specific extras belong in it, not in the hook itself; `go mod tidy` + verification in Go repos is one. If local testing is not possible in the repo, + it may skip `script/test` and run only `script/lint` and `script/fmt-check`. +- `script/cibuild` runs the CI build, and the Gitea workflow calls it. +- `script/projectname` (literally that filename) outputs the project's name. + Scripts that need the name call it, so they stay byte-identical across all + repos. `script/docker`, for example, assembles its image tag from it. + +From the standard's canonical set we use `bootstrap`, `setup`, `test`, and +`cibuild`. `check`, `precommit`, `install-precommit` and `projectname` are our +own extensions to the standard. + +The scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run +in minimal containers; alpine images, for example, have no bash. Locate the repo +root with `$(cd "$(dirname "$0")/.." && pwd -P)`, and `cd` there before acting. +A model Makefile is at +`https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`. Model scripts are +at `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/`. + +Run the repo's own entrypoints, never the underlying tools directly. The +`script/` scripts are the single source of truth for how each operation runs. +Calling them directly is fine, and so is calling the Makefile targets. The +Makefile targets must still exist, as the thin shims described above. + +An agent whose work is confined to one package may run `go test` directly on +that package's suite, or on individual tests in it, while iterating. This is the +only exception to this rule. `go build`, `go vet` and the linter still go +through the entrypoints. A run while iterating is not a gate, so the full +`make check` is still required before every push. + +The Makefile is authoritative documentation for how the repo is used. Beyond the +required targets above, it should have targets for every common operation: + +- running a local development server (`make run`, `make dev`) +- re-initializing or migrating the database (`make db-reset`, `make migrate`) +- building artifacts (`make build`) +- generating code +- seeding data +- anything else a developer would do regularly + +Typing `make` in a checkout should show every meaningful operation. A new +contributor should be able to understand the entire development workflow by +reading the Makefile. + +# Tests and checks + +- All repos with software must have tests that run via the platform-standard + test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). There is no + excuse for `make test` to be a no-op. If no meaningful tests exist yet, add + the most minimal one: importing the module under test to verify it + compiles/parses, for example. +- `make test` must complete in under 60 seconds. That is the hard cap, and a + suite that exceeds it fails. Under 20 seconds is the target. A suite between + 20 and 60 seconds is still green, but the overage must be filed as an + improvement bug against that repo. +- Add a 90-second timeout to the test invocation (`go test -timeout 90s`). This + backstop sits above the hard cap on purpose: it catches a hung test, not a + merely slow one. `-timeout` applies to each package, so several hung packages + can each use it. +- Docker builds must complete in under 5 minutes. +- `make check` must not modify any files in the repo. Tests may use temporary + directories. +- `main` must always pass `make check`, no exceptions. + +**The test command should use the conditional verbose rerun pattern.** Run tests +without `-v` (verbose) first. If they fail, rerun automatically with `-v`. +Success then leaves CI logs and `docker build` output clean, with just +package/suite summaries. Failure shows every test case and every assertion. + +- The rerun always ends in `exit 1`. The first run already proved the tests are + broken, so the build must not pass even if a flaky test succeeds on the second + attempt. The rerun exists solely for diagnostic output. +- A first run that timed out gets no rerun, which would only wait for the same + hang. A hung package therefore uses the 90-second backstop once. + +The command lives in the `test` phase of the `Dockerfile`, since `script/test` +builds that phase. The canonical Go `Dockerfile` and the Python `test` phase +example below carry it. The Makefile form below is the same pattern for any +repo-local invocation. In a Makefile, each `$` of those two commands is written +`$$`. + +```makefile +test: + @ || \ + { echo "--- Rerunning with -v for details ---"; \ + ; exit 1; } +``` + +# `script/bootstrap` + +`script/bootstrap` installs all dependencies idempotently and assumes nothing is +present. + +- A tool from the distribution has no pin, so it is installed only when it is + missing. make, git, node and yarn are such tools. They come from nix, apt, + brew, or apk, detected in that order; apt runs noninteractive. +- A tool from outside the distribution is never installed as "latest" or "lts"; + it is always an exact version. + +## node and yarn + +Use `yarn`, not `npm`. + +- `script/bootstrap` installs node, yarn and the node packages only when + `yarn.lock` is in the tree. A `Dockerfile` stage that needs no yarn therefore + copies no `yarn.lock`. The Go and Python stages below are such stages. +- It installs the node packages with `yarn install --immutable`, which fails + when a package's `checksum` is wrong, or when `package.json` is out of step + with `yarn.lock`. +- Each repository commits the yarn that runs. `yarnPath` in `.yarnrc.yml` points + at the release file in `.yarn/releases/`. The distribution's yarn only starts + that file, and that file does the work. So Alpine's yarn 1 and Debian 13's + yarn 4.1 run the same committed yarn. Corepack never runs, and no yarn is + downloaded. +- `.yarnrc.yml` holds `yarnPath`, `nodeLinker: node-modules` and + `enableTelemetry: false`. Without the last, yarn sends usage reports from any + host that is not CI. +- By default, yarn runs no dependency's install script. A dependency that must + run one is named in `package.json`, as + `"dependenciesMeta": { "": { "built": true } }`. +- `.gitattributes` carries `.yarn/releases/** binary`, so diffs and merges treat + the release file as one opaque file. +- yarn 1 ignores `--immutable` and exits 0, so on a tree with no `.yarnrc.yml` + the install is not frozen. A repository therefore re-vendors + `script/bootstrap` and converts its `yarn.lock` in one commit. +- Debian and Ubuntu name the yarn command `yarnpkg`. `script/bootstrap`, + `script/fmt` and `script/fmt-check` each carry the same small function that + runs whichever name is present. With neither name on `PATH`, `script/fmt` and + `script/fmt-check` print one line saying to run `script/bootstrap`, and + exit 1. +- A `Dockerfile` stage that needs yarn may start from any distribution that + packages node and yarn. Before it runs `script/bootstrap`, it copies + `package.json`, `yarn.lock`, `.yarnrc.yml` and `.yarn/releases/`. With only + the first two, the install fails, and Alpine's yarn 1 says to enable corepack. + Never do that; copy the missing files instead. + +## Pinned tools + +**`script/bootstrap` installs a pinned tool by comparing versions, never by +testing presence.** An `if ! command -v ; then install; fi` guard tests +`PATH` only. On an already-provisioned machine the pin is then inert, and a +version bump is a silent no-op. The Dockerfile's clean image still gets the +pinned version, so a local `make check` and `make docker` can disagree about +what the tool even is. The canonical form: + +- compares the installed version against the pin over the **whole** version + token. A parser that stops at the first `-` reports `2.12.2` for a host + running `2.12.2-rc1` and skips the install. +- treats `--version` output that is absent, non-zero, empty or unrecognised as a + mismatch. A failure then causes a redundant install, never a skipped one. +- after installing, re-resolves the binary the way callers do: `hash -r`, then + `PATH`, not the directory the installer wrote to. A shadowing binary can hide + an install that reported success. In that case it fails and names the resolved + path. +- is actually called, and prints the version on both success paths. A function + defined and never invoked has the same exit status and the same empty output + as one that worked. +- stays POSIX sh: no arrays, no `[[`, no `grep -P`. + +A Go tool a repo needs on the host is installed with `go install` pinned to a +commit hash (`go install @`). It is never tracked as a +`go.mod` tool dependency or through a `tools.go` file, either of which pulls the +tool's own dependencies into the repo's `go.mod` and `go.sum`. + +# Dockerfile + +Every repo should have a `Dockerfile`, and it carries the repo's gates: a `lint` +phase and a `test` phase. The final stage depends on both, so the image cannot +be built unless they pass. For non-server repos the final stage brings up a +development environment; for server repos it is the runtime image. + +**The lint and test gates run in Docker, as phases of the `Dockerfile`.** There +is no separate lint file. `script/lint` and `script/test` each build one phase +and nothing else: + +```sh +docker build --build-arg CACHEBUST="$(date +%s)-$$" \ + --target lint --output type=cacheonly . +docker build --build-arg CACHEBUST="$(date +%s)-$$" \ + --target test --output type=cacheonly . +``` + +- **A stage that is not the last one in the file is built only when the final + stage's chain depends on it, or when `--target` names it.** That is why the + two gates are always invoked by name here, and why the final stage carries a + `COPY --from=` of a harmless file from each of them. Without those copies, a + plain `docker build .` builds the last stage's chain alone and exits 0 having + linted and tested nothing. Keep the final stage last. If you add a stage after + it, give it the same two copies. +- **The gate builds write no image.** With `--output type=cacheonly` the phase + runs and a failing step fails the build, but the result is not exported. + Nothing uses those images, and writing one out is slow. A build given neither + `--output` nor `-t` leaves an untagged image dangling on every developer host + and CI runner. `script/cibuild` and `script/docker` tag the image that ships, + so each build replaces the previous image. +- **Inside a phase the tool is invoked directly:** `golangci-lint`, `go test`, + `eslint`, `prettier`. It is never invoked through `make lint` or + `script/test`, which are themselves a `docker build` and would recurse into a + daemon that does not exist in a build step. +- **Formatting is the exception and stays on the host.** `script/fmt` writes the + working tree, and `script/fmt-check` is its read-only twin. +- **No lint verdict may come from a host invocation of the linter.** On a shared + host golangci-lint reads a result cache keyed on file content rather than + location. A second checkout of the same content is therefore served the first + one's findings. A host-global lock in `$TMPDIR` also makes concurrent runs + exit non-zero with `parallel golangci-lint is running`, which a caller cannot + tell from real findings. A container has its own `TMPDIR`, out of the lock's + reach. The lint phase does keep golangci-lint's cache between check builds, + but it lints every checkout at `/src`. It reuses a finding only on the + conditions listed under the canonical Go `Dockerfile` below. + +What each stage installs: + +- The gate phases and the build stage start from their pinned base images. They + install what those images lack inline, as the canonical Go `Dockerfile` below + does for `git`. Or they run `script/bootstrap`, as the `prompts` repo's own + `Dockerfile` does for its yarn packages. +- The development environment stage installs development prerequisites by + running `script/bootstrap`. Only system packages may be installed inline above + it, so that check builds keep them; the `prompts` repo's own `Dockerfile` does + this. +- A stage that runs `script/bootstrap` first COPYs `script/` and the dependency + manifests it needs: `go.mod` and `go.sum`, for example, or the four yarn files + named above. + +## `CACHEBUST` + +**Any build that runs checks is given a new `CACHEBUST` value.** On an unchanged +tree Docker otherwise serves the check `RUN` from cache: nothing executes, and +the build still exits 0. + +Every `docker build` in `script/` therefore passes +`--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process id. +So no two builds share a value, not even the three in one `script/cibuild`. Four +scripts do this, and there is no fifth: `script/lint`, `script/test`, +`script/cibuild` and `script/docker`. `script/check` builds no image of its own. + +In every stage of the `Dockerfile`, every step that lints, tests or compiles +comes after `ARG CACHEBUST`. `ARG CACHEBUST` comes right after these steps: + +- the steps that install system packages +- the steps that download Go's modules +- the step that runs `script/bootstrap` to install a Python `test` phase's + packages from `Pipfile.lock` + +A stage with none of these steps declares `ARG CACHEBUST` right after `FROM`. A +step that also installs anything else comes after `ARG CACHEBUST` too. A +`script/bootstrap` that also installs node packages is one. + +- A new value reruns every `RUN` step after that line; `COPY` steps are compared + by content and need nothing. Docker keeps the layers above the line until the + base image, their own lines or the files they copy change. +- A stage without the `ARG` line is served from cache on an unchanged tree, so a + repository re-vendors the scripts and adds the `ARG` lines in one commit. +- The scripts do not pass `--no-cache`. That flag reruns every step too, but it + also rebuilds the layers above the line and empties every cache mount. The + canonical Go `Dockerfile` below mounts Go's build cache and golangci-lint's so + that they survive between check builds. +- A bare `docker build .` is not evidence that anything ran: a sub-second build + reporting success is a cache hit, not a result. +- Never invalidate by pruning: `docker builder prune` and friends destroy a + build cache shared with every other build on the host. +- When a check is added or changed, prove it works: plant a defect it must + catch, watch the run fail on it, then revert the defect. A green run alone + shows neither that the check ran nor that it covers what it should. + +## Canonical Go `Dockerfile` + +**The gate phases are separate stages, and the build stage depends on both.** +The lint phase is based on the `golangci/golangci-lint` image (pinned by hash), +so lint failures surface in seconds rather than after a full compile. That image +has both Go and the linter, so nothing needs installing. The test phase is based +on the Debian Go image. + +```dockerfile +# In every stage ARG CACHEBUST comes right after the steps that install +# system packages and download Go's modules, or right after FROM in a +# stage with neither, and every step that lints, tests or compiles +# comes after it. script/ passes a new value on every build, which +# reruns every RUN step after it. Docker keeps the layers above it +# until the base image, their own lines or go.mod and go.sum change. +# The steps that compile mount Go's build cache, and golangci-lint run +# also mounts golangci-lint's; both survive from build to build. + +# Lint phase +# golangci/golangci-lint:v2.x.x, YYYY-MM-DD +FROM golangci/golangci-lint@sha256:... AS lint +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +ARG CACHEBUST +COPY . . +RUN --mount=type=cache,target=/root/.cache/go-build \ + --mount=type=cache,target=/root/.cache/golangci-lint \ + golangci-lint run --config .golangci.yml ./... + +# Test phase. -race needs cgo and so a C compiler, which the Debian Go +# image ships and the alpine one does not. -count=1 because the mounted +# cache also holds test results, which go test would otherwise replay. +# A first run that timed out gets no rerun, because a rerun would only +# wait for the same hang. +# golang:1.x, YYYY-MM-DD +FROM golang@sha256:... AS test +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +ARG CACHEBUST +COPY . . +RUN --mount=type=cache,target=/root/.cache/go-build \ + status=0; \ + output=$(go test -count=1 -timeout 90s -race -cover ./... 2>&1) || status=$?; \ + printf '%s\n' "$output"; \ + test "$status" -eq 0 && exit 0; \ + case "$output" in \ + *"panic: test timed out"*|*"*** Test killed"*) exit 1 ;; \ + esac; \ + echo "--- Rerunning with -v for details ---"; \ + go test -count=1 -timeout 90s -race -v ./...; \ + exit 1 + +# Build stage. Nothing is wanted from either phase above; the copies +# are what make BuildKit build them first, so nothing after them can +# run unless lint and test passed. +# golang:1.x-alpine, YYYY-MM-DD +FROM golang@sha256:... AS builder +# git comes from the base image's own distribution package index, which +# is trusted, so it needs no version and no hash. +RUN apk add --no-cache git +WORKDIR /src +COPY go.mod go.sum ./ +RUN go mod download +ARG CACHEBUST +COPY --from=lint /src/go.sum /dev/null +COPY --from=test /src/go.sum /dev/null +# A tar-stream context keeps the sender's file owners, which git refuses. +RUN git config --system --add safe.directory /src +COPY . . + +# The VERSION build arg when one is given, otherwise +# `git describe --tags --always` on the .git in the build context. With +# .git present, a version that is still empty, dev or unknown fails the +# build: git is missing or could not read the checkout. +ARG VERSION +RUN --mount=type=cache,target=/root/.cache/go-build \ + VERSION="${VERSION:-$(git describe --tags --always)}"; \ + if [ -e .git ]; then \ + case "$VERSION" in ""|dev|unknown) \ + echo "version is '$VERSION' although .git is present" >&2; \ + exit 1 ;; \ + esac; \ + fi; \ + CGO_ENABLED=0 go build -trimpath \ + -ldflags="-s -w -X main.Version=${VERSION}" \ + -o /app ./cmd/app/ + +# Runtime stage, and the last one +FROM alpine@sha256:... +ARG CACHEBUST +COPY --from=builder /app /usr/local/bin/app +ENTRYPOINT ["app"] +``` + +Key points: + +- Check builds keep these, and none of them can hand back a stale result: + - The layers above `ARG CACHEBUST`: the system packages and Go's modules. Go + checks every module against `go.sum` as it downloads it, and no later + build can change a kept layer. + - Go's build cache (`/root/.cache/go-build`, where Go keeps it for root), + mounted on the steps that compile: `golangci-lint run`, both `go test` + runs and `go build`. Go reuses a compiled package only when all its inputs + are unchanged. + - golangci-lint's cache (`/root/.cache/golangci-lint`, where the pinned + image keeps it), mounted on `golangci-lint run`. golangci-lint reuses a + package's findings only when all of these are unchanged: + - the package's files and those of every package it imports + - `go.mod` + - the linters and their settings in `.golangci.yml` + - the golangci-lint version +- The modules are kept in a layer rather than a cache mount. A cache mount is + shared by every build on the host, and Go builds from whatever files it holds + without checking them again (only `go mod verify` does). A file changed there + would go unnoticed. +- The test command keeps the first run's output, to look in it for what Go + prints for a hang. `panic: test timed out` means the test's own timer fired. + `*** Test killed` means `go test` stopped a test that ignored that timer, + about a minute later. Go prints the goroutine stacks with either line, as long + as the test can still respond. On either line the command exits 1 without a + rerun. A failure with neither line reruns with `-v`. The kept output prints + when the first run ends, not while it runs. +- `-count=1` is required on both `go test` invocations. Go stores a passing + result in its cache directory (`GOCACHE`). When the same tests run again on + unchanged code, it prints that result, marked `(cached)`, without running + them. That directory lasts between runs on a developer's machine, and in the + `test` phase, which mounts it to keep compiled packages. +- `-count=1` defeats only that test _result_ cache. It leaves the build cache + alone, so it costs the runtime of the suite and no recompilation. The cache is + Go's own, separate from Docker's layer cache, which `CACHEBUST` handles. +- If the project uses `//go:embed` directives that reference build artifacts, + the lint phase must create placeholder files so the embed directives resolve. + A web frontend compiled in a separate stage is such an artifact. Example: + `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`. +- If the project requires CGO or system libraries for linting, install them in + the lint phase. The `golangci/golangci-lint` image is Debian-based and has no + `apk`. So install with `apt-get` under the Debian package name (`libvips-dev`, + where alpine says `vips-dev`), and delete the package lists in the same `RUN`, + so the layer does not keep them. The test phase, on the Debian Go image, + installs the libraries its tests need the same way. Either install goes right + after `FROM`, above `ARG CACHEBUST`: + + ```dockerfile + FROM golangci/golangci-lint@sha256:... AS lint + RUN apt-get update \ + && apt-get install -y --no-install-recommends libvips-dev \ + && rm -rf /var/lib/apt/lists/* + WORKDIR /src + COPY go.mod go.sum ./ + RUN go mod download + ARG CACHEBUST + ``` + +## Python `test` phase + +**A Python `test` phase that installs its packages from `Pipfile.lock` does it +above `ARG CACHEBUST`, so check builds keep them.** The steps are +`COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and +`RUN script/bootstrap`, with `ARG CACHEBUST` right after them. Docker keeps what +that `RUN` installed until the base image, `script/`, `Pipfile`, `Pipfile.lock` +or a line up to and including the `RUN` changes. + +The tests run after `ARG CACHEBUST`, on every check build. In every other stage, +a step that installs Python packages comes after `ARG CACHEBUST`. + +In that step, `script/bootstrap`: + +- installs nothing beyond system packages, pipenv and its dependencies, and the + packages from `Pipfile.lock`. If it installs anything else there, the step + goes after `ARG CACHEBUST`. +- installs no Python: the tests use the distribution's Python, which a step + above it installs. +- puts pipenv in a virtual environment at `$HOME/.pipenv`, because Debian's + Python refuses a system-wide `pip install`. It creates the environment with + `python3 -m venv` when it is missing, then uses that environment's `pip` and + `pipenv`. On a host, it first installs the distribution's python and venv + packages when they are missing. One script then works on a host and in a + `Dockerfile`. +- installs pipenv and its dependencies by hash, because PyPI is not trusted. It + runs `pip install --require-hashes -r` on a requirements file in `script/` + that lists each of them with `==` and its `--hash`. pip refuses to install a + dependency the file does not list. +- then runs `pipenv install --deploy`, with `--dev` when the tests need the + development packages. `--deploy` installs only what `Pipfile.lock` names, + checks every package against the lockfile's hashes, and fails when the + lockfile is out of date with `Pipfile`. +- sets `PIP_ONLY_BINARY=:all:` for both installs, and pipenv passes it on to + pip. A package with no ready-built wheel for the image then stops the install. + Without the setting, pip builds that package from source and downloads the + build tools without a hash check. + +`Pipfile` names no `python_version`, because `pipenv install --deploy` stops +when the distribution's Python differs from it. + +A kept layer holds nothing a new install would refuse. pip checked the hashes of +pipenv and its dependencies, and pipenv those of the lockfile's packages. pipenv +creates its virtual environment with the `virtualenv` package. The packages that +`virtualenv` adds come from wheels inside it, so the hash pip checked for it +covers them. System packages, Python among them, are not fixed by a hash. They +keep the versions the package manager gave when the layer was built; a new +install may get newer ones. + +As with Go's modules, the packages are kept in a layer, not a cache mount, which +every build on the host shares and nothing checks again. + +A Python repository's `script/bootstrap` adds these lines in `main`, above the +last `echo`. The `ensurepip` test fails when python is missing, and also when +Debian's venv package is missing: + +```sh +if [ ! -x "$HOME/.pipenv/bin/pip" ]; then + if ! python3 -m ensurepip --version >/dev/null 2>&1; then + pkg_install python3 python3-venv python3 python3 + fi + python3 -m venv "$HOME/.pipenv" +fi +export PIP_ONLY_BINARY=:all: +"$HOME/.pipenv/bin/pip" install --require-hashes -r script/requirements.txt +"$HOME/.pipenv/bin/pipenv" install --dev --deploy +``` + +Make `script/requirements.txt` once, with +`echo pipenv | pip-compile --generate-hashes --allow-unsafe -o script/requirements.txt -`. +`pip-compile` comes from Alpine's `py3-pip-tools` package; Debian and Ubuntu do +not package `pip-tools`. Make the first `Pipfile.lock` with +`$HOME/.pipenv/bin/pipenv lock`, after the `pip install` line has installed +pipenv. + +In the example below, `script/bootstrap` finds `make`, `git`, python and venv +already installed: + +```dockerfile +# Test phase. Both pytest runs share one 90-second timeout, so a hang +# uses it once and gets no rerun. +# debian:trixie-slim, YYYY-MM-DD +FROM debian@sha256:... AS test +# Python, pip and venv come from the distribution, which is trusted, so +# they need no version and no hash. make and git are base tools that +# script/bootstrap installs when they are missing. Installed here, they +# stay in their own layer when script/ changes. +RUN apt-get update \ + && apt-get install -y --no-install-recommends ca-certificates make git \ + python3 python3-venv \ + && rm -rf /var/lib/apt/lists/* +# script/bootstrap installs pipenv in a virtual environment at +# $HOME/.pipenv, and the tests below run that pipenv. +ENV PATH=/root/.pipenv/bin:$PATH +WORKDIR /app +COPY script/ script/ +COPY Pipfile Pipfile.lock ./ +RUN script/bootstrap +ARG CACHEBUST +COPY . . +RUN timeout 90 sh -c 'pipenv run pytest || { \ + echo "--- Rerunning with -v for details ---"; \ + pipenv run pytest -v; exit 1; }' +``` + +Run from a `make test` target on a host, that test command: + +- does not stop on Ctrl-C. It goes on until the tests end or the 90 seconds run + out. +- fails on a Mac without `timeout`, until it is installed. +- needs `$HOME/.pipenv/bin` on `PATH`. + +## Version + +**A plain `docker build .` of a clone stamps the version that +`git describe --tags --always` gives.** That gives: + +- the tag, on a tagged commit +- the tag, the number of commits since it and the short commit + (`v1.2.3-4-gabc1234`), on a later commit +- the short commit, when no tag is reachable + +The stage that compiles derives it from the `.git` in the build context, as the +canonical Go `Dockerfile` above shows: + +- It takes the version from the `VERSION` build argument when one is given, + otherwise from `git describe`. `ARG VERSION` is declared in that stage, + because `ARG` is stage-scoped, and has no default. +- It has `git`: the Debian Go image has it, and an alpine one needs + `apk add --no-cache git`. +- It marks its working directory safe for git + (`git config --system --add safe.directory /src`). A context sent as a tar + stream keeps the sender's file owners, and git refuses a checkout owned by + another user, so the version would come out empty. +- The build fails if the context carries `.git` and the version still comes out + empty, `dev` or `unknown`. Otherwise a missing `git` or an unreadable checkout + would leave `-X main.Version=` empty, and the build would still exit 0. +- A plain `docker build .` with no build arguments must succeed; a Dockerfile + that refuses an empty build argument drops that refusal and keeps the + argument. +- A checkout whose `.git` is a file is the exception: a linked worktree, or a + repository checked out as a submodule. That file points to a git directory + outside the build context, so the build cannot read the version and a plain + `docker build .` fails. Pass the version with `--build-arg VERSION=...`, as + `script/docker` and `script/cibuild` already do. + +`.dockerignore` lets `.git` into the build context. It keeps out every git +`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): + +- the repository's own +- each submodule's under `.git/modules/` +- that of a submodule keeping its own `.git` directory + +`git describe` does not need them, and each can hold a credential: a password in +a remote URL, or the token the CI checkout step stores there. A submodule whose +name has a `config` segment (`config`, `deploy/config`, `config/lib`) loses its +whole git directory to `**/.git/modules/**/config`, and Go's version stamping +then fails the build. Give it a name without that segment +(`git submodule add --name`). + +`script/docker` and `script/cibuild` pass the version they compute on the host; +it takes precedence. They do this byte-identically across repos: + +```sh +# The version and the tag each get their own line: a failing command +# substitution inside an argument does not trip `set -e`, so the inline +# form degrades to an empty constant. +version="$(git describe --tags --always --dirty 2>/dev/null || true)" +[ -n "$version" ] || version="unknown" +tag="$(script/projectname)" +docker build --build-arg CACHEBUST="$(date +%s)-$$" \ + --build-arg VERSION="$version" \ + -t "$tag" . +``` + +- `--always` makes an untagged repo yield an abbreviated commit hash rather than + failing. +- The `[ -n "$version" ]` line is the single place the fallback is applied. It + is a live check that fires on a build from an export with no `.git`, and on a + repository with no commits yet. Do not fold it into the substitution as + `|| echo unknown`, which makes the guard unreachable. +- Passing `VERSION` to a repo whose Dockerfile declares no such `ARG` is ignored + and costs nothing, which is why the scripts stay byte-identical. + +# CI + +Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that runs +`script/cibuild` on push. The workflow's only other step checks out the repo. +The canonical `.gitea/workflows/check.yml` sets these: + +- `persist-credentials: false` on the checkout step. `script/cibuild` needs no + token, and without that setting the checkout leaves the job's token in + `.git/config` for every later step. +- `fetch-depth: 0` on the checkout step, which fetches the tags `git describe` + needs. The default is a shallow clone with no tags, and a tagged repository's + CI build would then stamp a bare short commit id. +- a `concurrency` block that groups runs by workflow and branch + (`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`. + A new push then cancels the older run on the same branch, queued or running, + and no other: runs for replaced commits do not hold up the shared runner. +- `timeout-minutes: 20` on the `check` job, so a hung build frees the shared + runner after 20 minutes. That allows for the bootstrap and three Docker + builds, each held to the 5-minute limit. + +`script/cibuild` changes to the repo root, runs `script/bootstrap`, runs +`script/check`, and builds the image with the version: + +- **`script/cibuild` runs `script/bootstrap` first**, because the workflow + checks out the repo and runs nothing else, while `script/fmt-check` runs the + formatter on the host. Without it, a pristine checkout dies there, after the + containerised gates have passed. +- Its three Docker builds are the test phase, the lint phase, then the image. So + a successful run means every check passed, which a bare `docker build .` does + not show. +- The image build gets a `CACHEBUST` value of its own and so runs the gate + phases a second time. That price is worth paying: the image that ships is + built from a run of its own gates rather than from a cache entry. + +A separate workflow limited to `main` by a `branches` list under `on: push` +cannot be checked by review. To try a change to it, add the feature branch to +that list and push, then remove the branch from the list again before merging. +Keep any job in it that publishes behind `if: github.ref_name == 'main'`, so the +run from the feature branch publishes nothing. + +# `.gitignore` and `.dockerignore` + +`.gitignore` should be comprehensive from the start: + +- OS files (`.DS_Store`) +- editor files (`.swp`, `*~`) +- in-repo agent scratch directories (`.claude/`) +- `node_modules/` +- `.yarn/*`, with `!.yarn/releases` so that git keeps yarn's release file +- the repo's own build outputs + +Fetch the standard `.gitignore` from +`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up a +new repo. A repo's `.gitignore` is the standard file followed by the repo's own +entries, such as its binaries. A re-vendor replaces the standard part and keeps +those entries. + +**`.dockerignore` does not use `.gitignore` semantics, and copying patterns +across unmodified leaves secrets in the build context.** The standard +`.gitignore` patterns are written to `.gitignore`'s own semantics, in which an +unanchored pattern already matches at every depth. They must not be transplanted +into a `.dockerignore` unmodified. Docker matches with `moby/patternmatcher`: +`filepath.Match` semantics plus a `**` extension. So `*` does not cross `/`, and +a pattern without a leading `**/` is anchored at the build-context root. + +- A `.dockerignore` listing `.env`, `*.pem` and `*.key` therefore excludes only + the copies at the repository root. `config/.env` and `certs/server.key` still + reach the context and can land in an image layer. That is worse than a file + with no secret patterns at all, because it reads as solved and stops anyone + looking. +- Give every depth-independent pattern the `**/` prefix, and leave only + genuinely root-anchored entries unprefixed. Those are `.claude`, `.yarn/*`, `!.yarn/releases`, and the repo's own host-built binary. The binary is written `/myapp` and never `**/myapp`, which would also match `cmd/myapp/` and delete - the package directory from the context. Matching is case-sensitive, and an - all-caps copy of each pattern still misses `Server.Key`. So secret names use - character ranges: `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`, and likewise for - `.envrc` and the extensionless SSH keys. Where such a pattern also catches - something the build needs, re-include it with a negation - (`!docs/example.env`). Deleting the pattern reopens the exposure for every - other file it covers. Fetch the standard `.dockerignore` from + the package directory from the context. +- Matching is case-sensitive, and an all-caps copy of each pattern still misses + `Server.Key`. So secret names use character ranges: `**/*.[kK][eE][yY]`, + `**/*.[pP][eE][mM]`, and likewise for `.envrc` and the extensionless SSH keys. +- Where such a pattern also catches something the build needs, re-include it + with a negation (`!docs/example.env`). Deleting the pattern reopens the + exposure for every other file it covers. +- Fetch the standard `.dockerignore` from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend it with the repo's own artifacts. -- **In-repo agent scratch belongs in both files, written to each file's own - semantics.** `.claude/` holds one worktree per in-flight agent, an entire - additional checkout of the repo. Under `COPY . .` the build context therefore - inflates by a multiple of the repo, and another session's unreviewed work can - be copied into an image layer. In `.gitignore` the entry is `.claude/`, - unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/` - prefix, because the prefixed form would also delete any nested directory of - that name from the build. Anchoring carries a known gap. The canonical - `.dockerignore` states the gap in its own comment, since consuming repos - receive the file and not the tracker. The gap is that the directory is created - in the agent's working directory. A repo running agents in subdirectories - therefore still ships `services/api/.claude/` and must add its own anchored - entry there. +**In-repo agent scratch belongs in both files, written to each file's own +semantics.** `.claude/` holds one worktree per in-flight agent, an entire +additional checkout of the repo. Under `COPY . .` the build context therefore +inflates by a multiple of the repo, and another session's unreviewed work can be +copied into an image layer. -- **A plain `docker build .` of a clone stamps the version that - `git describe --tags --always` gives.** The version is derived from the `.git` - in the build context, as the canonical `Dockerfile` above shows. Without its - failure check, a missing `git` or an unreadable checkout would leave - `-X main.Version=` empty, and the build would still exit 0. `script/docker` - and `script/cibuild` pass the version they compute on the host; it takes - precedence. They do this byte-identically across repos: +- In `.gitignore` the entry is `.claude/`, unanchored. +- In `.dockerignore` it is `.claude`, anchored and with **no** `**/` prefix, + because the prefixed form would also delete any nested directory of that name + from the build. +- Anchoring carries a known gap, which the canonical `.dockerignore` states in + its own comment. The directory is created in the agent's working directory. A + repo running agents in subdirectories therefore still ships + `services/api/.claude/` and must add its own anchored entry there. - ```sh - # The version and the tag each get their own line: a failing command - # substitution inside an argument does not trip `set -e`, so the inline - # form degrades to an empty constant. - version="$(git describe --tags --always --dirty 2>/dev/null || true)" - [ -n "$version" ] || version="unknown" - tag="$(script/projectname)" - docker build --build-arg CACHEBUST="$(date +%s)-$$" \ - --build-arg VERSION="$version" \ - -t "$tag" . - ``` +**Verify `.dockerignore` by enumerating the image, not by reading the +patterns.** Plant files at the root _and_ at least two directories deep. Build a +probe image that does `COPY . .`, and list what actually landed +(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context` +size is not a substitute: a nested secret is a few bytes, and BuildKit transfers +only the delta from the previous build. - `--always` makes an untagged repo yield an abbreviated commit hash rather - than failing. The `[ -n "$version" ]` line is the single place the fallback - is applied. That line is a live check that fires on a build from an export - with no `.git`, and on a repository with no commits yet. Do not fold it into - the substitution as `|| echo unknown`, which makes the guard unreachable. - The Dockerfile's side is `ARG VERSION` in the stage that compiles, declared - there because `ARG` is stage-scoped. Passing `VERSION` to a repo whose - Dockerfile declares no such `ARG` is ignored and costs nothing, which is why - the scripts stay byte-identical. One consequence for CI: the standard - checkout action clones shallow and fetches no tags, so the canonical - `.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step. +# `.golangci.yml` -- **Verify `.dockerignore` by enumerating the image, not by reading the - patterns.** Plant files at the root _and_ at least two directories deep. Build - a probe image that does `COPY . .`, and list what actually landed - (`docker run --rm --entrypoint find IMAGE /app`). The `transferring context` - size is not a substitute: a nested secret is a few bytes, and BuildKit - transfers only the delta from the previous build. +`.golangci.yml` is standardized. The vendored copy in a consuming repo must +_NEVER_ be modified by an agent. Fetch it from +`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it +byte-identical, so that no repo can quietly loosen its own linting. -- **No build artifacts in version control.** Code-derived data must never be - committed to the repository if it can be avoided. Code-derived data includes: - - compiled bundles - - minified output - - generated assets - - The build process (e.g. Dockerfile, Makefile) should generate these at build - time. Notable exception: Go protobuf generated files (`.pb.go`) ARE - committed because repos need to work with `go get`, which downloads code but - does not execute code generation. - -- Never use `git add -A` or `git add .`. Always stage files explicitly by name. - -- Never force-push to `main`. - -- Make all changes on a feature branch. You can do whatever you want on a - feature branch. - -- `.golangci.yml` is standardized. The vendored copy in a consuming repo must - _NEVER_ be modified by an agent. Fetch it from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it - byte-identical, so that no repo can quietly loosen its own linting. Linter - configuration changes are made to the canonical copy in the `prompts` repo and - reach consuming repos by re-vendoring. An agent may open a PR against - canonical, which only the user merges. One list is exempt from byte-identity, - because it cannot be written once for every repo: the `deny` list of the - `test-support` depguard rule. There a repo names its own test-support packages - by full import path. A repo adds entries there and changes nothing else, and a - re-vendor carries its entries forward. The canonical golangci-lint version is - v2.15.0 (released 2026-10-16). The version is pinned as the digest of the lint - phase's base image, +- Linter configuration changes are made to the canonical copy in the `prompts` + repo and reach consuming repos by re-vendoring. An agent may open a PR against + canonical, which only the user merges. +- One list is exempt from byte-identity, because it cannot be written once for + every repo: the `deny` list of the `test-support` depguard rule. There a repo + names its own test-support packages by full import path. A repo adds entries + there and changes nothing else, and a re-vendor carries its entries forward. +- The canonical golangci-lint version is v2.15.0 (released 2026-10-16). The + version is pinned as the digest of the lint phase's base image, `golangci/golangci-lint@sha256:fbfcfd12a6f8d7f1b97b3362ca80fc2a244d46863148c73cb88e7793af96c481`, - which reports `2.15.0 built with go1.27.0 from 25c5369a`. A module's `go` - directive must not name a newer Go minor version than the one golangci-lint - was built with, or golangci-lint refuses to lint it. This release lints - `go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo - installs golangci-lint on the host. A repo sets the lint phase digest to the - one named here and re-vendors `.golangci.yml` in the same commit, whichever of - the two prompted the change. The canonical copy can name linters that an older - golangci-lint rejects. A newer golangci-lint can add linters that - `default: all` switches on until the canonical copy disables them. + which reports `2.15.0 built with go1.27.0 from 25c5369a`. That digest is the + only pin, since no repo installs golangci-lint on the host. +- A module's `go` directive must not name a newer Go minor version than the one + golangci-lint was built with, or golangci-lint refuses to lint it. This + release lints `go 1.27.1` but not `go 1.28`. +- A repo sets the lint phase digest to the one named here and re-vendors + `.golangci.yml` in the same commit, whichever of the two prompted the change. + The canonical copy can name linters that an older golangci-lint rejects. A + newer golangci-lint can add linters that `default: all` switches on until the + canonical copy disables them. -- **`script/bootstrap` installs a pinned tool by comparing versions, never by - testing presence.** An `if ! command -v ; then install; fi` guard tests - `PATH` only. On an already-provisioned machine the pin is then inert, and a - version bump is a silent no-op. The Dockerfile, installing into a clean image, - gets the pinned version, so a local `make check` and `make docker` can - disagree about what the tool even is. The canonical form: - - compares the installed version against the pin over the **whole** version - token. A parser that stops at the first `-` reports `2.12.2` for a host - running `2.12.2-rc1` and skips the install. - - treats `--version` output as a mismatch when it is any of these: - - absent - - non-zero - - empty - - unrecognised +# Formatting - So a failure causes a redundant install, never a skipped one. +Use platform-standard formatters: `black` for Python, `prettier` for +JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration, with +two exceptions: - - after installing, re-resolves the binary the way callers do, with - `hash -r` and then through `PATH`, not through the directory the installer - wrote to. A shadowing binary can hide an install, which then succeeds - while no caller sees any change. In that case it fails and names the - resolved path. - - is actually called, and prints the version on both success paths. A - function defined and never invoked has the same exit status and the same - empty output as one that worked. +- four-space indents (except Go) +- `proseWrap: always` for Markdown (hard-wrap at 80 columns) - A tool from the distribution has no pin, so `script/bootstrap` installs it - when it is missing. make, git, node and yarn are such tools. +Documentation and writing repos (Markdown, HTML, CSS) should also have +`.prettierrc` and `.prettierignore`. - Keep it POSIX sh: no arrays, no `[[`, no `grep -P`. +All repos should have an `.editorconfig` enforcing the project's indentation +settings. It is the standard file from +`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, followed by +the repo's own sections, such as one for another language it uses. The standard +file sets tabs for `Makefile` and Go files. A re-vendor replaces the standard +part and keeps those sections. - A Go tool a repo needs on the host is installed with `go install` pinned to - a commit hash (`go install @`). It is never tracked as - a `go.mod` tool dependency or through a `tools.go` file, either of which - pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`. +# Web services -- When pinning images or packages by hash, add a comment above the reference - with the version and date (YYYY-MM-DD). +Dockerized web services listen on port 8080 by default, overridable with `PORT`. -- Use `yarn`, not `npm`. +**HTTP/web services must be hardened for production internet exposure before +tagging 1.0.** This means full compliance with security best practices +including, without limitation, all of the following: + +- **Security headers** on every response: + - `Strict-Transport-Security` (HSTS) with `max-age` of at least one year and + `includeSubDomains`. + - `Content-Security-Policy` (CSP) with a restrictive default policy + (`default-src 'self'` as a baseline, tightened per-resource as needed). + Never use `unsafe-inline` or `unsafe-eval` unless unavoidable, and + document the reason. + - `X-Frame-Options: DENY` (or `SAMEORIGIN` if framing is required). Prefer + the `frame-ancestors` CSP directive as the primary control. + - `X-Content-Type-Options: nosniff`. + - `Referrer-Policy: strict-origin-when-cross-origin` (or stricter). + - `Permissions-Policy` restricting access to browser features the + application does not use (camera, microphone, geolocation, etc.). +- **Request and response limits:** + - Maximum request body size enforced on all endpoints (e.g. Go + `http.MaxBytesReader`). Choose a sane default per-route; never accept + unbounded input. + - Maximum response body size where applicable (e.g. paginated APIs). + - `ReadTimeout` and `ReadHeaderTimeout` on the `http.Server` to defend + against slowloris attacks. + - `WriteTimeout` and `IdleTimeout` on the `http.Server`. + - Per-handler execution time limits via `context.WithTimeout` or chi/stdlib + `middleware.Timeout`. +- **Authentication and session security:** + - Rate limiting on password-based authentication endpoints. API keys are + high-entropy and not susceptible to brute force, so they are exempt. + - CSRF tokens on all state-mutating HTML forms. API endpoints authenticated + via `Authorization` header (Bearer token, API key) are exempt because the + browser does not attach these automatically. + - Passwords stored using bcrypt, scrypt, or argon2 — never plain-text, MD5, + or SHA. + - Session cookies set with `HttpOnly`, `Secure`, and `SameSite=Lax` (or + `Strict`) attributes. +- **Reverse proxy awareness:** True client IP detection when behind a reverse + proxy (`X-Forwarded-For`, `X-Real-IP`). The application must accept forwarded + headers only from a configured set of trusted proxy addresses, never + unconditionally. +- **CORS:** Authenticated endpoints must restrict `Access-Control-Allow-Origin` + to an explicit allowlist of known origins. Wildcard (`*`) is acceptable only + for public, unauthenticated read-only APIs. +- **Error handling:** Internal errors must never leak implementation details to + the client: stack traces, SQL queries, file paths. Return generic error + messages in production; detailed errors only when `DEBUG` is enabled. +- **TLS:** Services never terminate TLS directly. They are always deployed + behind a TLS-terminating reverse proxy. The service itself listens on plain + HTTP. However, HSTS headers and `Secure` cookie flags must still be set by the + application so that the browser enforces HTTPS end-to-end. + +Apply defense-in-depth: if a standard security hardening measure exists for HTTP +services and is not listed here, it is still expected. When in doubt, harden. + +# Database migrations + +Database migrations live in `internal/db/migrations/` and must be embedded in +the binary. + +- `000_migration.sql` — contains ONLY the creation of the migrations tracking + table itself. Nothing else. +- `001_schema.sql` — the full application schema. +- **Pre-1.0.0:** never add additional migration files (002, 003, etc.). There is + no installed base to migrate. Edit `001_schema.sql` directly. +- **Post-1.0.0:** add new numbered migration files for each schema change. Never + edit existing migrations after release. + +# README + +`README.md` is the primary documentation. Required sections: + +- **Description**: First line must include the project name, purpose, category + (web server, SPA, CLI tool, etc.), license and author. Example: "µPaaS is an + MIT-licensed Go web application by @sneak that receives git-frontend webhooks + and deploys applications via Docker in realtime." +- **Getting Started**: Copy-pasteable install/usage code block. +- **Entrypoints**: Opens by stating that the repo adheres to the + [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) + standard (with that link), then documents each provided `script/` entrypoint + and its purpose. +- **Rationale**: Why does this exist? +- **Design**: How is the program structured? +- **TODO**: Update meticulously, even between commits. When planning, put the + todo list in the README so a new agent can pick up where the last one left + off. +- **License**: MIT, GPL, or WTFPL. Ask the user for new projects. Include a + `LICENSE` file in the repo root. +- **Author**: [@sneak](https://sneak.berlin). + +# Repository layout + +Avoid putting files in the repo root unless necessary. Root should contain only +project-level config files: `README.md`, `AGENTS.md`, `Makefile`, `Dockerfile`, +`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md` and +language-specific config. Everything else goes in a subdirectory. Canonical +subdirectory names: + +- `bin/` — executable scripts and tools +- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose + body is a single call into `internal/` or `pkg/`, no project logic in `cmd/` +- `configs/` — configuration templates and examples +- `deploy/` — deployment manifests (k8s, compose, terraform) +- `docs/` — documentation and markdown (README.md stays in root) +- `internal/` — Go internal packages +- `internal/db/migrations/` — database migrations +- `pkg/` — Go library packages +- `share/` — systemd units, data files +- `static/` — static assets (images, fonts, etc.) +- `web/` — web frontend source + +Guidance for coding agents lives in one `AGENTS.md` at the repository root. It +is never committed under a file or directory named after one agent tool, such as +`CLAUDE.md` or `.claude/`, and never split into separate memory files. + +New repos must contain at minimum: + +- `README.md`, `.git`, `.gitignore`, `.editorconfig` +- `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo) +- `Makefile` +- `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`, `lint`, + `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, + `install-precommit`) +- `Dockerfile`, `.dockerignore` +- `.gitea/workflows/check.yml` +- Go: `go.mod`, `go.sum`, `.golangci.yml` +- JS: `package.json`, `yarn.lock`, `.yarnrc.yml`, `.yarn/releases/`, + `.gitattributes`, `.prettierrc`, `.prettierignore` +- Python: `pyproject.toml` + +When setting up a new repo, files from the `prompts` repo may be used as +templates. Fetch them from +`https://git.eeqj.de/sneak/prompts/raw/branch/main/`. First commit of a +new repo should contain only `README.md`. + +# Other conventions - Write all dates as YYYY-MM-DD (ISO 8601). - -- Simple projects should be configured with environment variables. - -- Dockerized web services listen on port 8080 by default, overridable with - `PORT`. - -- **HTTP/web services must be hardened for production internet exposure before - tagging 1.0.** This means full compliance with security best practices - including, without limitation, all of the following: - - **Security headers** on every response: - - `Strict-Transport-Security` (HSTS) with `max-age` of at least one year - and `includeSubDomains`. - - `Content-Security-Policy` (CSP) with a restrictive default policy - (`default-src 'self'` as a baseline, tightened per-resource as - needed). Never use `unsafe-inline` or `unsafe-eval` unless - unavoidable, and document the reason. - - `X-Frame-Options: DENY` (or `SAMEORIGIN` if framing is required). - Prefer the `frame-ancestors` CSP directive as the primary control. - - `X-Content-Type-Options: nosniff`. - - `Referrer-Policy: strict-origin-when-cross-origin` (or stricter). - - `Permissions-Policy` restricting access to browser features the - application does not use (camera, microphone, geolocation, etc.). - - **Request and response limits:** - - Maximum request body size enforced on all endpoints (e.g. Go - `http.MaxBytesReader`). Choose a sane default per-route; never accept - unbounded input. - - Maximum response body size where applicable (e.g. paginated APIs). - - `ReadTimeout` and `ReadHeaderTimeout` on the `http.Server` to defend - against slowloris attacks. - - `WriteTimeout` on the `http.Server`. - - `IdleTimeout` on the `http.Server`. - - Per-handler execution time limits via `context.WithTimeout` or - chi/stdlib `middleware.Timeout`. - - **Authentication and session security:** - - Rate limiting on password-based authentication endpoints. API keys are - high-entropy and not susceptible to brute force, so they are exempt. - - CSRF tokens on all state-mutating HTML forms. API endpoints - authenticated via `Authorization` header (Bearer token, API key) are - exempt because the browser does not attach these automatically. - - Passwords stored using bcrypt, scrypt, or argon2 — never plain-text, - MD5, or SHA. - - Session cookies set with `HttpOnly`, `Secure`, and `SameSite=Lax` (or - `Strict`) attributes. - - **Reverse proxy awareness:** - - True client IP detection when behind a reverse proxy - (`X-Forwarded-For`, `X-Real-IP`). The application must accept - forwarded headers only from a configured set of trusted proxy - addresses. Never trust `X-Forwarded-For` unconditionally. - - **CORS:** - - Authenticated endpoints must restrict `Access-Control-Allow-Origin` to - an explicit allowlist of known origins. Wildcard (`*`) is acceptable - only for public, unauthenticated read-only APIs. - - **Error handling:** - - Internal errors must never leak implementation details to the client. - Those details include: - - stack traces - - SQL queries - - file paths - - Return generic error messages in production; detailed errors only - when `DEBUG` is enabled. - - - **TLS:** - - Services never terminate TLS directly. They are always deployed behind - a TLS-terminating reverse proxy. The service itself listens on plain - HTTP. However, HSTS headers and `Secure` cookie flags must still be - set by the application so that the browser enforces HTTPS end-to-end. - - This list is non-exhaustive. Apply defense-in-depth: if a standard security - hardening measure exists for HTTP services and is not listed here, it is - still expected. When in doubt, harden. - -- `README.md` is the primary documentation. Required sections: - - **Description**: First line must include: - - the project name - - purpose - - category (web server, SPA, CLI tool, etc.) - - license - - author - - Example: "µPaaS is an MIT-licensed Go web application by @sneak that - receives git-frontend webhooks and deploys applications via Docker in - realtime." - - - **Getting Started**: Copy-pasteable install/usage code block. - - **Entrypoints**: Opens by stating that the repo adheres to the - [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) - standard (with that link), then documents each provided `script/` - entrypoint and its purpose. - - **Rationale**: Why does this exist? - - **Design**: How is the program structured? - - **TODO**: Update meticulously, even between commits. When planning, put - the todo list in the README so a new agent can pick up where the last one - left off. - - **License**: MIT, GPL, or WTFPL. Ask the user for new projects. Include a - `LICENSE` file in the repo root and a License section in the README. - - **Author**: [@sneak](https://sneak.berlin). - -- First commit of a new repo should contain only `README.md`. - +- Cross-project documentation (such as this file) must include + `last_modified: YYYY-MM-DD` in the YAML front matter so it can be kept in sync + with the authoritative source as policies evolve. +- Use SemVer. - Go module root: `sneak.berlin/go/`. Always run `go mod tidy` before committing. - -- Use SemVer. - -- Database migrations live in `internal/db/migrations/` and must be embedded in - the binary. - - `000_migration.sql` — contains ONLY the creation of the migrations - tracking table itself. Nothing else. - - `001_schema.sql` — the full application schema. - - **Pre-1.0.0:** never add additional migration files (002, 003, etc.). - There is no installed base to migrate. Edit `001_schema.sql` directly. - - **Post-1.0.0:** add new numbered migration files for each schema change. - Never edit existing migrations after release. - -- All repos should have an `.editorconfig` enforcing the project's indentation - settings. It is the standard file from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, followed by - the repo's own sections, such as one for another language it uses. The - standard file sets tabs for `Makefile` and Go files. A re-vendor replaces the - standard part and keeps those sections. - -- Avoid putting files in the repo root unless necessary. Root should contain - only project-level config files: - - `README.md` - - `AGENTS.md` - - `Makefile` - - `Dockerfile` - - `LICENSE` - - `.gitignore` - - `.editorconfig` - - `REPO_POLICIES.md` - - language-specific config - - Everything else goes in a subdirectory. Canonical subdirectory names: - - `bin/` — executable scripts and tools - - `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose - body is a single call into `internal/` or `pkg/`, no project logic in - `cmd/` - - `configs/` — configuration templates and examples - - `deploy/` — deployment manifests (k8s, compose, terraform) - - `docs/` — documentation and markdown (README.md stays in root) - - `internal/` — Go internal packages - - `internal/db/migrations/` — database migrations - - `pkg/` — Go library packages - - `share/` — systemd units, data files - - `static/` — static assets (images, fonts, etc.) - - `web/` — web frontend source - -- When setting up a new repo, files from the `prompts` repo may be used as - templates. Fetch them from - `https://git.eeqj.de/sneak/prompts/raw/branch/main/`. - -- New repos must contain at minimum: - - `README.md`, `.git`, `.gitignore`, `.editorconfig` - - `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo) - - `Makefile` - - `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`, - `lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, - `install-precommit`) - - `Dockerfile`, `.dockerignore` - - `.gitea/workflows/check.yml` - - Go: `go.mod`, `go.sum`, `.golangci.yml` - - JS: `package.json`, `yarn.lock`, `.yarnrc.yml`, `.yarn/releases/`, - `.gitattributes`, `.prettierrc`, `.prettierignore` - - Python: `pyproject.toml` - -- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It - is never committed under a file or directory named after one agent tool, such - as `CLAUDE.md` or `.claude/`, and never split into separate memory files. +- Simple projects should be configured with environment variables. -- 2.54.0 From 2e98be476aa68baeb992ff2d89499f6f06f02193 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Fri, 6 Nov 2026 01:25:27 +0000 Subject: [PATCH 2/3] REPO_POLICIES: fix the review findings on the compaction (#158) Fixes the findings of the review of https://git.eeqj.de/sneak/prompts/pulls/159. - CI: "The workflow sets these:", so the four settings bind every repo's workflow again. - node and yarn: a stage that needs no yarn copies no `yarn.lock` before it runs `script/bootstrap`, which is what the examples do. - Dockerfile: names the stage to keep last, the one that ships. - Version: "Without that check" replaces "Otherwise". - The lists the review named are bulleted again, with the wording on `next`. The submodule examples are in their own sentence, as in both checklists. - `TODO.md`: the entry carries the new size, 1,062 lines and 7,794 words. Model: fable-5-1 --- TODO.md | 2 +- prompts/REPO_POLICIES.md | 126 +++++++++++++++++++++++++++------------ 2 files changed, 89 insertions(+), 39 deletions(-) diff --git a/TODO.md b/TODO.md index acb2991..12caf64 100644 --- a/TODO.md +++ b/TODO.md @@ -22,7 +22,7 @@ fmt-check, and commit. # Completed Steps - 2026-11-06: Compacted `prompts/REPO_POLICIES.md` from 1,145 lines and 8,427 - words to 1,012 lines and 7,709 words + words to 1,062 lines and 7,794 words (https://git.eeqj.de/sneak/prompts/issues/158). Every rule keeps its meaning. The rules now sit under headings, the pinning rule first. Each rule that was stated in several places is stated once. The Makefile copies of the Go and diff --git a/prompts/REPO_POLICIES.md b/prompts/REPO_POLICIES.md index f2f5ad7..7a1a01e 100644 --- a/prompts/REPO_POLICIES.md +++ b/prompts/REPO_POLICIES.md @@ -73,11 +73,15 @@ possible: - Never force-push to `main`. - Never use `git add -A` or `git add .`. Always stage files explicitly by name. - **No build artifacts in version control.** Code-derived data must never be - committed if it can be avoided: compiled bundles, minified output, generated - assets. The build process (e.g. Dockerfile, Makefile) should generate these at - build time. Notable exception: Go protobuf generated files (`.pb.go`) ARE - committed, because repos need to work with `go get`, which downloads code but - does not execute code generation. + committed to the repository if it can be avoided. Code-derived data includes: + - compiled bundles + - minified output + - generated assets + + The build process (e.g. Dockerfile, Makefile) should generate these at build + time. Notable exception: Go protobuf generated files (`.pb.go`) ARE + committed, because repos need to work with `go get`, which downloads code + but does not execute code generation. # Entrypoints @@ -206,8 +210,9 @@ present. Use `yarn`, not `npm`. - `script/bootstrap` installs node, yarn and the node packages only when - `yarn.lock` is in the tree. A `Dockerfile` stage that needs no yarn therefore - copies no `yarn.lock`. The Go and Python stages below are such stages. + `yarn.lock` is in the tree. A `Dockerfile` stage that needs no yarn copies no + `yarn.lock` before it runs `script/bootstrap`, so `script/bootstrap` installs + no node and no yarn there. The Go and Python stages below are such stages. - It installs the node packages with `yarn install --immutable`, which fails when a package's `checksum` is wrong, or when `package.json` is out of step with `yarn.lock`. @@ -250,8 +255,14 @@ what the tool even is. The canonical form: - compares the installed version against the pin over the **whole** version token. A parser that stops at the first `-` reports `2.12.2` for a host running `2.12.2-rc1` and skips the install. -- treats `--version` output that is absent, non-zero, empty or unrecognised as a - mismatch. A failure then causes a redundant install, never a skipped one. +- treats `--version` output as a mismatch when it is any of these: + - absent + - non-zero + - empty + - unrecognised + + So a failure causes a redundant install, never a skipped one. + - after installing, re-resolves the binary the way callers do: `hash -r`, then `PATH`, not the directory the installer wrote to. A shadowing binary can hide an install that reported success. In that case it fails and names the resolved @@ -289,7 +300,8 @@ docker build --build-arg CACHEBUST="$(date +%s)-$$" \ two gates are always invoked by name here, and why the final stage carries a `COPY --from=` of a harmless file from each of them. Without those copies, a plain `docker build .` builds the last stage's chain alone and exits 0 having - linted and tested nothing. Keep the final stage last. If you add a stage after + linted and tested nothing. Keep the stage that ships last: the runtime image, + or the development environment in a non-server repo. If you add a stage after it, give it the same two copies. - **The gate builds write no image.** With `--output type=cacheonly` the phase runs and a failing step fails the build, but the result is not exported. @@ -352,8 +364,12 @@ step that also installs anything else comes after `ARG CACHEBUST` too. A `script/bootstrap` that also installs node packages is one. - A new value reruns every `RUN` step after that line; `COPY` steps are compared - by content and need nothing. Docker keeps the layers above the line until the - base image, their own lines or the files they copy change. + by content and need nothing. Docker keeps the layers above the line until one + of these changes: + - the base image + - their own lines + - the files they copy + - A stage without the `ARG` line is served from cache on an unchanged tree, so a repository re-vendors the scripts and adds the `ARG` lines in one commit. - The scripts do not pass `--no-cache`. That flag reruns every step too, but it @@ -529,17 +545,26 @@ Key points: above `ARG CACHEBUST`, so check builds keep them.** The steps are `COPY script/ script/`, `COPY Pipfile Pipfile.lock ./` and `RUN script/bootstrap`, with `ARG CACHEBUST` right after them. Docker keeps what -that `RUN` installed until the base image, `script/`, `Pipfile`, `Pipfile.lock` -or a line up to and including the `RUN` changes. +that `RUN` installed until one of these changes: + +- the base image +- `script/` +- `Pipfile` +- `Pipfile.lock` +- a line up to and including the `RUN` The tests run after `ARG CACHEBUST`, on every check build. In every other stage, a step that installs Python packages comes after `ARG CACHEBUST`. In that step, `script/bootstrap`: -- installs nothing beyond system packages, pipenv and its dependencies, and the - packages from `Pipfile.lock`. If it installs anything else there, the step - goes after `ARG CACHEBUST`. +- installs nothing beyond these: + - system packages + - pipenv and its dependencies + - the packages from `Pipfile.lock` + + If it installs anything else there, the step goes after `ARG CACHEBUST`. + - installs no Python: the tests use the distribution's Python, which a step above it installs. - puts pipenv in a virtual environment at `$HOME/.pipenv`, because Debian's @@ -553,9 +578,11 @@ In that step, `script/bootstrap`: that lists each of them with `==` and its `--hash`. pip refuses to install a dependency the file does not list. - then runs `pipenv install --deploy`, with `--dev` when the tests need the - development packages. `--deploy` installs only what `Pipfile.lock` names, - checks every package against the lockfile's hashes, and fails when the - lockfile is out of date with `Pipfile`. + development packages. `--deploy` does all of these: + - installs only what `Pipfile.lock` names + - checks every package against the lockfile's hashes + - fails when the lockfile is out of date with `Pipfile` + - sets `PIP_ONLY_BINARY=:all:` for both installs, and pipenv passes it on to pip. A package with no ready-built wheel for the image then stops the install. Without the setting, pip builds that package from source and downloads the @@ -658,8 +685,9 @@ canonical Go `Dockerfile` above shows: stream keeps the sender's file owners, and git refuses a checkout owned by another user, so the version would come out empty. - The build fails if the context carries `.git` and the version still comes out - empty, `dev` or `unknown`. Otherwise a missing `git` or an unreadable checkout - would leave `-X main.Version=` empty, and the build would still exit 0. + empty, `dev` or `unknown`. Without that check, a missing `git` or an + unreadable checkout would leave `-X main.Version=` empty, and the build would + still exit 0. - A plain `docker build .` with no build arguments must succeed; a Dockerfile that refuses an empty build argument drops that refusal and keeps the argument. @@ -677,11 +705,11 @@ canonical Go `Dockerfile` above shows: - that of a submodule keeping its own `.git` directory `git describe` does not need them, and each can hold a credential: a password in -a remote URL, or the token the CI checkout step stores there. A submodule whose -name has a `config` segment (`config`, `deploy/config`, `config/lib`) loses its -whole git directory to `**/.git/modules/**/config`, and Go's version stamping -then fails the build. Give it a name without that segment -(`git submodule add --name`). +a remote URL, or the token the CI checkout step stores there. Some submodule +names have a `config` segment, as in `config`, `deploy/config` or `config/lib`. +Such a submodule loses its whole git directory to `**/.git/modules/**/config`, +and Go's version stamping then fails the build. Give it a name without that +segment (`git submodule add --name`). `script/docker` and `script/cibuild` pass the version they compute on the host; it takes precedence. They do this byte-identically across repos: @@ -711,7 +739,7 @@ docker build --build-arg CACHEBUST="$(date +%s)-$$" \ Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that runs `script/cibuild` on push. The workflow's only other step checks out the repo. -The canonical `.gitea/workflows/check.yml` sets these: +The workflow sets these: - `persist-credentials: false` on the checkout step. `script/cibuild` needs no token, and without that setting the checkout leaves the job's token in @@ -910,8 +938,14 @@ including, without limitation, all of the following: to an explicit allowlist of known origins. Wildcard (`*`) is acceptable only for public, unauthenticated read-only APIs. - **Error handling:** Internal errors must never leak implementation details to - the client: stack traces, SQL queries, file paths. Return generic error - messages in production; detailed errors only when `DEBUG` is enabled. + the client. Those details include: + - stack traces + - SQL queries + - file paths + + Return generic error messages in production; detailed errors only when + `DEBUG` is enabled. + - **TLS:** Services never terminate TLS directly. They are always deployed behind a TLS-terminating reverse proxy. The service itself listens on plain HTTP. However, HSTS headers and `Secure` cookie flags must still be set by the @@ -937,10 +971,17 @@ the binary. `README.md` is the primary documentation. Required sections: -- **Description**: First line must include the project name, purpose, category - (web server, SPA, CLI tool, etc.), license and author. Example: "µPaaS is an - MIT-licensed Go web application by @sneak that receives git-frontend webhooks - and deploys applications via Docker in realtime." +- **Description**: First line must include: + - the project name + - purpose + - category (web server, SPA, CLI tool, etc.) + - license + - author + + Example: "µPaaS is an MIT-licensed Go web application by @sneak that + receives git-frontend webhooks and deploys applications via Docker in + realtime." + - **Getting Started**: Copy-pasteable install/usage code block. - **Entrypoints**: Opens by stating that the repo adheres to the [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) @@ -958,10 +999,19 @@ the binary. # Repository layout Avoid putting files in the repo root unless necessary. Root should contain only -project-level config files: `README.md`, `AGENTS.md`, `Makefile`, `Dockerfile`, -`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md` and -language-specific config. Everything else goes in a subdirectory. Canonical -subdirectory names: +project-level config files: + +- `README.md` +- `AGENTS.md` +- `Makefile` +- `Dockerfile` +- `LICENSE` +- `.gitignore` +- `.editorconfig` +- `REPO_POLICIES.md` +- language-specific config + +Everything else goes in a subdirectory. Canonical subdirectory names: - `bin/` — executable scripts and tools - `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose -- 2.54.0 From f01ca24dc2ea9042cdc8a3adc6a655d0c5f89a98 Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Fri, 6 Nov 2026 01:53:49 +0000 Subject: [PATCH 3/3] REPO_POLICIES: fix the second review's findings on the compaction (#158) Fixes the seven findings of https://git.eeqj.de/sneak/prompts/pulls/159#issuecomment-169529, with the wording each one gives. - Go key points: "whatever files the mount holds". - `.dockerignore`: "Such a `.dockerignore` is more dangerous than a short file with no secret patterns at all". - Python: "pipenv checked the hashes of the lockfile's packages". - Python: "In that step" leads only the two bullets about that step. The other four sit under "A Python repository's `script/bootstrap`:". - Entrypoints: "Three more scripts have no required target:". - Dockerfile: the two ways a stage installs what its image lacks are bulleted again, as on `next`. - Version: the sentence ends at "stage-scoped", then "It has no default." - `TODO.md`: the entry carries the new size, 1,066 lines and 7,812 words. Model: fable-5-1 --- TODO.md | 2 +- prompts/REPO_POLICIES.md | 38 +++++++++++++++++++++----------------- 2 files changed, 22 insertions(+), 18 deletions(-) diff --git a/TODO.md b/TODO.md index 12caf64..60db56a 100644 --- a/TODO.md +++ b/TODO.md @@ -22,7 +22,7 @@ fmt-check, and commit. # Completed Steps - 2026-11-06: Compacted `prompts/REPO_POLICIES.md` from 1,145 lines and 8,427 - words to 1,062 lines and 7,794 words + words to 1,066 lines and 7,812 words (https://git.eeqj.de/sneak/prompts/issues/158). Every rule keeps its meaning. The rules now sit under headings, the pinning rule first. Each rule that was stated in several places is stated once. The Makefile copies of the Go and diff --git a/prompts/REPO_POLICIES.md b/prompts/REPO_POLICIES.md index 7a1a01e..207d20c 100644 --- a/prompts/REPO_POLICIES.md +++ b/prompts/REPO_POLICIES.md @@ -103,7 +103,7 @@ where its implementation lives: - `make docker`: `script/docker` - `make hooks`: `script/install-precommit` installs the git pre-commit hook. -Three more scripts have no target: +Three more scripts have no required target: - `script/precommit` is what the pre-commit hook runs. It calls `script/check`. Repo-type-specific extras belong in it, not in the hook itself; `go mod tidy` @@ -328,9 +328,10 @@ docker build --build-arg CACHEBUST="$(date +%s)-$$" \ What each stage installs: - The gate phases and the build stage start from their pinned base images. They - install what those images lack inline, as the canonical Go `Dockerfile` below - does for `git`. Or they run `script/bootstrap`, as the `prompts` repo's own - `Dockerfile` does for its yarn packages. + install what those images lack in either of these ways: + - inline, as the canonical Go `Dockerfile` below does for `git` + - by running `script/bootstrap`, as the `prompts` repo's own `Dockerfile` + does for its yarn packages - The development environment stage installs development prerequisites by running `script/bootstrap`. Only system packages may be installed inline above it, so that check builds keep them; the `prompts` repo's own `Dockerfile` does @@ -498,9 +499,9 @@ Key points: - the linters and their settings in `.golangci.yml` - the golangci-lint version - The modules are kept in a layer rather than a cache mount. A cache mount is - shared by every build on the host, and Go builds from whatever files it holds - without checking them again (only `go mod verify` does). A file changed there - would go unnoticed. + shared by every build on the host, and Go builds from whatever files the mount + holds without checking them again (only `go mod verify` does). A file changed + there would go unnoticed. - The test command keeps the first run's output, to look in it for what Go prints for a hang. `panic: test timed out` means the test's own timer fired. `*** Test killed` means `go test` stopped a test that ignored that timer, @@ -567,6 +568,9 @@ In that step, `script/bootstrap`: - installs no Python: the tests use the distribution's Python, which a step above it installs. + +A Python repository's `script/bootstrap`: + - puts pipenv in a virtual environment at `$HOME/.pipenv`, because Debian's Python refuses a system-wide `pip install`. It creates the environment with `python3 -m venv` when it is missing, then uses that environment's `pip` and @@ -592,12 +596,12 @@ In that step, `script/bootstrap`: when the distribution's Python differs from it. A kept layer holds nothing a new install would refuse. pip checked the hashes of -pipenv and its dependencies, and pipenv those of the lockfile's packages. pipenv -creates its virtual environment with the `virtualenv` package. The packages that -`virtualenv` adds come from wheels inside it, so the hash pip checked for it -covers them. System packages, Python among them, are not fixed by a hash. They -keep the versions the package manager gave when the layer was built; a new -install may get newer ones. +pipenv and its dependencies, and pipenv checked the hashes of the lockfile's +packages. pipenv creates its virtual environment with the `virtualenv` package. +The packages that `virtualenv` adds come from wheels inside it, so the hash pip +checked for it covers them. System packages, Python among them, are not fixed by +a hash. They keep the versions the package manager gave when the layer was +built; a new install may get newer ones. As with Go's modules, the packages are kept in a layer, not a cache mount, which every build on the host shares and nothing checks again. @@ -677,7 +681,7 @@ canonical Go `Dockerfile` above shows: - It takes the version from the `VERSION` build argument when one is given, otherwise from `git describe`. `ARG VERSION` is declared in that stage, - because `ARG` is stage-scoped, and has no default. + because `ARG` is stage-scoped. It has no default. - It has `git`: the Debian Go image has it, and an alpine one needs `apk add --no-cache git`. - It marks its working directory safe for git @@ -802,9 +806,9 @@ a pattern without a leading `**/` is anchored at the build-context root. - A `.dockerignore` listing `.env`, `*.pem` and `*.key` therefore excludes only the copies at the repository root. `config/.env` and `certs/server.key` still - reach the context and can land in an image layer. That is worse than a file - with no secret patterns at all, because it reads as solved and stops anyone - looking. + reach the context and can land in an image layer. Such a `.dockerignore` is + more dangerous than a short file with no secret patterns at all, because it + reads as solved and stops anyone looking. - Give every depth-independent pattern the `**/` prefix, and leave only genuinely root-anchored entries unprefixed. Those are `.claude`, `.yarn/*`, `!.yarn/releases`, and the repo's own host-built binary. The binary is written -- 2.54.0