Author SHA1 Message Date
clawbot b5819282f3 Re-vendor the canonical files from sneak/prompts at dd4027b (closes #95)
check / check (push) Waiting to run
The shared files are the copies at sneak/prompts commit dd4027b, with this
repository's own entries kept after them. golangci-lint is v2.14.0 and
raises no findings. Lint and test are phases of the Dockerfile, built by
script/lint and script/test; the tests run under the race detector as
nobody, so Dockerfile.lint, script/verify-lint-image-pin and make
test-race are gone. Every docker build in script/ passes --no-cache in
place of the old cache-busting build argument, and script/cibuild runs
script/bootstrap and script/check before the image build.
.claude/settings.json is deleted.

Deviation: the workflow keeps fetch-depth: 0 for the tag-derived version.
Deviation: .gitignore keeps the scan database patterns.
Deviation: prettier still runs in Docker, not on the host.
Over the cap: make test takes about 60 seconds on this host.

Model: opus-5-5
2026-10-07 21:46:20 +00:00
25 changed files with 320 additions and 674 deletions
-3
View File
@@ -13,6 +13,3 @@ indent_style = tab
[*.go] [*.go]
indent_style = tab indent_style = tab
# This repository's own sections, such as one for another language it
# uses, go below this comment, and a re-vendor keeps them.
-9
View File
@@ -1,20 +1,11 @@
name: check name: check
on: [push] on: [push]
# Free the shared runner: a new push cancels only the same branch's older run.
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs: jobs:
check: check:
runs-on: ubuntu-latest runs-on: ubuntu-latest
# Free the shared runner from a hung build.
timeout-minutes: 20
steps: steps:
# actions/checkout v4.2.2, 2026-02-22 # actions/checkout v4.2.2, 2026-02-22
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
# script/cibuild needs no token, so none is left in .git/config.
with: with:
persist-credentials: false
# All history and tags, so git describe finds the version tag.
fetch-depth: 0 fetch-depth: 0
- run: script/cibuild - run: script/cibuild
+3 -5
View File
@@ -27,7 +27,7 @@ node_modules/
# Environment files. `*.env` covers bare `.env` and the `prod.env` # Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are # convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds # re-included below. A repository that commits any other template adds
# its own negation at the end of this file, for example `!.env.example`. # its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV] *.[eE][nN][vV]
.[eE][nN][vV].* .[eE][nN][vV].*
.[eE][nN][vV][rR][cC] .[eE][nN][vV][rR][cC]
@@ -46,13 +46,11 @@ node_modules/
[iI][dD]_[eE][dD]25519 [iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK] [iI][dD]_[eE][dD]25519_[sS][kK]
# This repository's own entries, such as its build outputs, go below # This repository's own entries, kept after the canonical content above.
# this comment, and a re-vendor keeps them. Anchor a binary built at the
# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
/sfdupes
*.log *.log
*.out *.out
*.test *.test
/sfdupes
# A scan database lists every path it scanned. # A scan database lists every path it scanned.
*.sqlite *.sqlite
-2
View File
@@ -25,8 +25,6 @@ linters:
# silenced by disabling that name, not by enabling the successor. # silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5 - wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2 - gomodguard # Deprecated, replaced by gomodguard_v2
# Misses findings at random in v2.14.0; back once a pinned release fixes it
- canonicalheader
settings: settings:
lll: lll:
line-length: 88 line-length: 88
+36 -12
View File
@@ -1,6 +1,6 @@
# Lint phase, built alone by script/lint. The tools are invoked directly # Lint phase, built alone by script/lint. The tools are invoked directly
# rather than through `make lint`, which runs docker itself and so cannot # rather than through `make lint` or `make fmt-check`, which run docker
# run inside a build step. # themselves and so cannot run inside a build step.
# golangci/golangci-lint:v2.14.0, 2026-10-07 # golangci/golangci-lint:v2.14.0, 2026-10-07
FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
WORKDIR /src WORKDIR /src
@@ -8,9 +8,9 @@ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
# The gofmt half of `make fmt-check`. gofmt's output is assigned to a # The gofmt half of `make fmt-check`; the markdown stage is its prettier
# variable first so that its own exit status, as when it cannot parse a # half. gofmt's output is assigned to a variable first so that its own
# file, still fails the step. # exit status, as when it cannot parse a file, still fails the step.
RUN files="$(gofmt -s -l .)" && \ RUN files="$(gofmt -s -l .)" && \
if [ -n "$files" ]; then \ if [ -n "$files" ]; then \
echo "gofmt: files not formatted:" >&2; echo "$files" >&2; exit 1; \ echo "gofmt: files not formatted:" >&2; echo "$files" >&2; exit 1; \
@@ -38,19 +38,43 @@ RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \ { echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; } go test -timeout 90s -race -v ./...; exit 1; }
# Build stage. Nothing is wanted from either phase above; the copies are # Prettier stage: the prettier that formats this repository's Markdown,
# what make BuildKit build them first, so this stage cannot run unless # never installed on a host. script/fmt and script/fmt-check build this
# lint and test passed. # stage alone and run it with the repository mounted on /src. prettier
# is installed in /tools so that the repository, mounted or copied onto
# /src, cannot hide it.
# node:22-alpine, 2026-02-22
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS prettier
WORKDIR /tools
# yarn.lock pins prettier by hash, and --frozen-lockfile fails rather
# than install anything yarn.lock does not name.
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile
ENV PATH=/tools/node_modules/.bin:$PATH
WORKDIR /src
# Markdown stage: the Markdown half of `make fmt-check`, as a gate.
FROM prettier AS markdown
COPY . .
RUN prettier --check '**/*.md' --tab-width 4 --prose-wrap always
# Build stage. Nothing is wanted from the lint, test and markdown stages;
# the copies are what make BuildKit build them first, so this stage
# cannot run unless all three passed.
# golang:1.25-alpine, 2026-07-23 # golang:1.25-alpine, 2026-07-23
FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git make COPY --from=markdown /src/go.sum /dev/null
WORKDIR /src
# script/bootstrap installs the git and make this image lacks, and ends
# in `go mod download`.
COPY script/ script/
COPY go.mod go.sum ./
RUN script/bootstrap
# A tar-stream context keeps the sender's file owners, which git refuses. # A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . . COPY . .
# The version stamped into the binary: the VERSION build argument when # The version stamped into the binary: the VERSION build argument when
+53 -56
View File
@@ -312,34 +312,29 @@ All three subcommands operate on a single SQLite database file:
```sql ```sql
CREATE TABLE files ( CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat mtime INTEGER NOT NULL, -- Unix seconds, from lstat
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999 head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB
head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB
tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples
content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples
) WITHOUT ROWID; ) WITHOUT ROWID;
CREATE INDEX files_signature ON files (size, head, tail, content); CREATE INDEX files_signature ON files (size, head, tail, content);
``` ```
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
UTF-8. `mtime` and `mtime_nsec` hold the file's mtime to the nanosecond: UTF-8. `mtime` is used only for change detection; it is not part of the
`mtime` the whole Unix seconds, rounded down, and `mtime_nsec` the duplicate key. For a file under 10 MiB `head`, `tail`, and `content` all
nanoseconds past that second. Split this way they hold any mtime a hold the whole-file hash (that range is hashed in full, with no end
filesystem can record, one before 1678 or after 2262 included, which a windows); for a larger file `head` and `tail` hold the first- and last-64
single 64-bit count of nanoseconds cannot. They are used only for change KiB hashes and `content` the whole-file or sampled hash. All three are empty
detection and are not part of the duplicate key. For a file under 10 MiB strings when the file has never been hashed because its size was unique as
`head`, `tail`, and `content` all hold the whole-file hash (that range is of the last scan that covered it. For a file of 10 MiB or more, `content`
hashed in full, with no end windows); for a larger file `head` and `tail` stays empty until the content phase of a scan (see "`scan` mode" below) has
hold the first- and last-64 KiB hashes and `content` the whole-file or read the file. A record with an empty `content` is never part of a duplicate
sampled hash. All three are empty strings when the file has never been group, though it still defines the file for tree reconstruction. The
hashed because its size was unique as of the last scan that covered it. For `files_signature` index lets SQLite group the records by signature for
a file of 10 MiB or more, `content` stays empty until the content phase of a `report` without sorting the whole table.
scan (see "`scan` mode" below) has read the file. A record with an empty
`content` is never part of a duplicate group, though it still defines the
file for tree reconstruction. The `files_signature` index lets SQLite group
the records by signature for `report` without sorting the whole table.
### Duplicate detection ### Duplicate detection
@@ -426,9 +421,6 @@ operands:
once its size, `head`, and `tail` match another record's. once its size, `head`, and `tail` match another record's.
- A file whose mtime is newer than recorded, or whose size differs, is processed - A file whose mtime is newer than recorded, or whose size differs, is processed
as if new: re-hashed, or recorded without hashes, per the shared-size rule. as if new: re-hashed, or recorded without hashes, per the shared-size rule.
Change detection compares the mtime to the nanosecond, as finely as the
filesystem records it, so a same-size rewrite counts as a change whenever the
filesystem gives it a later mtime than recorded, even within the same second.
- A database record whose path lies under one of the scanned operands but was - A database record whose path lies under one of the scanned operands but was
not successfully processed this run is deleted. This removes records for not successfully processed this run is deleted. This removes records for
deleted files. It also removes records for paths that failed to stat or hash deleted files. It also removes records for paths that failed to stat or hash
@@ -746,48 +738,50 @@ entrypoints are:
- `script/bootstrap` — install everything needed to build and develop this - `script/bootstrap` — install everything needed to build and develop this
repository, idempotently, assuming nothing is present. `git`, `make`, and `go` repository, idempotently, assuming nothing is present. `git`, `make`, and `go`
come from the first of nix, apt, brew, or apk found on the host, and are come from the first of nix, apt, brew, or apk found on the host, and are
presence-checked only. An installed node is used as it is; otherwise node presence-checked only. `golangci-lint` and prettier are deliberately **not**
22.17.0 is installed through nvm, which comes from a release archive whose installed: they run in Docker (see `script/lint` and `script/fmt`) and never
sha256 the script checks. A yarn already on `PATH` is used as it is; otherwise from a host install, so there is no host copy to drift from the pin. A missing
yarn 1.22.22 is activated through corepack, or installed with `npm` when there `docker` is warned about rather than installed or treated as fatal:
is no corepack. `yarn install --frozen-lockfile` then installs the prettier `make build` works without it, and testing, linting, formatting and the image
that `package.json` and `yarn.lock` pin. `golangci-lint` is never installed: build need it. Ends with `go mod download`.
it runs in Docker (see `script/lint`). `docker` is not installed either;
testing, linting and the image build need it. Ends with `go mod download`.
- `script/setup` — make a fresh clone ready for development: runs - `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`. `script/bootstrap`, then `script/install-precommit`.
- `script/projectname` — print this project's name (`sfdupes`). Scripts that - `script/projectname` — print this project's name (`sfdupes`). Scripts that
need the name call it, so they stay identical across repositories. need the name call it, so they stay identical across repositories.
- `script/test` — run the test suite under the race detector, with a 90-second - `script/test` — run the test suite under the race detector, with a 90-second
timeout and coverage enabled, rerunning verbosely on failure so the logs show timeout and coverage enabled, rerunning verbosely on failure so the logs show
which test failed. It builds the `Dockerfile`'s `test` phase alone and writes which test failed. It builds the `Dockerfile`'s `test` phase alone, tagged
no image. The race detector needs cgo and a C compiler, which the build never `sfdupes-test`. The race detector needs cgo and a C compiler, which the build
uses, so the phase starts from a digest-pinned Debian `golang` image, which never uses, so the phase starts from a digest-pinned Debian `golang` image,
has `gcc`. The tests run as `nobody`, because several of them make a file which has `gcc`. The tests run as `nobody`, because several of them make a
unreadable and root reads it anyway. file unreadable and root reads it anyway.
- `script/lint` — run the linter. It builds the `Dockerfile`'s `lint` phase - `script/lint` — run the linter. It builds the `Dockerfile`'s `lint` phase
alone and writes no image: in the digest-pinned `golangci/golangci-lint` alone, tagged `sfdupes-lint`: in the digest-pinned `golangci/golangci-lint`
image, the gofmt check, `golangci-lint config verify` and `golangci-lint run` image, the gofmt check, `golangci-lint config verify` and `golangci-lint run`
run as build steps, so a successful build is a clean lint. The linter is never run as build steps, so a successful build is a clean lint. The linter is never
run on the host, which makes a working `docker` the one prerequisite for run on the host, which makes a working `docker` the one prerequisite for
linting. linting.
- `script/fmt` — format in place: the Go sources with `gofmt -s -w`, and every - `script/fmt` — format in place: the Go sources with `gofmt -s -w`, and every
Markdown file with prettier, at the settings in `.prettierrc` (4-space Markdown file with prettier, at the settings in `.prettierrc` (4-space
indents, prose wrapped at 80 columns). Both run on the host, prettier through indents, prose wrapped at 80 columns). prettier is pinned by hash through
yarn at the version `yarn.lock` pins. When yarn is not on `PATH`, this loads `package.json` and `yarn.lock` and never installed on the host: this builds
the node 22.17.0 that `script/bootstrap` installed through nvm. the `Dockerfile`'s `prettier` stage, a digest-pinned node image into which
- `script/fmt-check` — the read-only counterpart of `script/fmt`: prints any `yarn install --frozen-lockfile` installs it, tagged `sfdupes-prettier`, and
unformatted file and exits non-zero instead of writing. gofmt and prettier runs that with the repository mounted, as the calling user. Needs `docker`,
both run every time, and each names itself when it fails. The `Dockerfile`'s and because of the mount, unlike `script/lint`, a local docker daemon.
`lint` phase runs the same gofmt check. - `script/fmt-check` — the read-only counterpart of `script/fmt`, with the
repository mounted read-only: prints any unformatted file and exits non-zero
instead of writing. gofmt and prettier both run every time, and each names
itself when it fails. The `Dockerfile` runs the same two checks as gates: the
gofmt check in its `lint` phase, prettier in its `markdown` stage.
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in - `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in
that order. Modifies nothing. The first two need `docker`, the third what that order. Modifies nothing. Needs `docker`, because all three do.
`script/bootstrap` installs.
- `script/docker` — build the Docker image, tagged with the name from - `script/docker` — build the Docker image, tagged with the name from
`script/projectname`, passing the output of `script/projectname`, passing the output of
`git describe --tags --always --dirty` (or `unknown` when that is empty) as `git describe --tags --always --dirty` (or `unknown` when that is empty) as
the `VERSION` build argument. The `Dockerfile`'s build stage depends on its the `VERSION` build argument. The `Dockerfile`'s build stage depends on its
`lint` and `test` phases, so this also runs the tests and the linter. `lint`, `test` and `markdown` stages, so this is also the check a developer or
reviewer runs by hand.
- `script/cibuild` — run `script/bootstrap`, then `script/check`, then build the - `script/cibuild` — run `script/bootstrap`, then `script/check`, then build the
image as `script/docker` does. This is what the Gitea workflow runs on push; a image as `script/docker` does. This is what the Gitea workflow runs on push; a
successful run means every check passed. The tests and the linter run twice: successful run means every check passed. The tests and the linter run twice:
@@ -795,13 +789,16 @@ entrypoints are:
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a - `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a
no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then
`script/check`. `script/check`.
- `script/install-precommit` — install the git pre-commit hook, as - `script/install-precommit` — install the git pre-commit hook that runs
`.git/hooks/pre-commit`, that runs `script/precommit`. `script/precommit`. The hook is written to the common git directory, so the
main checkout and every worktree share it.
Every `docker build` in `script/` passes `--no-cache`. On an unchanged tree Every `docker build` in `script/` passes `--no-cache`. On an unchanged tree
Docker would serve the gate steps from its cache, and the build would pass Docker would serve the gate steps from its cache, and the build would pass
having run no test and no linter. Every run therefore downloads the Go modules having run no test and no linter. Every run therefore downloads the Go modules
again and needs the network. again and needs the network. `script/test` and `script/lint` send the daemon
only a build context, so they work against a remote docker daemon; `script/fmt`
and `script/fmt-check` mount the repository and need a local one.
## Build ## Build
@@ -818,10 +815,10 @@ compile recipe:
- `make lint` — run `golangci-lint` with the repo config and the gofmt check, in - `make lint` — run `golangci-lint` with the repo config and the gofmt check, in
Docker (see `script/lint`); requires `docker`. Docker (see `script/lint`); requires `docker`.
- `make fmt` / `make fmt-check` — format the Go sources and the Markdown / - `make fmt` / `make fmt-check` — format the Go sources and the Markdown /
verify formatting without writing, on the host; requires `go` and what verify formatting without writing; requires `docker`, for prettier (see
`make bootstrap` installs (see `script/fmt`). `script/fmt`).
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires - `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires
`docker` and what `make bootstrap` installs. `docker`.
- `make docker` — build the Docker image, which runs the gates as build stages. - `make docker` — build the Docker image, which runs the gates as build stages.
- `make hooks` — install the pre-commit hook. - `make hooks` — install the pre-commit hook.
- `make clean` — remove the binary. - `make clean` — remove the binary.
+25 -58
View File
@@ -1,6 +1,6 @@
--- ---
title: Repository Policies title: Repository Policies
last_modified: 2026-10-07 last_modified: 2026-10-04
--- ---
This document covers repository structure, tooling, and workflow standards. Code This document covers repository structure, tooling, and workflow standards. Code
@@ -118,8 +118,8 @@ style conventions are in separate documents:
and nothing else: and nothing else:
```sh ```sh
docker build --no-cache --target lint --output type=cacheonly . docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test --output type=cacheonly . docker build --no-cache --target test -t "$(script/projectname)-test" .
``` ```
**A stage that is not the last one in the file is built only when the final **A stage that is not the last one in the file is built only when the final
@@ -129,15 +129,10 @@ style conventions are in separate documents:
plain `docker build .` builds the last stage alone and exits 0 having linted plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing. and tested nothing.
**The gate builds write no image.** With `--output type=cacheonly` the phase **Every `docker build` in `script/` is tagged**, here and in
runs and a failing step fails the build, but the result is not exported. `script/cibuild` and `script/docker`. An untagged build leaves a dangling
Nothing uses those images, and writing one out is slow: a Go test phase's image behind on every invocation, on every developer host and every CI
image holds the toolchain and every compiled package. A build given neither runner; a tagged one replaces the previous image.
`--output` nor `-t` writes an untagged image and leaves it dangling, on
every developer host and every CI runner. `script/cibuild` and
`script/docker` build the image that ships and tag it, so each build
replaces the previous image; each assigns the tag on its own line before the
build, so `set -e` stops it where `script/projectname` fails.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`, Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are `eslint`, `prettier` — never through `make lint` or `script/test`, which are
@@ -283,39 +278,20 @@ style conventions are in separate documents:
`.git` and the version still comes out empty, `dev` or `unknown`. A plain `.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that `docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument. refuses an empty build argument drops that refusal and keeps the argument.
A checkout whose `.git` is a file (a linked worktree, or a repository
checked out as a submodule) is the exception: that file points to a git
directory outside the build context, so the build cannot read the version
and a plain `docker build .` fails; pass the version with
`--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
do.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` on push, and checks out the repo as its only other step, runs `script/cibuild` on push, and checks out the repo as its only other step.
with `persist-credentials: false`: `script/cibuild` needs no token, and That script bootstraps, runs the gate phases, and then builds the image, so a
without it the checkout leaves the job's token in `.git/config` for every successful run means every check passed; a bare `docker build .` does not
later step. The checkout step also sets `fetch-depth: 0`, which fetches the
tags `git describe` needs: by default it clones shallow with no tags, and a
tagged repository's CI build would stamp a bare short commit id. The
workflow's `concurrency` block groups runs by workflow and branch
(`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
so a new push cancels the older run on the same branch, queued or running, and
no other: runs for replaced commits do not hold up the shared runner.
`script/cibuild` bootstraps, runs the gate phases, and then builds the image,
so a successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. The `check` job from a run of its own gates rather than from a cache entry. A separate
sets `timeout-minutes: 20`, so a hung build frees the shared runner after 20 workflow limited to `main` by a `branches` list under `on: push` cannot be
minutes. That allows for the three Docker builds described above (the test checked by review: to try a change to it, add the feature branch to that list
phase, the lint phase, then the image), each held to the 5-minute Docker build and push, then remove the branch from the list again before merging. Keep any
limit below, plus the bootstrap. A separate workflow limited to `main` by a job in it that publishes behind `if: github.ref_name == 'main'`, so the run
`branches` list under `on: push` cannot be checked by review: to try a change from the feature branch publishes nothing.
to it, add the feature branch to that list and push, then remove the branch
from the list again before merging. Keep any job in it that publishes behind
`if: github.ref_name == 'main'`, so the run from the feature branch publishes
nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for - Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -408,12 +384,9 @@ style conventions are in separate documents:
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`), - `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`), editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
`node_modules/`, and the repo's own build outputs. Fetch the standard language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
`.gitignore` from from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up setting up a new repo. These patterns are written to `.gitignore`'s own
a new repo. A repo's `.gitignore` is the standard file followed by the repo's
own entries, such as its binaries; a re-vendor replaces the standard part and
keeps those entries. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified. are not a `.dockerignore` and must not be transplanted into one unmodified.
@@ -461,15 +434,13 @@ style conventions are in separate documents:
byte-identically across repos: byte-identically across repos:
```sh ```sh
# The version and the tag each get their own line: a failing command # Own line: a failing command substitution inside an argument does not
# substitution inside an argument does not trip `set -e`, so the inline # trip `set -e`, so the inline form degrades to an empty constant.
# form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
tag="$(script/projectname)"
docker build --no-cache \ docker build --no-cache \
--build-arg VERSION="$version" \ --build-arg VERSION="$version" \
-t "$tag" . -t "$(script/projectname)" .
``` ```
`--always` makes an untagged repo yield an abbreviated commit hash rather `--always` makes an untagged repo yield an abbreviated commit hash rather
@@ -481,8 +452,8 @@ style conventions are in separate documents:
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so the canonical checkout action clones shallow and fetches no tags, so a repo that embeds a
`.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step. tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the - **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build patterns.** Plant files at the root _and_ at least two directories deep, build
@@ -665,11 +636,7 @@ style conventions are in separate documents:
Never edit existing migrations after release. Never edit existing migrations after release.
- All repos should have an `.editorconfig` enforcing the project's indentation - All repos should have an `.editorconfig` enforcing the project's indentation
settings: the standard file from settings.
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
tabs for `Makefile` and Go files, followed by the repo's own sections, such as
one for another language it uses. A re-vendor replaces the standard part and
keeps those sections.
- Avoid putting files in the repo root unless necessary. Root should contain - Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`, only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
+82 -64
View File
@@ -29,43 +29,27 @@
# Completed Steps # Completed Steps
- re-vendor the canonical files and model scripts from `sneak/prompts` `next` at - re-vendor the canonical files from `sneak/prompts` commit `dd4027b`:
`c55a0cb`: golangci-lint v2.14.0; lint and test are phases of the `Dockerfile` golangci-lint v2.14.0; lint and test are phases of the `Dockerfile`, and
that write no image, and `make test` runs the suite under the race detector, `make test` runs the suite under the race detector, so `Dockerfile.lint`,
so `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race` are `script/verify-lint-image-pin` and `make test-race` are gone; every
gone; every `docker build` in `script/` passes `--no-cache`; prettier runs on `docker build` in `script/` passes `--no-cache`; `.claude/settings.json` is
the host, from the node and yarn `script/bootstrap` installs, so the deleted (2026-10-07, https://git.eeqj.de/sneak/sfdupes/issues/95)
`prettier` and `markdown` stages are gone and the build stage installs `git`
and `make` itself; a new push cancels the workflow's older run on the same
branch, and a run stops after 20 minutes; `.claude/settings.json` is deleted
(2026-10-08, https://git.eeqj.de/sneak/sfdupes/issues/95)
- `scan` records mtime to the nanosecond, as whole seconds in `mtime` plus
`mtime_nsec`, and compares it at that resolution, so a same-size rewrite
within the same second is re-hashed (2026-10-07,
https://git.eeqj.de/sneak/sfdupes/issues/12)
- cut the narration from `TODO.md` Completed Steps and from the comments in - cut the narration from `TODO.md` Completed Steps and from the comments in
`script/`, `Dockerfile` and `Dockerfile.lint` (gone since `script/` and both Dockerfiles; §Workflow now branches from and merges to
https://git.eeqj.de/sneak/sfdupes/issues/95); §Workflow now branches from and `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
merges to `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
- `make test-race` ran the test suite under the race detector in a cgo-enabled - `make test-race` runs the test suite under the race detector in a cgo-enabled
container, outside `make check` (2026-10-04, container, outside `make check` (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/18). Since https://git.eeqj.de/sneak/sfdupes/issues/18)
https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
under the race detector, in the `Dockerfile`'s Debian-based `test` phase, and
`make test-race` is gone
- a bare `docker build .` failed, naming `script/cibuild` and `script/docker`, - a bare `docker build .` fails with a message naming `script/cibuild` and
rather than serve the gates from cache (2026-10-04, `script/docker` instead of serving the gates from cache (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/39). Since https://git.eeqj.de/sneak/sfdupes/issues/39)
https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
`REPO_POLICIES.md` requires, and may serve the gates from cache; the builds in
`script/` pass `--no-cache`, so theirs always run
- `make fmt` and `make fmt-check` run prettier over all Markdown, and CI checks - `make fmt` and `make fmt-check` run prettier over all Markdown, in Docker, and
it; all Markdown reformatted (2026-10-04, CI checks it; all Markdown reformatted (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/19) https://git.eeqj.de/sneak/sfdupes/issues/19)
- `script/lint` writes no image, so a run no longer leaves an untagged one - `script/lint` writes no image, so a run no longer leaves an untagged one
@@ -108,12 +92,9 @@
- README documents install, Docker, a daily cron scan and how to read and check - README documents install, Docker, a daily cron scan and how to read and check
the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54) the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
- the `Dockerfile` build stage kept the Go module cache out of `builder`'s home - the `Dockerfile` build stage keeps the Go module cache out of `builder`'s home
and copied the sources with `--chown`, so no `chown -R` walked them and copies the sources with `--chown`, so no `chown -R` walks them
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43). Since (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43)
https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
nothing changes owner: the build stage only compiles, as root, and the tests
run as `nobody` in the `test` phase
- `--version` prints `sfdupes VERSION` to stdout; README documents it and - `--version` prints `sfdupes VERSION` to stdout; README documents it and
`--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15) `--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
@@ -177,45 +158,82 @@
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and - fix the lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` (2026-08-10, branch `next`, closes `Dockerfile.lint` (2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/25): both pins became the policy https://git.eeqj.de/sneak/sfdupes/issues/25): both pins are now the policy
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`, `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
without the false `(Debian-based)` note or the tag. Since without the false `(Debian-based)` note or the tag; digest unchanged.
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase `script/verify-lint-image-pin` still matches the tagless form, and a tag on
holds the only golangci-lint pin, in that form, so `Dockerfile.lint` and one side only is caught as a plain mismatch.
`script/verify-lint-image-pin`, which compared the two pins, are gone.
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10, - run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
owner ruling the linter is never installed on a host, and owner ruling the linter is never installed on a host. `Dockerfile.lint` copies
`golangci-lint config verify` runs before `golangci-lint run`. the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
`script/bootstrap` stopped installing or pinning the linter, and runs `golangci-lint config verify` and `golangci-lint run` as build steps;
`script/verify-linter-pin` was retired. Since `script/lint` builds it. `script/bootstrap` no longer installs or pins the
https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the linter, and warns rather than fails when `docker` is absent;
`Dockerfile`'s `lint` phase, which `script/lint` builds alone and the build `ENV PATH=/home/builder/go/bin:$PATH` went with its `go install`.
stage depends on through `COPY --from=lint /src/go.sum /dev/null`; `script/verify-linter-pin` is retired; `script/verify-lint-image-pin`, a gate
`Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside in both files, compares their two `FROM` lines and restates neither pin.
an image build may run docker, so the phase calls `golangci-lint` directly. Traps: nothing inside an image build may shell out to docker, so the
`Dockerfile` lint stage calls `golangci-lint` directly and the build stage
runs `make test` and `make fmt-check` instead of `make check`, through `make`
because the Makefile's `export CGO_ENABLED = 0` only reaches what it invokes.
`COPY --from=lint /src/go.sum /dev/null` replaces the copied linter binary as
the only edge making the build stage wait for lint; dropping it would end
fail-fast linting under a still-green build. `golangci-lint config verify`,
included per the ruling, validates from an embedded schema with no network
call, but `go mod download` above the gates still needs the network on a cold
cache. Verified: `make lint` green with no `golangci-lint` on `PATH`; two
back-to-back `script/lint` runs on an untouched tree both ran the linter
(27.7s and 28.7s in the lint step, `COPY . .` `CACHED` above); a planted
unused variable failed `script/lint`, and failed `make docker` at `[lint 9/9]`
with the build stage stopped at `[builder 3/12]`; the drift guard fails on a
tag-only, a digest-only and an unreadable reference, naming both sides; under
`--network none` config verify passes a valid config and rejects an invalid
one; `make docker` green in 5m35s with all six gates run (lint 37.6s, test
25.2s reporting `ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not
`(cached)`); in the builder image with the Go test cache off, `--user 0:0`
still fails `TestScanHardlinkRunFailsTogether` where the unprivileged user
passes. Noted for follow-up, not fixed here: `golangci-lint` warns that
`gomodguard` is deprecated since v2.12.0 in favour of `gomodguard_v2`.
- install the Docker build stage's prerequisites by running `script/bootstrap` - install the Docker build stage's prerequisites by running `script/bootstrap`
instead of `apk add --no-cache make` inline (2026-08-09, branch instead of `apk add --no-cache make` inline (2026-08-09, branch
`dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42), `dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42):
with `script/verify-linter-pin` failing the build unless the linter copied the stage copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
from the lint stage was the version `script/bootstrap` pinned. Builds then which ends in `go mod download`, so the separate call to it is gone.
took up to 5m14s cold, mostly in a `chown -R` of the module cache, filed as `COPY --from=lint /usr/bin/golangci-lint` stays and moves above the bootstrap
https://git.eeqj.de/sneak/sfdupes/issues/43. Since layer: it is the only edge making this stage depend on the lint stage, so
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and deleting it would end fail-fast linting silently. A new
`make` with `apk add --no-cache` and only compiles; the `lint` and `test` `script/verify-linter-pin`, run in the build stage before bootstrap, fails the
phases are the gates build naming both versions unless that copied binary is the version
`script/bootstrap` pins; a pin it cannot read is a hard failure, not a skip.
`$GOPATH/bin` joins `PATH`, where bootstrap's `go install` lands. The `chown`
and `USER builder` still precede `make check`. Verified: the guard fails the
build with both versions named when the lint stage's linter is faked to
another version, and passes an unmodified build; bootstrap runs clean under
Alpine's `sh` and `apk`, finding the copied linter already at the pin; a
second build served the bootstrap and dependency layers `CACHED` while both
gates ran; a planted `unused` finding failed the build at the lint gate in
48.9s with the build stage's `make check` never starting; and the suite run in
the image as `--user 0:0` fails `TestScanHardlinkRunFailsTogether`, so the
drop to the unprivileged user is still needed. That last check needs the Go
test cache off: as root it first reported `ok ... (cached)`, reusing the
build-time result. Build times on a noisy shared host: 2m13s on an unchanged
tree, 2m17s and 4m29s after a source change, 5m14s cold, which breaches the
policy ceiling; `chown -R builder:builder /src /home/builder` walks the module
cache and alone varied from 77s to 210s across those builds, and `main`
measured 5m03s cold with a 209s `chown`. Filed as
https://git.eeqj.de/sneak/sfdupes/issues/43
- bust the Docker layer cache for the gate steps, so `script/cibuild` and - bust the Docker layer cache for the gate steps, so `script/cibuild` and
`script/docker` cannot report a green they did not earn (2026-08-09, branch `script/docker` cannot report a green they did not earn (2026-08-09, branch
`cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the `cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the
`Dockerfile` copies the tree before its gates, so on an unchanged tree Docker `Dockerfile` copies the tree before its gates, so on an unchanged tree Docker
served them from cache and the build exited 0 having run nothing. The fix was served them from cache and the build exited 0 having run nothing. Every
a `CHECK_EPOCH` build argument; since `docker build` in `script/` now passes `--no-cache` instead
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/` (https://git.eeqj.de/sneak/sfdupes/issues/95). Run as root, the tests fail
passes `--no-cache` instead. Run as root, the tests fail
`TestScanHardlinkRunFailsTogether`, because root reads through the `chmod(0)` `TestScanHardlinkRunFailsTogether`, because root reads through the `chmod(0)`
the test relies on, so the `test` phase runs them as `nobody` the test relies on, so they run as an unprivileged user
- check the installed golangci-lint version in `script/bootstrap` instead of - check the installed golangci-lint version in `script/bootstrap` instead of
only its presence (2026-08-09, branch `bootstrap-version-check`, closes only its presence (2026-08-09, branch `bootstrap-version-check`, closes
https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
+28 -38
View File
@@ -11,7 +11,6 @@ import (
"path/filepath" "path/filepath"
"slices" "slices"
"strconv" "strconv"
"time"
"golang.org/x/sys/unix" "golang.org/x/sys/unix"
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo // The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
@@ -41,18 +40,15 @@ const dbDirPerm = 0o755
const lockFilePerm = 0o600 const lockFilePerm = 0o600
// createTableSQL is the schema applied to a fresh database. Paths are // createTableSQL is the schema applied to a fresh database. Paths are
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime // BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
// holds whole Unix seconds and mtime_nsec the nanoseconds within that
// second.
const createTableSQL = ` const createTableSQL = `
CREATE TABLE files ( CREATE TABLE files (
path BLOB PRIMARY KEY, path BLOB PRIMARY KEY,
size INTEGER NOT NULL, size INTEGER NOT NULL,
mtime INTEGER NOT NULL, mtime INTEGER NOT NULL,
mtime_nsec INTEGER NOT NULL, head TEXT NOT NULL,
head TEXT NOT NULL, tail TEXT NOT NULL,
tail TEXT NOT NULL, content TEXT NOT NULL
content TEXT NOT NULL
) WITHOUT ROWID ) WITHOUT ROWID
` `
@@ -65,11 +61,10 @@ CREATE INDEX files_signature ON files (size, head, tail, content)
// upsertSQL inserts one file record, replacing any existing record for // upsertSQL inserts one file record, replacing any existing record for
// the same path. // the same path.
const upsertSQL = ` const upsertSQL = `
INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content) INSERT INTO files (path, size, mtime, head, tail, content)
VALUES (?, ?, ?, ?, ?, ?, ?) VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT (path) DO UPDATE SET ON CONFLICT (path) DO UPDATE SET
size = excluded.size, mtime = excluded.mtime, size = excluded.size, mtime = excluded.mtime,
mtime_nsec = excluded.mtime_nsec,
head = excluded.head, tail = excluded.tail, head = excluded.head, tail = excluded.tail,
content = excluded.content content = excluded.content
` `
@@ -361,8 +356,8 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
// which is the order of the primary key, so SQLite does not sort. // which is the order of the primary key, so SQLite does not sort.
func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error { func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
rows, err := db.QueryContext(ctx, rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+ "SELECT path, size, mtime, head, tail, content FROM files "+
"FROM files ORDER BY path") "ORDER BY path")
if err != nil { if err != nil {
return fmt.Errorf("read records: %w", err) return fmt.Errorf("read records: %w", err)
} }
@@ -371,19 +366,17 @@ func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
for rows.Next() { for rows.Next() {
var ( var (
path []byte path []byte
sec, nsec int64 r scanRec
r scanRec
) )
err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail, err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail,
&r.content) &r.content)
if err != nil { if err != nil {
return fmt.Errorf("read record: %w", err) return fmt.Errorf("read record: %w", err)
} }
r.path = string(path) r.path = string(path)
r.mtime = time.Unix(sec, nsec)
fn(r) fn(r)
} }
@@ -473,10 +466,10 @@ func loadDupeRows(ctx context.Context, db *sql.DB,
// values, and skipping the hash columns keeps the scan's in-memory // values, and skipping the hash columns keeps the scan's in-memory
// index small on multi-million-file databases. // index small on multi-million-file databases.
func loadFileMeta(ctx context.Context, db *sql.DB, func loadFileMeta(ctx context.Context, db *sql.DB,
fn func(path string, size int64, mtime time.Time, hashed bool), fn func(path string, size, mtime int64, hashed bool),
) error { ) error {
rows, err := db.QueryContext(ctx, rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, mtime_nsec, head <> '' FROM files") "SELECT path, size, mtime, head <> '' FROM files")
if err != nil { if err != nil {
return fmt.Errorf("read records: %w", err) return fmt.Errorf("read records: %w", err)
} }
@@ -485,17 +478,17 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
for rows.Next() { for rows.Next() {
var ( var (
path []byte path []byte
size, sec, nsec int64 size, mtime int64
hashed int64 hashed int64
) )
err = rows.Scan(&path, &size, &sec, &nsec, &hashed) err = rows.Scan(&path, &size, &mtime, &hashed)
if err != nil { if err != nil {
return fmt.Errorf("read record: %w", err) return fmt.Errorf("read record: %w", err)
} }
fn(string(path), size, time.Unix(sec, nsec), hashed != 0) fn(string(path), size, mtime, hashed != 0)
} }
err = rows.Err() err = rows.Err()
@@ -514,7 +507,7 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
// memory; the rows come ordered by size, head, and tail, so each // memory; the rows come ordered by size, head, and tail, so each
// group's rows arrive together. // group's rows arrive together.
const contentCandidatesSQL = ` const contentCandidatesSQL = `
SELECT f.path, f.size, f.mtime, f.mtime_nsec, f.head, f.tail, f.content <> '' SELECT f.path, f.size, f.mtime, f.head, f.tail, f.content <> ''
FROM files AS f FROM files AS f
JOIN ( JOIN (
SELECT size, head, tail SELECT size, head, tail
@@ -540,20 +533,17 @@ func loadContentCandidates(ctx context.Context, db *sql.DB,
for rows.Next() { for rows.Next() {
var ( var (
path []byte path []byte
sec, nsec int64 r scanRec
r scanRec hashed int64
hashed int64
) )
err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail, err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail, &hashed)
&hashed)
if err != nil { if err != nil {
return fmt.Errorf("read record: %w", err) return fmt.Errorf("read record: %w", err)
} }
r.path = string(path) r.path = string(path)
r.mtime = time.Unix(sec, nsec)
fn(r, hashed != 0) fn(r, hashed != 0)
} }
@@ -637,8 +627,8 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
defer func() { _ = st.Close() }() defer func() { _ = st.Close() }()
for _, r := range upserts { for _, r := range upserts {
_, err = st.ExecContext(ctx, []byte(r.path), r.size, _, err = st.ExecContext(ctx,
r.mtime.Unix(), r.mtime.Nanosecond(), r.head, r.tail, r.content) []byte(r.path), r.size, r.mtime, r.head, r.tail, r.content)
if err != nil { if err != nil {
return fmt.Errorf("upsert %s: %w", r.path, err) return fmt.Errorf("upsert %s: %w", r.path, err)
} }
+5 -10
View File
@@ -10,7 +10,6 @@ import (
"slices" "slices"
"strings" "strings"
"testing" "testing"
"time"
) )
// testDBPath returns a database path inside a fresh temp dir. // testDBPath returns a database path inside a fresh temp dir.
@@ -261,13 +260,10 @@ func TestApplyChangesRoundTrip(t *testing.T) {
// written. // written.
recs := []scanRec{ recs := []scanRec{
{ {
size: 2, mtime: time.Unix(20, 999_999_999), head: "h2", tail: "t2", size: 2, mtime: 20, head: "h2", tail: "t2", content: "c2",
content: "c2", path: "/a/tab\tnew\nline", path: "/a/tab\tnew\nline",
},
{
size: 1, mtime: time.Unix(10, 0), head: "h1", tail: "t1",
content: "c1", path: "/a/x",
}, },
{size: 1, mtime: 10, head: "h1", tail: "t1", content: "c1", path: "/a/x"},
} }
err := applyChanges(t.Context(), db, recs, nil, err := applyChanges(t.Context(), db, recs, nil,
@@ -285,8 +281,7 @@ func TestApplyChangesRoundTrip(t *testing.T) {
// An upsert for an existing path updates in place; a delete // An upsert for an existing path updates in place; a delete
// removes exactly its path. // removes exactly its path.
upd := scanRec{ upd := scanRec{
size: 3, mtime: time.Unix(30, 0), head: "h3", tail: "t3", content: "c3", size: 3, mtime: 30, head: "h3", tail: "t3", content: "c3", path: "/a/x",
path: "/a/x",
} }
err = applyChanges(t.Context(), db, []scanRec{upd}, err = applyChanges(t.Context(), db, []scanRec{upd},
@@ -313,7 +308,7 @@ func TestApplyChangesBatching(t *testing.T) {
recs := make([]scanRec, 0, n) recs := make([]scanRec, 0, n)
for i := range n { for i := range n {
recs = append(recs, scanRec{ recs = append(recs, scanRec{
size: int64(i), mtime: time.Unix(1, 0), head: "h", tail: "t", size: int64(i), mtime: 1, head: "h", tail: "t",
path: fmt.Sprintf("/batch/%07d", i), path: fmt.Sprintf("/batch/%07d", i),
}) })
} }
+1 -2
View File
@@ -7,7 +7,6 @@ import (
"io" "io"
"os" "os"
"strings" "strings"
"time"
) )
// ioBufSize is the buffer size for the buffered stdout writers. // ioBufSize is the buffer size for the buffered stdout writers.
@@ -22,7 +21,7 @@ const minGroupSize = 2
// only and used by scan for change detection. // only and used by scan for change detection.
type scanRec struct { type scanRec struct {
size int64 size int64
mtime time.Time mtime int64
head string head string
tail string tail string
content string content string
+2 -9
View File
@@ -11,7 +11,6 @@ import (
"slices" "slices"
"strings" "strings"
"testing" "testing"
"time"
) )
// awkwardDir is a directory name holding every byte the reports escape. // awkwardDir is a directory name holding every byte the reports escape.
@@ -314,14 +313,8 @@ func TestDupeGroupsMtimeExcluded(t *testing.T) {
// mtime is informational only; records differing only in mtime // mtime is informational only; records differing only in mtime
// still group together. // still group together.
recs := []scanRec{ recs := []scanRec{
{ {size: 9, mtime: 100, head: "h", tail: "t", content: "c", path: "/m/1"},
size: 9, mtime: time.Unix(100, 0), head: "h", tail: "t", {size: 9, mtime: 200, head: "h", tail: "t", content: "c", path: "/m/2"},
content: "c", path: "/m/1",
},
{
size: 9, mtime: time.Unix(200, 0), head: "h", tail: "t",
content: "c", path: "/m/2",
},
} }
groups := dupeGroupsOf(t, recs) groups := dupeGroupsOf(t, recs)
+7 -18
View File
@@ -17,7 +17,6 @@ import (
"strings" "strings"
"sync" "sync"
"syscall" "syscall"
"time"
) )
// The duplicate ladder (see hashSignature and README "Duplicate // The duplicate ladder (see hashSignature and README "Duplicate
@@ -69,7 +68,7 @@ var errInterrupted = errors.New("scan interrupted")
type fileRec struct { type fileRec struct {
path string path string
size int64 size int64
mtime time.Time mtime int64
dev uint64 dev uint64
ino uint64 ino uint64
} }
@@ -80,7 +79,7 @@ type fileRec struct {
// they would dominate the scan's memory. // they would dominate the scan's memory.
type fileMeta struct { type fileMeta struct {
size int64 size int64
mtime time.Time mtime int64
hashed bool hashed bool
} }
@@ -370,7 +369,7 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
s.existing = make(map[string]fileMeta) s.existing = make(map[string]fileMeta)
return loadFileMeta(ctx, s.db, return loadFileMeta(ctx, s.db,
func(path string, size int64, mtime time.Time, hashed bool) { func(path string, size, mtime int64, hashed bool) {
prog.increment() prog.increment()
if underAnyRoot(path, roots) { if underAnyRoot(path, roots) {
@@ -416,7 +415,7 @@ func (s *scanState) walkPhase(
old, ok := s.existing[ev.rec.path] old, ok := s.existing[ev.rec.path]
switch { switch {
case !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime): case !ok || old.size != ev.rec.size || old.mtime < ev.rec.mtime:
changed = append(changed, ev.rec) changed = append(changed, ev.rec)
case old.hashed: case old.hashed:
delete(s.existing, ev.rec.path) delete(s.existing, ev.rec.path)
@@ -808,7 +807,7 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
} }
if !fi.Mode().IsRegular() || fi.Size() != r.size || if !fi.Mode().IsRegular() || fi.Size() != r.size ||
mtimeAfter(fi.ModTime(), r.mtime) { fi.ModTime().Unix() > r.mtime {
return fileRec{}, false, nil return fileRec{}, false, nil
} }
@@ -819,16 +818,6 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
}, true, nil }, true, nil
} }
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under // underAnyRoot reports whether path is any of the roots or lies under
// one of them. // one of them.
func underAnyRoot(path string, roots []string) bool { func underAnyRoot(path string, roots []string) bool {
@@ -977,7 +966,7 @@ func seedRoot(ctx context.Context, root string,
sendEvent(ctx, events, walkEvent{rec: fileRec{ sendEvent(ctx, events, walkEvent{rec: fileRec{
path: root, path: root,
size: fi.Size(), size: fi.Size(),
mtime: fi.ModTime(), mtime: fi.ModTime().Unix(),
dev: dev, dev: dev,
ino: ino, ino: ino,
}}) }})
@@ -1135,7 +1124,7 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
sendEvent(ctx, events, walkEvent{rec: fileRec{ sendEvent(ctx, events, walkEvent{rec: fileRec{
path: p, path: p,
size: info.Size(), size: info.Size(),
mtime: info.ModTime(), mtime: info.ModTime().Unix(),
dev: dev, dev: dev,
ino: ino, ino: ino,
}}) }})
+7 -227
View File
@@ -20,8 +20,6 @@ import (
"syscall" "syscall"
"testing" "testing"
"time" "time"
"golang.org/x/sys/unix"
) )
// writeFile creates a file with the given content and returns its path. // writeFile creates a file with the given content and returns its path.
@@ -583,65 +581,6 @@ func TestScanContentHashedStalePartners(t *testing.T) {
} }
} }
// TestScanContentSameSecondRewrite is TestScanContentStalePartners for
// a stored file rewritten in place at the same size with an mtime later
// in the same second than recorded: the file counts as changed, so
// neither it nor its match inside the operand is read.
func TestScanContentSameSecondRewrite(t *testing.T) {
t.Parallel()
db := openTestDB(t)
dirA := t.TempDir()
changed := sparseFileWithoutMatch(t, dirA, "changed", headTailMin)
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(changed, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dirA)
before := dbRecords(t, db)
// Rewrite one byte in place, keeping the size.
pokeAt(t, changed, headTailMin/2, []byte{1})
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(changed, later, later)
if err != nil {
t.Fatal(err)
}
dirB := t.TempDir()
sparseFile(t, dirB, "changed-copy", headTailMin)
st := syncTree(t, db, dirB)
if st != (scanStats{walked: 1, added: 1}) {
t.Errorf("stats = %+v, want 1 added and nothing skipped", st)
}
recs := dbRecords(t, db)
for _, r := range recs {
if r.content != "" {
t.Errorf("%s: content = %q, want none: its only match is stale",
r.path, r.content)
}
}
for _, old := range before {
if r := recordByPath(t, recs, old.path); r != old {
t.Errorf("record = %+v, want it left as %+v", r, old)
}
}
if groups := dupeGroups(t, db); len(groups) != 0 {
t.Errorf("groups = %+v, want none", groups)
}
}
// TestScanContentReadFailure checks that a failed content read is // TestScanContentReadFailure checks that a failed content read is
// counted as skipped and leaves the record without a content hash, and // counted as skipped and leaves the record without a content hash, and
// that a later scan tries the read again. // that a later scan tries the read again.
@@ -844,8 +783,8 @@ func TestWalk(t *testing.T) {
t.Errorf("%s: size = %d, want 1..3", r.path, r.size) t.Errorf("%s: size = %d, want 1..3", r.path, r.size)
} }
if r.mtime.Unix() <= 0 { if r.mtime <= 0 {
t.Errorf("%s: mtime = %v, want after 1970", r.path, r.mtime) t.Errorf("%s: mtime = %d, want positive", r.path, r.mtime)
} }
} }
} }
@@ -1346,172 +1285,11 @@ func TestSyncScanMtimeBump(t *testing.T) {
t.Fatalf("mtime-bump stats = %+v, want 1 updated", st) t.Fatalf("mtime-bump stats = %+v, want 1 updated", st)
} }
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(future) { if r := recordByPath(t, dbRecords(t, db), a); r.mtime != future.Unix() {
t.Fatalf("mtime = %v, want %v", r.mtime, future) t.Fatalf("mtime = %d, want %d", r.mtime, future.Unix())
} }
} }
// assertWholeFileHashed fails unless the record for path holds the
// whole-file hash of data as its head, tail, and content.
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
data []byte,
) {
t.Helper()
r := recordByPath(t, dbRecords(t, db), path)
if want := hexSum(data); r.head != want || r.tail != want ||
r.content != want {
t.Fatalf("head, tail, content = %q, %q, %q, want %q for each",
r.head, r.tail, r.content, want)
}
}
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
// size with an mtime later in the same second as the recorded one: the
// next scan must notice the change and re-hash the file.
func TestSyncScanSameSecondRewrite(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
// b.bin shares the size of a.bin, so a.bin is hashed.
writeFile(t, dir, "b.bin", pattern(2, 500))
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(a, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(a, later, later)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st)
}
assertWholeFileHashed(t, db, a, rewritten)
}
// TestSyncScanOperandSameSecondRewrite is TestSyncScanSameSecondRewrite
// for files given to scan as operands, which scan stats without reading
// their directory.
func TestSyncScanOperandSameSecondRewrite(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
// b.bin shares the size of a.bin, so a.bin is hashed.
b := writeFile(t, dir, "b.bin", pattern(2, 500))
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(a, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, a, b)
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(a, later, later)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, a, b)
if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st)
}
assertWholeFileHashed(t, db, a, rewritten)
}
// TestSyncScanRewriteAfter2262 runs assertLateRewriteRehashed with an
// mtime after 2262, a time too late to count in nanoseconds in an int64.
func TestSyncScanRewriteAfter2262(t *testing.T) {
t.Parallel()
assertLateRewriteRehashed(t, time.Date(2300, 1, 2, 3, 4, 5, 0, time.UTC))
}
// TestSyncScanRewritePastTimeLimit runs assertLateRewriteRehashed with an
// mtime one second past the latest a time.Time holds without wrapping it
// to a time far in the past.
func TestSyncScanRewritePastTimeLimit(t *testing.T) {
t.Parallel()
assertLateRewriteRehashed(t, time.Unix(9223371974719179008, 0))
}
// assertLateRewriteRehashed scans a directory, rewrites a file in it in
// place at the same size, sets its mtime to late, and fails unless the
// next scan re-hashes the file. It skips where late does not fit the
// platform's timespec or the filesystem does not store it.
func assertLateRewriteRehashed(t *testing.T, late time.Time) {
t.Helper()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
// b.bin shares the size of a.bin, so a.bin is hashed.
writeFile(t, dir, "b.bin", pattern(2, 500))
syncTree(t, db, dir)
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
// os.Chtimes cannot set such a time: it converts through UnixNano.
ts, err := unix.TimeToTimespec(late)
if err != nil {
t.Skipf("an mtime %d seconds after 1970 does not fit this platform's "+
"timespec: %v", late.Unix(), err)
}
err = unix.UtimesNano(a, []unix.Timespec{ts, ts})
if err != nil {
t.Fatal(err)
}
fi, err := os.Lstat(a)
if err != nil {
t.Fatal(err)
}
if fi.ModTime().Unix() != late.Unix() {
t.Skipf("the filesystem stored the mtime as %d seconds after 1970, "+
"not %d", fi.ModTime().Unix(), late.Unix())
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 2, updated: 1, unchanged: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated 1 unchanged", st)
}
assertWholeFileHashed(t, db, a, rewritten)
}
func TestSyncScanAddRemove(t *testing.T) { func TestSyncScanAddRemove(t *testing.T) {
t.Parallel() t.Parallel()
@@ -1557,7 +1335,9 @@ func TestSyncScanSizeChange(t *testing.T) {
writeFile(t, dir, "f", pattern(1, 200)) writeFile(t, dir, "f", pattern(1, 200))
err := os.Chtimes(p, old.mtime, old.mtime) mt := time.Unix(old.mtime, 0)
err := os.Chtimes(p, mt, mt)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+16 -84
View File
@@ -1,22 +1,13 @@
#!/bin/sh #!/bin/sh
# script/bootstrap: install all dependencies needed to build and develop # script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent; assumes nothing is present (not git, make, or
# installed tools are skipped. Base tooling comes from nix, apt, brew, # go). Base tooling comes from nix, apt, brew, or apk (detected in that
# or apk (detected in that order); assumes nothing is present. Node is # order). golangci-lint and prettier are never installed: they run via
# used directly if installed; otherwise it is installed at a pinned # docker only (script/lint, script/fmt, script/fmt-check).
# version via nvm (installing nvm itself first, from a hash-verified
# release archive, never curl | sh).
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-07-06
NODE_VERSION="22.17.0"
NVM_VERSION="0.40.3"
# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
YARN_VERSION="1.22.22"
PKGMGR="" PKGMGR=""
SUDO="" SUDO=""
APT_UPDATED="" APT_UPDATED=""
@@ -49,7 +40,6 @@ pkg_install() {
case "$PKGMGR" in case "$PKGMGR" in
nix) nix-env -iA "nixpkgs.$1" ;; nix) nix-env -iA "nixpkgs.$1" ;;
apt) apt)
# Package lists may be empty (fresh images); refresh once per run.
if [ -z "$APT_UPDATED" ]; then if [ -z "$APT_UPDATED" ]; then
$SUDO env DEBIAN_FRONTEND=noninteractive apt-get update $SUDO env DEBIAN_FRONTEND=noninteractive apt-get update
APT_UPDATED=1 APT_UPDATED=1
@@ -65,82 +55,24 @@ missing() {
! command -v "$1" >/dev/null 2>&1 ! command -v "$1" >/dev/null 2>&1
} }
# verify_sha256 <file> <expected-hash>
verify_sha256() {
if command -v sha256sum >/dev/null 2>&1; then
actual="$(sha256sum "$1" | cut -d' ' -f1)"
else
actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
fi
if [ "$actual" != "$2" ]; then
echo "bootstrap: sha256 mismatch for $1" >&2
echo " expected: $2" >&2
echo " actual: $actual" >&2
exit 1
fi
}
# nvm is a bash script; run a command in a bash with nvm loaded
nvm_sh() {
bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
}
ensure_nvm() {
[ -s "$HOME/.nvm/nvm.sh" ] && return 0
# nvm prerequisites; nvm itself requires bash
if missing bash; then pkg_install bash bash bash bash; fi
if missing curl; then pkg_install curl curl curl curl; fi
if missing git; then pkg_install git git git git; fi
tmp="$(mktemp -d)"
curl -fsSL -o "$tmp/nvm.tar.gz" \
"https://github.com/nvm-sh/nvm/archive/refs/tags/v${NVM_VERSION}.tar.gz"
verify_sha256 "$tmp/nvm.tar.gz" "$NVM_SHA256"
mkdir -p "$HOME/.nvm"
tar -xzf "$tmp/nvm.tar.gz" -C "$HOME/.nvm" --strip-components=1
rm -rf "$tmp"
}
ensure_node() {
if ! missing node; then return 0; fi
ensure_nvm
nvm_sh "nvm install $NODE_VERSION"
}
ensure_yarn() {
if ! missing yarn; then return 0; fi
if ! missing corepack; then
corepack enable
corepack prepare "yarn@$YARN_VERSION" --activate
elif [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
corepack prepare yarn@$YARN_VERSION --activate"
else
npm install -g "yarn@$YARN_VERSION"
fi
}
install_js_deps() {
if missing yarn && [ -s "$HOME/.nvm/nvm.sh" ]; then
nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
yarn install --frozen-lockfile"
else
yarn install --frozen-lockfile
fi
}
main() { main() {
cd "$ROOT" cd "$ROOT"
if missing make; then pkg_install gnumake make make make; fi # Deliberately unpinned, so presence is the whole check: go.mod
# governs the Go version, and reproducible builds run in the
# digest-pinned Docker images.
if missing git; then pkg_install git git git git; fi if missing git; then pkg_install git git git git; fi
# Go builds the binary and runs gofmt. Presence is the whole check: if missing make; then pkg_install gnumake make make make; fi
# go.mod names the Go version, and the tests and the linter run in
# digest-pinned images.
if missing go; then pkg_install go golang go go; fi if missing go; then pkg_install go golang go go; fi
ensure_node # Warn, do not fail: only the targets named below, and the
ensure_yarn # pre-commit hook, need docker.
install_js_deps if missing docker; then
echo "bootstrap: WARNING: docker not found; make test, make lint," >&2
echo "bootstrap: make fmt, make fmt-check, make check and" >&2
echo "bootstrap: make docker require it." >&2
fi
go mod download go mod download
echo "bootstrap complete" echo "bootstrap complete"
+1 -3
View File
@@ -1,8 +1,6 @@
#!/bin/sh #!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own # script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. test and lint are Docker # extension to scripts-to-rule-them-all. Must not modify any files.
# phases; fmt-check is native, because a formatter writes the working
# tree. Must not modify any files.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+5 -7
View File
@@ -14,17 +14,15 @@ main() {
cd "$ROOT" cd "$ROOT"
"$SCRIPT_DIR/bootstrap" "$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check" "$SCRIPT_DIR/check"
# The version and the tag each get their own line: a failing # Own line: a failing command substitution inside an argument does
# command substitution inside an argument does not trip `set -e`, # not trip `set -e`, so the inline form degrades silently to an
# so the inline form degrades silently to an empty constant. The # empty constant. The VERSION build argument takes precedence over
# VERSION build argument takes precedence over the version a build # the version a build stage derives from the .git in the context.
# stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")"
docker build --no-cache \ docker build --no-cache \
--build-arg VERSION="$version" \ --build-arg VERSION="$version" \
-t "$tag" . -t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+5 -7
View File
@@ -10,17 +10,15 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
# The version and the tag each get their own line: a failing # Own line: a failing command substitution inside an argument does
# command substitution inside an argument does not trip `set -e`, # not trip `set -e`, so the inline form degrades silently to an
# so the inline form degrades silently to an empty constant. The # empty constant. The VERSION build argument takes precedence over
# VERSION build argument takes precedence over the version a build # the version a build stage derives from the .git in the context.
# stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)" version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown" [ -n "$version" ] || version="unknown"
tag="$("$SCRIPT_DIR/projectname")"
docker build --no-cache \ docker build --no-cache \
--build-arg VERSION="$version" \ --build-arg VERSION="$version" \
-t "$tag" . -t "$("$SCRIPT_DIR/projectname")" .
} }
main "$@" main "$@"
+13 -22
View File
@@ -1,32 +1,23 @@
#!/bin/sh #!/bin/sh
# script/fmt: format all files (writes). # script/fmt: format all files (writes): the Go sources with gofmt, the
# Markdown with prettier. prettier is never installed on the host: it
# runs from the Dockerfile's prettier stage with the repository mounted,
# as the calling user so the files it rewrites keep their owner. The
# build passes --no-cache, as every docker build in script/ does. The tag
# makes each build replace the previous image instead of leaving another
# one behind.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() { main() {
cd "$ROOT" cd "$ROOT"
gofmt -s -w . gofmt -s -w .
run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always image="$("$SCRIPT_DIR/projectname")-prettier"
docker build -q --no-cache --target prettier -t "$image" . >/dev/null
docker run --rm --user "$(id -u):$(id -g)" -v "$ROOT:/src" "$image" \
prettier --write '**/*.md' --tab-width 4 --prose-wrap always
} }
main "$@" main "$@"
+15 -26
View File
@@ -1,35 +1,19 @@
#!/bin/sh #!/bin/sh
# script/fmt-check: check formatting (read-only). # script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing. gofmt and prettier both run
# every time and each reports its own failure, so the output says which
# one failed.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt-check: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() { main() {
cd "$ROOT" cd "$ROOT"
status=0 status=0
# gofmt and prettier both run every time, so the output names each # Under set -e a bare assignment would end the script when gofmt
# one that fails. Under set -e a bare assignment would end the # fails (a Go file it cannot parse), and prettier would never run.
# script when gofmt fails (a Go file it cannot parse).
if ! files="$(gofmt -s -l .)"; then if ! files="$(gofmt -s -l .)"; then
echo "gofmt: failed; see its errors above" >&2 echo "gofmt: failed; see its errors above" >&2
status=1 status=1
@@ -40,9 +24,14 @@ main() {
status=1 status=1
fi fi
# run_yarn ends in exec; the subshell returns here afterwards. # Same image as script/fmt; see there.
(run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always) || image="$("$SCRIPT_DIR/projectname")-prettier"
docker build -q --no-cache --target prettier -t "$image" . >/dev/null
if ! docker run --rm -v "$ROOT:/src:ro" "$image" \
prettier --check '**/*.md' --tab-width 4 --prose-wrap always; then
echo "prettier: Markdown not formatted; run make fmt" >&2
status=1 status=1
fi
exit "$status" exit "$status"
} }
+7 -3
View File
@@ -1,15 +1,19 @@
#!/bin/sh #!/bin/sh
# script/install-precommit: install the git pre-commit hook that runs # script/install-precommit: install the git pre-commit hook that runs
# script/precommit. Our own extension to scripts-to-rule-them-all. # script/precommit. Our own extension to scripts-to-rule-them-all.
# Hooks are shared between the main checkout and all worktrees, so
# resolve the common git dir instead of assuming .git is a directory.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
hook=".git/hooks/pre-commit" hooks_dir="$(git rev-parse --git-common-dir)/hooks"
printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit mkdir -p "$hooks_dir"
chmod +x .git/hooks/pre-commit hook="$hooks_dir/pre-commit"
printf '#!/bin/sh\nset -e\nscript/precommit\n' > "$hook"
chmod +x "$hook"
echo "pre-commit hook installed: runs script/precommit" echo "pre-commit hook installed: runs script/precommit"
} }
+3 -3
View File
@@ -6,8 +6,8 @@
# #
# The phase is not the last stage in the file, so it is built only when # The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint # --target names it. --no-cache because a cached lint layer is a lint
# that did not run. --output type=cacheonly writes no image, since # that did not run. The tag makes each build replace the previous image
# nothing uses one. # instead of leaving a dangling one behind.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -17,7 +17,7 @@ main() {
cd "$ROOT" cd "$ROOT"
docker build --no-cache \ docker build --no-cache \
--target lint \ --target lint \
--output type=cacheonly . -t "$("$SCRIPT_DIR/projectname")-lint" .
} }
main "$@" main "$@"
+2 -2
View File
@@ -1,13 +1,13 @@
#!/bin/sh #!/bin/sh
# script/precommit: run by the git pre-commit hook; fails the commit if # script/precommit: run by the git pre-commit hook; fails the commit if
# checks fail. Our own extension to scripts-to-rule-them-all. # checks fail. Our own extension to scripts-to-rule-them-all. Go extra:
# go mod tidy must be a no-op before the checks run.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)" ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
# Go extra: go mod tidy must be a no-op before the checks run.
cd "$ROOT" cd "$ROOT"
go mod tidy go mod tidy
if ! git diff --exit-code -- go.mod go.sum; then if ! git diff --exit-code -- go.mod go.sum; then
+1 -1
View File
@@ -1,6 +1,6 @@
#!/bin/sh #!/bin/sh
# script/setup: set up the repo for development after a fresh clone: # script/setup: set up the repo for development after a fresh clone:
# installs dependencies and the git pre-commit hook. # installs dependencies (script/bootstrap) and the git pre-commit hook.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+3 -3
View File
@@ -2,8 +2,8 @@
# script/test: run the test suite. Testing is a phase of the Dockerfile # script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint: # and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when # --target because a phase that is not the last stage is built only when
# named, and --no-cache because a cached test layer is a test that did # named, --no-cache because a cached test layer is a test that did not
# not run. --output type=cacheonly writes no image, since nothing uses one. # run, and a tag so each build replaces the previous image.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -13,7 +13,7 @@ main() {
cd "$ROOT" cd "$ROOT"
docker build --no-cache \ docker build --no-cache \
--target test \ --target test \
--output type=cacheonly . -t "$("$SCRIPT_DIR/projectname")-test" .
} }
main "$@" main "$@"