Run all linting in Docker via Dockerfile.lint (closes #46)

Per the owner ruling, the linter runs inside a container invoked
through the script/ entrypoint and is never installed on a host. A new
root Dockerfile.lint COPYs the repo into the digest-pinned
golangci/golangci-lint:v2.12.2 image and runs `golangci-lint config
verify` and `golangci-lint run` as build steps, so a successful build
IS a clean lint. script/lint is reduced to building it, which also
works when the docker daemon is remote and bind mounts are impossible.

script/bootstrap loses the `go install`, the pin constants, the
version parser and verify_golangci_lint outright rather than being
hardened: with nothing linting on the host, the $GOPATH/bin versus
PATH shadowing problem those existed to diagnose has no subject. It
keeps the git/make/go presence checks and `go mod download`, and warns
rather than fails when docker is absent.

Two traps.

A lint build on an unchanged tree returns success in well under a
second having run no linter, which is #32 and #39 over again. Caching
is waived by ruling, so Dockerfile.lint carries ARG CHECK_EPOCH
referenced inside every gate RUN -- BuildKit hashes the expanded
command, not the declaration, so a declared but unreferenced ARG
invalidates nothing -- and script/lint passes "$(date +%s)-$$". The
PID is in that value because two lint runs land inside the same second
easily and a bare epoch would cache the second one.

Nothing inside an image build may shell out to docker. The main
Dockerfile's lint stage therefore invokes golangci-lint directly
instead of `make lint`, and its build stage runs `make test` and
`make fmt-check` instead of the `make check` aggregate, which reaches
script/lint. Those two remain `make` invocations rather than the bare
scripts because the Makefile's `export CGO_ENABLED = 0` only applies
to what it invokes, and today's `make check` gets it.

COPY --from=lint /usr/bin/golangci-lint is replaced by
COPY --from=lint /src/go.sum /dev/null. The copied binary was the only
edge forcing BuildKit to finish linting before the build stage starts;
dropping it without replacing the edge would have ended fail-fast
linting silently under a still-green build. That no-op copy is the
ordering edge canonical REPO_POLICIES.md prescribes. Nothing in the
build stage runs the linter any more, so the binary itself is not
wanted there, and ENV PATH=/home/builder/go/bin:$PATH goes with the
`go install` that justified it.

script/verify-linter-pin is retired -- deleted along with its README
entry -- because both of its subjects ceased to exist in this same
change: it compared a linter binary against GOLANGCI_LINT_VERSION in
script/bootstrap, and there is now neither a binary crossing between
stages nor a version pin in bootstrap. The drift it guarded has not
gone away, it has moved. The linter is still pinned twice, now as the
FROM line of Dockerfile.lint and the FROM line of the Dockerfile lint
stage, with nothing syncing them, which is exactly what #42 made a
build failure. Its replacement is one new script/verify-lint-image-pin
that compares those two references to each other and deliberately
restates neither pin: a hardcoded expected digest would be a third
copy and the same drift one file further out. It runs as a gate in
both files, so `make lint`, `make check` and `make docker` all catch
drift, and an unreadable reference is a hard failure rather than a
vacuous pass.

`golangci-lint config verify` is included per the ruling. The concern
about its unpinned live HTTPS schema fetch was measured rather than
assumed: under --network none the pinned binary both passes a valid
config and rejects an invalid one with the jsonschema error, so it
validates against a schema it embeds and linting needs no network
beyond pulling the image. The README states that rather than a
requirement that does not exist.

Verified. `make lint` green with every PATH directory containing a
golangci-lint removed and `command -v golangci-lint` empty. Two
consecutive script/lint runs on an untouched tree both executed the
linter, 27.7s and 28.7s in the lint step under distinct epochs with
the COPY layer CACHED above them. A planted unused variable failed
script/lint with that exact finding and failed `make docker` at the
lint stage with the build stage stopped before its COPY --from=lint,
then reverted clean. The drift guard fails on tag-only, digest-only
and unreadable-reference cases, naming both sides. `make check` green;
`make docker` green in 5m35s with all six gates executing and the test
gate reporting real coverage rather than a cached ok. In the builder
image with the Go test cache off, --user 0:0 still fails
TestScanHardlinkRunFailsTogether where the unprivileged user passes,
so the non-root quirk is intact.

.claude/settings.json

@@ -0,0 +1,5 @@
+{
+    "worktree": {
+        "bgIsolation": "none"
+    }
+}

.gitea/workflows/check.yml

@@ -6,4 +6,4 @@ jobs:
         steps:
             # actions/checkout v4.2.2, 2026-02-22
             - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-            - run: docker build .
+            - run: script/cibuild

.gitignore

@@ -28,6 +28,9 @@ node_modules/
 
 # Local scan data
 files.dat
+*.sqlite
+*.sqlite-shm
+*.sqlite-wal
 
 # Agent worktrees
 .claude/worktrees/

.golangci.yml

@@ -1,5 +1,9 @@
 version: "2"
 
+# Config schema uses the golangci-lint v2 layout (settings live under
+# linters.settings, not top-level linters-settings) so that the
+# thresholds below are actually applied by golangci-lint >= v2.
+
 run:
   timeout: 5m
   modules-download-mode: readonly
@@ -14,19 +18,17 @@ linters:
     - wsl              # Deprecated, replaced by wsl_v5
     - wrapcheck        # Too verbose for internal packages
     - varnamelen       # Short names like db, id are idiomatic Go
-
-linters-settings:
-  lll:
-    line-length: 88
-  funlen:
-    lines: 80
-    statements: 50
-  cyclop:
-    max-complexity: 15
-  dupl:
-    threshold: 100
+  settings:
+    lll:
+      line-length: 88
+    funlen:
+      lines: 80
+      statements: 50
+    cyclop:
+      max-complexity: 15
+    dupl:
+      threshold: 100
 
 issues:
-  exclude-use-default: false
   max-issues-per-linter: 0
   max-same-issues: 0

Dockerfile

@@ -1,31 +1,112 @@
 # Lint stage — fast feedback on formatting and lint issues
-# golangci/golangci-lint:v2.12.1, 2026-07-23
-FROM golangci/golangci-lint@sha256:c9843d374ca80ecbac86081ec4dd7fe2bb6187b03224f59a0cc2f80759e1845b AS lint
+# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
+FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
 WORKDIR /src
 COPY go.mod go.sum ./
 RUN go mod download
 COPY . .
-RUN make fmt-check
-RUN make lint
+
+# Cache-buster for the gate layers, and only for them. Docker
+# invalidates COPY only when the copied content changes, so on an
+# unchanged tree the gates below would be served from cache and the
+# build would exit 0 having run nothing. script/cibuild and
+# script/docker pass a fresh CHECK_EPOCH on every invocation.
+#
+# Two properties this depends on. ARG is per-stage, so the build stage
+# below declares it again; one declaration here would leave that
+# stage's gate cacheable. And each gate RUN must reference the value,
+# because BuildKit hashes the expanded command: a declared but
+# unreferenced ARG invalidates nothing.
+#
+# It sits below the dependency layers deliberately. Everything above it
+# (the pinned base image, go mod download) keeps its cache; only the
+# gates go cold.
+ARG CHECK_EPOCH
+
+# The linter is invoked directly here, not through `make lint`. That
+# target now runs `docker build -f Dockerfile.lint`, and a docker build
+# cannot run a docker build: routing the gate through make would mean
+# nesting docker inside this image. Same reason `make check` is gone
+# from the build stage below. `make fmt-check` stays as it is — it is a
+# gate, not the aggregate, and it shells out to nothing.
+RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
+
+# The FROM above and the one in Dockerfile.lint pin the same linter
+# twice, and nothing else keeps them in sync; this fails the build when
+# they disagree. See the script for why it restates neither pin.
+RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
+    script/verify-lint-image-pin
+
+# Same config-schema check Dockerfile.lint runs, kept here so this build
+# gates on exactly what script/lint gates on. It validates against a
+# schema the pinned binary embeds, so it needs no network.
+RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
+    golangci-lint config verify --config .golangci.yml
+
+RUN echo "gate lint, epoch ${CHECK_EPOCH}" && \
+    golangci-lint run --config .golangci.yml ./...
 
 # Build stage
 # golang:1.25-alpine, 2026-07-23
 FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder
 
-RUN apk add --no-cache make
+# We never build or run as root. Create an unprivileged user and point
+# HOME and the Go caches at its home so go build and go test can write
+# their caches when we drop to it below. $GOPATH/bin is deliberately not
+# on PATH: script/bootstrap no longer `go install`s anything (the linter
+# runs from a pinned image, never from a host install), so nothing lands
+# there and adding it would only widen what this image resolves.
+RUN adduser -D -u 1000 builder
+ENV HOME=/home/builder
+ENV GOPATH=/home/builder/go
+ENV GOCACHE=/home/builder/.cache/go-build
 
 WORKDIR /src
 
-# Reuse the linter binary from the lint stage; the copy also forces
-# BuildKit to complete linting before this stage proceeds.
-COPY --from=lint /usr/bin/golangci-lint /usr/local/bin/golangci-lint
+# No-op file copy whose only purpose is the build-graph edge: it is what
+# makes this stage depend on the lint stage, and so what forces BuildKit
+# to finish fmt-check, the pin guard and lint before compilation and
+# tests start. Remove it and the fail-fast design dies silently — the
+# build stops gating on lint and still exits 0. It replaces a copy of
+# the linter binary itself, which is no longer wanted here: nothing in
+# this stage runs the linter, because `make lint` is now a docker build
+# and a docker build cannot run inside one.
+COPY --from=lint /src/go.sum /dev/null
 
+# Install development prerequisites the same way a developer does,
+# rather than duplicating the installs inline. Only script/ and the
+# dependency manifests are copied first, nothing else, so this layer
+# stays cached until the scripts or the dependencies change — bootstrap
+# ends in `go mod download`, which is why there is no separate
+# invocation of it here.
+COPY script/ script/
 COPY go.mod go.sum ./
-RUN go mod download
+RUN script/bootstrap
+
 COPY . .
 
-# Fail the build unless the branch is green.
-RUN make check
+# Hand the sources and caches to the unprivileged user, then drop root
+# before running any checks or builds.
+RUN chown -R builder:builder /src /home/builder
+USER builder
+
+# Fail the build unless the branch is green. Runs as non-root so the
+# permission-denied test paths are exercised legitimately (root would
+# bypass the chmod(0) the tests rely on).
+#
+# The gates are the individual targets, not `make check`: that aggregate
+# runs `script/lint`, which is now a docker build, and nothing inside an
+# image build may shell out to docker. Lint is not skipped by this — it
+# ran in the lint stage above, which this stage's COPY --from makes a
+# prerequisite. `make`, not the scripts directly, because the Makefile's
+# `export CGO_ENABLED = 0` applies only to what it invokes.
+#
+# Second per-stage declaration of the gate cache-buster; see the lint
+# stage above for why one is not enough. It is placed after USER so the
+# drop to the unprivileged user still happens before the checks run.
+ARG CHECK_EPOCH
+RUN echo "gate test, epoch ${CHECK_EPOCH}" && make test
+RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
 
 RUN make build
 

Dockerfile.lint

@@ -0,0 +1,59 @@
+# Lint-only image: this is how the linter runs, everywhere. The repo is
+# COPYed into the pinned golangci-lint image and the linter runs as a
+# build step, so a successful build IS a clean lint. golangci-lint is
+# never installed on a host — one toolchain, pinned by digest, identical
+# on a laptop and in CI — and this works even when the docker daemon is
+# remote and bind mounts are impossible.
+#
+# script/lint builds this file. It is a separate image from the lint
+# stage of the main Dockerfile because script/lint must not depend on
+# the rest of that build; the two FROM lines are kept identical by
+# script/verify-lint-image-pin, run as a gate below.
+# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
+FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240
+
+WORKDIR /src
+
+# Dependency layers first, so they stay cached across lint runs.
+COPY go.mod go.sum ./
+RUN go mod download
+
+COPY . .
+
+# Cache-buster for the gate layers, and only for them. Caching of the
+# lint run is waived by ruling: COPY is invalidated only by changed
+# content, so on an unchanged tree the gates below would be served from
+# cache and this build would exit 0 in under a second having run no
+# linter at all. That exact false green has bitten this repo twice
+# already (#32, #39). script/lint passes a fresh value on every
+# invocation.
+#
+# Each gate RUN must reference the value, because BuildKit hashes the
+# expanded command and not the ARG declaration: a declared but
+# unreferenced ARG invalidates nothing. The ARG sits below the
+# dependency layers deliberately — everything above it keeps its cache,
+# only the gates go cold.
+ARG CHECK_EPOCH
+
+# The linter version is pinned in two places, here and in the main
+# Dockerfile's lint stage. Nothing else keeps them in sync, so a
+# half-applied bump is a build failure; see the script.
+RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
+    script/verify-lint-image-pin
+
+# Validates .golangci.yml against golangci-lint's JSON schema. The
+# concern about this step was that it fetches that schema over a live,
+# unpinned HTTPS call; measured on the pinned image, it does not. The
+# binary carries the schema for its own version, so under
+# `--network none` this both passes on a valid config and still rejects
+# an invalid one with the jsonschema error. That holds for the gate
+# steps generally — none of them makes a network call — but not for
+# this build as a whole: `go mod download` above needs the network on a
+# cold cache, and under `--network none` a first build fails there
+# before reaching any gate. That layer stays cached, so only a warm
+# cache lints offline, until go.mod or go.sum changes.
+RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
+    golangci-lint config verify --config .golangci.yml
+
+RUN echo "gate lint, epoch ${CHECK_EPOCH}" && \
+    golangci-lint run --config .golangci.yml ./...

Makefile

@@ -6,43 +6,44 @@ BINARY := sfdupes
 VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
 LDFLAGS := -X main.Version=$(VERSION)
 
-.PHONY: all build test lint fmt fmt-check check docker hooks clean
+.PHONY: sfdupes build bootstrap setup test lint fmt fmt-check check docker hooks clean
 
-all: check build
+# Standard targets are thin shims; the implementations live in script/
+# per the scripts-to-rule-them-all pattern.
 
-build:
+# Default target: build the binary. Phony so go build (which has its
+# own build cache) always decides what to recompile.
+sfdupes:
 	go build -ldflags "$(LDFLAGS)" -o $(BINARY)
 
+build: sfdupes
+
+bootstrap:
+	@script/bootstrap
+
+setup:
+	@script/setup
+
 test:
-	@go test -timeout 30s -cover ./... || \
-		{ echo "--- Rerunning with -v for details ---"; \
-		  go test -timeout 30s -v ./...; exit 1; }
+	@script/test
 
 lint:
-	golangci-lint run --config .golangci.yml ./...
+	@script/lint
 
 fmt:
-	gofmt -s -w .
+	@script/fmt
 
 fmt-check:
-	@files="$$(gofmt -l -s .)"; if [ -n "$$files" ]; then \
-		echo "gofmt: files not formatted:"; echo "$$files"; exit 1; fi
+	@script/fmt-check
 
-check: test lint fmt-check
+check:
+	@script/check
 
 docker:
-	docker build -t $(BINARY) .
-
-# Hooks are shared between the main checkout and all worktrees, so
-# resolve the common git dir instead of assuming .git is a directory.
-HOOKS_DIR := $(shell git rev-parse --git-common-dir)/hooks
+	@script/docker
 
 hooks:
-	@printf '#!/bin/sh\nset -e\n' > $(HOOKS_DIR)/pre-commit
-	@printf 'go mod tidy\ngo fmt ./...\n' >> $(HOOKS_DIR)/pre-commit
-	@printf 'git diff --exit-code -- go.mod go.sum || { echo "go mod tidy changed files; please stage and retry"; exit 1; }\n' >> $(HOOKS_DIR)/pre-commit
-	@printf 'make check\n' >> $(HOOKS_DIR)/pre-commit
-	@chmod +x $(HOOKS_DIR)/pre-commit
+	@script/install-precommit
 
 clean:
 	rm -f $(BINARY) files.dat

README.md

@@ -12,7 +12,9 @@ SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
 content (the middle of the file is never read); the intended use is
 finding duplicate downloads and duplicated directory trees on
 multi-terabyte ZFS servers where reading every byte is prohibitively
-expensive.
+expensive. `scan` maintains a persistent SQLite database of file
+signatures that survives between runs, so it can be run from cron and
+the reports can be generated at any time from the most recent scan.
 
 This README is the complete and authoritative specification.
 
@@ -20,29 +22,41 @@ This README is the complete and authoritative specification.
 
 ```sh
 make build
-./sfdupes scan /srv > files.dat
-./sfdupes report files.dat > dupes.tsv
-./sfdupes trees files.dat > dupetrees.tsv
+export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
+./sfdupes scan /srv
+./sfdupes report > dupes.tsv
+./sfdupes trees > dupetrees.tsv
 ```
 
-`scan` walks one or more filesystem trees and emits one record per
-regular file (path, size, mtime, head hash, tail hash). `report`
-ingests that stream and prints the file-level duplicates report.
-`trees` ingests the same stream and prints the duplicate-tree report. A
-missing/invalid subcommand — or a `scan` invocation with no `PATH`
-operand — prints a usage message and exits 2.
+`scan` walks one or more filesystem trees and maintains one database
+record per regular file (path, size, mtime, head hash, tail hash). The
+database persists between runs; a rescan only hashes files that are new
+or changed, and removes records for files that no longer exist.
+`report` reads the database and prints the file-level duplicates
+report. `trees` reads the same database and prints the duplicate-tree
+report. A missing/invalid subcommand — or a `scan` invocation with no
+`PATH` operand — prints a usage message and exits 2.
+
+The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed
+anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a
+daily `sfdupes scan` cron job, with the reporting commands run
+interactively whenever needed; their results are as fresh as the last
+completed scan.
 
 ## Rationale
 
 Duplicate finders that hash entire files do not scale to the target
 environment: ~10 million files and ~150 TB on possibly slow or busy
-disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes
-a full-filesystem sweep tractable, and the resulting scan stream is
-self-contained, so the expensive filesystem pass runs exactly once and
-all analysis happens offline. The end goal is not individual files but
-whole duplicated trees — duplicate extractions, duplicate downloads,
-copied project trees — which an operator can consider removing as a
-unit.
+disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
+only from files whose size at least one other file shares, since a
+size-unique file cannot be a duplicate — makes a full-filesystem sweep
+tractable, and the signatures are kept in a persistent database, so
+the expensive filesystem pass is incremental: a rescan re-hashes only
+files whose recorded mtime or size changed, and all analysis happens
+offline from the database alone. The end goal is
+not individual files but whole duplicated trees — duplicate
+extractions, duplicate downloads, copied project trees — which an
+operator can consider removing as a unit.
 
 ## Design
 
@@ -55,14 +69,19 @@ Goals, in order:
     removing an entire subtree at once. File-level duplicate detection is
     the foundation; tree-level detection is built on top of it.
 2.  **Never read full file contents.** At most 2 KiB is read per file
-    (first and last 1024 bytes). Scale target: ~10 million files, ~150 TB
+    (first and last 1024 bytes), and only files whose size at least
+    one other file shares are read at all — a size-unique file cannot
+    be a duplicate. Scale target: tens of millions of files, ~150 TB
     filesystem, possibly slow or busy disks (ZFS pool under resilver).
-    Holding the full file list in memory is acceptable; reading file
-    contents beyond 2 KiB per file is not.
-3.  **Scan once, analyze offline.** The expensive filesystem scan
-    produces a self-contained stream; all analysis (`report`, `trees`)
-    works from that stream alone and must never touch the scanned
-    filesystem again.
+    Holding one small record (path, size, mtime) per file in memory
+    during a scan is acceptable; holding every file's hashes is not
+    (they stay in the database).
+3.  **Scan incrementally, analyze offline.** The expensive filesystem
+    scan maintains a persistent database; an unchanged file is never
+    read again on a rescan. All analysis (`report`, `trees`) works from
+    the database alone and must never touch the scanned filesystem
+    again. `scan` is designed to be cronned; the reports run at any
+    time against the last completed scan.
 4.  **Clean stream separation.** Everything on stdout is machine-readable
     data. All progress, warnings, and summaries go to stderr. Never mix
     them.
@@ -71,54 +90,162 @@ Goals, in order:
 
 -   Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
     `sfdupes`.
--   Dependencies: standard library, `github.com/spf13/cobra` for the CLI,
-    and **one progress-bar library**
-    (`github.com/schollz/progressbar/v3`). `github.com/spf13/viper` is
-    permitted if configuration-file support is ever needed, but is not
-    currently used. No other third-party deps.
+-   Dependencies: standard library, `github.com/spf13/cobra` for the
+    CLI, **one progress-bar library**
+    (`github.com/schollz/progressbar/v3`), and **one SQLite driver**
+    (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
+    `github.com/spf13/viper` is permitted if configuration-file support
+    is ever needed, but is not currently used. No other third-party
+    deps.
 -   Cross-compilation is not a concern. Builds run with cgo disabled (the
     `Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
 -   Analysis modes (`report`, `trees`) must be deterministic: identical
-    input stream, identical output, regardless of record order.
+    database contents, identical output, regardless of the order in
+    which records were inserted.
 
 ### Subcommands
 
 Three subcommands, all implemented:
 
-1.  `scan` — walk the filesystem and emit one signature record per
-    regular file.
-2.  `report` — file-level duplicate report from the scan stream.
+1.  `scan` — walk the filesystem and synchronize the database: one
+    signature record per regular file.
+2.  `report` — file-level duplicate report from the database.
 3.  `trees` — tree-level duplicate report: reconstruct the directory
-    hierarchy from the scan stream, compute a Merkle-style digest per
-    directory, and report maximal groups of identical trees.
+    hierarchy from the database records, compute a Merkle-style digest
+    per directory, and report maximal groups of identical trees.
 
 ```
-sfdupes scan [--workers N] [-x] PATH... > files.dat
-sfdupes report [files.dat|-] > dupes.tsv
-sfdupes trees [files.dat|-] > dupetrees.tsv
+sfdupes scan [--workers N] [-x] PATH...
+sfdupes report > dupes.tsv
+sfdupes trees > dupetrees.tsv
 ```
 
+### Database
+
+All three subcommands operate on a single SQLite database file:
+
+-   Location: the value of the `SFDUPES_DATABASE` environment variable
+    when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`.
+    There is no command-line flag.
+-   `scan` creates the database (and its parent directory) on first
+    use. `report` and `trees` require an existing database; a missing
+    database file is a fatal error (exit 1) telling the user to run
+    `scan` first.
+-   The database uses WAL journal mode and a busy timeout, so running a
+    report while a cron `scan` is in progress is safe. The filesystem
+    is authoritative; the database is an eventually-consistent
+    reflection of it. Hashed records are committed in batched
+    transactions while the scan is still running (keeping the WAL
+    small and letting concurrent reports observe progress), so a
+    report may see a scan's changes partially applied, and a scan
+    that dies partway leaves a valid database holding everything
+    hashed so far; the next scan skips those records and converges
+    toward the filesystem.
+-   Schema (`PRAGMA user_version` is the schema version, currently 1; a
+    database with any other version is a fatal error):
+
+    ```sql
+    CREATE TABLE files (
+        path  BLOB PRIMARY KEY,  -- absolute path, raw bytes
+        size  INTEGER NOT NULL,  -- bytes, from lstat
+        mtime INTEGER NOT NULL,  -- Unix seconds, from lstat
+        head  TEXT NOT NULL,     -- lowercase-hex SHA-256, first 1 KiB
+        tail  TEXT NOT NULL      -- lowercase-hex SHA-256, last 1 KiB
+    ) WITHOUT ROWID;
+    ```
+
+    Paths are stored as BLOBs because Unix paths are raw bytes, not
+    guaranteed UTF-8. `mtime` is used only for change detection; it is
+    not part of the duplicate key. `head` and `tail` are empty strings
+    when the file has never been hashed because its size was unique as
+    of the last scan that covered it; such records still define the
+    file for tree reconstruction but never participate in duplicate
+    groups.
+
 ### `scan` mode
 
 `scan` requires one or more `PATH` operands naming the trees to scan.
 There is no default path; invoking `scan` with no operand is a usage
 error (usage message on stderr, exit 2). An operand may be a directory
 or a regular file; an operand that does not exist is a fatal error
-(exit 1). Operands are walked in the order given; overlapping operands
-(one containing another) emit their common files once per operand, so
-callers should pass disjoint paths.
+(exit 1). Because database records persist between runs and are keyed
+by absolute path, each operand is resolved to an absolute, lexically
+cleaned path (symlinks are not resolved) before walking, so results do
+not depend on the working directory. All operands belong to a single
+scan and are enumerated concurrently: every operand seeds the shared
+walk worker pool. Overlapping operands are harmless — an operand that
+duplicates another or lies under another is dropped before walking,
+so every file is reached exactly once and produces one database
+record.
 
-`scan` runs **three sequential passes**, in this order, so that every
-expensive pass has an exact total for meaningful progress and ETA:
+`scan` synchronizes the database with the filesystem state under the
+scanned operands:
 
-1.  **walk** — recursively enumerate the tree under each `PATH` in
-    turn, collecting the list of regular-file paths. Total unknown
-    while running: show a live count, not a percentage.
-2.  **stat** — `lstat` every collected path, recording size and mtime.
-3.  **hash** — for each file, read the first `min(1024, size)` bytes and
-    the last `min(1024, size)` bytes (the two reads overlap when
-    `size < 2048`; for `size == 0` hash the empty input) and compute the
-    SHA-256 of each. Emit the output record.
+-   Only a file whose size at least one other file shares is ever
+    read: a size-unique file cannot be a duplicate, so it is recorded
+    without hashes (`head` and `tail` empty). The size census covers
+    every file walked this scan plus every database record outside
+    the scanned operands, so a possible duplicate of a separately
+    scanned tree is still recognized.
+-   A file not yet in the database is inserted: hashed when its size
+    is shared, without hashes otherwise.
+-   A file already in the database is **skipped without reading its
+    contents** when its lstat size equals the recorded size and its
+    lstat mtime is not newer than the recorded mtime. This is what
+    makes a daily rescan cheap. Exception: an unchanged file whose
+    record lacks hashes is hashed — and its record updated — once its
+    size becomes shared, so hashing deferred by size-uniqueness
+    happens as soon as it could matter.
+-   A file whose mtime is newer than recorded, or whose size differs,
+    is processed as if new: re-hashed, or recorded without hashes,
+    per the shared-size rule.
+-   A database record whose path lies under one of the scanned operands
+    but was not successfully processed this run is deleted. This
+    removes records for deleted files. It also removes records for
+    paths that failed to stat or hash this run: the database only ever
+    contains signatures verified by the most recent scan that covered
+    them (a subsequent successful scan re-adds such files).
+-   Database records outside the scanned operands are untouched, so
+    disjoint trees can be scanned on different schedules into the same
+    database.
+
+`scan` runs **three sequential phases over the whole scan**.
+Parallelism lives inside each phase; batched database writes begin
+during the hash phase:
+
+1.  **walk + stat** — enumerate the trees under all `PATH` operands
+    concurrently with the walk worker pool: every operand seeds the
+    shared queue, and each worker reads one directory at a time,
+    handing discovered subdirectories back to the queue and running
+    `lstat` on each regular file as it is discovered (while the
+    directory's metadata is still hot). Sequential directory
+    enumeration is metadata-latency-bound and takes hours at tens of
+    millions of files; per-directory parallelism is what makes the
+    walk tractable on large or busy pools. The walk builds the size
+    census and resolves unchanged already-hashed files on the fly;
+    every other file is carried to the hash phase as a (path, size,
+    mtime) record.
+2.  **hash** — with the census complete, each carried file's size
+    decides its fate. Size-unique files are never read: new or
+    changed ones are recorded without hashes in the update phase,
+    unchanged unhashed ones simply keep their records. Every file
+    with a shared size is hashed by the worker pool: read the first
+    `min(1024, size)` bytes and the last `min(1024, size)` bytes
+    (one read when `size <= 1024`, since the two windows coincide)
+    and compute the SHA-256 of each. Zero-length files have constant
+    hashes and are never opened. Files are hashed in **inode order**
+    (minimizing seeks on spinning disks), and paths that are hard
+    links to the same inode are **read once**, all sharing the one
+    result — a hard-link backup farm costs one read per inode, not
+    per path. The phase total counts actual reads, so progress and
+    ETA are meaningful. Completed records are committed in batched
+    transactions **while hashing runs**, so a scan interrupted after
+    hours keeps everything hashed so far and the next scan resumes
+    cheaply, skipping records already written.
+3.  **update** — commit the final partial batch, the hash-less
+    records for size-unique new and changed files, and the deletions
+    for records the scan did not verify (vanished files, plus paths
+    that failed to stat or hash).
 
 Rules for the walk:
 
@@ -134,52 +261,55 @@ Rules for the walk:
 -   On any per-path error (permission denied, file vanished between
     passes, unreadable): print a one-line warning to stderr, skip the
     path, and continue. Per-file errors never abort the run; the final
-    summary reports how many were skipped.
+    summary reports how many were skipped. As specified above, a
+    skipped path that has a database record from an earlier scan loses
+    that record; an unreadable directory subtree likewise loses its
+    records (accepted: the database mirrors what the latest scan could
+    actually verify).
 
-Concurrency: the stat and hash passes use a worker pool (`--workers`,
-default `runtime.NumCPU()`). The main goroutine owns stdout writing and
-progress rendering; progress display must never block the workers.
+Concurrency: the walk phase (which also stats files) and the hash
+phase each use a worker pool of `--workers` workers (default
+`runtime.NumCPU()`); the walk parallelizes across directories,
+hashing across files. Both phases are seek-bound on spinning disks,
+so raising `--workers` well past the core count can help on pools
+with many spindles. The main goroutine owns partitioning, database
+writes, and progress rendering; progress display must never block
+the workers.
 
-#### Output record format
-
-One record per file on stdout, NUL-terminated (`\x00`), with
-tab-separated fields, **path last** so tabs or newlines embedded in
-paths cannot corrupt the record structure:
+`scan` writes nothing to stdout. The summary line on stderr reports the
+files seen this run broken down by disposition, plus skips:
 
 ```
-<size>\t<mtime_unix>\t<sha256_first1k_hex>\t<sha256_last1k_hex>\t<path>\x00
+scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
 ```
 
--   `size`: decimal bytes, from the stat pass.
--   `mtime_unix`: decimal Unix seconds. Informational only; not part of
-    the duplicate key.
--   Hashes: lowercase hex, 64 chars each.
--   Record order is unspecified (workers complete out of order); the
-    analysis modes must not depend on ordering.
+(`removed` counts deleted database records, which are not part of the
+files-seen total.)
 
 ### `report` mode
 
-`report` reads the scan stream from the file named in its first
-positional argument, or from stdin if the argument is absent or `-`.
+`report` reads every record from the database and takes no positional
+arguments.
 
 **`report` must never touch the filesystem being analyzed.** It does not
 stat, open, or otherwise access any path that appears in the records; its
-only I/O is reading the scan file/stdin and writing stdout/stderr. It must
+only I/O is reading the database and writing stdout/stderr. It must
 produce identical output whether or not the scanned filesystem is still
 mounted.
 
 Processing:
 
--   Parse records; a record that does not have exactly 5 fields or whose
-    size is non-numeric is counted as malformed and skipped (warn once
-    with the total malformed count in the summary, not per record).
--   Group records by the key `(size, head_hash, tail_hash)`.
+-   Records without hashes (size-unique when last scanned) are
+    excluded: their content is unknown, so they are never reported as
+    duplicates.
+-   Group the remaining records by the key
+    `(size, head_hash, tail_hash)`.
 -   Every group with two or more paths is a duplicate group.
 -   Within each group, sort paths lexicographically (byte order). The
     first path is the group's `first`; every other path is a `dupe`.
 -   Order groups by size descending (biggest reclaimable space first),
     tie-broken by `first` path ascending. Output must be fully
-    deterministic for a given input.
+    deterministic for a given database state.
 
 #### Report output format
 
@@ -192,16 +322,16 @@ first	dupe	size
 /srv/a/big.iso	/srv/c/big-copy2.iso	4294967296
 ```
 
-Summary to stderr: records read, malformed count (if any), number of
-duplicate groups, number of dupe files, and total reclaimable bytes
-(sum of `size` over all dupe rows) in human units.
+Summary to stderr: records read, number of duplicate groups, number of
+dupe files, and total reclaimable bytes (sum of `size` over all dupe
+rows) in human units.
 
 ### `trees` mode
 
-`trees` reads the same scan stream as `report` (same argument handling,
-same parsing and malformed-record rules) and reports **entire duplicate
-directory trees**: directories under which the exact same set of relative
-paths exists with the exact same file signatures.
+`trees` reads the same database as `report` (no positional arguments)
+and reports **entire duplicate directory trees**: directories under
+which the exact same set of relative paths exists with the exact same
+file signatures.
 
 **`trees` must never touch the filesystem being analyzed** — the same
 rule as `report`. The directory hierarchy is reconstructed purely from
@@ -210,7 +340,10 @@ the paths in the records, split on `/`.
 Definitions:
 
 -   A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
-    informational and excluded.
+    informational and excluded. An unhashed record (empty hashes) has
+    unknown content: its signature is treated as unique to that file,
+    so a tree containing an unhashed file never compares equal to any
+    other tree.
 -   A directory's **digest** is a SHA-256 Merkle digest computed
     bottom-up: serialize the directory's child entries — for a file
     child, its name and signature; for a subdirectory child, its name
@@ -222,10 +355,16 @@ Definitions:
     equal. Equal digests imply equal recursive file count and equal
     total byte size.
 
-Known limitation (accepted): only regular files that appear in the scan
-stream define a tree. Empty directories are invisible, and a file skipped
-during the scan (e.g. permission error) in one copy but not the other
-will make otherwise-identical trees compare as different.
+Known limitation (accepted): hard-linked paths are reported as
+duplicates by `report` and count toward duplicate trees — their
+content is genuinely identical — even though they share storage, so
+removing one reclaims no space. Inode identity is used during the
+scan to avoid redundant reads but is not persisted in the database.
+
+Known limitation (accepted): only regular files that appear in the
+database define a tree. Empty directories are invisible, and a file
+skipped during the scan (e.g. permission error) in one copy but not the
+other will make otherwise-identical trees compare as different.
 
 Processing:
 
@@ -254,32 +393,35 @@ first	dupe	files	size
 /srv/a/project	/srv/backup/project	3417	104857600
 ```
 
-Summary to stderr: records read, malformed count (if any), number of
-duplicate-tree groups, number of dupe trees, and total reclaimable bytes
-(sum of `size` over all dupe rows) in human units.
+Summary to stderr: records read, number of duplicate-tree groups,
+number of dupe trees, and total reclaimable bytes (sum of `size` over
+all dupe rows) in human units.
 
 ### Progress
 
-Use the progress-bar library for all scan-pass progress; rendering in the
+Use the progress-bar library for all scan progress; rendering in the
 style of `pv` is the model. All progress goes to stderr.
 
-Each scan pass gets its own bar. Required elements for the stat and hash
-passes (known totals):
+Each phase gets its own display, rendered the moment the phase
+starts — a scan must never look hung. Loading the existing-record
+index (`load`) and the walk have no known totals while running: show
+a live count, rate, and elapsed time (spinner-style, no percentage or
+ETA). The hash and update phases
+have exact totals — only files that actually need hashing appear in
+the hash total, so its ETA is meaningful. Required elements for the
+bars with known totals:
 
 -   elapsed time
 -   estimated time remaining
--   a `[m/n] x%` display (files processed / total files, percent)
--   current rate (files/s)
+-   a `[m/n] x%` display (items processed / total items, percent)
+-   current rate (items/s)
 
 Example shape (exact layout is flexible, content is not):
 
 ```
-hash: [1234567/9876543] 12% |████      | 8123 files/s elapsed 2:32 eta 17:54
+hash: [12345/98765] 12% |████      | 92 files/s elapsed 2:32 eta 17:54
 ```
 
-The walk pass has no known total: show a live file count and elapsed time
-(spinner-style, no percentage or ETA).
-
 Additional requirements:
 
 -   When stderr is not a TTY, do not emit ANSI redraws: print a plain
@@ -292,25 +434,137 @@ Additional requirements:
 ### Error handling and exit codes
 
 -   `0`: success, even if individual files were skipped with warnings.
--   `1`: fatal error (e.g., a `PATH` operand does not exist, cannot
-    read the scan input, stdout write failure).
--   `2`: usage error (including `scan` with no `PATH` operand).
+-   `1`: fatal error (e.g., a `PATH` operand does not exist, the
+    database cannot be created/opened/read/written, a missing database
+    for `report`/`trees`, stdout write failure).
+-   `2`: usage error (including `scan` with no `PATH` operand and
+    `report`/`trees` with any positional argument).
+
+## Entrypoints
+
+This repository adheres to the
+[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
+standard: the normalized executables in `script/` are the entrypoints
+for the development workflow, and the `Makefile` targets are thin
+shims that call them. Every script is POSIX `sh`, resolves the
+repository root itself so it can be run from any working directory,
+and may be invoked directly. The provided entrypoints are:
+
+-   `script/bootstrap` — install everything needed to build and
+    develop this repository, idempotently, assuming nothing is
+    present. `git`, `make`, and `go` come from the first of nix, apt,
+    brew, or apk found on the host, and are presence-checked only.
+    `golangci-lint` is deliberately **not** installed: it runs from a
+    digest-pinned image via `script/lint` and never from a host
+    install, so there is no host copy to drift from the pin. A missing
+    `docker` is warned about rather than installed or treated as
+    fatal — everything except linting works without it. Ends with
+    `go mod download`.
+-   `script/setup` — make a fresh clone ready for development: runs
+    `script/bootstrap`, then `script/install-precommit`.
+-   `script/projectname` — print this project's name (`sfdupes`).
+    Scripts that need the name call it, so they stay identical across
+    repositories.
+-   `script/test` — run the test suite with a 30-second timeout and
+    coverage enabled, rerunning verbosely on failure so the logs show
+    which test failed.
+-   `script/lint` — run the linter. It builds `Dockerfile.lint`, which
+    copies the repository into the digest-pinned
+    `golangci/golangci-lint` image and runs
+    `golangci-lint config verify` and `golangci-lint run` as build
+    steps, so a successful build is a clean lint. The linter is never
+    run on the host, which makes a working `docker` the one
+    prerequisite for linting — and therefore for `make check` and the
+    pre-commit hook. Offline machines: the gate steps themselves make
+    no network calls. `golangci-lint run` does not, and neither does
+    `golangci-lint config verify` — it validates against a schema the
+    pinned binary embeds, measured under `--network none` to both
+    pass a valid config and reject an invalid one. The build around
+    them does. `Dockerfile.lint` runs `go mod download` before the
+    gates and this module has external dependencies, so a first lint
+    on a machine with a cold BuildKit cache reaches the network there
+    (as well as pulling the pinned image); under `--network none` it
+    fails at that step, before any gate. That layer sits above the
+    gates and stays cached, so once it is warm `script/lint` — and
+    with it `make check` — runs entirely offline, until `go.mod` or
+    `go.sum` changes and the download layer goes cold again. Because
+    the daemon only ever sees a build context, this works when the
+    docker daemon is remote and bind mounts are impossible.
+-   `script/fmt` — format the Go sources in place (`gofmt -s -w`).
+    Markdown is not formatted.
+-   `script/fmt-check` — the read-only counterpart of `script/fmt`:
+    prints any unformatted file and exits non-zero instead of writing.
+-   `script/check` — run `script/test`, `script/lint`, and
+    `script/fmt-check`, in that order. Modifies nothing. Needs
+    `docker`, because `script/lint` does.
+-   `script/docker` — build the Docker image, tagged with the name
+    from `script/projectname`. The `Dockerfile` runs the gates as
+    build steps, so this is also the check a developer or reviewer
+    runs by hand.
+-   `script/cibuild` — build the Docker image untagged. This is what
+    the Gitea workflow runs on push; because the gates run as build
+    steps, a successful build implies the repository is green.
+-   `script/precommit` — run by the git pre-commit hook: `go mod tidy`
+    must be a no-op (a resulting change to `go.mod` or `go.sum` fails
+    the commit), then `script/check`.
+-   `script/install-precommit` — install the git pre-commit hook that
+    runs `script/precommit`. The hook is written to the common git
+    directory, so the main checkout and every worktree share it.
+-   `script/verify-lint-image-pin` — fail unless the
+    `golangci/golangci-lint` reference in `Dockerfile.lint` and the
+    one in the `Dockerfile` lint stage are the same image at the same
+    digest, naming both if not. The linter is pinned in those two
+    files and nothing else keeps them in sync, so a bump applied to
+    one alone would leave `make lint` and the `Dockerfile`'s
+    fail-fast lint stage checking the same tree against different
+    rulesets, both green. The guard restates neither pin — a third
+    copy would be the same drift one file further out — and runs as a
+    gate in both files, so `make lint`, `make check` and `make docker`
+    all catch it.
+
+`script/verify-linter-pin` used to live here. It compared a linter
+binary against a version pin in `script/bootstrap`, and both of its
+subjects are gone: no linter binary is copied between build stages any
+more, and bootstrap pins no version because it installs no linter. The
+drift it existed to catch has moved from binary-versus-pin to
+pin-versus-pin, which is what `script/verify-lint-image-pin` above
+checks.
+
+`script/lint`, `script/docker` and `script/cibuild` all pass a freshly
+computed `CHECK_EPOCH` build argument, and the gate steps in
+`Dockerfile.lint` and `Dockerfile` reference it. Without that, an
+unchanged tree lets Docker serve the gate layers from cache and the
+build exits 0 having executed no tests and no lint — a green it never
+earned, and one this repository has produced twice. `CHECK_EPOCH`
+invalidates the gate layers on every run while leaving the pinned base
+images and the dependency layers cached. `script/lint`'s value carries
+the process id as well as the epoch, because two lint runs land inside
+the same second easily and a bare epoch would cache the second one.
 
 ## Build
 
-The `Makefile` is the single source of truth for all operations:
+The `script/` entrypoints above are where the implementations live;
+the `Makefile` targets are shims onto them, except `build`, which
+carries the compile recipe:
 
--   `make build` — build the `sfdupes` binary (cgo disabled).
+-   `make` / `make build` — build the `sfdupes` binary (cgo
+    disabled); building is the default target.
+-   `make bootstrap` — install the build and development
+    dependencies.
+-   `make setup` — prepare a fresh clone: `bootstrap` plus the
+    pre-commit hook.
 -   `make test` — run the test suite (30-second timeout; reruns with
     `-v` on failure).
--   `make lint` — run `golangci-lint` with the repo config.
+-   `make lint` — run `golangci-lint` with the repo config, in Docker
+    (see `script/lint`); requires `docker`.
 -   `make fmt` / `make fmt-check` — format Go sources / verify
     formatting without writing.
 -   `make check` — `test`, `lint`, and `fmt-check`; modifies nothing.
--   `make docker` — build the Docker image, which runs `make check` as
-    a build stage.
+    Requires `docker`, via `lint`.
+-   `make docker` — build the Docker image, which runs the gates as
+    build stages.
 -   `make hooks` — install the pre-commit hook.
--   `make clean` — remove the binary and any local `files.dat`.
+-   `make clean` — remove the binary.
 
 ### Definition of done
 
@@ -323,6 +577,7 @@ All of the following, run in this directory, must pass:
 
     ```sh
     d=$(mktemp -d)
+    export SFDUPES_DATABASE="$d/db.sqlite"
     mkdir -p "$d/a" "$d/b"
     head -c 2000 /dev/urandom > "$d/a/one.bin"
     cp "$d/a/one.bin" "$d/b/copy.bin"
@@ -339,28 +594,41 @@ All of the following, run in this directory, must pass:
     cp "$d/t1/sub/f2" "$d/t2/sub/f2"
     cp "$d/t1/f1" "$d/t3/f1"
     cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
-    ./sfdupes scan "$d" > files.dat
-    ./sfdupes report files.dat
-    ./sfdupes trees files.dat
+    ./sfdupes scan "$d"
+    ./sfdupes report
+    ./sfdupes trees
+    # incremental behavior (scan a subtree; records elsewhere persist):
+    ./sfdupes scan "$d/a"                  # everything unchanged, nothing hashed
+    printf 'z' >> "$d/a/one.bin"           # modify: next scan re-hashes it
+    rm "$d/a/unique.bin"                   # delete: next scan removes its record
+    ./sfdupes scan "$d/a"                  # 1 updated, 1 removed
+    ./sfdupes report
     ```
 
-    Expected from `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
-    group (two dupe rows, `first` is the lexicographically smallest
-    path); `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
-    `t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
-    `empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere;
-    groups ordered by size descending; piping scan directly into report
-    (`./sfdupes scan "$d" | ./sfdupes report`) gives the same
-    rows.
+    (The scan database lives inside `$d` here purely for test hygiene;
+    scanning `$d` therefore also records the SQLite file itself, which
+    is harmless.)
+
+    Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin`
+    form one group (two dupe rows, `first` is the lexicographically
+    smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group;
+    `t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group;
+    `tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and
+    `tiny3` appear nowhere; groups ordered by size descending.
 
     Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`
     `$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is
     suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
     appears nowhere (its file set differs by name).
 
+    Expected from the second `report` (after the modify/delete rescan):
+    `one.bin` has left its group (its content changed), so
+    `copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is
+    gone from the database.
+
     The test suite automates this scenario (see `scan_test.go`), plus a
-    negative check: `report` and `trees` operate on the captured stream
-    alone and never touch the scanned filesystem.
+    negative check: `report` and `trees` operate on the database alone
+    and never touch the scanned filesystem.
 
 ## TODO
 
@@ -371,7 +639,9 @@ Tracked in [TODO.md](TODO.md).
 -   No full-content verification, no byte-for-byte compare, no deletion
     or linking of duplicates. The reports are advisory; acting on them is
     the user's job.
--   No persistence formats beyond the scan stream described above.
+-   No persistence beyond the SQLite database described above; no
+    export/import formats.
+-   No daemon or filesystem watcher; scheduling rescans is cron's job.
 
 ## License
 

REPO_POLICIES.md

@@ -34,10 +34,46 @@ style conventions are in separate documents:
   every file before committing. There are zero exceptions to this rule.
 
 - Every repo with software must have a root `Makefile` with these targets:
-  `make test`, `make lint`, `make fmt` (writes), `make fmt-check` (read-only),
-  `make check` (prereqs: `test`, `lint`, `fmt-check`), `make docker`, and
-  `make hooks` (installs pre-commit hook). A model Makefile is at
-  `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
+  `make bootstrap`, `make setup`, `make test`, `make lint`, `make fmt` (writes),
+  `make fmt-check` (read-only), `make check` (runs `test`, `lint`, `fmt-check`),
+  `make docker`, and `make hooks` (installs pre-commit hook). A model Makefile
+  is at `https://git.eeqj.de/sneak/prompts/raw/branch/main/Makefile`.
+
+- Repos follow the
+  [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
+  pattern: the implementation of each Makefile target lives in an executable
+  script in `script/` (`script/bootstrap`, `script/setup`, `script/test`,
+  `script/lint`, `script/fmt`, `script/fmt-check`, `script/check`,
+  `script/docker`), and the Makefile targets are thin shims that call them. The
+  scripts must be POSIX sh (`#!/bin/sh`, `set -eu`, no bashisms) so they run in
+  minimal containers (e.g. alpine images have no bash); locate the repo root
+  with `$(cd "$(dirname "$0")/.." && pwd -P)` and `cd` there before acting. From
+  the standard's canonical set we use `bootstrap`, `setup` (make the repo ready
+  for development after a fresh clone: runs `bootstrap`, then
+  `install-precommit`, plus any repo-specific initialization), `test`, and
+  `cibuild`. `script/bootstrap` installs all dependencies idempotently and
+  assumes nothing is present: base tools come from nix, apt, brew, or apk
+  (detected in that order; apt runs noninteractive). For node it uses the
+  installed node if present; otherwise it installs a PINNED node version via
+  nvm, first installing nvm itself if missing — from a hash-verified GitHub
+  release archive (never `curl | sh`), with bash installed as an explicit
+  prerequisite since nvm requires bash. yarn is then pinned via
+  `corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
+  always exact versions. `script/cibuild` runs the CI build: it changes to the
+  repo root and runs `docker build .`; the Gitea workflow calls it. Four further
+  scripts are our own extensions to the standard: `script/check` runs
+  `script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
+  what the git pre-commit hook runs, and it calls `script/check`;
+  `script/install-precommit` installs the git pre-commit hook (the `make hooks`
+  target shims to it); and `script/projectname` (literally that filename) simply
+  outputs the project's name. Scripts that need the name call
+  `script/projectname` — e.g. `script/docker` assembles its image tag from it —
+  so those scripts stay byte-identical across all repos. Repo-type-specific
+  pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
+  `script/precommit`, not in the hook itself. Model scripts are at
+  `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
+  must document the provided scripts in an **Entrypoints** section (see the
+  README requirements below).
 
 - Always use Makefile targets (`make fmt`, `make test`, `make lint`, etc.)
   instead of invoking the underlying tools directly. The Makefile is the single
@@ -57,7 +93,11 @@ style conventions are in separate documents:
   as a build step so the build fails if the branch is not green. For non-server
   repos, the Dockerfile should bring up a development environment and run
   `make check`. For server repos, `make check` should run as an early build
-  stage before the final image is assembled.
+  stage before the final image is assembled. Dockerfiles install development
+  prerequisites by running `script/bootstrap` rather than duplicating installs
+  inline; COPY `script/` and the dependency manifests (`package.json` +
+  `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
+  layer stays cached until dependencies change.
 
 - **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
   repos use a multistage build where linting runs in an independent stage based
@@ -127,8 +167,9 @@ style conventions are in separate documents:
       artifacts or heavier dependencies.
 
 - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
-  runs `docker build .` on push. Since the Dockerfile already runs `make check`,
-  a successful build implies all checks pass.
+  runs `script/cibuild` (which runs `docker build .`) on push. Since the
+  Dockerfile already runs `make check`, a successful build implies all checks
+  pass.
 
 - Use platform-standard formatters: `black` for Python, `prettier` for
   JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -136,9 +177,11 @@ style conventions are in separate documents:
   Markdown (hard-wrap at 80 columns). Documentation and writing repos (Markdown,
   HTML, CSS) should also have `.prettierrc` and `.prettierignore`.
 
-- Pre-commit hook: `make check` if local testing is possible, otherwise
-  `make lint && make fmt-check`. The Makefile should provide a `make hooks`
-  target to install the pre-commit hook.
+- Pre-commit hook: runs `script/precommit`, which calls `script/check`. If local
+  testing is not possible in the repo, `script/precommit` may skip `script/test`
+  and run only `script/lint` and `script/fmt-check`. The hook is installed by
+  `script/install-precommit`; the Makefile must provide a `make hooks` target
+  that shims to it.
 
 - All repos with software must have tests that run via the platform-standard
   test framework (`go test`, `pytest`, `jest`/`vitest`, etc.). If no meaningful
@@ -297,6 +340,10 @@ style conventions are in separate documents:
       "µPaaS is an MIT-licensed Go web application by @sneak that receives
       git-frontend webhooks and deploys applications via Docker in realtime."
     - **Getting Started**: Copy-pasteable install/usage code block.
+    - **Entrypoints**: Opens by stating that the repo adheres to the
+      [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
+      standard (with that link), then documents each provided `script/`
+      entrypoint and its purpose.
     - **Rationale**: Why does this exist?
     - **Design**: How is the program structured?
     - **TODO**: Update meticulously, even between commits. When planning, put
@@ -326,16 +373,6 @@ style conventions are in separate documents:
 - All repos should have an `.editorconfig` enforcing the project's indentation
   settings.
 
-- **Claude Code repo memory is versioned in the repo**, not left only in
-  `~/.claude` on one machine. Each memory is one file at
-  `.claude/memory/<memory>.md`, and every memory file must be `@`-imported
-  from `.claude/CLAUDE.md` (one `- @memory/<memory>.md` list line per file;
-  relative import paths resolve against `.claude/`, and Claude Code expands
-  the imports into context at session launch). When adding a memory, add both
-  the file and its import line. A root `MEMORY.md` is a violation — Claude
-  Code never auto-loads it; split it into `.claude/memory/` files. Repos with
-  no memories yet need no `.claude/` scaffolding.
-
 - Avoid putting files in the repo root unless necessary. Root should contain
   only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
   `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
@@ -361,6 +398,9 @@ style conventions are in separate documents:
     - `README.md`, `.git`, `.gitignore`, `.editorconfig`
     - `LICENSE`, `REPO_POLICIES.md` (copy from the `prompts` repo)
     - `Makefile`
+    - `script/` entrypoints (`bootstrap`, `setup`, `projectname`, `test`,
+      `lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
+      `install-precommit`)
     - `Dockerfile`, `.dockerignore`
     - `.gitea/workflows/check.yml`
     - Go: `go.mod`, `go.sum`, `.golangci.yml`

TODO.md

@@ -1,23 +1,364 @@
 # Workflow
 
+- take an issue from the `1.0.0` milestone on the tracker; work not
+  yet on the tracker gets filed as an issue first
 - branch (from `main`)
-- do the work in Next Step
-- move Next Step to the top of Completed Steps
-- move the top item of Future Steps into Next Step
-- commit (`TODO.md` changes in the same commit as the work)
-- merge to `main` if the branch is not protected, otherwise open a PR
-- push
+- do the work, with tests, in small focused commits
+- record it at the top of Completed Steps (`TODO.md` changes in the
+  same commit as the work)
+- push the branch and open a PR whose title ends with
+  ` (closes #N)`
+- an independent review gates the merge; every finding is addressed
+  or explicitly rebutted on the PR
+- merge to `main` once the review passes
 
 # Status
 
 - pre-1.0
+- the Gitea tracker is authoritative for the pre-1.0 backlog: the
+  open issues under the `1.0.0` milestone are what remains before
+  the tag, and this file records history and process, not the queue
 
 # Next Step
 
-- add a remote on git.eeqj.de and push (`main` plus tags)
+- take the next issue from the `1.0.0` milestone on the tracker:
+  https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
+  the source of truth for what is left before 1.0.0. Individual
+  issues are deliberately not restated here; a copy in this file
+  drifts out of date the moment the tracker moves
 
 # Completed Steps
 
+- run all linting in Docker via `Dockerfile.lint` and `script/lint`
+  (2026-08-10, branch `next`, closes
+  https://git.eeqj.de/sneak/sfdupes/issues/46): per the owner ruling, the
+  linter runs inside a container invoked through the `script/`
+  entrypoint and is never installed on a host. New root
+  `Dockerfile.lint` COPYs the repo into the digest-pinned
+  `golangci/golangci-lint:v2.12.2` image and runs
+  `golangci-lint config verify` and `golangci-lint run` as build
+  steps, so a successful build IS a clean lint; `script/lint` is
+  reduced to building it. `script/bootstrap` loses the `go install`,
+  the pin constants, the version parser and `verify_golangci_lint`
+  outright rather than hardening them — with nothing linting on the
+  host, the `$GOPATH/bin` versus `PATH` problem that motivated them has
+  no subject — and now warns rather than fails when `docker` is absent.
+  Two traps handled. A lint build on an unchanged tree returns success
+  in well under a second having run no linter, which is
+  https://git.eeqj.de/sneak/sfdupes/issues/32 and
+  https://git.eeqj.de/sneak/sfdupes/issues/39 again, so
+  `Dockerfile.lint` carries `ARG CHECK_EPOCH` referenced
+  inside every gate `RUN` (BuildKit hashes the expanded command, not
+  the declaration) and `script/lint` passes `"$(date +%s)-$$"` — the
+  PID matters because two lint runs land inside the same second easily.
+  And nothing inside an image build may shell out to docker, so the
+  main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
+  instead of `make lint`, and its build stage runs `make test` and
+  `make fmt-check` instead of the `make check` aggregate (`make`, not
+  the scripts bare, because the Makefile's `export CGO_ENABLED = 0`
+  only reaches what it invokes). `COPY --from=lint`
+  `/usr/bin/golangci-lint` is replaced by
+  `COPY --from=lint /src/go.sum /dev/null`: the copied binary was the
+  only edge forcing BuildKit to finish linting before the build stage
+  starts, and dropping it without replacing the edge would have ended
+  fail-fast linting silently under a still-green build. That is
+  canonical `REPO_POLICIES.md:107`'s ordering edge, restored.
+  `ENV PATH=/home/builder/go/bin:$PATH` is gone with the `go install`
+  that justified it. `script/verify-linter-pin` is retired, deleted
+  along with its README entry, because both of its subjects ceased to
+  exist in the same change: it compared a linter binary against
+  `GOLANGCI_LINT_VERSION` in `script/bootstrap`, and there is now
+  neither a binary crossing between stages nor a version pin in
+  bootstrap. The drift it guarded has not gone away, it has moved — the
+  linter is still pinned twice, now as the `FROM` line of
+  `Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage,
+  with nothing syncing them, which is exactly what
+  https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
+  replacement is one new `script/verify-lint-image-pin`,
+  run as a gate in both files, which compares the two references to
+  each other and deliberately restates neither: a hardcoded expected
+  digest would be a third copy and the same drift one file further out.
+  `golangci-lint config verify` is included per the ruling, and the
+  concern about its unpinned live HTTPS schema fetch was measured
+  rather than assumed — under `--network none` the pinned binary both
+  passes a valid config and rejects an invalid one with the jsonschema
+  error, so it validates from an embedded schema and makes no network
+  call of its own. The README scopes that to the gate steps rather
+  than to linting as a whole: `Dockerfile.lint` runs `go mod download`
+  above them, so a cold cache still needs the network and only a warm
+  one lints offline. Verified: `make lint` green with every `PATH`
+  directory containing a `golangci-lint` removed
+  (`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
+  `command -v golangci-lint` empty); two consecutive `script/lint` runs
+  on an untouched tree both executed the linter, 27.7s and 28.7s in the
+  lint step under distinct epochs with the `COPY . .` layer `CACHED`
+  above them, at 42.2s and 41.8s wall clock — the no-cache rule was not
+  weakened to shorten that. Negative control: a planted
+  `var unusedIssue46Sentinel = 1` failed `script/lint` with
+  `report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and
+  failed `make docker` at `[lint 9/9]` with the build stage stopped at
+  `[builder 3/12]` — `COPY --from=lint`, `script/bootstrap`, the test
+  gate and `make build` all zero occurrences — then reverted clean. The
+  drift guard fails on a tag-only disagreement, on a digest-only
+  disagreement, and on an unreadable reference, naming both sides.
+  `make docker` green in 5m35s with all six gates executing under one
+  epoch (lint 37.6s, test 25.2s reporting
+  `ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`).
+  The non-root quirk still holds: in the builder image with the Go test
+  cache off, `--user 0:0` fails `TestScanHardlinkRunFailsTogether`
+  (exit 1) where the unprivileged user passes (exit 0). Noted for
+  follow-up, not fixed here: `golangci-lint` warns that the
+  `gomodguard` linter is deprecated since v2.12.0 in favour of
+  `gomodguard_v2`.
+
+- install the Docker build stage's prerequisites by running
+  `script/bootstrap` instead of `apk add --no-cache make` inline
+  (2026-08-09, branch `dockerfile-bootstrap`, closes #42): canonical
+  `REPO_POLICIES.md:97` requires it, and the inline install left the
+  build stage maintaining its own notion of the toolchain — exactly
+  the divergence #24 exists to close, one layer down. The stage now
+  copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
+  which ends in `go mod download`, so the separate invocation of that
+  is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, and moves
+  above the bootstrap layer. It is the only edge making this stage
+  depend on the lint stage, so deleting it as redundant would end
+  fail-fast linting silently. Letting bootstrap install its own linter
+  here would have reintroduced the second toolchain and paid for a
+  from-source build of it. What makes the two stages provably one
+  toolchain rather than two that happen to agree is a new
+  `script/verify-linter-pin`, run in the build stage on the binary
+  that arrives from the lint stage, before bootstrap: it fails the
+  build naming both versions unless that binary is the version
+  `script/bootstrap` pins. Bootstrap's own check could not serve that
+  purpose — it reinstalls its pin from source and then verifies
+  whatever `PATH` resolves, so drift self-heals silently and a lint
+  stage image bumped on its own would lint at the new version while
+  `make check` ran at the old one, green. The linter version is pinned
+  in two independent places (the lint stage image digest and
+  `GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
+  half-applied bump is now a build failure. The pin is read out of
+  `script/bootstrap`, which stays the single source of truth; a pin
+  that cannot be read is a hard failure, not a skip. The check needs
+  no `CHECK_EPOCH`: its only inputs are the copied binary and
+  `script/`, so Docker invalidates the layer exactly when a cached
+  result would stop being true, and it is documented with the other
+  entrypoints in the README. `$GOPATH/bin` joins `PATH` because
+  that is where bootstrap's `go install` lands and bootstrap verifies
+  its installs against what `PATH` resolves — nothing in the image is
+  shadowed by it, the directory does not exist until bootstrap runs.
+  Everything added sits above `ARG CHECK_EPOCH`, and the `chown` and
+  `USER builder` still precede `make check`. Verified: the guard fails
+  the build with both versions named when the lint stage's linter is
+  faked to a different version, and an unmodified build still passes
+  it; bootstrap runs clean under Alpine's `sh` and its `apk` branch,
+  installing `git` and `make` and finding the copied
+  linter already at the pin; a second build served the bootstrap and
+  dependency layers `CACHED` while both gates ran with a fresh epoch;
+  a planted `unused` finding failed the build at the lint gate in
+  48.9s with the build stage's `make check` never starting; and the
+  suite run in the image as `--user 0:0` fails
+  `TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged
+  user is still load-bearing. That last check needs the Go test cache
+  disabled — the first attempt reported `ok ... (cached)` as root,
+  reusing the result the build-time run had left in the shared cache,
+  which would have read as a pass. Build wall time, on a shared host
+  running many concurrent builds and so noisy: 2m13s on an unchanged
+  tree, 2m17s and 4m29s for two builds after a source change, 5m14s
+  cold. Only the cold one breaches the policy ceiling, and not because
+  of this change — `chown -R builder:builder /src /home/builder` walks
+  the module cache and re-runs on every source change, and it alone
+  varied between 77s and 210s across those four builds, which is also
+  the whole spread in the totals. The same cold measurement against
+  `main` is 5m03s with a 209s `chown`. Filed as #43
+- bust the Docker layer cache for the gate steps, so `script/cibuild`
+  and `script/docker` cannot report a green they did not earn
+  (2026-08-09, branch `cibuild-cache-bust`, closes #32): both scripts
+  were bare `docker build` invocations with no cache control, and the
+  `Dockerfile` copies the tree before running its gates, so on an
+  unchanged tree Docker served those layers from cache and the build
+  exited 0 having executed nothing. That is not hypothetical here —
+  every merge this repo has done is a non-fast-forward merge of an
+  undiverged branch, so each merge commit's tree is byte-identical to
+  the branch head's and each merge CI run was almost certainly a full
+  cache hit; and PR #31's reviewer found `make docker` returning
+  success as a 17-layer cache hit, catching it only by being
+  suspicious. The fix is `ARG CHECK_EPOCH` with the scripts passing
+  `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or break
+  it. `ARG` is scoped per stage and this `Dockerfile` has three gates
+  across two — `make fmt-check` and `make lint` in the lint stage,
+  `make check` in the build stage — so a single declaration would have
+  left one stage silently cacheable; it is declared in both. And
+  BuildKit hashes the expanded command, not the declaration, so a
+  declared-but-unreferenced `ARG` invalidates nothing: each gate `RUN`
+  echoes the epoch, which also puts the value in the build log as
+  evidence the layer really ran. Placement is below the dependency
+  layers on purpose — a build that goes cold every time would be a
+  different bug, not a fix. Verified by running each script twice back
+  to back on an unchanged tree under `BUILDKIT_PROGRESS=plain`: all
+  three gates executed on all four runs, each with a fresh epoch in
+  the log (`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s
+  then 53.4s), and twelve steps were still served `CACHED` in the
+  steady state — both `go mod download`s, `apk add`, `adduser`, the
+  `chown`, every `go.mod`/`go.sum` and source copy, the linter copy
+  out of the lint stage, and the binary copy into the runtime stage.
+  The lint stage still gates the build stage: with a deliberate
+  `unused` finding planted in the tree, the build failed at
+  `make lint` in 36.1s and the build-stage `make check` never started.
+  The build stage also still drops to the unprivileged `builder` user
+  before `make check`, which the suite depends on rather than merely
+  prefers: forcing the same image to run the tests as root fails
+  `TestScanHardlinkRunFailsTogether`, because root reads straight
+  through the `chmod(0)` the test uses to prove hard links are read
+  once. This is the local fix only; propagating it to the canonical
+  templates is `prompts` #26
+- check the installed golangci-lint version in `script/bootstrap`
+  instead of only its presence (2026-08-09, branch
+  `bootstrap-version-check`, closes #24): `missing golangci-lint` meant
+  any linter already on `PATH` satisfied the check, so the pin was never
+  consulted and the v2.12.2 bump from #3 was inert on every host that
+  already had one — this host ran v2.10.1 against a v2.12.2 pin,
+  `make check` went green, and `make docker` then rejected the same
+  commit with findings the local gate never saw. The version now lives
+  in one place, `GOLANGCI_LINT_VERSION`, with the `go install` module
+  ref derived from it so a bump cannot half-apply; a
+  `golangci_lint_version` helper parses `golangci-lint --version`
+  (taking the field after the word `version` and tolerating an optional
+  leading `v`, which the module ref carries and the binary's output does
+  not), and any version that is not the pin — older, newer, absent or
+  unparseable — is reinstalled. The install is then verified against the
+  binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
+  `GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes
+  first on `PATH`, so a wrong-version one sitting ahead of it — nix,
+  apt, brew, apk, or the `/usr/local/bin` copy the `Dockerfile` builder
+  stage makes — would swallow the install and leave the local gate
+  disagreeing with CI under an affirmative `bootstrap complete`.
+  Bootstrap now re-reads the effective version after installing and, on
+  a mismatch, prints both paths and both versions to stderr and exits
+  non-zero instead of claiming success; it does not reorder anyone's
+  `PATH` or delete their binary. The `--version` call keeps its stderr
+  connected, so a present-but-broken binary says why rather than
+  reinstalling forever in silence, and is bounded by `timeout(1)` where
+  that exists, so a wedged binary cannot hang bootstrap. `git`, `make`
+  and `go` keep their presence-only checks and now say why in a
+  comment: they are host package-manager tools the repo deliberately
+  does not pin, with `go.mod` governing the language version and the
+  digest-pinned images covering reproducible builds. Verified on this
+  host by bootstrapping from v2.10.1 to v2.12.2 and running it again to
+  a no-op, plus stub runs of the real script under `dash` covering a
+  thirteen-input parse matrix (absent, older, newer, host-style,
+  image-style, leading-`v`, stderr-only, empty, non-zero exit, impostor
+  binary, `(devel)`, trailing `version`), a shadowed install that must
+  exit non-zero, an install destination not on `PATH` at all, `GOBIN`
+  set, and a wedged binary that must hit the timeout; `make check` and
+  `make lint` are clean at v2.12.2, so v2.10.1 was not hiding any
+  findings on `main`
+- unwind the hash worker pool on the error path (2026-08-09, branch
+  `hash-pool-cleanup`, closes #6): `hashPhase` used to return the
+  moment `recordRun` failed and abandon the pool — the feeder parked
+  forever on a full `jobs` channel and every worker on a full
+  `results` channel. That only stopped being invisible when #4 landed
+  and `runScan` began unwinding instead of calling `os.Exit`. The
+  pool is now an owned, context-aware `hashPool`: every blocking send
+  in the feeder and the workers selects on `ctx.Done()`, `jobs` is
+  closed on every path out, and `hashPhase` defers `pool.stop()`,
+  which cancels and then drains `results` until the last goroutine
+  has exited — draining is what frees a worker already parked on a
+  send. `ctx` is threaded from `cmd.Context()` through `runScan`,
+  `syncScan`, both worker pools and the whole database layer (it is
+  the first parameter everywhere), so #5 can hand this path a signal
+  and needs to add nothing else. The walk pool never leaked, because
+  `walkPhase` always drains its events to close, but it has the same
+  unbounded-send shape and #5 will give it an early return, so it
+  gets the same treatment plus a `ctx.Err()` guard after the walk: a
+  cancelled walk yields a partial size census, and every file it never
+  reached looks vanished to the update phase. That phase's own
+  `BeginTx` fails on the same cancelled context before deleting
+  anything, so the guard is defence in depth rather than the only
+  barrier — but it is the one that survives #5 deciding an interrupted
+  scan may commit what it has. Tests drive `run(scan)` against a
+  database whose insert trigger aborts, and assert both that the scan
+  fails instead of hanging and that `runtime.NumGoroutine()` polls
+  back to its pre-scan baseline; a second set cancels a scan part-way
+  through the walk — deterministically, by counting the scan's own
+  consultations of `ctx.Done()` rather than racing a timer — and
+  asserts that it stops at the guard holding a partial census and a
+  still-populated record index, with every record intact. The
+  remaining cancellation branches of both pools are covered by direct
+  tests of `sendEvent`, the walk workers, `dispatchDirs`,
+  `feedHashJobs`, `hashWorker` and `hashPhase`
+- guarantee the database is closed on every fatal exit path
+  (2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
+  its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
+  it the SQLite WAL checkpoint — now actually runs when a subcommand
+  fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
+  `resolveRoots` return errors instead. The single exit point is `run`
+  in `main.go`: it maps a `fatalError` (anything a subcommand
+  returned) to exit 1 and cobra's own argument and flag errors to exit
+  2, which keeps a runtime failure from being reported as a usage
+  error or printing the usage text. New `main_test.go` drives the CLI
+  in-process and asserts the exit codes from README §Error handling
+  plus the stdout/stderr split, including that a fatal error raised
+  after the database is open leaves no `-wal`/`-shm` sidecar behind
+  for `scan`, `report` or `trees`
+- update golangci-lint to v2.12.2 with the canonical config
+  (2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
+  closes #3): bumped the pinned linter in the `Dockerfile` lint
+  stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
+  `.golangci.yml` with the canonical file — the linter settings
+  (`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
+  `linters.settings` per the v2 schema, so they are actually
+  applied; no new lint findings surfaced
+- convert Makefile targets to scripts-to-rule-them-all `script/`
+  entrypoints like the other managed repos (2026-07-26, commit
+  `3abeacf`, closes #1): all 12 `script/` entrypoints exist
+  (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
+  `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
+  `install-precommit`) and every Makefile target is now a thin shim
+  over them, matching the other managed repos
+- make the binary the default Make target (2026-07-24, branch
+  `make-default-target`): plain `make` now builds `sfdupes`
+  (previously it ran `check` plus `build`); `make build` remains as
+  an alias
+- scan-wide phases, concurrent operands, batched updates (2026-07-24,
+  branch `scan-wide-phases`): all operands seed the shared walk pool
+  and every pass runs once over the whole scan, so totals and ETAs
+  are scan-global; the per-operand walk/hash/update cycles and their
+  stderr announcements are gone; the update pass commits in batched
+  transactions — the filesystem is authoritative and the database an
+  eventually-consistent reflection, so scan-level atomicity is not
+  required
+- split the stat pass back out of the walk (2026-07-24, branch
+  `parallel-phases`): phases are strictly sequential again — walk,
+  stat, hash, update per operand — with parallelism only inside each
+  phase; the walk enumerates paths with per-directory workers and the
+  stat pass lstats them with per-file workers, restoring the exact
+  total/ETA stat bar
+- announce each operand on stderr before its passes (2026-07-24,
+  branch `scan-operand-progress`): with per-operand walk/hash/update
+  cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
+  that looked like the whole run's — an operator watching operand 3 of
+  14 hash 300k files concluded 20M files were being skipped
+- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
+  was a single goroutine and took hours at ~20M files on a busy pool
+  (observed: 22M files in 4h on a ZFS server); it is now a
+  per-directory worker-pool traversal that records size/mtime during
+  the walk (folding away the separate stat pass, halving metadata
+  I/O), and each `PATH` operand commits in its own transaction so an
+  interrupted scan keeps completed operands
+
+- persistent scan database (2026-07-24, branch `persistent-database`):
+  `scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
+  Go, cgo stays disabled) keyed by absolute path that survives between
+  runs — a rescan hashes only new or changed files (by mtime/size),
+  deletes records for files vanished from under the scanned operands,
+  and leaves records outside them untouched, so `scan` can be cronned
+  daily; `report` and `trees` read the database (no positional
+  arguments) instead of a scan stream. Database at
+  `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
+  WAL journaling plus a single-transaction update keep a report run
+  during a scan safe
+- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
+  `v0.0.1`, and push `main` plus tags (2026-07-23)
 - `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
   `PATH...` operands via cobra flags replacing the `/srv` `-root`
   default; new `-x`/`--one-file-system` flag (GNU convention) to stop
@@ -30,9 +371,6 @@
 
 # Future Steps
 
-- convert Makefile targets to scripts-to-rule-them-all `script/`
-  entrypoints like the other managed repos
-- tag `v0.0.1` once the compliance branch is merged
 - possible later features (explicitly out of scope per README):
   full-content verification of candidates, removal-script helpers
 

cancel_test.go

@@ -0,0 +1,489 @@
+package main
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"os"
+	"path/filepath"
+	"strconv"
+	"sync"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+// poolUnwind bounds how long a goroutine is given to leave a pool
+// after its context is cancelled. Only a failing run ever waits this
+// long: a pool that ignored its cancellation parks forever, and this
+// is what turns that into a failed assertion instead of a suite that
+// hangs until the test binary's own timeout.
+const poolUnwind = 2 * time.Second
+
+// walkClock is a context whose cancellation is driven by the scan's
+// own progress rather than by the wall clock: it cancels itself the
+// moment its Done method has been consulted n times. That is what
+// makes "cancel in the middle of the walk" reproducible instead of a
+// race against a timer.
+//
+// The accounting behind the n chosen by each test: every blocking
+// channel operation in the walk selects on Done, so the walk spends
+// one consultation per file event plus a couple per directory, while
+// the index load that runs ahead of it spends a small fixed number
+// (three) whatever the record count.
+type walkClock struct {
+	n    int64
+	seen atomic.Int64
+	once sync.Once
+	done chan struct{}
+}
+
+// newWalkClock returns a context that cancels itself on the nth
+// consultation of its Done method.
+func newWalkClock(n int64) *walkClock {
+	return &walkClock{n: n, done: make(chan struct{})}
+}
+
+// Done returns the cancellation channel, cancelling the context on the
+// nth call and on every call after it. The same channel is returned
+// throughout, so a caller that took it before the cancellation still
+// observes the close.
+func (c *walkClock) Done() <-chan struct{} {
+	if c.seen.Add(1) >= c.n {
+		c.once.Do(func() { close(c.done) })
+	}
+
+	return c.done
+}
+
+// Err reports the cancellation without consuming a consultation, which
+// is what lets the post-walk guard read it without disturbing the
+// count.
+func (c *walkClock) Err() error {
+	select {
+	case <-c.done:
+		return context.Canceled
+	default:
+		return nil
+	}
+}
+
+// Deadline reports no deadline: this context is cancelled by progress,
+// never by time.
+func (c *walkClock) Deadline() (time.Time, bool) {
+	return time.Time{}, false
+}
+
+// Value carries nothing.
+func (c *walkClock) Value(_ any) any {
+	return nil
+}
+
+// walkCancelDirs and walkCancelFilesPerDir shape the fixture for the
+// mid-walk cancellation test. Spreading the files over directories is
+// load-bearing: it is what bounds how much of the tree can still be
+// walked after the cancellation, since the workers drop every
+// directory still queued and only the handful already in flight can
+// emit anything more.
+const (
+	walkCancelDirs         = 100
+	walkCancelFilesPerDir  = 20
+	walkCancelFiles        = walkCancelDirs * walkCancelFilesPerDir
+	walkCancelWorkers      = 4
+	walkCancelInFlightDirs = walkCancelWorkers * walkCancelFilesPerDir
+)
+
+// walkCancelAtDone is the consultation on which the fixture's context
+// cancels itself. A quarter of the file count is far past the index
+// load's fixed handful and far short of the walk's total, so the
+// cancellation lands deep inside the walk and nowhere near either end
+// of it.
+const walkCancelAtDone = walkCancelFiles / 4
+
+// buildWalkCancelTree writes walkCancelFiles empty files spread over
+// walkCancelDirs subdirectories. Zero-length files are never opened by
+// the hasher, so the fixture costs directory entries and no read I/O
+// while still giving the walk thousands of events to emit.
+func buildWalkCancelTree(t *testing.T) string {
+	t.Helper()
+
+	dir := t.TempDir()
+
+	for i := range walkCancelDirs {
+		sub := filepath.Join(dir, "d"+strconv.Itoa(i))
+
+		err := os.Mkdir(sub, 0o750)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		writeEmptyFiles(t, sub, walkCancelFilesPerDir)
+	}
+
+	return dir
+}
+
+// assertRecordsIntact fails when the database no longer holds exactly
+// the records it held before, reporting the first difference rather
+// than dumping thousands of paths.
+func assertRecordsIntact(t *testing.T, db *sql.DB, before []string) {
+	t.Helper()
+
+	got := recordPaths(dbRecords(t, db))
+	if len(got) != len(before) {
+		t.Fatalf("%d records after the cancelled scan, want %d",
+			len(got), len(before))
+	}
+
+	for i := range got {
+		if got[i] != before[i] {
+			t.Fatalf("record %d = %q after the cancelled scan, want %q",
+				i, got[i], before[i])
+		}
+	}
+}
+
+// TestSyncScanCancelledMidWalkKeepsRecords is the regression net under
+// the post-walk guard. The scan is cancelled part-way through the
+// walk, so it reaches the guard holding a genuinely partial size
+// census and a still-populated index of records the walk never got to.
+// Every one of those records would look vanished to the update phase.
+// The guard is what stops the scan there, and this test is what
+// notices if it stops doing so: deleting the guard, or making it
+// unreachable, makes the scan carry its truncated view into a later
+// phase and fail there instead, with a wrapped error rather than the
+// bare cancellation.
+//
+//nolint:paralleltest // counts goroutines: must not run beside others
+func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
+	dir := buildWalkCancelTree(t)
+	db := openTestDB(t)
+
+	st := syncTree(t, db, dir)
+	if st.added != walkCancelFiles {
+		t.Fatalf("setup scan added %d records, want %d",
+			st.added, walkCancelFiles)
+	}
+
+	before := recordPaths(dbRecords(t, db))
+	base := baselineGoroutines(t)
+
+	st, err := syncScan(newWalkClock(walkCancelAtDone), db,
+		[]string{dir}, walkCancelWorkers, false)
+
+	assertWalkGuardAborted(t, st, err)
+	assertRecordsIntact(t, db, before)
+
+	if got := settledGoroutines(t, base); got > base {
+		t.Errorf("goroutines = %d after the cancelled scan, want %d back",
+			got, base)
+	}
+}
+
+// assertWalkGuardAborted checks that the scan stopped at the post-walk
+// guard: with a census that is neither empty (the walk really ran)
+// nor complete (it really was cut short), and with the guard's own
+// bare cancellation as the error. A wrapped error means the partial
+// census was carried past the guard into the hash or update phase,
+// which is the failure this test exists to catch.
+func assertWalkGuardAborted(t *testing.T, st scanStats, err error) {
+	t.Helper()
+
+	if !errors.Is(err, context.Canceled) {
+		t.Fatalf("syncScan cancelled mid-walk = %v, want %v",
+			err, context.Canceled)
+	}
+
+	if errors.Unwrap(err) != nil {
+		t.Errorf("syncScan reported %q, want the guard's bare "+
+			"cancellation: a wrapped error means the truncated census "+
+			"reached a later phase", err)
+	}
+
+	if st.unchanged == 0 {
+		t.Fatalf("stats = %+v: the census is empty, so the walk never "+
+			"ran and the guard was reached for the wrong reason", st)
+	}
+
+	if st.unchanged >= walkCancelFiles {
+		t.Fatalf("stats = %+v: the census covers the whole tree, so the "+
+			"walk was not cut short", st)
+	}
+
+	// The workers drop every directory still queued once the scan is
+	// cancelled, so only the directories already in flight can add to
+	// the census after the fact. A census beyond that bound would mean
+	// the cancellation was not observed where it should have been.
+	limit := walkCancelAtDone + walkCancelInFlightDirs
+	if st.unchanged > limit {
+		t.Errorf("census covers %d files, want at most %d: the walk kept "+
+			"taking directories off the queue after cancellation",
+			st.unchanged, limit)
+	}
+
+	if st.removed != 0 {
+		t.Errorf("stats = %+v: the scan counted records for removal from "+
+			"a partial census", st)
+	}
+}
+
+// TestSyncScanCancelledBeforeLoadIndex covers the trivial end of the
+// cancellation path: a scan handed a context that is already cancelled
+// fails in the index load, before the walk pool is ever started. It
+// says nothing about the post-walk guard — nothing downstream of
+// loadIndex runs at all — only that the failure surfaces as a
+// cancellation and that no record is touched on the way out.
+func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
+	t.Parallel()
+
+	dir := buildSmokeTree(t)
+	db := openTestDB(t)
+
+	syncTree(t, db, dir)
+
+	before := recordPaths(dbRecords(t, db))
+
+	ctx, cancel := context.WithCancel(t.Context())
+	cancel()
+
+	st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false)
+	if !errors.Is(err, context.Canceled) {
+		t.Fatalf("syncScan on a cancelled context = %v, want %v",
+			err, context.Canceled)
+	}
+
+	if st != (scanStats{}) {
+		t.Errorf("stats = %+v, want none: the scan gave up in the index "+
+			"load, before any phase ran", st)
+	}
+
+	assertRecordsIntact(t, db, before)
+}
+
+// drainClosed counts the values received from ch until it closes,
+// failing the test if it does not close within poolUnwind. A pool that
+// ignored its cancellation leaves its channel open with its goroutines
+// parked, and this is what reports that as an assertion.
+func drainClosed[T any](t *testing.T, ch <-chan T, what string) int {
+	t.Helper()
+
+	counted := make(chan int, 1)
+
+	go func() {
+		n := 0
+		for range ch {
+			n++
+		}
+
+		counted <- n
+	}()
+
+	select {
+	case n := <-counted:
+		return n
+	case <-time.After(poolUnwind):
+		t.Fatalf("%s stayed open after cancellation", what)
+
+		return 0
+	}
+}
+
+// awaitReturn fails the test if done is not closed within poolUnwind.
+func awaitReturn(t *testing.T, done <-chan struct{}, what string) {
+	t.Helper()
+
+	select {
+	case <-done:
+	case <-time.After(poolUnwind):
+		t.Fatalf("%s did not return after cancellation", what)
+	}
+}
+
+// cancelledContext returns a context that is already cancelled.
+func cancelledContext(t *testing.T) context.Context {
+	t.Helper()
+
+	ctx, cancel := context.WithCancel(t.Context())
+	cancel()
+
+	return ctx
+}
+
+// TestSendEventAbandonsBlockedSend checks that a walk goroutine with an
+// event to deliver and nobody to deliver it to leaves on cancellation
+// instead of holding the pool open. The channel here is unbuffered and
+// unread, so the send can never complete.
+func TestSendEventAbandonsBlockedSend(t *testing.T) {
+	t.Parallel()
+
+	done := make(chan struct{})
+	events := make(chan walkEvent)
+
+	go func() {
+		defer close(done)
+
+		sendEvent(cancelledContext(t), events, walkEvent{})
+	}()
+
+	awaitReturn(t, done, "sendEvent")
+}
+
+// TestWalkWorkersDropQueuedDirs checks that cancelled walk workers keep
+// reading jobs and drop the directories rather than stopping their
+// read: the range over jobs has to run out for the pool to tear down
+// and close its event stream.
+func TestWalkWorkersDropQueuedDirs(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	writeEmptyFiles(t, dir, walkCancelFilesPerDir)
+
+	jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false)
+
+	for range 4 {
+		jobs <- dirJob{path: dir}
+	}
+
+	close(jobs)
+
+	if n := drainClosed(t, events, "the walk event stream"); n != 0 {
+		t.Errorf("cancelled walk workers emitted %d events, want none", n)
+	}
+}
+
+// TestWalkWorkerAbandonsSubdirHandoff checks the other blocking send a
+// walk worker makes: handing discovered subdirectories back to the
+// dispatcher. Once the dispatcher has left, nothing drains that
+// channel, and a worker parked on it would hold the pool open forever.
+func TestWalkWorkerAbandonsSubdirHandoff(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	writeEmptyFiles(t, dir, 1)
+
+	err := os.Mkdir(filepath.Join(dir, "sub"), 0o750)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	ctx, cancel := context.WithCancel(t.Context())
+	defer cancel()
+
+	jobs, subdirs, events := startWalkWorkers(ctx, 1, false)
+
+	// Fill the hand-back channel to its capacity — one slot per worker
+	// — so the worker's own hand-back is certain to block.
+	subdirs <- nil
+
+	jobs <- dirJob{path: dir}
+
+	// The file event proves the worker has read the directory and has
+	// nothing left to do but the blocked hand-back.
+	ev := <-events
+	if ev.fail {
+		t.Fatalf("walk event = %+v, want the fixture file", ev)
+	}
+
+	cancel()
+	close(jobs)
+	drainClosed(t, events, "the walk event stream")
+}
+
+// TestDispatchDirsClosesJobsWhenCancelled checks that a dispatcher
+// leaving on cancellation closes the job channel on its way out. The
+// workers range over that channel; a dispatcher that returned without
+// closing it would strand every one of them.
+func TestDispatchDirsClosesJobsWhenCancelled(t *testing.T) {
+	t.Parallel()
+
+	// Unbuffered and unread: with no worker pool behind it, the
+	// dispatcher can only leave through its cancellation case.
+	jobs := make(chan dirJob)
+	subdirs := make(chan []dirJob)
+	initial := []dirJob{{path: "/a"}, {path: "/b"}}
+
+	dispatchDirs(cancelledContext(t), initial, jobs, subdirs)
+
+	if n := drainClosed(t, jobs, "the walk job queue"); n > len(initial) {
+		t.Errorf("dispatcher queued %d jobs, want at most %d",
+			n, len(initial))
+	}
+}
+
+// TestFeedHashJobsClosesJobsWhenCancelled checks that the hash feeder
+// abandons the runs it has not queued yet and still closes the job
+// channel, which is what lets the workers' range terminate.
+func TestFeedHashJobsClosesJobsWhenCancelled(t *testing.T) {
+	t.Parallel()
+
+	done := make(chan struct{})
+	// Unbuffered and unread until the feeder has returned, so the only
+	// way out of the feeder is its cancellation case.
+	jobs := make(chan []fileRec)
+	runs := [][]fileRec{{{path: "a"}}, {{path: "b"}}}
+
+	go func() {
+		defer close(done)
+
+		feedHashJobs(cancelledContext(t), runs, jobs)
+	}()
+
+	awaitReturn(t, done, "feedHashJobs")
+
+	if _, ok := <-jobs; ok {
+		t.Error("the hash job channel was left open after cancellation")
+	}
+}
+
+// TestHashWorkerDropsQueuedRuns checks that a cancelled hash worker
+// keeps reading jobs and drops the runs rather than reading files
+// nobody wants the hashes of — while still letting the range run out
+// so the pool tears down. The queued run names a file that does not
+// exist, so a worker that hashed it anyway would produce a result.
+func TestHashWorkerDropsQueuedRuns(t *testing.T) {
+	t.Parallel()
+
+	done := make(chan struct{})
+	jobs := make(chan []fileRec, 1)
+	results := make(chan hashResult, 1)
+
+	run := []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}}
+
+	jobs <- run
+
+	close(jobs)
+
+	go func() {
+		defer close(done)
+
+		hashWorker(cancelledContext(t), jobs, results)
+	}()
+
+	awaitReturn(t, done, "hashWorker")
+
+	select {
+	case r := <-results:
+		t.Errorf("cancelled hash worker produced %+v, want the run dropped",
+			r)
+	default:
+	}
+}
+
+// TestHashPhaseCancelledReturnsContextError checks the result loop's
+// own exit: with the pool cancelled, no result will ever arrive, and
+// the loop must leave through the cancellation rather than wait for a
+// receive that cannot happen.
+func TestHashPhaseCancelledReturnsContextError(t *testing.T) {
+	t.Parallel()
+
+	s := &scanState{
+		db:     openTestDB(t),
+		toHash: []fileRec{{path: "a", size: 1, dev: 1, ino: 1}},
+	}
+
+	err := s.hashPhase(cancelledContext(t), 2)
+	if !errors.Is(err, context.Canceled) {
+		t.Fatalf("hashPhase on a cancelled context = %v, want %v",
+			err, context.Canceled)
+	}
+}

db.go

@@ -0,0 +1,383 @@
+package main
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"fmt"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"slices"
+	"strconv"
+
+	// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
+	// disabled.
+	_ "modernc.org/sqlite"
+)
+
+// defaultDatabasePath is where the persistent scan database lives when
+// SFDUPES_DATABASE is not set.
+const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
+
+// databaseEnv is the environment variable that overrides the database
+// path.
+const databaseEnv = "SFDUPES_DATABASE"
+
+// schemaVersion is the database schema version this build reads and
+// writes, stored in PRAGMA user_version.
+const schemaVersion = 1
+
+// dbDirPerm is the mode for a database parent directory created by
+// scan.
+const dbDirPerm = 0o755
+
+// createTableSQL is the schema applied to a fresh database. Paths are
+// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
+const createTableSQL = `
+CREATE TABLE files (
+    path  BLOB PRIMARY KEY,
+    size  INTEGER NOT NULL,
+    mtime INTEGER NOT NULL,
+    head  TEXT NOT NULL,
+    tail  TEXT NOT NULL
+) WITHOUT ROWID
+`
+
+// upsertSQL inserts one file record, replacing any existing record for
+// the same path.
+const upsertSQL = `
+INSERT INTO files (path, size, mtime, head, tail)
+VALUES (?, ?, ?, ?, ?)
+ON CONFLICT (path) DO UPDATE SET
+    size = excluded.size, mtime = excluded.mtime,
+    head = excluded.head, tail = excluded.tail
+`
+
+// errNoDatabase reports a missing database file for report/trees.
+var errNoDatabase = errors.New(
+	"no database (run \"sfdupes scan\" first, or set " + databaseEnv + ")")
+
+// errSchemaVersion reports a database whose schema version this build
+// does not understand.
+var errSchemaVersion = errors.New("unsupported database schema version")
+
+// databasePath resolves the database location: SFDUPES_DATABASE when
+// set and non-empty, the compiled-in default otherwise.
+func databasePath() string {
+	if p := os.Getenv(databaseEnv); p != "" {
+		return p
+	}
+
+	return defaultDatabasePath
+}
+
+// openDB opens the SQLite database at path with WAL journaling and a
+// busy timeout, so a report can run while a cron scan is in progress.
+// It does not create or verify the schema.
+func openDB(path string) (*sql.DB, error) {
+	dsn := "file:" + path +
+		"?_pragma=busy_timeout(10000)" +
+		"&_pragma=journal_mode(WAL)" +
+		"&_pragma=synchronous(NORMAL)"
+
+	db, err := sql.Open("sqlite", dsn)
+	if err != nil {
+		return nil, fmt.Errorf("open database %s: %w", path, err)
+	}
+
+	// A single connection avoids SQLITE_BUSY between this process's
+	// own connections; concurrency lives in the worker pools, not in
+	// parallel database access.
+	db.SetMaxOpenConns(1)
+
+	return db, nil
+}
+
+// openScanDatabase opens the database for the scan subcommand, creating
+// the file, its parent directory, and the schema as needed.
+func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
+	err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
+	if err != nil {
+		return nil, fmt.Errorf("create database directory: %w", err)
+	}
+
+	db, err := openDB(path)
+	if err != nil {
+		return nil, err
+	}
+
+	err = initSchema(ctx, db)
+	if err != nil {
+		_ = db.Close()
+
+		return nil, fmt.Errorf("database %s: %w", path, err)
+	}
+
+	return db, nil
+}
+
+// openReportDatabase opens an existing database for the report and
+// trees subcommands. A missing database file is an error directing the
+// user to run scan first; the schema version must match exactly.
+func openReportDatabase(ctx context.Context,
+	path string,
+) (*sql.DB, error) {
+	_, err := os.Stat(path)
+	if errors.Is(err, fs.ErrNotExist) {
+		return nil, fmt.Errorf("%s: %w", path, errNoDatabase)
+	}
+
+	if err != nil {
+		return nil, fmt.Errorf("database: %w", err)
+	}
+
+	db, err := openDB(path)
+	if err != nil {
+		return nil, err
+	}
+
+	v, err := userVersion(ctx, db)
+	if err != nil {
+		_ = db.Close()
+
+		return nil, fmt.Errorf("database %s: %w", path, err)
+	}
+
+	if v != schemaVersion {
+		_ = db.Close()
+
+		return nil, fmt.Errorf("database %s: version %d, want %d: %w",
+			path, v, schemaVersion, errSchemaVersion)
+	}
+
+	return db, nil
+}
+
+// initSchema creates the schema on a fresh database and verifies the
+// schema version on an existing one.
+func initSchema(ctx context.Context, db *sql.DB) error {
+	v, err := userVersion(ctx, db)
+	if err != nil {
+		return err
+	}
+
+	switch v {
+	case 0:
+		return createSchema(ctx, db)
+	case schemaVersion:
+		return nil
+	default:
+		return fmt.Errorf("version %d, want %d: %w",
+			v, schemaVersion, errSchemaVersion)
+	}
+}
+
+// createSchema applies the schema to a fresh database and stamps the
+// schema version.
+func createSchema(ctx context.Context, db *sql.DB) error {
+	_, err := db.ExecContext(ctx, createTableSQL)
+	if err != nil {
+		return fmt.Errorf("create schema: %w", err)
+	}
+
+	_, err = db.ExecContext(ctx,
+		"PRAGMA user_version = "+strconv.Itoa(schemaVersion))
+	if err != nil {
+		return fmt.Errorf("set schema version: %w", err)
+	}
+
+	return nil
+}
+
+// userVersion reads the database's PRAGMA user_version.
+func userVersion(ctx context.Context, db *sql.DB) (int, error) {
+	var v int
+
+	err := db.QueryRowContext(ctx, "PRAGMA user_version").Scan(&v)
+	if err != nil {
+		return 0, fmt.Errorf("read schema version: %w", err)
+	}
+
+	return v, nil
+}
+
+// loadFileRows reads every record from the files table.
+func loadFileRows(ctx context.Context, db *sql.DB) ([]scanRec, error) {
+	rows, err := db.QueryContext(ctx,
+		"SELECT path, size, mtime, head, tail FROM files")
+	if err != nil {
+		return nil, fmt.Errorf("read records: %w", err)
+	}
+
+	defer func() { _ = rows.Close() }()
+
+	var recs []scanRec
+
+	for rows.Next() {
+		var (
+			path []byte
+			r    scanRec
+		)
+
+		err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail)
+		if err != nil {
+			return nil, fmt.Errorf("read record: %w", err)
+		}
+
+		r.path = string(path)
+		recs = append(recs, r)
+	}
+
+	err = rows.Err()
+	if err != nil {
+		return nil, fmt.Errorf("read records: %w", err)
+	}
+
+	return recs, nil
+}
+
+// loadFileMeta streams every record's path, size, mtime, and whether
+// it carries hashes to fn. Scan change detection needs no hash
+// values, and skipping the hash columns keeps the scan's in-memory
+// index small on multi-million-file databases.
+func loadFileMeta(ctx context.Context, db *sql.DB,
+	fn func(path string, size, mtime int64, hashed bool),
+) error {
+	rows, err := db.QueryContext(ctx,
+		"SELECT path, size, mtime, head <> '' FROM files")
+	if err != nil {
+		return fmt.Errorf("read records: %w", err)
+	}
+
+	defer func() { _ = rows.Close() }()
+
+	for rows.Next() {
+		var (
+			path        []byte
+			size, mtime int64
+			hashed      int64
+		)
+
+		err = rows.Scan(&path, &size, &mtime, &hashed)
+		if err != nil {
+			return fmt.Errorf("read record: %w", err)
+		}
+
+		fn(string(path), size, mtime, hashed != 0)
+	}
+
+	err = rows.Err()
+	if err != nil {
+		return fmt.Errorf("read records: %w", err)
+	}
+
+	return nil
+}
+
+// updateBatchSize is the number of record changes committed per
+// transaction during the update pass. The filesystem is authoritative
+// and the database an eventually-consistent reflection of it, so
+// scan-level atomicity is not required; smaller transactions keep the
+// WAL small and let concurrent reports observe progress.
+const updateBatchSize = 10000
+
+// applyChanges writes one scan's database changes — upserts for new and
+// changed files, deletes for vanished ones — in batched transactions.
+// Progress is rendered on prog (one increment per change).
+func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
+	deletes []string, prog *progress,
+) error {
+	for batch := range slices.Chunk(upserts, updateBatchSize) {
+		err := applyBatch(ctx, db, batch, nil, prog)
+		if err != nil {
+			return err
+		}
+	}
+
+	for batch := range slices.Chunk(deletes, updateBatchSize) {
+		err := applyBatch(ctx, db, nil, batch, prog)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// applyBatch commits one batch of upserts and deletes in a single
+// transaction.
+func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
+	deletes []string, prog *progress,
+) error {
+	tx, err := db.BeginTx(ctx, nil)
+	if err != nil {
+		return fmt.Errorf("begin transaction: %w", err)
+	}
+
+	defer func() { _ = tx.Rollback() }()
+
+	err = execUpserts(ctx, tx, upserts, prog)
+	if err != nil {
+		return err
+	}
+
+	err = execDeletes(ctx, tx, deletes, prog)
+	if err != nil {
+		return err
+	}
+
+	err = tx.Commit()
+	if err != nil {
+		return fmt.Errorf("commit: %w", err)
+	}
+
+	return nil
+}
+
+// execUpserts inserts or updates one record per new or changed file.
+func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
+	prog *progress,
+) error {
+	st, err := tx.PrepareContext(ctx, upsertSQL)
+	if err != nil {
+		return fmt.Errorf("prepare upsert: %w", err)
+	}
+
+	defer func() { _ = st.Close() }()
+
+	for _, r := range upserts {
+		_, err = st.ExecContext(ctx,
+			[]byte(r.path), r.size, r.mtime, r.head, r.tail)
+		if err != nil {
+			return fmt.Errorf("upsert %s: %w", r.path, err)
+		}
+
+		prog.increment()
+	}
+
+	return nil
+}
+
+// execDeletes removes the records for paths no longer present.
+func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string,
+	prog *progress,
+) error {
+	st, err := tx.PrepareContext(ctx, "DELETE FROM files WHERE path = ?")
+	if err != nil {
+		return fmt.Errorf("prepare delete: %w", err)
+	}
+
+	defer func() { _ = st.Close() }()
+
+	for _, p := range deletes {
+		_, err = st.ExecContext(ctx, []byte(p))
+		if err != nil {
+			return fmt.Errorf("delete %s: %w", p, err)
+		}
+
+		prog.increment()
+	}
+
+	return nil
+}

db_test.go

@@ -0,0 +1,227 @@
+package main
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"fmt"
+	"path/filepath"
+	"slices"
+	"strings"
+	"testing"
+)
+
+// testDBPath returns a database path inside a fresh temp dir.
+func testDBPath(t *testing.T) string {
+	t.Helper()
+
+	return filepath.Join(t.TempDir(), "db.sqlite")
+}
+
+// openTestDB creates a fresh scan database in a temp dir.
+func openTestDB(t *testing.T) *sql.DB {
+	t.Helper()
+
+	db, err := openScanDatabase(t.Context(), testDBPath(t))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	t.Cleanup(func() { _ = db.Close() })
+
+	return db
+}
+
+func TestDatabasePath(t *testing.T) {
+	t.Setenv(databaseEnv, "")
+
+	if got := databasePath(); got != defaultDatabasePath {
+		t.Errorf("databasePath() = %q, want %q", got, defaultDatabasePath)
+	}
+
+	t.Setenv(databaseEnv, "/custom/place.sqlite")
+
+	if got := databasePath(); got != "/custom/place.sqlite" {
+		t.Errorf("databasePath() = %q, want the env override", got)
+	}
+}
+
+func TestOpenScanDatabaseCreates(t *testing.T) {
+	t.Parallel()
+
+	// The parent directory does not exist yet; scan must create it.
+	path := filepath.Join(t.TempDir(), "nested", "dir", "db.sqlite")
+
+	db, err := openScanDatabase(t.Context(), path)
+	if err != nil {
+		t.Fatalf("openScanDatabase: %v", err)
+	}
+
+	v, err := userVersion(t.Context(), db)
+	if err != nil || v != schemaVersion {
+		t.Fatalf("userVersion = %d, %v; want %d, nil", v, err, schemaVersion)
+	}
+
+	_ = db.Close()
+
+	// Reopening an existing database must succeed and find the schema.
+	db, err = openScanDatabase(t.Context(), path)
+	if err != nil {
+		t.Fatalf("reopen: %v", err)
+	}
+
+	defer func() { _ = db.Close() }()
+
+	recs, err := loadFileRows(t.Context(), db)
+	if err != nil || len(recs) != 0 {
+		t.Fatalf("loadFileRows = %v, %v; want empty, nil", recs, err)
+	}
+}
+
+func TestOpenReportDatabaseMissing(t *testing.T) {
+	t.Parallel()
+
+	_, err := openReportDatabase(t.Context(), testDBPath(t))
+	if !errors.Is(err, errNoDatabase) {
+		t.Fatalf("err = %v, want errNoDatabase", err)
+	}
+}
+
+func TestOpenReportDatabaseVersionMismatch(t *testing.T) {
+	t.Parallel()
+
+	path := testDBPath(t)
+
+	db, err := openScanDatabase(t.Context(), path)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	_, err = db.ExecContext(context.Background(), "PRAGMA user_version = 99")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	_ = db.Close()
+
+	_, err = openReportDatabase(t.Context(), path)
+	if !errors.Is(err, errSchemaVersion) {
+		t.Fatalf("err = %v, want errSchemaVersion", err)
+	}
+}
+
+func TestOpenReportDatabaseOK(t *testing.T) {
+	t.Parallel()
+
+	path := testDBPath(t)
+
+	db, err := openScanDatabase(t.Context(), path)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	_ = db.Close()
+
+	db, err = openReportDatabase(t.Context(), path)
+	if err != nil {
+		t.Fatalf("openReportDatabase: %v", err)
+	}
+
+	_ = db.Close()
+}
+
+func TestApplyChangesRoundTrip(t *testing.T) {
+	t.Parallel()
+
+	db := openTestDB(t)
+
+	// Paths may contain tabs and newlines; the database must store
+	// them byte-exactly.
+	recs := []scanRec{
+		{size: 2, mtime: 20, head: "h2", tail: "t2", path: "/a/tab\tnew\nline"},
+		{size: 1, mtime: 10, head: "h1", tail: "t1", path: "/a/x"},
+	}
+
+	err := applyChanges(t.Context(), db, recs, nil,
+		newProgress("update", 2))
+	if err != nil {
+		t.Fatalf("applyChanges: %v", err)
+	}
+
+	got, err := loadFileRows(t.Context(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	slices.SortFunc(got, func(a, b scanRec) int {
+		return strings.Compare(a.path, b.path)
+	})
+
+	if !slices.Equal(got, recs) {
+		t.Fatalf("rows = %+v, want %+v", got, recs)
+	}
+
+	// An upsert for an existing path updates in place; a delete
+	// removes exactly its path.
+	upd := scanRec{size: 3, mtime: 30, head: "h3", tail: "t3", path: "/a/x"}
+
+	err = applyChanges(t.Context(), db, []scanRec{upd},
+		[]string{"/a/tab\tnew\nline"}, newProgress("update", 2))
+	if err != nil {
+		t.Fatalf("applyChanges: %v", err)
+	}
+
+	got, err = loadFileRows(t.Context(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if len(got) != 1 || got[0] != upd {
+		t.Fatalf("rows = %+v, want just %+v", got, upd)
+	}
+}
+
+func TestApplyChangesBatching(t *testing.T) {
+	t.Parallel()
+
+	db := openTestDB(t)
+
+	// One more change than the batch size, so the update spans two
+	// transactions.
+	n := updateBatchSize + 1
+
+	recs := make([]scanRec, 0, n)
+	for i := range n {
+		recs = append(recs, scanRec{
+			size: int64(i), mtime: 1, head: "h", tail: "t",
+			path: fmt.Sprintf("/batch/%07d", i),
+		})
+	}
+
+	err := applyChanges(t.Context(), db, recs, nil,
+		newProgress("update", int64(n)))
+	if err != nil {
+		t.Fatalf("applyChanges: %v", err)
+	}
+
+	got, err := loadFileRows(t.Context(), db)
+	if err != nil || len(got) != n {
+		t.Fatalf("loadFileRows = %d rows, %v; want %d", len(got), err, n)
+	}
+
+	deletes := make([]string, 0, n)
+	for _, r := range recs {
+		deletes = append(deletes, r.path)
+	}
+
+	err = applyChanges(t.Context(), db, nil, deletes,
+		newProgress("update", int64(n)))
+	if err != nil {
+		t.Fatalf("applyChanges deletes: %v", err)
+	}
+
+	got, err = loadFileRows(t.Context(), db)
+	if err != nil || len(got) != 0 {
+		t.Fatalf("loadFileRows = %d rows, %v; want 0", len(got), err)
+	}
+}

go.mod

@@ -5,13 +5,22 @@ go 1.25.7
 require (
 	github.com/schollz/progressbar/v3 v3.19.1
 	github.com/spf13/cobra v1.10.2
+	modernc.org/sqlite v1.54.0
 )
 
 require (
+	github.com/dustin/go-humanize v1.0.1 // indirect
+	github.com/google/uuid v1.6.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
+	github.com/mattn/go-isatty v0.0.22 // indirect
 	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
+	github.com/ncruces/go-strftime v1.0.0 // indirect
+	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	golang.org/x/sys v0.46.0 // indirect
 	golang.org/x/term v0.44.0 // indirect
+	modernc.org/libc v1.74.1 // indirect
+	modernc.org/mathutil v1.7.1 // indirect
+	modernc.org/memory v1.11.0 // indirect
 )

go.sum

@@ -3,14 +3,28 @@ github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
+github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
+github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
+github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
+github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
 github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
 github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
+github.com/mattn/go-isatty v0.0.22 h1:j8l17JJ9i6VGPUFUYoTUKPSgKe/83EYU2zBC7YNKMw4=
+github.com/mattn/go-isatty v0.0.22/go.mod h1:ZXfXG4SQHsB/w3ZeOYbR0PrPwLy+n6xiMrJlRFqopa4=
 github.com/mattn/go-runewidth v0.0.16 h1:E5ScNMtiwvlvB5paMFdw9p4kSQzbXFikJ5SQO6TULQc=
 github.com/mattn/go-runewidth v0.0.16/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
 github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db h1:62I3jR2EmQ4l5rM/4FEfDWcRD+abF5XlKShorW5LRoQ=
 github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db/go.mod h1:l0dey0ia/Uv7NcFFVbCLtqEBQbrT4OCwCSKTEv6enCw=
+github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
+github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
+github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
+github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
 github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
 github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -23,10 +37,44 @@ github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An
 github.com/stretchr/testify v1.9.0 h1:HtqpIVDClZ4nwg75+f6Lvsy/wHu+3BoSGCbBAcpTsTg=
 github.com/stretchr/testify v1.9.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
 go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
+golang.org/x/mod v0.37.0 h1:vF1DjpVEshcIqoEaauuHebaLk1O1forxjxBaVn884JQ=
+golang.org/x/mod v0.37.0/go.mod h1:m8S8VeM9r4dzDwjrKO0a1sZP3YjeMamRRlD+fmR2Q/0=
+golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
+golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
 golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw=
 golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
 golang.org/x/term v0.44.0 h1:0rLvDRCtNj0gZkyIXhCyOb2OAzEhLVqc4B+hrsBhrmc=
 golang.org/x/term v0.44.0/go.mod h1:7ze4MdzUzLXpSAoFP1H0bOI9aXDqveSvatT5vKcFh2Y=
+golang.org/x/tools v0.47.0 h1:7Kn5x/d1svx/PzryTsqeoZN4TZwqeH5pGWjefhLi/1Q=
+golang.org/x/tools v0.47.0/go.mod h1:dFHnyTvFWY212G+h7ZY4Vsp/K3U4/7W9TyVaAul8uCA=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
 gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+modernc.org/cc/v4 v4.29.0 h1:CXgwL8cvxmyzBQZzbSl/6xFtMCryb6u8IOqDci39cgc=
+modernc.org/cc/v4 v4.29.0/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
+modernc.org/ccgo/v4 v4.34.6 h1:sBgfIwyN0TQ9C5hwIeuqyeAKyMWnbvj2fvpF4L11uzU=
+modernc.org/ccgo/v4 v4.34.6/go.mod h1:SZ8YcN9NG7XVsQYdm6jYBvi8PQP1qi+kqB6OhjqI3Fk=
+modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
+modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
+modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
+modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
+modernc.org/gc/v3 v3.1.4 h1:2g65LGVSmFQrXeITAw97x7hCRvZFcyE1uDP+7Vng7JI=
+modernc.org/gc/v3 v3.1.4/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
+modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
+modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
+modernc.org/libc v1.74.1 h1:bdR4VTKFMC4966QSNZ05XLGI/VwzVa2kTUX51Dm0riQ=
+modernc.org/libc v1.74.1/go.mod h1:uH4t5bOx3G3g9Xcmj10YKlTcVISlRDwv8VoQJG9n8Os=
+modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
+modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
+modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
+modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
+modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
+modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
+modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
+modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
+modernc.org/sqlite v1.54.0 h1:JCxR4qwkJvOaqAoYcgDoO25Nc+ROg6EJ2LfBVzdrgog=
+modernc.org/sqlite v1.54.0/go.mod h1:4ntCLuNmnH8+GNqjka1wNg7KJd5/Hi5FYp8K+XQ7GZw=
+modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
+modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
+modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
+modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=

main.go

@@ -2,32 +2,50 @@
 // very large filesystems without reading full file contents. Files are
 // considered duplicates when they have identical size, identical SHA-256
 // of their first 1024 bytes, and identical SHA-256 of their last 1024
-// bytes.
+// bytes. scan maintains a persistent SQLite database of file signatures
+// (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) that the
+// reporting subcommands read.
 //
 // Usage:
 //
-//	sfdupes scan [--workers N] [-x] PATH... > files.dat
-//	sfdupes report [files.dat|-] > dupes.tsv
-//	sfdupes trees [files.dat|-] > dupetrees.tsv
+//	sfdupes scan [--workers N] [-x] PATH...
+//	sfdupes report > dupes.tsv
+//	sfdupes trees > dupetrees.tsv
 //
 // See README.md for the complete specification.
 package main
 
 import (
+	"context"
+	"errors"
 	"fmt"
+	"io"
 	"os"
 	"runtime"
 
 	"github.com/spf13/cobra"
 )
 
-// Exit codes: 0 is success (even with per-file warnings), exitFatal is
-// a fatal error, exitUsage is a usage error.
+// Exit codes: exitOK is success (even with per-file warnings),
+// exitFatal is a fatal error, exitUsage is a usage error.
 const (
+	exitOK    = 0
 	exitFatal = 1
 	exitUsage = 2
 )
 
+// The subcommand names, as typed on the command line.
+const (
+	cmdScan   = "scan"
+	cmdReport = "report"
+	cmdTrees  = "trees"
+)
+
+// errNoSubcommand is returned by the root command when it is invoked
+// without a subcommand. That is a usage error, and the usage text
+// cobra prints for it is the whole message.
+var errNoSubcommand = errors.New("no subcommand")
+
 // Version is the build version, injected at link time via -ldflags
 // (see the Makefile); "dev" for a plain go build.
 //
@@ -35,22 +53,64 @@ const (
 var Version = "dev"
 
 func main() {
+	os.Exit(run(os.Args[1:], os.Stderr))
+}
+
+// run executes args against the command tree and returns the process
+// exit code. It is the program's single exit point: the subcommands
+// return their errors instead of exiting, so every deferred cleanup —
+// above all closing the database, which checkpoints the SQLite WAL —
+// runs before the process ends.
+func run(args []string, stderr io.Writer) int {
+	// A nil slice makes cobra fall back to os.Args, which would let a
+	// test binary's own flags reach the command tree.
+	if args == nil {
+		args = []string{}
+	}
+
+	root := newRootCommand(stderr)
+	root.SetArgs(args)
+
+	err := root.Execute()
+
+	var fatal fatalError
+
+	switch {
+	case err == nil:
+		return exitOK
+	case errors.As(err, &fatal):
+		// The command ran and failed: a runtime error, reported
+		// without the usage text that a usage error gets.
+		_, _ = fmt.Fprintf(stderr, "sfdupes: %v\n", err)
+
+		return exitFatal
+	default:
+		// A usage error: cobra has already printed the message and
+		// the usage text.
+		return exitUsage
+	}
+}
+
+// newRootCommand builds the command tree. Everything on stdout is
+// machine-readable data; all human-facing output (help, usage, errors)
+// goes to stderr.
+func newRootCommand(stderr io.Writer) *cobra.Command {
 	root := &cobra.Command{
 		Use:     "sfdupes",
 		Short:   "Find candidate duplicate files by size and head/tail SHA-256",
 		Version: Version,
 		Args:    cobra.NoArgs,
-		Run: func(cmd *cobra.Command, _ []string) {
-			// A missing subcommand prints usage and exits 2.
-			_ = cmd.Usage()
+		RunE: func(cmd *cobra.Command, _ []string) error {
+			// A missing subcommand prints usage and exits 2: cobra
+			// prints the usage text for the returned error, and run
+			// maps everything that is not a fatal error to exit 2.
+			cmd.SilenceErrors = true
 
-			os.Exit(exitUsage)
+			return errNoSubcommand
 		},
 	}
-	// Everything on stdout is machine-readable data; all human-facing
-	// output (help, usage, errors) goes to stderr.
-	root.SetOut(os.Stderr)
-	root.SetErr(os.Stderr)
+	root.SetOut(stderr)
+	root.SetErr(stderr)
 	root.CompletionOptions.DisableDefaultCmd = true
 
 	var (
@@ -59,48 +119,72 @@ func main() {
 	)
 
 	scanCmd := &cobra.Command{
-		Use:   "scan [--workers N] [-x] PATH...",
-		Short: "Walk trees and emit one record per regular file on stdout",
+		Use:   cmdScan + " [--workers N] [-x] PATH...",
+		Short: "Walk trees and synchronize the scan database",
 		Args:  cobra.MinimumNArgs(1),
-		Run: func(_ *cobra.Command, args []string) {
-			runScan(args, scanWorkers, scanOneFS)
-		},
+		RunE: runE(func(ctx context.Context, args []string) error {
+			return runScan(ctx, args, scanWorkers, scanOneFS)
+		}),
 	}
 	scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
-		"concurrent workers for the stat and hash passes")
+		"concurrent workers for the walk and hash phases")
 	scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
 		"do not cross filesystem boundaries")
 
 	reportCmd := &cobra.Command{
-		Use:   "report [files.dat|-]",
-		Short: "Read a scan stream and print the file-level duplicates report",
-		Args:  cobra.MaximumNArgs(1),
-		Run: func(_ *cobra.Command, args []string) {
-			runReport(args)
-		},
+		Use:   cmdReport,
+		Short: "Read the scan database and print the file-level duplicates report",
+		Args:  cobra.NoArgs,
+		RunE: runE(func(ctx context.Context, _ []string) error {
+			return runReport(ctx)
+		}),
 	}
 
 	treesCmd := &cobra.Command{
-		Use:   "trees [files.dat|-]",
-		Short: "Read a scan stream and print the duplicate-tree report",
-		Args:  cobra.MaximumNArgs(1),
-		Run: func(_ *cobra.Command, args []string) {
-			runTrees(args)
-		},
+		Use:   cmdTrees,
+		Short: "Read the scan database and print the duplicate-tree report",
+		Args:  cobra.NoArgs,
+		RunE: runE(func(ctx context.Context, _ []string) error {
+			return runTrees(ctx)
+		}),
 	}
 
 	root.AddCommand(scanCmd, reportCmd, treesCmd)
 
-	err := root.Execute()
-	if err != nil {
-		// Cobra has already printed the error and usage to stderr;
-		// an invalid subcommand or bad arguments is a usage error.
-		os.Exit(exitUsage)
+	return root
+}
+
+// runE adapts a subcommand implementation to cobra's RunE. Cobra
+// prints the error and the command's usage text for every error RunE
+// returns, but a subcommand that ran and failed has no usage problem
+// to report: both are silenced here, and the error is marked fatal so
+// that run reports it on stderr and exits 1 rather than 2. The command's
+// context is handed to the implementation: cancelling it unwinds the
+// scan's worker pools.
+func runE(
+	fn func(ctx context.Context, args []string) error,
+) func(*cobra.Command, []string) error {
+	return func(cmd *cobra.Command, args []string) error {
+		cmd.SilenceUsage = true
+		cmd.SilenceErrors = true
+
+		err := fn(cmd.Context(), args)
+		if err != nil {
+			return fatalError{err: err}
+		}
+
+		return nil
 	}
 }
 
-// fatalf reports a fatal error and exits 1.
-func fatalf(format string, args ...any) {
-	fmt.Fprintf(os.Stderr, "sfdupes: "+format+"\n", args...)
-	os.Exit(exitFatal)
+// fatalError marks a runtime failure, as opposed to the usage errors
+// cobra itself produces while parsing arguments and flags. Both come
+// out of Execute as plain errors, so the wrapper is what tells run to
+// report this one as "sfdupes: ..." and exit 1.
+type fatalError struct {
+	err error
 }
+
+func (e fatalError) Error() string { return e.err.Error() }
+
+func (e fatalError) Unwrap() error { return e.err }

main_test.go

@@ -0,0 +1,397 @@
+package main
+
+import (
+	"bytes"
+	"context"
+	"errors"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"testing"
+)
+
+// usageMarker is the first line of cobra's usage text, which a usage
+// error prints and a runtime failure must not.
+const usageMarker = "Usage:"
+
+// walSuffixes are the SQLite sidecar files a WAL-mode database keeps
+// while it is open. A clean close checkpoints the WAL and removes
+// both; finding either afterwards means the database was never closed.
+//
+//nolint:gochecknoglobals // a constant list, immutable by convention
+var walSuffixes = []string{"-wal", "-shm"}
+
+// assertNoSidecars fails when a WAL sidecar is still present beside the
+// database at path.
+func assertNoSidecars(t *testing.T, path string) {
+	t.Helper()
+
+	for _, suffix := range walSuffixes {
+		_, err := os.Stat(path + suffix)
+		if err == nil {
+			t.Errorf("%s%s still present: the database was not closed",
+				path, suffix)
+
+			continue
+		}
+
+		if !errors.Is(err, fs.ErrNotExist) {
+			t.Fatal(err)
+		}
+	}
+}
+
+// captureStdout redirects os.Stdout to a file for the rest of the test
+// and returns a function reading back everything written to it. Only
+// machine-readable data belongs on stdout (README design goal 4), so
+// the tests assert on it directly.
+func captureStdout(t *testing.T) func() string {
+	t.Helper()
+
+	f, err := os.Create(filepath.Join(t.TempDir(), "stdout"))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	saved := os.Stdout
+	os.Stdout = f
+
+	t.Cleanup(func() {
+		os.Stdout = saved
+
+		_ = f.Close()
+	})
+
+	return func() string {
+		// Read what has been written without disturbing the write
+		// offset, so the capture can be inspected more than once.
+		size, err := f.Seek(0, io.SeekCurrent)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		if size == 0 {
+			return ""
+		}
+
+		b := make([]byte, size)
+
+		_, err = f.ReadAt(b, 0)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		return string(b)
+	}
+}
+
+// brokenDatabase writes a database that opens cleanly and passes the
+// schema-version check but has no files table, so the first query
+// fails with the database already open: a fatal error on a path that
+// owns an open database.
+func brokenDatabase(t *testing.T) string {
+	t.Helper()
+
+	path := testDBPath(t)
+
+	db, err := openDB(path)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	_, err = db.ExecContext(context.Background(),
+		"PRAGMA user_version = "+strconv.Itoa(schemaVersion))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = db.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	return path
+}
+
+func TestOpenDatabaseKeepsWALWhileOpen(t *testing.T) {
+	t.Parallel()
+
+	// The premise of the fatal-path tests below: an open database has
+	// a -wal sidecar, so its absence afterwards is evidence that the
+	// database was closed and its WAL checkpointed.
+	path := testDBPath(t)
+
+	db, err := openScanDatabase(t.Context(), path)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	_, err = os.Stat(path + "-wal")
+	if err != nil {
+		t.Fatalf("no -wal beside an open database: %v", err)
+	}
+
+	err = db.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	assertNoSidecars(t, path)
+}
+
+func TestRunFatalAfterOpenClosesDatabase(t *testing.T) {
+	// Every subcommand that owns an open database must close it when
+	// it fails: no os.Exit between the open and the return.
+	cases := map[string][]string{
+		cmdScan:   {cmdScan},
+		cmdReport: {cmdReport},
+		cmdTrees:  {cmdTrees},
+	}
+
+	for name, args := range cases {
+		t.Run(name, func(t *testing.T) {
+			path := brokenDatabase(t)
+			t.Setenv(databaseEnv, path)
+
+			if name == cmdScan {
+				args = append(args, t.TempDir())
+			}
+
+			var stderr bytes.Buffer
+
+			stdout := captureStdout(t)
+
+			code := run(args, &stderr)
+			if code != exitFatal {
+				t.Errorf("run(%v) = %d, want %d", args, code, exitFatal)
+			}
+
+			assertNoSidecars(t, path)
+			assertFatalOutput(t, stderr.String(), stdout())
+
+			// Proof that the failure happened after the open: only a
+			// query against the opened database can report this.
+			if !strings.Contains(stderr.String(), "no such table: files") {
+				t.Errorf("stderr = %q, want the failure to come from a "+
+					"query on the open database", stderr.String())
+			}
+		})
+	}
+}
+
+func TestRunMissingOperandIsFatalNotUsage(t *testing.T) {
+	// README §Error handling: a PATH operand that does not exist is a
+	// fatal error (1), not a usage error (2) — and a runtime failure
+	// must not dump the usage text.
+	t.Setenv(databaseEnv, testDBPath(t))
+
+	var stderr bytes.Buffer
+
+	stdout := captureStdout(t)
+
+	missing := filepath.Join(t.TempDir(), "nope")
+
+	code := run([]string{cmdScan, missing}, &stderr)
+	if code != exitFatal {
+		t.Errorf("run(scan %s) = %d, want %d", missing, code, exitFatal)
+	}
+
+	assertFatalOutput(t, stderr.String(), stdout())
+}
+
+// assertFatalOutput checks that a fatal error was reported the way
+// README §Error handling and design goal 4 require: the message on
+// stderr, prefixed with the program name, no usage text, and nothing
+// at all on stdout.
+func assertFatalOutput(t *testing.T, stderr, stdout string) {
+	t.Helper()
+
+	if !strings.Contains(stderr, "sfdupes: ") {
+		t.Errorf("stderr = %q, want a \"sfdupes: \" error report", stderr)
+	}
+
+	if strings.Contains(stderr, usageMarker) {
+		t.Errorf("stderr = %q, want no usage text for a runtime failure",
+			stderr)
+	}
+
+	if stdout != "" {
+		t.Errorf("stdout = %q, want nothing (data only)", stdout)
+	}
+}
+
+func TestRunUsageErrors(t *testing.T) {
+	// Usage errors keep exiting 2 with cobra's own report on stderr.
+	cases := map[string]struct {
+		args []string
+		want string
+	}{
+		"no subcommand":      {[]string{}, usageMarker},
+		"scan without paths": {[]string{cmdScan}, usageMarker},
+		"report with args":   {[]string{cmdReport, "x"}, usageMarker},
+		"trees with args":    {[]string{cmdTrees, "x"}, usageMarker},
+		"unknown flag":       {[]string{cmdScan, "--nope", "/"}, usageMarker},
+		"unknown subcommand": {[]string{"nope"}, "unknown command"},
+	}
+
+	for name, tc := range cases {
+		t.Run(name, func(t *testing.T) {
+			// No usage error may reach the database, so point it at a
+			// path that does not exist.
+			t.Setenv(databaseEnv, testDBPath(t))
+
+			var stderr bytes.Buffer
+
+			stdout := captureStdout(t)
+
+			code := run(tc.args, &stderr)
+			if code != exitUsage {
+				t.Errorf("run(%v) = %d, want %d", tc.args, code, exitUsage)
+			}
+
+			if !strings.Contains(stderr.String(), tc.want) {
+				t.Errorf("stderr = %q, want %q", stderr.String(), tc.want)
+			}
+
+			if got := stdout(); got != "" {
+				t.Errorf("stdout = %q, want nothing (data only)", got)
+			}
+		})
+	}
+}
+
+// TestRunHelpAndVersionSucceed checks that the two informational flags
+// exit 0 and keep their human-facing output on stderr.
+//
+//nolint:paralleltest // captureStdout replaces the process-wide os.Stdout
+func TestRunHelpAndVersionSucceed(t *testing.T) {
+	assertHumanOutput(t, "--help")
+	assertHumanOutput(t, "--version")
+}
+
+// assertHumanOutput runs sfdupes with one informational flag and checks
+// that it succeeds with its output on stderr and stdout untouched
+// (README design goal 4).
+func assertHumanOutput(t *testing.T, arg string) {
+	t.Helper()
+
+	var stderr bytes.Buffer
+
+	stdout := captureStdout(t)
+
+	code := run([]string{arg}, &stderr)
+	if code != exitOK {
+		t.Errorf("run(%s) = %d, want %d", arg, code, exitOK)
+	}
+
+	if stderr.Len() == 0 {
+		t.Errorf("run(%s) wrote nothing to stderr", arg)
+	}
+
+	if got := stdout(); got != "" {
+		t.Errorf("stdout = %q, want nothing (data only)", got)
+	}
+}
+
+// scanFixture builds a small tree holding one duplicate pair and one
+// unreadable file and scans it into the database the caller has
+// pointed SFDUPES_DATABASE at. The unreadable file makes the scan warn
+// and skip, which README §Error handling still calls a successful run.
+// It returns the duplicate pair's paths.
+func scanFixture(t *testing.T) []string {
+	t.Helper()
+
+	dir := t.TempDir()
+
+	dupes := []string{
+		writeFile(t, dir, "one/a.bin", pattern(1, 300)),
+		writeFile(t, dir, "two/a.bin", pattern(1, 300)),
+	}
+
+	// Same size as the pair, so the scan queues it for hashing and the
+	// read fails.
+	unreadable := writeFile(t, dir, "unreadable.bin", pattern(2, 300))
+
+	err := os.Chmod(unreadable, 0)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	var stderr bytes.Buffer
+
+	stdout := captureStdout(t)
+
+	code := run([]string{cmdScan, dir}, &stderr)
+	if code != exitOK {
+		t.Fatalf("run(scan) = %d, want %d; stderr: %s",
+			code, exitOK, stderr.String())
+	}
+
+	if got := stdout(); got != "" {
+		t.Errorf("scan stdout = %q, want nothing (data only)", got)
+	}
+
+	return dupes
+}
+
+func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	scanFixture(t)
+	assertNoSidecars(t, path)
+}
+
+func TestRunReportSucceeds(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dupes := scanFixture(t)
+
+	var stderr bytes.Buffer
+
+	stdout := captureStdout(t)
+
+	code := run([]string{cmdReport}, &stderr)
+	if code != exitOK {
+		t.Fatalf("run(report) = %d, want %d; stderr: %s",
+			code, exitOK, stderr.String())
+	}
+
+	want := "first\tdupe\tsize\n" + dupes[0] + "\t" + dupes[1] + "\t300\n"
+	if got := stdout(); got != want {
+		t.Errorf("stdout = %q, want %q", got, want)
+	}
+
+	assertNoSidecars(t, path)
+}
+
+func TestRunTreesSucceeds(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dupes := scanFixture(t)
+
+	var stderr bytes.Buffer
+
+	stdout := captureStdout(t)
+
+	code := run([]string{cmdTrees}, &stderr)
+	if code != exitOK {
+		t.Fatalf("run(trees) = %d, want %d; stderr: %s",
+			code, exitOK, stderr.String())
+	}
+
+	// The two directories holding the duplicate pair are duplicate
+	// trees of each other.
+	want := "first\tdupe\tfiles\tsize\n" +
+		filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
+	if got := stdout(); got != want {
+		t.Errorf("stdout = %q, want %q", got, want)
+	}
+
+	assertNoSidecars(t, path)
+}

progress.go

@@ -38,7 +38,10 @@ func stderrIsTTY() bool {
 // stderr is not a TTY it emits no ANSI redraws: it prints a plain
 // one-line update no more often than every plainInterval.
 //
-// All methods must be called from the main goroutine only.
+// All methods must be called from the main goroutine only. A nil
+// *progress is a valid no-display receiver: every method is a no-op,
+// so batched database flushes during the streaming pass can reuse the
+// update-pass helpers without rendering anything.
 type progress struct {
 	label string
 	total int64 // -1 when unknown (walk pass)
@@ -62,6 +65,9 @@ func newProgress(label string, total int64) *progress {
 		progressbar.OptionSetItsString("files"),
 		progressbar.OptionSetElapsedTime(true),
 		progressbar.OptionThrottle(barThrottle),
+		// Render at zero immediately: a phase must be visible the
+		// moment it starts, even before its first item completes.
+		progressbar.OptionSetRenderBlankState(true),
 	}
 	if total >= 0 {
 		opts = append(opts,
@@ -82,6 +88,10 @@ func newProgress(label string, total int64) *progress {
 
 // increment records one completed item and refreshes the display.
 func (p *progress) increment() {
+	if p == nil {
+		return
+	}
+
 	p.count++
 	if p.bar != nil {
 		_ = p.bar.Add(1)
@@ -97,6 +107,10 @@ func (p *progress) increment() {
 
 // warnf prints a one-line warning to stderr without corrupting the bar.
 func (p *progress) warnf(format string, args ...any) {
+	if p == nil {
+		return
+	}
+
 	if p.bar != nil {
 		_ = p.bar.Clear()
 	}
@@ -106,6 +120,10 @@ func (p *progress) warnf(format string, args ...any) {
 
 // finish terminates the pass's display.
 func (p *progress) finish() {
+	if p == nil {
+		return
+	}
+
 	if p.bar != nil {
 		_ = p.bar.Finish()
 

report.go

@@ -2,106 +2,53 @@ package main
 
 import (
 	"bufio"
-	"bytes"
+	"context"
 	"fmt"
-	"io"
 	"os"
 	"slices"
-	"strconv"
 	"strings"
 )
 
-// ioBufSize is the buffer size for the buffered scan-stream reader and
-// the buffered stdout writers.
+// ioBufSize is the buffer size for the buffered stdout writers.
 const ioBufSize = 1 << 20
 
-// recordFieldCount is the number of tab-separated fields in a scan
-// record: size, mtime, head hash, tail hash, path.
-const recordFieldCount = 5
-
-// scanInitBufSize and scanMaxRecordSize bound the scanner buffer used
-// to read scan records; a record longer than scanMaxRecordSize is a
-// fatal read error.
-const (
-	scanInitBufSize   = 64 << 10
-	scanMaxRecordSize = 4 << 20
-)
-
 // minGroupSize is the smallest number of members that makes a
 // duplicate group.
 const minGroupSize = 2
 
-// scanRec is one well-formed record parsed from a scan stream. The
-// signature (size, head, tail) is the duplicate key; mtime is
-// informational and not retained.
+// scanRec is one file record from the database. The signature (size,
+// head, tail) is the duplicate key; mtime is informational only and
+// used by scan for change detection.
 type scanRec struct {
-	size int64
-	head string
-	tail string
-	path string
+	size  int64
+	mtime int64
+	head  string
+	tail  string
+	path  string
 }
 
-// openScanInput resolves the analysis-mode input: the file named by the
-// single optional positional argument, or stdin when it is absent or
-// "-". The returned closer must be called when reading is done.
-func openScanInput(args []string) (io.Reader, string, func()) {
-	if len(args) == 1 && args[0] != "-" {
-		f, err := os.Open(args[0])
-		if err != nil {
-			fatalf("open %s: %v", args[0], err)
-		}
+// loadRecords opens the database and reads every file record for the
+// report and trees subcommands. Any database problem — including a
+// missing database — is fatal. The error is returned rather than
+// exiting, so that the deferred close — which checkpoints the SQLite
+// WAL — always runs; the database is closed before the caller formats
+// its output, so it stays closed even if that output fails.
+func loadRecords(ctx context.Context) ([]scanRec, error) {
+	dbPath := databasePath()
 
-		return f, args[0], func() { _ = f.Close() }
-	}
-
-	return os.Stdin, "stdin", func() {}
-}
-
-// parseScanStream reads every NUL-terminated record from in. A record
-// that does not have exactly 5 fields or whose size is non-numeric is
-// counted as malformed and skipped. Total records read is
-// len(recs) + malformed.
-func parseScanStream(in io.Reader, name string) ([]scanRec, int) {
-	sc := bufio.NewScanner(bufio.NewReaderSize(in, ioBufSize))
-	sc.Buffer(make([]byte, 0, scanInitBufSize), scanMaxRecordSize)
-	sc.Split(splitNUL)
-
-	var (
-		recs      []scanRec
-		malformed int
-	)
-
-	for sc.Scan() {
-		// The path is the last field and may itself contain tabs,
-		// so split into at most recordFieldCount fields.
-		fields := strings.SplitN(sc.Text(), "\t", recordFieldCount)
-		if len(fields) != recordFieldCount {
-			malformed++
-
-			continue
-		}
-
-		size, err := strconv.ParseInt(fields[0], 10, 64)
-		if err != nil {
-			malformed++
-
-			continue
-		}
-
-		recs = append(recs, scanRec{
-			size: size,
-			head: fields[2],
-			tail: fields[3],
-			path: fields[4],
-		})
-	}
-
-	err := sc.Err()
+	db, err := openReportDatabase(ctx, dbPath)
 	if err != nil {
-		fatalf("read %s: %v", name, err)
+		return nil, err
 	}
 
-	return recs, malformed
+	defer func() { _ = db.Close() }()
+
+	recs, err := loadFileRows(ctx, db)
+	if err != nil {
+		return nil, fmt.Errorf("database %s: %w", dbPath, err)
+	}
+
+	return recs, nil
 }
 
 // dupeGroup is one set of candidate-duplicate files: identical size,
@@ -112,25 +59,23 @@ type dupeGroup struct {
 	paths []string
 }
 
-// runReport implements the report subcommand: it reads a scan stream
-// from the named file (or stdin when absent or "-") and prints the
-// file-level duplicates report as TSV on stdout. It never touches the
-// scanned filesystem; its only I/O is the scan input, stdout, and
-// stderr. args holds the positional arguments already validated by
-// cobra (at most one).
-func runReport(args []string) {
-	in, name, closer := openScanInput(args)
-	defer closer()
+// runReport implements the report subcommand: it reads every record
+// from the database and prints the file-level duplicates report as TSV
+// on stdout. It never touches the scanned filesystem; its only I/O is
+// the database, stdout, and stderr.
+func runReport(ctx context.Context) error {
+	recs, err := loadRecords(ctx)
+	if err != nil {
+		return err
+	}
 
-	recs, malformed := parseScanStream(in, name)
-	records := len(recs) + malformed
 	dupes := collectDupeGroups(recs)
 
 	out := bufio.NewWriterSize(os.Stdout, ioBufSize)
 
-	_, err := fmt.Fprintln(out, "first\tdupe\tsize")
+	_, err = fmt.Fprintln(out, "first\tdupe\tsize")
 	if err != nil {
-		fatalf("write stdout: %v", err)
+		return fmt.Errorf("write stdout: %w", err)
 	}
 
 	dupeFiles := 0
@@ -142,7 +87,7 @@ func runReport(args []string) {
 			_, err = fmt.Fprintf(out, "%s\t%s\t%d\n",
 				g.paths[0], p, g.size)
 			if err != nil {
-				fatalf("write stdout: %v", err)
+				return fmt.Errorf("write stdout: %w", err)
 			}
 
 			dupeFiles++
@@ -152,13 +97,15 @@ func runReport(args []string) {
 
 	err = out.Flush()
 	if err != nil {
-		fatalf("write stdout: %v", err)
+		return fmt.Errorf("write stdout: %w", err)
 	}
 
 	fmt.Fprintf(os.Stderr,
-		"report: %d records read%s, %d duplicate groups, %d dupe files, %s reclaimable\n",
-		records, malformedNote(malformed), len(dupes), dupeFiles,
-		humanBytes(reclaimable))
+		"report: %d records read, %d duplicate groups, %d dupe files, "+
+			"%s reclaimable\n",
+		len(recs), len(dupes), dupeFiles, humanBytes(reclaimable))
+
+	return nil
 }
 
 // collectDupeGroups groups records by signature and returns every group
@@ -168,6 +115,13 @@ func collectDupeGroups(recs []scanRec) []dupeGroup {
 	groups := make(map[fileSig][]string)
 
 	for _, r := range recs {
+		// A record without hashes (its size was unique when last
+		// scanned) has unknown content and is never reported as a
+		// duplicate.
+		if r.head == "" {
+			continue
+		}
+
 		k := fileSig{size: r.size, head: r.head, tail: r.tail}
 		groups[k] = append(groups[k], r.path)
 	}
@@ -199,30 +153,6 @@ func collectDupeGroups(recs []scanRec) []dupeGroup {
 	return dupes
 }
 
-// malformedNote formats the optional malformed-record note for the
-// stderr summaries.
-func malformedNote(malformed int) string {
-	if malformed == 0 {
-		return ""
-	}
-
-	return fmt.Sprintf(" (%d malformed, skipped)", malformed)
-}
-
-// splitNUL is a bufio.SplitFunc for NUL-terminated records. Trailing
-// data without a terminator at EOF is returned as a final record.
-func splitNUL(data []byte, atEOF bool) (int, []byte, error) {
-	if i := bytes.IndexByte(data, 0); i >= 0 {
-		return i + 1, data[:i], nil
-	}
-
-	if atEOF && len(data) > 0 {
-		return len(data), data, nil
-	}
-
-	return 0, nil, nil
-}
-
 // humanBytes formats a byte count in human units (binary prefixes).
 func humanBytes(n int64) string {
 	const unit = 1024

report_test.go

@@ -1,92 +1,10 @@
 package main
 
 import (
-	"bufio"
-	"fmt"
 	"slices"
-	"strings"
 	"testing"
 )
 
-// mkRecord serializes one scan record in the on-the-wire format.
-func mkRecord(size, mtime int64, head, tail, path string) string {
-	return fmt.Sprintf("%d\t%d\t%s\t%s\t%s\x00",
-		size, mtime, head, tail, path)
-}
-
-func TestSplitNULScanner(t *testing.T) {
-	t.Parallel()
-
-	sc := bufio.NewScanner(strings.NewReader("a\x00bb\x00\x00tail"))
-	sc.Split(splitNUL)
-
-	var got []string
-	for sc.Scan() {
-		got = append(got, sc.Text())
-	}
-
-	err := sc.Err()
-	if err != nil {
-		t.Fatalf("scanner error: %v", err)
-	}
-
-	want := []string{"a", "bb", "", "tail"}
-	if !slices.Equal(got, want) {
-		t.Fatalf("tokens = %q, want %q", got, want)
-	}
-}
-
-func TestParseScanStream(t *testing.T) {
-	t.Parallel()
-
-	stream := mkRecord(10, 1, "h1", "t1", "/a/x") +
-		"garbage-without-tabs\x00" +
-		"notanumber\t1\th\tt\t/a/bad\x00" +
-		mkRecord(20, 2, "h2", "t2", "/a/tab\tin\tname") +
-		"30\t3\th3\tt3\t/trailing/no-nul"
-
-	recs, malformed := parseScanStream(strings.NewReader(stream), "test")
-
-	if malformed != 2 {
-		t.Errorf("malformed = %d, want 2", malformed)
-	}
-
-	want := []scanRec{
-		{size: 10, head: "h1", tail: "t1", path: "/a/x"},
-		{size: 20, head: "h2", tail: "t2", path: "/a/tab\tin\tname"},
-		{size: 30, head: "h3", tail: "t3", path: "/trailing/no-nul"},
-	}
-	if !slices.Equal(recs, want) {
-		t.Fatalf("recs = %+v, want %+v", recs, want)
-	}
-}
-
-func TestParseScanStreamPathWithNewline(t *testing.T) {
-	t.Parallel()
-
-	in := strings.NewReader(mkRecord(5, 9, "h", "t", "/a/new\nline"))
-
-	recs, malformed := parseScanStream(in, "test")
-	if malformed != 0 || len(recs) != 1 {
-		t.Fatalf("got %d recs, %d malformed, want 1, 0",
-			len(recs), malformed)
-	}
-
-	if recs[0].path != "/a/new\nline" {
-		t.Fatalf("path = %q, want %q", recs[0].path, "/a/new\nline")
-	}
-}
-
-func TestParseScanStreamEmpty(t *testing.T) {
-	t.Parallel()
-
-	recs, malformed := parseScanStream(strings.NewReader(""), "test")
-	if len(recs) != 0 || malformed != 0 {
-		t.Fatalf("got %d recs, %d malformed, want 0, 0",
-			len(recs), malformed)
-	}
-}
-
 func TestCollectDupeGroups(t *testing.T) {
 	t.Parallel()
 
@@ -120,6 +38,22 @@ func TestCollectDupeGroups(t *testing.T) {
 	}
 }
 
+func TestCollectDupeGroupsMtimeExcluded(t *testing.T) {
+	t.Parallel()
+
+	// mtime is informational only; records differing only in mtime
+	// still group together.
+	recs := []scanRec{
+		{size: 9, mtime: 100, head: "h", tail: "t", path: "/m/1"},
+		{size: 9, mtime: 200, head: "h", tail: "t", path: "/m/2"},
+	}
+
+	groups := collectDupeGroups(recs)
+	if len(groups) != 1 {
+		t.Fatalf("len(groups) = %d, want 1", len(groups))
+	}
+}
+
 func TestCollectDupeGroupsTieBreak(t *testing.T) {
 	t.Parallel()
 
@@ -190,15 +124,3 @@ func TestHumanBytes(t *testing.T) {
 		}
 	}
 }
-
-func TestMalformedNote(t *testing.T) {
-	t.Parallel()
-
-	if got := malformedNote(0); got != "" {
-		t.Errorf("malformedNote(0) = %q, want empty", got)
-	}
-
-	if got := malformedNote(3); got != " (3 malformed, skipped)" {
-		t.Errorf("malformedNote(3) = %q", got)
-	}
-}

script/bootstrap

@@ -0,0 +1,90 @@
+#!/bin/sh
+# script/bootstrap: install all dependencies needed to build and develop
+# this repo. Idempotent: every install is guarded by a check so already
+# installed tools are skipped. Base tooling comes from nix, apt, brew,
+# or apk (detected in that order); assumes nothing is present (not git,
+# make, or go). The linter is NOT installed: golangci-lint runs via
+# docker only (script/lint), pinned by image digest, so the only lint
+# prerequisite is a working docker — which is warned about, not
+# installed, because everything except linting works without it.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+PKGMGR=""
+SUDO=""
+APT_UPDATED=""
+
+detect_pkgmgr() {
+    [ -n "$PKGMGR" ] && return 0
+    if command -v nix-env >/dev/null 2>&1; then
+        PKGMGR="nix"
+    elif command -v apt-get >/dev/null 2>&1; then
+        PKGMGR="apt"
+    elif command -v brew >/dev/null 2>&1; then
+        PKGMGR="brew"
+    elif command -v apk >/dev/null 2>&1; then
+        PKGMGR="apk"
+    else
+        echo "bootstrap: no supported package manager (nix, apt, brew, apk)" >&2
+        exit 1
+    fi
+    if [ "$PKGMGR" = "apt" ]; then
+        export DEBIAN_FRONTEND=noninteractive
+        if [ "$(id -u)" != "0" ]; then
+            SUDO="sudo"
+        fi
+    fi
+}
+
+# pkg_install <nix-attr> <apt-pkg> <brew-formula> <apk-pkg>
+pkg_install() {
+    detect_pkgmgr
+    case "$PKGMGR" in
+        nix) nix-env -iA "nixpkgs.$1" ;;
+        apt)
+            if [ -z "$APT_UPDATED" ]; then
+                $SUDO env DEBIAN_FRONTEND=noninteractive apt-get update
+                APT_UPDATED=1
+            fi
+            $SUDO env DEBIAN_FRONTEND=noninteractive apt-get install -y "$2"
+            ;;
+        brew) brew install "$3" ;;
+        apk) apk add --no-cache "$4" ;;
+    esac
+}
+
+missing() {
+    ! command -v "$1" >/dev/null 2>&1
+}
+
+main() {
+    cd "$ROOT"
+
+    # System tooling, deliberately unpinned: these come from the host
+    # package manager and whatever version it ships is what the host
+    # gets, so a presence check is the right check. The repo pins no
+    # system toolchain versions — the Go language version is governed by
+    # go.mod, and builds that must be reproducible run in the Docker
+    # image, whose base images are pinned by digest.
+    if missing git; then pkg_install git git git git; fi
+    if missing make; then pkg_install gnumake make make make; fi
+    if missing go; then pkg_install go golang go go; fi
+
+    # Linting runs via docker only (script/lint), so docker is a lint
+    # prerequisite rather than something bootstrap installs. Warn, do
+    # not fail: everything except `make lint` — and, through it,
+    # `make check`, `make docker` and the pre-commit hook — works
+    # without it.
+    if missing docker; then
+        echo "bootstrap: WARNING: docker not found; make lint, make check" >&2
+        echo "bootstrap: and make docker require it. Install docker to" >&2
+        echo "bootstrap: run the linter." >&2
+    fi
+
+    go mod download
+
+    echo "bootstrap complete"
+}
+
+main "$@"

script/check

@@ -0,0 +1,14 @@
+#!/bin/sh
+# script/check: run all checks (test, lint, fmt-check). Our own
+# extension to scripts-to-rule-them-all. Must not modify any files.
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+
+main() {
+    "$SCRIPT_DIR/test"
+    "$SCRIPT_DIR/lint"
+    "$SCRIPT_DIR/fmt-check"
+}
+
+main "$@"

script/cibuild

@@ -0,0 +1,34 @@
+#!/bin/sh
+# script/cibuild: run the CI build. The Gitea workflow runs this on
+# push.
+#
+# The Dockerfile runs the gates individually as build steps, not the
+# make check aggregate: the lint stage runs make fmt-check,
+# script/verify-lint-image-pin, golangci-lint config verify and
+# golangci-lint run; the build stage, dropped to an unprivileged user,
+# runs make test and make fmt-check. Neither make lint nor make check
+# appears, because both reach script/lint, which is itself a docker
+# build, and a docker build cannot run inside one. Lint is not skipped
+# by that — the linter is invoked directly in the lint stage, and the
+# build stage's COPY --from=lint makes that stage a prerequisite, so
+# BuildKit must finish it first. Between the two stages everything
+# make check would run has run, which is why a successful build here
+# implies the repo is green.
+#
+# That implication holds only because of CHECK_EPOCH. A COPY layer is
+# invalidated by changed content, and a merge commit's tree is
+# byte-identical to the branch head it merges, so without a fresh value
+# here Docker serves the gate layers from cache and the build reports a
+# green it never earned. Passing the current epoch invalidates the gate
+# layers on every run while leaving the pinned base images and
+# go mod download cached; see the Dockerfile for the placement.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    docker build --build-arg CHECK_EPOCH="$(date +%s)" .
+}
+
+main "$@"

script/docker

@@ -0,0 +1,25 @@
+#!/bin/sh
+# script/docker: build the Docker image tagged with the project name.
+# The tag comes from script/projectname.
+#
+# CHECK_EPOCH is passed for the same reason script/cibuild passes it:
+# without it Docker serves the Dockerfile's gate layers from cache on an
+# unchanged tree and this exits 0 having run neither the lint stage's
+# gates nor the builder stage's test and fmt-check gates. This is the
+# set of gates a developer or reviewer runs by hand, so a cached pass
+# here is the most misleading result the repo can produce. Dependency
+# layers sit above the ARG and stay cached.
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    docker build \
+        --build-arg CHECK_EPOCH="$(date +%s)" \
+        -t "$("$SCRIPT_DIR/projectname")" \
+        .
+}
+
+main "$@"

script/fmt

@@ -0,0 +1,12 @@
+#!/bin/sh
+# script/fmt: format all files (writes).
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    gofmt -s -w .
+}
+
+main "$@"

script/fmt-check

@@ -0,0 +1,18 @@
+#!/bin/sh
+# script/fmt-check: check formatting (read-only). Same scope as
+# script/fmt, but fails instead of writing.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    files="$(gofmt -s -l .)"
+    if [ -n "$files" ]; then
+        echo "gofmt: files not formatted:" >&2
+        echo "$files" >&2
+        exit 1
+    fi
+}
+
+main "$@"

script/install-precommit

@@ -0,0 +1,20 @@
+#!/bin/sh
+# script/install-precommit: install the git pre-commit hook that runs
+# script/precommit. Our own extension to scripts-to-rule-them-all.
+# Hooks are shared between the main checkout and all worktrees, so
+# resolve the common git dir instead of assuming .git is a directory.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    hooks_dir="$(git rev-parse --git-common-dir)/hooks"
+    mkdir -p "$hooks_dir"
+    hook="$hooks_dir/pre-commit"
+    printf '#!/bin/sh\nset -e\nscript/precommit\n' > "$hook"
+    chmod +x "$hook"
+    echo "pre-commit hook installed: runs script/precommit"
+}
+
+main "$@"

script/lint

@@ -0,0 +1,29 @@
+#!/bin/sh
+# script/lint: run the linter. golangci-lint is never installed on a
+# host: it runs via docker only, one way, everywhere — this builds
+# Dockerfile.lint, which COPYs the repo into the digest-pinned
+# golangci-lint image and lints as a build step, so a successful build
+# is a clean lint. The only prerequisite is a working docker. The gate
+# steps make no network calls of their own, but Dockerfile.lint runs
+# `go mod download` above them, so a cold cache does reach the network
+# (as does pulling the pinned image); that layer stays cached, and once
+# it is warm this runs offline until go.mod or go.sum changes.
+#
+# CHECK_EPOCH is what makes the result mean anything. Without it docker
+# serves the gate layers from cache on an unchanged tree and this exits
+# 0 in well under a second having run no linter. The PID is in the value
+# as well as the epoch because two lint runs land inside the same second
+# easily, and `date +%s` alone would cache the second one.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    docker build \
+        --build-arg CHECK_EPOCH="$(date +%s)-$$" \
+        -f Dockerfile.lint \
+        .
+}
+
+main "$@"

script/precommit

@@ -0,0 +1,21 @@
+#!/bin/sh
+# script/precommit: run by the git pre-commit hook; fails the commit if
+# checks fail. Our own extension to scripts-to-rule-them-all. Go extra:
+# go mod tidy must be a no-op before the checks run.
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    go mod tidy
+    if ! git diff --exit-code -- go.mod go.sum; then
+        echo "precommit: go mod tidy changed go.mod/go.sum;" \
+            "stage the changes and retry" >&2
+        exit 1
+    fi
+    "$SCRIPT_DIR/check"
+}
+
+main "$@"

script/projectname

@@ -0,0 +1,12 @@
+#!/bin/sh
+# script/projectname: output the name of this project. Our own
+# extension to scripts-to-rule-them-all. Other scripts that need the
+# name (e.g. script/docker) call this, so they can stay identical
+# across all repos.
+set -eu
+
+main() {
+    echo "sfdupes"
+}
+
+main "$@"

script/setup

@@ -0,0 +1,13 @@
+#!/bin/sh
+# script/setup: set up the repo for development after a fresh clone:
+# installs dependencies (script/bootstrap) and the git pre-commit hook.
+set -eu
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+
+main() {
+    "$SCRIPT_DIR/bootstrap"
+    "$SCRIPT_DIR/install-precommit"
+}
+
+main "$@"

script/test

@@ -0,0 +1,17 @@
+#!/bin/sh
+# script/test: run the test suite. Reruns verbosely on failure so CI
+# logs show which test failed.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+main() {
+    cd "$ROOT"
+    go test -timeout 30s -cover ./... || {
+        echo "--- Rerunning with -v for details ---"
+        go test -timeout 30s -v ./...
+        exit 1
+    }
+}
+
+main "$@"

script/verify-lint-image-pin

@@ -0,0 +1,84 @@
+#!/bin/sh
+# script/verify-lint-image-pin: fail unless the golangci-lint image
+# referenced by Dockerfile.lint and the one referenced by the main
+# Dockerfile's lint stage are the same image at the same digest. Our own
+# extension to scripts-to-rule-them-all, not one of its entrypoints.
+#
+# The linter version is pinned in two independent files. That is the
+# shape #42 turned into a build failure rather than tolerate: nothing
+# else keeps the two in sync, and a bump applied to one file alone would
+# leave `make lint` and the fail-fast lint stage of `make docker`
+# linting the same tree against different rulesets, both green. This is
+# the single guard that stops it, run as a gate in both files.
+#
+# It deliberately restates neither pin. A hardcoded expected digest here
+# would be a third copy — one more thing to bump, and the same drift one
+# file further out. It compares the two files to each other and knows
+# nothing about which version is correct.
+#
+# A reference that cannot be read is a hard failure, not a skip: a
+# comparison of two empty strings succeeds, which would turn this guard
+# into exactly the unearned green it exists to prevent.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+LINT_DOCKERFILE="Dockerfile.lint"
+MAIN_DOCKERFILE="Dockerfile"
+
+# Echo the single golangci-lint image reference in the named Dockerfile.
+# Scans every argument of every FROM instruction rather than assuming a
+# field position, so `FROM --platform=... img AS stage` reads correctly.
+# Exits non-zero, with a diagnosis, unless there is exactly one.
+lint_image_ref() {
+    file="$1"
+
+    if [ ! -f "$file" ]; then
+        echo "verify-lint-image-pin: $file: not found" >&2
+        return 1
+    fi
+
+    refs="$(
+        awk '
+            toupper($1) == "FROM" {
+                for (i = 2; i <= NF; i++) {
+                    if ($i ~ /^golangci\/golangci-lint[:@]/) {
+                        print $i
+                    }
+                }
+            }
+        ' "$file"
+    )"
+
+    count="$(printf '%s' "$refs" | grep -c . || true)"
+    if [ "$count" -ne 1 ]; then
+        echo "verify-lint-image-pin: $file: expected exactly one" \
+            "golangci/golangci-lint FROM reference, found $count" >&2
+        return 1
+    fi
+
+    printf '%s\n' "$refs"
+}
+
+main() {
+    cd "$ROOT"
+
+    lint_ref="$(lint_image_ref "$LINT_DOCKERFILE")"
+    main_ref="$(lint_image_ref "$MAIN_DOCKERFILE")"
+
+    if [ "$lint_ref" != "$main_ref" ]; then
+        echo "verify-lint-image-pin: the linter image is pinned twice and" \
+            "the two pins disagree:" >&2
+        echo "verify-lint-image-pin:   $LINT_DOCKERFILE: $lint_ref" >&2
+        echo "verify-lint-image-pin:   $MAIN_DOCKERFILE: $main_ref" >&2
+        echo "verify-lint-image-pin: bump both FROM lines together, tag and" \
+            "digest, so script/lint and the Dockerfile lint stage keep" \
+            "running the same linter" >&2
+        exit 1
+    fi
+
+    echo "verify-lint-image-pin: $LINT_DOCKERFILE and $MAIN_DOCKERFILE" \
+        "agree on $lint_ref"
+}
+
+main "$@"

trees.go

@@ -2,6 +2,7 @@ package main
 
 import (
 	"bufio"
+	"context"
 	"crypto/sha256"
 	"fmt"
 	"os"
@@ -28,19 +29,17 @@ type treeNode struct {
 	totalSize int64
 }
 
-// runTrees implements the trees subcommand: it reads a scan stream from
-// the named file (or stdin when absent or "-"), reconstructs the
-// directory hierarchy from the record paths, computes a Merkle-style
-// digest per directory, and prints maximal duplicate-tree groups as TSV
-// on stdout. It never touches the scanned filesystem; its only I/O is
-// the scan input, stdout, and stderr. args holds the positional
-// arguments already validated by cobra (at most one).
-func runTrees(args []string) {
-	in, name, closer := openScanInput(args)
-	defer closer()
-
-	recs, malformed := parseScanStream(in, name)
-	records := len(recs) + malformed
+// runTrees implements the trees subcommand: it reads every record from
+// the database, reconstructs the directory hierarchy from the record
+// paths, computes a Merkle-style digest per directory, and prints
+// maximal duplicate-tree groups as TSV on stdout. It never touches the
+// scanned filesystem; its only I/O is the database, stdout, and
+// stderr.
+func runTrees(ctx context.Context) error {
+	recs, err := loadRecords(ctx)
+	if err != nil {
+		return err
+	}
 
 	super, allDirs := buildHierarchy(recs)
 	super.compute()
@@ -49,9 +48,9 @@ func runTrees(args []string) {
 
 	out := bufio.NewWriterSize(os.Stdout, ioBufSize)
 
-	_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
+	_, err = fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
 	if err != nil {
-		fatalf("write stdout: %v", err)
+		return fmt.Errorf("write stdout: %w", err)
 	}
 
 	dupeTrees := 0
@@ -64,7 +63,7 @@ func runTrees(args []string) {
 			_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\n",
 				first.path, n.path, first.fileCount, first.totalSize)
 			if err != nil {
-				fatalf("write stdout: %v", err)
+				return fmt.Errorf("write stdout: %w", err)
 			}
 
 			dupeTrees++
@@ -74,13 +73,15 @@ func runTrees(args []string) {
 
 	err = out.Flush()
 	if err != nil {
-		fatalf("write stdout: %v", err)
+		return fmt.Errorf("write stdout: %w", err)
 	}
 
 	fmt.Fprintf(os.Stderr,
-		"trees: %d records read%s, %d duplicate tree groups, %d dupe trees, %s reclaimable\n",
-		records, malformedNote(malformed), len(dupes), dupeTrees,
-		humanBytes(reclaimable))
+		"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
+			"%s reclaimable\n",
+		len(recs), len(dupes), dupeTrees, humanBytes(reclaimable))
+
+	return nil
 }
 
 // buildHierarchy reconstructs the directory hierarchy from the record
@@ -121,9 +122,17 @@ func buildHierarchy(recs []scanRec) (*treeNode, []*treeNode) {
 			node.files = make(map[string]fileSig)
 		}
 
-		node.files[comps[len(comps)-1]] = fileSig{
-			size: r.size, head: r.head, tail: r.tail,
+		sig := fileSig{size: r.size, head: r.head, tail: r.tail}
+
+		// An unhashed record (its size was unique when last scanned)
+		// has unknown content: give it a signature no other file can
+		// share, so trees containing it never compare equal. Real
+		// heads are hex, so the NUL-prefixed form cannot collide.
+		if sig.head == "" {
+			sig.head = "unhashed\x00" + r.path
 		}
+
+		node.files[comps[len(comps)-1]] = sig
 	}
 
 	return super, allDirs