Run all linting in Docker via Dockerfile.lint (closes #46)

Per the owner ruling, the linter runs inside a container invoked
through the script/ entrypoint and is never installed on a host. A new
root Dockerfile.lint COPYs the repo into the digest-pinned
golangci/golangci-lint:v2.12.2 image and runs `golangci-lint config
verify` and `golangci-lint run` as build steps, so a successful build
IS a clean lint. script/lint is reduced to building it, which also
works when the docker daemon is remote and bind mounts are impossible.

script/bootstrap loses the `go install`, the pin constants, the
version parser and verify_golangci_lint outright rather than being
hardened: with nothing linting on the host, the $GOPATH/bin versus
PATH shadowing problem those existed to diagnose has no subject. It
keeps the git/make/go presence checks and `go mod download`, and warns
rather than fails when docker is absent.

Two traps.

A lint build on an unchanged tree returns success in well under a
second having run no linter, which is #32 and #39 over again. Caching
is waived by ruling, so Dockerfile.lint carries ARG CHECK_EPOCH
referenced inside every gate RUN -- BuildKit hashes the expanded
command, not the declaration, so a declared but unreferenced ARG
invalidates nothing -- and script/lint passes "$(date +%s)-$$". The
PID is in that value because two lint runs land inside the same second
easily and a bare epoch would cache the second one.

Nothing inside an image build may shell out to docker. The main
Dockerfile's lint stage therefore invokes golangci-lint directly
instead of `make lint`, and its build stage runs `make test` and
`make fmt-check` instead of the `make check` aggregate, which reaches
script/lint. Those two remain `make` invocations rather than the bare
scripts because the Makefile's `export CGO_ENABLED = 0` only applies
to what it invokes, and today's `make check` gets it.

COPY --from=lint /usr/bin/golangci-lint is replaced by
COPY --from=lint /src/go.sum /dev/null. The copied binary was the only
edge forcing BuildKit to finish linting before the build stage starts;
dropping it without replacing the edge would have ended fail-fast
linting silently under a still-green build. That no-op copy is the
ordering edge canonical REPO_POLICIES.md prescribes. Nothing in the
build stage runs the linter any more, so the binary itself is not
wanted there, and ENV PATH=/home/builder/go/bin:$PATH goes with the
`go install` that justified it.

script/verify-linter-pin is retired -- deleted along with its README
entry -- because both of its subjects ceased to exist in this same
change: it compared a linter binary against GOLANGCI_LINT_VERSION in
script/bootstrap, and there is now neither a binary crossing between
stages nor a version pin in bootstrap. The drift it guarded has not
gone away, it has moved. The linter is still pinned twice, now as the
FROM line of Dockerfile.lint and the FROM line of the Dockerfile lint
stage, with nothing syncing them, which is exactly what #42 made a
build failure. Its replacement is one new script/verify-lint-image-pin
that compares those two references to each other and deliberately
restates neither pin: a hardcoded expected digest would be a third
copy and the same drift one file further out. It runs as a gate in
both files, so `make lint`, `make check` and `make docker` all catch
drift, and an unreadable reference is a hard failure rather than a
vacuous pass.

`golangci-lint config verify` is included per the ruling. The concern
about its unpinned live HTTPS schema fetch was measured rather than
assumed: under --network none the pinned binary both passes a valid
config and rejects an invalid one with the jsonschema error, so it
validates against a schema it embeds and linting needs no network
beyond pulling the image. The README states that rather than a
requirement that does not exist.

Verified. `make lint` green with every PATH directory containing a
golangci-lint removed and `command -v golangci-lint` empty. Two
consecutive script/lint runs on an untouched tree both executed the
linter, 27.7s and 28.7s in the lint step under distinct epochs with
the COPY layer CACHED above them. A planted unused variable failed
script/lint with that exact finding and failed `make docker` at the
lint stage with the build stage stopped before its COPY --from=lint,
then reverted clean. The drift guard fails on tag-only, digest-only
and unreadable-reference cases, naming both sides. `make check` green;
`make docker` green in 5m35s with all six gates executing and the test
gate reporting real coverage rather than a cached ok. In the builder
image with the Go test cache off, --user 0:0 still fails
TestScanHardlinkRunFailsTogether where the unprivileged user passes,
so the non-root quirk is intact.

.claude/settings.json

@@ -0,0 +1,5 @@
+{
+    "worktree": {
+        "bgIsolation": "none"
+    }
+}

.dockerignore

@@ -1,79 +1,8 @@
-# .dockerignore does NOT use .gitignore semantics. Docker matches with
+.git
-# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
-# `/` and an unprefixed pattern is anchored at the context root. Every
-# depth-independent pattern therefore needs `**/`, or `config/.env` and
-# `certs/server.key` still ship while this file reads as solved. Only
-# genuinely root-anchored entries go unprefixed. Never transplant these
-# into .gitignore, where `**/` is wrong.
-#
-# Matching is case-sensitive, so secrets use character ranges rather
-# than an ALL-CAPS twin, which would still miss `Server.Key`.
-#
-# Extend with this repo's own host-built artifacts, written anchored:
-# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
-# deletes the package directory from the context.
-
-# .git is sent without its config. Without a VERSION build argument the
-# stage that compiles runs `git describe --tags --always` on .git, which
-# does not need .git/config; that file can hold a credential, such as a
-# password in a remote URL or the token the CI checkout step stores there.
-# Each submodule keeps a config with the same exposure in its git directory
-# under .git/modules/, nested again for a submodule's own submodules, or in
-# its own .git directory when it keeps one.
-# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
-# `deploy/config`, `config/lib`) loses its whole git directory, because
-# `**/.git/modules/**/config` also matches that segment's directory
-# under .git/modules/. Go's version stamping then fails the build;
-# nothing leaks. Name such a submodule without that segment:
-# `git submodule add --name`.
-**/.git/config
-**/.git/modules/**/config
-
-# Agent scratch: one full checkout of the repo per in-flight agent.
-# Anchored because it occurs once where agents run at the repo root.
-# KNOWN GAP: a repo running agents in subdirectories still ships
-# `services/api/.claude/` and must add its own anchored entry.
 .claude
-
+.DS_Store
-# Environment files. `*.env` covers bare `.env` and the `prod.env`
+sfdupes
-# convention. Re-include a committed template with a negation if the
+files.dat
-# build needs one: `!docs/example.env`.
+*.log
-**/*.[eE][nN][vV]
+*.out
-**/.[eE][nN][vV].*
+*.test
-**/.[eE][nN][vV][rR][cC]
-
-# Private keys and the bundles carrying them. Public certificates
-# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
-**/*.[pP][eE][mM]
-**/*.[kK][eE][yY]
-**/*.[pP]12
-**/*.[pP][fF][xX]
-**/[iI][dD]_[rR][sS][aA]
-**/[iI][dD]_[dD][sS][aA]
-**/[iI][dD]_[eE][cC][dD][sS][aA]
-**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
-**/[iI][dD]_[eE][dD]25519
-**/[iI][dD]_[eE][dD]25519_[sS][kK]
-
-# Dependencies: restored inside the image, never copied in.
-**/node_modules
-
-# OS metadata.
-**/.DS_Store
-**/Thumbs.db
-
-# Editor state: never a build input, and it churns COPY.
-**/*.swp
-**/*.swo
-**/*~
-**/*.bak
-**/.idea
-**/.vscode
-**/*.sublime-*
-
-# This repository's host-built artifacts: the binary `make build` writes,
-# and test binaries, coverage output and logs.
-/sfdupes
-/*.test
-/*.out
-/*.log

.editorconfig

@@ -13,6 +13,3 @@ indent_style = tab
 
 [*.go]
 indent_style = tab
-
-# This repository's own sections, such as one for another language it
-# uses, go below this comment, and a re-vendor keeps them.

.gitea/workflows/check.yml

@@ -1,20 +1,9 @@
 name: check
 on: [push]
-# Free the shared runner: a new push cancels only the same branch's older run.
-concurrency:
-    group: ${{ github.workflow }}-${{ github.ref }}
-    cancel-in-progress: true
 jobs:
     check:
         runs-on: ubuntu-latest
-        # Free the shared runner from a hung build.
-        timeout-minutes: 20
         steps:
             # actions/checkout v4.2.2, 2026-02-22
             - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683
-              # script/cibuild needs no token, so none is left in .git/config.
-              with:
-                  persist-credentials: false
-                  # All history and tags, so git describe finds the version tag.
-                  fetch-depth: 0
             - run: script/cibuild

.gitignore

@@ -11,50 +11,26 @@ Thumbs.db
 .vscode/
 *.sublime-*
 
-# Agent scratch (worktrees of this repo, created and destroyed by
-# in-flight tooling). Unanchored: .gitignore patterns already match at
-# every depth, so no prefix is wanted here. This is not a .dockerignore
-# entry and must not be given a `**/` prefix on the way into one.
-.claude/
-
 # Node
 node_modules/
 
-# Secrets. Unanchored like every entry above, so each matches at every
+# Environment / secrets
-# depth. Matching is case-sensitive on Linux, so names use character
+.env
-# ranges rather than a lowercase form that misses `Server.Key`.
+.env.*
+*.pem
+*.key
 
-# Environment files. `*.env` covers bare `.env` and the `prod.env`
+# Go build artifacts
-# convention. Only the templates `example.env` and `sample.env` are
-# re-included below. A repository that commits any other template adds
-# its own negation at the end of this file, for example `!.env.example`.
-*.[eE][nN][vV]
-.[eE][nN][vV].*
-.[eE][nN][vV][rR][cC]
-!example.env
-!sample.env
-
-# Private keys and the bundles carrying them.
-*.[pP][eE][mM]
-*.[kK][eE][yY]
-*.[pP]12
-*.[pP][fF][xX]
-[iI][dD]_[rR][sS][aA]
-[iI][dD]_[dD][sS][aA]
-[iI][dD]_[eE][cC][dD][sS][aA]
-[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
-[iI][dD]_[eE][dD]25519
-[iI][dD]_[eE][dD]25519_[sS][kK]
-
-# This repository's own entries, such as its build outputs, go below
-# this comment, and a re-vendor keeps them. Anchor a binary built at the
-# root: `/myapp`, never `myapp`, which also ignores `cmd/myapp/`.
 /sfdupes
-*.log
-*.out
 *.test
+*.out
+*.log
 
-# A scan database lists every path it scanned.
+# Local scan data
+files.dat
 *.sqlite
 *.sqlite-shm
 *.sqlite-wal
+
+# Agent worktrees
+.claude/worktrees/

.golangci.yml

@@ -10,23 +10,14 @@ run:
 
 linters:
   default: all
-  enable:
-    # Successor to the deprecated gomodguard. Named explicitly, rather than
-    # left to `default: all`, because it carries the module policy below.
-    - gomodguard_v2
   disable:
     # Genuinely incompatible with project patterns
     - exhaustruct      # Requires all struct fields
-    - exhaustruct_v5   # Requires all struct fields (successor to exhaustruct)
+    - depguard         # Dependency allow/block lists
     - godot            # Requires comments to end with periods
+    - wsl              # Deprecated, replaced by wsl_v5
     - wrapcheck        # Too verbose for internal packages
     - varnamelen       # Short names like db, id are idiomatic Go
-    # Deprecated: the warning is attached to the old name, so it is
-    # silenced by disabling that name, not by enabling the successor.
-    - wsl              # Deprecated, replaced by wsl_v5
-    - gomodguard       # Deprecated, replaced by gomodguard_v2
-    # Misses findings at random in v2.14.0; back once a pinned release fixes it
-    - canonicalheader
   settings:
     lll:
       line-length: 88
@@ -37,64 +28,6 @@ linters:
       max-complexity: 15
     dupl:
       threshold: 100
-    depguard:
-      # Test-support code must not be compiled into the shipped binary. A
-      # test-support package exists to hand a test privileges the program
-      # itself must never have, so a file that is not a test must not import
-      # one. Test files, and the files inside a package whose directory name
-      # ends in `test`, are where that code belongs, and are exempt.
-      #
-      # The deny list below is the one part of this file a repository is
-      # expected to extend, and the only part it may. depguard matches an
-      # import path against a list of prefixes, so it cannot be told "any path
-      # whose last segment ends in test"; a repository's own test-support
-      # packages have to be named here one at a time, by full import path,
-      # under a module path that differs from repository to repository. Add
-      # them; change nothing else.
-      rules:
-        test-support:
-          list-mode: lax
-          files:
-            - "$all"
-            - "!$test"
-            - "!**/*test/**"
-          deny:
-            - pkg: net/http/httptest
-              desc: >-
-                Test-support code belongs in test files and in packages whose
-                directory name ends in test, not in the shipped binary.
-    # Only decisions already recorded in the Go package defaults are
-    # listed here. Every entry matches the module path exactly.
-    gomodguard_v2:
-      blocked:
-        - module: github.com/rs/zerolog
-          recommendations:
-            - log/slog
-          reason: "Structured logging is stdlib log/slog."
-        # One entry per pre-fork module path, because the later releases
-        # are separate paths. A prefix match would be shorter but would
-        # also reach github.com/go-redis/redismock, the test double for
-        # the successor these entries recommend.
-        - module: github.com/go-redis/redis
-          recommendations:
-            - github.com/redis/go-redis/v9
-          reason: "Pre-fork module; use the maintained go-redis v9."
-        - module: github.com/go-redis/redis/v7
-          recommendations:
-            - github.com/redis/go-redis/v9
-          reason: "Pre-fork module; use the maintained go-redis v9."
-        - module: github.com/go-redis/redis/v8
-          recommendations:
-            - github.com/redis/go-redis/v9
-          reason: "Pre-fork module; use the maintained go-redis v9."
-        - module: github.com/sergi/go-diff
-          recommendations:
-            - github.com/aymanbagabas/go-udiff
-          reason: "No unified diff output; use go-udiff."
-        - module: github.com/hexops/gotextdiff
-          recommendations:
-            - github.com/aymanbagabas/go-udiff
-          reason: "Unmaintained fork; use go-udiff."
 
 issues:
   max-issues-per-linter: 0

.prettierignore

@@ -1,2 +0,0 @@
-node_modules/
-yarn.lock

.prettierrc

@@ -1,4 +0,0 @@
-{
-    "tabWidth": 4,
-    "proseWrap": "always"
-}

Dockerfile

@@ -1,92 +1,118 @@
-# Every stage declares ARG CACHEBUST first: script/ passes a new value
+# Lint stage — fast feedback on formatting and lint issues
-# on every build, which reruns every RUN step after it. The steps that
+# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
-# compile mount Go's build cache, which survives from build to build.
+FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
-
-# Lint phase, built alone by script/lint. The tools are invoked directly
-# rather than through `make lint`, which runs docker itself and so cannot
-# run inside a build step.
-# golangci/golangci-lint:v2.14.0, 2026-10-07
-FROM golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f AS lint
-ARG CACHEBUST
 WORKDIR /src
 COPY go.mod go.sum ./
 RUN go mod download
 COPY . .
 
-# The gofmt half of `make fmt-check`. gofmt's output is assigned to a
+# Cache-buster for the gate layers, and only for them. Docker
-# variable first so that its own exit status, as when it cannot parse a
+# invalidates COPY only when the copied content changes, so on an
-# file, still fails the step.
+# unchanged tree the gates below would be served from cache and the
-RUN files="$(gofmt -s -l .)" && \
+# build would exit 0 having run nothing. script/cibuild and
-    if [ -n "$files" ]; then \
+# script/docker pass a fresh CHECK_EPOCH on every invocation.
-        echo "gofmt: files not formatted:" >&2; echo "$files" >&2; exit 1; \
+#
-    fi
+# Two properties this depends on. ARG is per-stage, so the build stage
+# below declares it again; one declaration here would leave that
+# stage's gate cacheable. And each gate RUN must reference the value,
+# because BuildKit hashes the expanded command: a declared but
+# unreferenced ARG invalidates nothing.
+#
+# It sits below the dependency layers deliberately. Everything above it
+# (the pinned base image, go mod download) keeps its cache; only the
+# gates go cold.
+ARG CHECK_EPOCH
 
-# Validates .golangci.yml against the schema the pinned binary embeds.
+# The linter is invoked directly here, not through `make lint`. That
-RUN golangci-lint config verify --config .golangci.yml
+# target now runs `docker build -f Dockerfile.lint`, and a docker build
-RUN --mount=type=cache,target=/root/.cache/go-build \
+# cannot run a docker build: routing the gate through make would mean
+# nesting docker inside this image. Same reason `make check` is gone
+# from the build stage below. `make fmt-check` stays as it is — it is a
+# gate, not the aggregate, and it shells out to nothing.
+RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
+
+# The FROM above and the one in Dockerfile.lint pin the same linter
+# twice, and nothing else keeps them in sync; this fails the build when
+# they disagree. See the script for why it restates neither pin.
+RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
+    script/verify-lint-image-pin
+
+# Same config-schema check Dockerfile.lint runs, kept here so this build
+# gates on exactly what script/lint gates on. It validates against a
+# schema the pinned binary embeds, so it needs no network.
+RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
+    golangci-lint config verify --config .golangci.yml
+
+RUN echo "gate lint, epoch ${CHECK_EPOCH}" && \
     golangci-lint run --config .golangci.yml ./...
 
-# Test phase, built alone by script/test. -race needs cgo and so a C
+# Build stage
-# compiler, which the Debian Go image ships and the alpine one does not.
-# -count=1 because the mounted cache also holds test results, which go
-# test would otherwise replay.
-#
-# The tests run as nobody: several of them make a file unreadable and
-# expect reading it to fail, and root reads it anyway. nobody has no home
-# directory, so HOME is /tmp, and Go keeps its build cache in
-# /tmp/.cache/go-build. That is where the cache is mounted, owned by
-# nobody (65534) so the tests can write to it.
-# golang:1.25-trixie, 2026-10-04
-FROM golang@sha256:2c4c60ef415fbfa5e90300722293bef36c5e63fae17570ce18f580af933dbd73 AS test
-ARG CACHEBUST
-USER nobody
-ENV HOME=/tmp
-WORKDIR /src
-COPY go.mod go.sum ./
-RUN go mod download
-COPY . .
-RUN --mount=type=cache,target=/tmp/.cache/go-build,uid=65534,gid=65534 \
-    go test -count=1 -timeout 90s -race -cover ./... || \
-    { echo "--- Rerunning with -v for details ---"; \
-      go test -count=1 -timeout 90s -race -v ./...; exit 1; }
-
-# Build stage. Nothing is wanted from either phase above; the copies are
-# what make BuildKit build them first, so this stage cannot run unless
-# lint and test passed.
 # golang:1.25-alpine, 2026-07-23
 FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder
-ARG CACHEBUST
+
-COPY --from=lint /src/go.sum /dev/null
+# We never build or run as root. Create an unprivileged user and point
-COPY --from=test /src/go.sum /dev/null
+# HOME and the Go caches at its home so go build and go test can write
-RUN apk add --no-cache git make
+# their caches when we drop to it below. $GOPATH/bin is deliberately not
-# A tar-stream context keeps the sender's file owners, which git refuses.
+# on PATH: script/bootstrap no longer `go install`s anything (the linter
-RUN git config --system --add safe.directory /src
+# runs from a pinned image, never from a host install), so nothing lands
+# there and adding it would only widen what this image resolves.
+RUN adduser -D -u 1000 builder
+ENV HOME=/home/builder
+ENV GOPATH=/home/builder/go
+ENV GOCACHE=/home/builder/.cache/go-build
+
 WORKDIR /src
+
+# No-op file copy whose only purpose is the build-graph edge: it is what
+# makes this stage depend on the lint stage, and so what forces BuildKit
+# to finish fmt-check, the pin guard and lint before compilation and
+# tests start. Remove it and the fail-fast design dies silently — the
+# build stops gating on lint and still exits 0. It replaces a copy of
+# the linter binary itself, which is no longer wanted here: nothing in
+# this stage runs the linter, because `make lint` is now a docker build
+# and a docker build cannot run inside one.
+COPY --from=lint /src/go.sum /dev/null
+
+# Install development prerequisites the same way a developer does,
+# rather than duplicating the installs inline. Only script/ and the
+# dependency manifests are copied first, nothing else, so this layer
+# stays cached until the scripts or the dependencies change — bootstrap
+# ends in `go mod download`, which is why there is no separate
+# invocation of it here.
+COPY script/ script/
 COPY go.mod go.sum ./
-RUN go mod download
+RUN script/bootstrap
+
 COPY . .
 
-# The version stamped into the binary: the VERSION build argument when
+# Hand the sources and caches to the unprivileged user, then drop root
-# one is given, otherwise `git describe --tags --always` of the .git in
+# before running any checks or builds.
-# the build context. A context that carries .git and still yields no
+RUN chown -R builder:builder /src /home/builder
-# version fails the build; with neither, as from a source tarball, it is
+USER builder
-# "dev". `make build` rather than `go build`, so the image and a host
-# build share one compile recipe, cgo disabled included.
-ARG VERSION
-RUN --mount=type=cache,target=/root/.cache/go-build \
-    version="${VERSION:-$(git describe --tags --always || echo dev)}"; \
-    if [ -e .git ] && { [ -z "$version" ] || [ "$version" = dev ] || \
-        [ "$version" = unknown ]; }; then \
-        echo "no version could be derived although the build context carries .git" >&2; \
-        exit 1; \
-    fi; \
-    make build VERSION="$version"
 
-# Runtime stage, and the last one: a plain `docker build .` builds this
+# Fail the build unless the branch is green. Runs as non-root so the
-# stage's chain and nothing else.
+# permission-denied test paths are exercised legitimately (root would
+# bypass the chmod(0) the tests rely on).
+#
+# The gates are the individual targets, not `make check`: that aggregate
+# runs `script/lint`, which is now a docker build, and nothing inside an
+# image build may shell out to docker. Lint is not skipped by this — it
+# ran in the lint stage above, which this stage's COPY --from makes a
+# prerequisite. `make`, not the scripts directly, because the Makefile's
+# `export CGO_ENABLED = 0` applies only to what it invokes.
+#
+# Second per-stage declaration of the gate cache-buster; see the lint
+# stage above for why one is not enough. It is placed after USER so the
+# drop to the unprivileged user still happens before the checks run.
+ARG CHECK_EPOCH
+RUN echo "gate test, epoch ${CHECK_EPOCH}" && make test
+RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
+
+RUN make build
+
+# Runtime stage
 # alpine:3.22, 2026-07-23
 FROM alpine@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce
-ARG CACHEBUST
 
 COPY --from=builder /src/sfdupes /usr/local/bin/sfdupes
 

Dockerfile.lint

@@ -0,0 +1,55 @@
+# Lint-only image: this is how the linter runs, everywhere. The repo is
+# COPYed into the pinned golangci-lint image and the linter runs as a
+# build step, so a successful build IS a clean lint. golangci-lint is
+# never installed on a host — one toolchain, pinned by digest, identical
+# on a laptop and in CI — and this works even when the docker daemon is
+# remote and bind mounts are impossible.
+#
+# script/lint builds this file. It is a separate image from the lint
+# stage of the main Dockerfile because script/lint must not depend on
+# the rest of that build; the two FROM lines are kept identical by
+# script/verify-lint-image-pin, run as a gate below.
+# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07
+FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240
+
+WORKDIR /src
+
+# Dependency layers first, so they stay cached across lint runs.
+COPY go.mod go.sum ./
+RUN go mod download
+
+COPY . .
+
+# Cache-buster for the gate layers, and only for them. Caching of the
+# lint run is waived by ruling: COPY is invalidated only by changed
+# content, so on an unchanged tree the gates below would be served from
+# cache and this build would exit 0 in under a second having run no
+# linter at all. That exact false green has bitten this repo twice
+# already (#32, #39). script/lint passes a fresh value on every
+# invocation.
+#
+# Each gate RUN must reference the value, because BuildKit hashes the
+# expanded command and not the ARG declaration: a declared but
+# unreferenced ARG invalidates nothing. The ARG sits below the
+# dependency layers deliberately — everything above it keeps its cache,
+# only the gates go cold.
+ARG CHECK_EPOCH
+
+# The linter version is pinned in two places, here and in the main
+# Dockerfile's lint stage. Nothing else keeps them in sync, so a
+# half-applied bump is a build failure; see the script.
+RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
+    script/verify-lint-image-pin
+
+# Validates .golangci.yml against golangci-lint's JSON schema. The
+# concern about this step was that it fetches that schema over a live,
+# unpinned HTTPS call; measured on the pinned image, it does not. The
+# binary carries the schema for its own version, so under
+# `--network none` this both passes on a valid config and still rejects
+# an invalid one with the jsonschema error. Linting therefore needs no
+# network beyond pulling the pinned image.
+RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
+    golangci-lint config verify --config .golangci.yml
+
+RUN echo "gate lint, epoch ${CHECK_EPOCH}" && \
+    golangci-lint run --config .golangci.yml ./...

Makefile

@@ -46,4 +46,4 @@ hooks:
 	@script/install-precommit
 
 clean:
-	rm -f $(BINARY)
+	rm -f $(BINARY) files.dat

REPO_POLICIES.md

@@ -1,6 +1,6 @@
 ---
 title: Repository Policies
-last_modified: 2026-10-15
+last_modified: 2026-07-06
 ---
 
 This document covers repository structure, tooling, and workflow standards. Code
@@ -54,37 +54,21 @@ style conventions are in separate documents:
   `cibuild`. `script/bootstrap` installs all dependencies idempotently and
   assumes nothing is present: base tools come from nix, apt, brew, or apk
   (detected in that order; apt runs noninteractive). For node it uses the
-  installed node only when its major version is the pinned one; otherwise (node
+  installed node if present; otherwise it installs a PINNED node version via
-  missing, or another major, such as the node 24 the CI runner image ships) it
+  nvm, first installing nvm itself if missing — from a hash-verified GitHub
-  installs the PINNED node version via nvm and installs yarn and the packages
+  release archive (never `curl | sh`), with bash installed as an explicit
-  under it. nvm itself is installed first if missing, from a hash-verified
+  prerequisite since nvm requires bash. yarn is then pinned via
-  GitHub release archive (never `curl | sh`), with bash installed as an explicit
+  `corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
-  prerequisite since nvm requires bash. Only the major version is compared
+  always exact versions. `script/cibuild` runs the CI build: it changes to the
-  because the `Dockerfile` stages start from a node image whose exact version is
+  repo root and runs `docker build .`; the Gitea workflow calls it. Four further
-  not the pin, and nvm cannot install a prebuilt node on alpine. yarn is pinned
+  scripts are our own extensions to the standard: `script/check` runs
-  via `corepack prepare yarn@<version> --activate`. Never install "latest" or
+  `script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
-  "lts"; always exact versions. `script/cibuild` runs the CI build: it changes
+  what the git pre-commit hook runs, and it calls `script/check`;
-  to the repo root, runs `script/bootstrap`, runs `script/check`, and builds the
+  `script/install-precommit` installs the git pre-commit hook (the `make hooks`
-  image with the version; the Gitea workflow calls it. **`script/cibuild` runs
+  target shims to it); and `script/projectname` (literally that filename) simply
-  `script/bootstrap` first**, because the workflow checks out the repo and runs
+  outputs the project's name. Scripts that need the name call
-  nothing else, while `script/fmt-check` runs the formatter on the host: on a
+  `script/projectname` — e.g. `script/docker` assembles its image tag from it —
-  pristine checkout with nothing installed the run dies there, after the
+  so those scripts stay byte-identical across all repos. Repo-type-specific
-  containerised gates have passed. **The bootstrap alone is not enough**: when
-  `script/bootstrap` installs node and yarn under nvm it leaves neither on the
-  `PATH` of the shell that called it, so a bare `yarn` either exits 127 or runs
-  under another node. The host entrypoints that need yarn — `script/fmt` and
-  `script/fmt-check` — therefore use the same test as `script/bootstrap`: when
-  the node on `PATH` has the pinned major version they run the `yarn` on `PATH`,
-  and otherwise they load nvm through `$HOME/.nvm/nvm.sh` and run yarn under the
-  pinned node. A runner carrying nothing but docker and git then gets through
-  `script/check`. Four further scripts are our own extensions to the standard:
-  `script/check` runs `script/test`, `script/lint` and `script/fmt-check`;
-  `script/precommit` is what the git pre-commit hook runs, and it calls
-  `script/check`; `script/install-precommit` installs the git pre-commit hook
-  (the `make hooks` target shims to it); and `script/projectname` (literally
-  that filename) simply outputs the project's name. Scripts that need the name
-  call `script/projectname` — e.g. `script/docker` assembles its image tag from
-  it — so those scripts stay byte-identical across all repos. Repo-type-specific
   pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
   `script/precommit`, not in the hook itself. Model scripts are at
   `https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
@@ -105,257 +89,87 @@ style conventions are in separate documents:
   contributor should be able to understand the entire development workflow by
   reading the Makefile.
 
-- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
+- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
-  `lint` phase and a `test` phase, with the final stage depending on both so the
+  as a build step so the build fails if the branch is not green. For non-server
-  image cannot be built unless they pass. For non-server repos the final stage
+  repos, the Dockerfile should bring up a development environment and run
-  brings up a development environment; for server repos it is the runtime image.
+  `make check`. For server repos, `make check` should run as an early build
-  The gate phases and the build stage start from their pinned base images and
+  stage before the final image is assembled. Dockerfiles install development
-  install what those images lack either inline, as the canonical Go `Dockerfile`
+  prerequisites by running `script/bootstrap` rather than duplicating installs
-  below does for `git`, or by running `script/bootstrap`, as the `prompts`
+  inline; COPY `script/` and the dependency manifests (`package.json` +
-  repo's own `Dockerfile` does for its yarn packages. The development
+  `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
-  environment stage installs development prerequisites by running
+  layer stays cached until dependencies change.
-  `script/bootstrap` rather than duplicating its installs inline. A stage that
-  runs `script/bootstrap` COPYs `script/` and the dependency manifests
-  (`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
 
-- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
+- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
-  no separate lint file. `script/lint` and `script/test` each build one phase
+  repos use a multistage build where linting runs in an independent stage based
-  and nothing else:
+  on the `golangci/golangci-lint` image (pinned by hash). This stage runs
+  `make fmt-check` and `make lint` before the full build begins. The build stage
+  then declares an explicit dependency on the lint stage via
+  `COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
+  linting before proceeding to compilation and tests. This ensures lint failures
+  surface in seconds rather than minutes, without blocking on dependency
+  download or compilation in the build stage.
 
-    ```sh
+    The standard pattern for a Go repo Dockerfile is:
-    docker build --build-arg CACHEBUST="$(date +%s)-$$" \
-        --target lint --output type=cacheonly .
-    docker build --build-arg CACHEBUST="$(date +%s)-$$" \
-        --target test --output type=cacheonly .
-    ```
-
-    **A stage that is not the last one in the file is built only when the final
-    stage's chain depends on it, or when `--target` names it.** That is why the
-    two gates are always invoked by name here, and why the final stage carries a
-    `COPY --from=` of a harmless file from each of them: without that edge a
-    plain `docker build .` builds the last stage alone and exits 0 having linted
-    and tested nothing.
-
-    **The gate builds write no image.** With `--output type=cacheonly` the phase
-    runs and a failing step fails the build, but the result is not exported.
-    Nothing uses those images, and writing one out is slow: a Go test phase's
-    image holds the toolchain. A build given neither `--output` nor `-t` writes
-    an untagged image and leaves it dangling, on every developer host and every
-    CI runner. `script/cibuild` and `script/docker` build the image that ships
-    and tag it, so each build replaces the previous image; each assigns the tag
-    on its own line before the build, so `set -e` stops it where
-    `script/projectname` fails.
-
-    Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
-    `eslint`, `prettier` — never through `make lint` or `script/test`, which are
-    themselves a `docker build` and would recurse into a daemon that does not
-    exist in a build step. Formatting is the exception and stays on the host:
-    `script/fmt` writes the working tree, and `script/fmt-check` is its
-    read-only twin.
-
-    **No lint verdict may come from a host invocation of the linter.** On a
-    shared host golangci-lint reads a result cache keyed on file content rather
-    than location, so a second checkout of the same content is served the first
-    one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
-    exit non-zero with `parallel golangci-lint is running` — a status a caller
-    cannot tell from real findings. Both have produced wrong verdicts in this
-    org, in both directions. A container has its own golangci-lint cache, its
-    own `TMPDIR` and a digest-pinned binary, so neither is reachable.
-
-- **Any build that runs checks is given a new `CACHEBUST` value.** Docker
-  invalidates a `COPY` layer only when the copied content changes, so on an
-  unchanged tree the check `RUN` is served from cache, nothing executes, and the
-  build still exits 0. Every `docker build` in `script/` therefore passes
-  `--build-arg CACHEBUST="$(date +%s)-$$"`, the time and the script's process
-  id, so no two builds share a value, not even the three in one
-  `script/cibuild`. `script/lint`, `script/test`, `script/cibuild` and
-  `script/docker` are the four, and there is no fifth — `script/check` runs the
-  two gate phases and `script/fmt-check`, and builds no image of its own. Every
-  stage of the `Dockerfile` declares `ARG CACHEBUST` as its first line after
-  `FROM`, and a new value reruns every `RUN` step after that line; `COPY` steps
-  are compared by content and need nothing. A stage without the `ARG` line is
-  served from cache on an unchanged tree, so a repository re-vendors the scripts
-  and adds the `ARG` lines in one commit. The scripts do not pass `--no-cache`:
-  it reruns every step too, but it also empties every cache mount, and Go's
-  build cache, which the canonical Go `Dockerfile` below mounts so that compiled
-  packages survive between check builds, would start empty every time. A bare
-  `docker build .` is not evidence that anything ran: a sub-second build
-  reporting success is a cache hit, not a result. Never invalidate by pruning —
-  `docker builder prune` and friends destroy a build cache shared with every
-  other build on the host. When a check is added or changed, prove it works by
-  planting a defect it must catch and watching the run fail on it, then revert
-  the defect. A green run alone shows neither that the check ran nor that it
-  covers what it should.
-
-- **The gate phases are separate stages, and the build stage depends on both.**
-  The lint phase is based on the `golangci/golangci-lint` image (pinned by
-  hash), so lint failures surface in seconds rather than after a full compile,
-  and the test phase is based on the Debian Go image. The canonical Go repo
-  `Dockerfile`:
 
     ```dockerfile
-    # Every stage declares ARG CACHEBUST first: script/ passes a new value
+    # Lint stage — fast feedback on formatting and lint issues
-    # on every build, which reruns every RUN step after it. The steps that
-    # compile mount Go's build cache, which survives from build to build.
-
-    # Lint phase
     # golangci/golangci-lint:v2.x.x, YYYY-MM-DD
     FROM golangci/golangci-lint@sha256:... AS lint
-    ARG CACHEBUST
     WORKDIR /src
     COPY go.mod go.sum ./
     RUN go mod download
     COPY . .
-    RUN --mount=type=cache,target=/root/.cache/go-build \
+    RUN make fmt-check
-        golangci-lint run --config .golangci.yml ./...
+    RUN make lint
 
-    # Test phase. -race needs cgo and so a C compiler, which the Debian Go
+    # Build stage
-    # image ships and the alpine one does not. -count=1 because the mounted
-    # cache also holds test results, which go test would otherwise replay.
-    # golang:1.x, YYYY-MM-DD
-    FROM golang@sha256:... AS test
-    ARG CACHEBUST
-    WORKDIR /src
-    COPY go.mod go.sum ./
-    RUN go mod download
-    COPY . .
-    RUN --mount=type=cache,target=/root/.cache/go-build \
-        go test -count=1 -timeout 90s -race -cover ./... || \
-        { echo "--- Rerunning with -v for details ---"; \
-          go test -count=1 -timeout 90s -race -v ./...; exit 1; }
-
-    # Build stage. Nothing is wanted from either phase above; the copies
-    # are what make BuildKit build them first, so this stage cannot run
-    # unless lint and test passed.
     # golang:1.x-alpine, YYYY-MM-DD
     FROM golang@sha256:... AS builder
-    ARG CACHEBUST
-    COPY --from=lint /src/go.sum /dev/null
-    COPY --from=test /src/go.sum /dev/null
-    RUN apk add --no-cache git
-    # A tar-stream context keeps the sender's file owners, which git refuses.
-    RUN git config --system --add safe.directory /src
     WORKDIR /src
+
+    # Force BuildKit to run the lint stage before proceeding
+    COPY --from=lint /src/go.sum /dev/null
+
     COPY go.mod go.sum ./
     RUN go mod download
     COPY . .
+    RUN make test
 
-    # The VERSION build arg when one is given, otherwise
+    ARG VERSION=dev
-    # `git describe --tags --always` on the .git in the build context. With
+    RUN CGO_ENABLED=0 go build -trimpath \
-    # .git present, a version that is still empty, dev or unknown fails the
-    # build: git is missing or could not read the checkout.
-    ARG VERSION
-    RUN --mount=type=cache,target=/root/.cache/go-build \
-        VERSION="${VERSION:-$(git describe --tags --always)}"; \
-        if [ -e .git ]; then \
-            case "$VERSION" in ""|dev|unknown) \
-                echo "version is '$VERSION' although .git is present" >&2; \
-                exit 1 ;; \
-            esac; \
-        fi; \
-        CGO_ENABLED=0 go build -trimpath \
         -ldflags="-s -w -X main.Version=${VERSION}" \
         -o /app ./cmd/app/
 
-    # Runtime stage, and the last one
+    # Runtime stage
     FROM alpine@sha256:...
-    ARG CACHEBUST
     COPY --from=builder /app /usr/local/bin/app
     ENTRYPOINT ["app"]
     ```
 
     Key points:
-    - The lint phase uses the `golangci/golangci-lint` image directly (it has
+    - The lint stage uses the `golangci/golangci-lint` image directly (it
-      both Go and the linter), so nothing needs installing.
+      includes both Go and the linter), so there is no need to install the
-    - `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
+      linter separately.
-      purpose is the ordering edge. BuildKit runs stages in parallel by default,
+    - `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
-      and a stage nothing depends on is not built at all, so without these two
+      a stage dependency. BuildKit runs stages in parallel by default; without
-      lines a red gate would not fail the build.
+      this line, the build stage would not wait for lint to finish and a lint
-    - Keep the runtime stage last, and if you add a stage after it, give it the
+      failure might not fail the overall build.
-      same two copies. A plain `docker build .` builds the last stage's chain
-      and nothing else.
-    - Only Go's build cache is mounted (`/root/.cache/go-build`, where Go keeps
-      it for root), and only on the steps that compile: `golangci-lint run`,
-      both `go test` runs and `go build`. It keeps compiled packages between
-      check builds, and Go reuses one only when all its inputs are unchanged, so
-      it saves the compile without skipping a check. Nothing else is kept:
-      golangci-lint's own cache would replay stored findings in place of a lint
-      run, and Go builds from whatever files a kept module cache holds without
-      checking them again (only `go mod verify` does), so a file changed there
-      would go unnoticed. Go's build cache also holds test results, which is why
-      both `go test` runs carry `-count=1`.
     - If the project uses `//go:embed` directives that reference build artifacts
-      (e.g. a web frontend compiled in a separate stage), the lint phase must
+      (e.g. a web frontend compiled in a separate stage), the lint stage must
       create placeholder files so the embed directives resolve. Example:
       `RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
-    - If the project requires CGO or system libraries for linting, install them
+      The lint stage should not depend on the actual build output — it exists to
-      in the lint phase. The `golangci/golangci-lint` image is Debian-based and
+      fail fast.
-      has no `apk`, so install with `apt-get` under the Debian package name
+    - If the project requires CGO or system libraries for linting (e.g.
-      (`libvips-dev`, where alpine says `vips-dev`), and delete the package
+      `vips-dev`), install them in the lint stage with `apk add`.
-      lists in the same `RUN`, so the layer does not keep them:
+    - The build stage runs `make test` after compilation setup. Tests run in the
-
+      build stage, not the lint stage, because they may require compiled
-        ```dockerfile
+      artifacts or heavier dependencies.
-        RUN apt-get update \
-            && apt-get install -y --no-install-recommends libvips-dev \
-            && rm -rf /var/lib/apt/lists/*
-        ```
-
-    - `.dockerignore` lets `.git` into the build context. It keeps out every git
-      `config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
-      repository's own, each submodule's under `.git/modules/`, and that of a
-      submodule keeping its own `.git` directory. `git describe` does not need
-      them, and each can hold a credential: a password in a remote URL, or the
-      token the CI checkout step stores there. A submodule whose name has a
-      `config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
-      git directory to `**/.git/modules/**/config`, and Go's version stamping
-      then fails the build: give it a name without that segment
-      (`git submodule add --name`). The stage that compiles has `git` (the
-      Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
-      takes the version from the `VERSION` build argument when one is given,
-      otherwise from `git describe --tags --always`. That gives the tag on a
-      tagged commit; on a later commit, the tag, the number of commits since it
-      and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
-      tag is reachable. The stage that compiles also marks its working directory
-      safe for git (`git config --system --add safe.directory /src`): a context
-      sent as a tar stream keeps the sender's file owners, and git refuses a
-      checkout owned by another user, so the version would come out empty.
-      `ARG VERSION` has no default, and the build fails if the context carries
-      `.git` and the version still comes out empty, `dev` or `unknown`. A plain
-      `docker build .` with no build arguments must succeed; a Dockerfile that
-      refuses an empty build argument drops that refusal and keeps the argument.
-      A checkout whose `.git` is a file (a linked worktree, or a repository
-      checked out as a submodule) is the exception: that file points to a git
-      directory outside the build context, so the build cannot read the version
-      and a plain `docker build .` fails; pass the version with
-      `--build-arg VERSION=...`, as `script/docker` and `script/cibuild` already
-      do.
 
 - Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
-  runs `script/cibuild` on push, and checks out the repo as its only other step,
+  runs `script/cibuild` (which runs `docker build .`) on push. Since the
-  with `persist-credentials: false`: `script/cibuild` needs no token, and
+  Dockerfile already runs `make check`, a successful build implies all checks
-  without it the checkout leaves the job's token in `.git/config` for every
+  pass.
-  later step. The checkout step also sets `fetch-depth: 0`, which fetches the
-  tags `git describe` needs: by default it clones shallow with no tags, and a
-  tagged repository's CI build would stamp a bare short commit id. The
-  workflow's `concurrency` block groups runs by workflow and branch
-  (`${{ github.workflow }}-${{ github.ref }}`) with `cancel-in-progress: true`,
-  so a new push cancels the older run on the same branch, queued or running, and
-  no other: runs for replaced commits do not hold up the shared runner.
-  `script/cibuild` bootstraps, runs the gate phases, and then builds the image,
-  so a successful run means every check passed; a bare `docker build .` does not
-  carry the same guarantee, because its gate phases may come from the cache. The
-  image build gets a `CACHEBUST` value of its own and so runs the gate phases a
-  second time. That is the price of the rule above, and it is worth paying: the
-  image that ships is built from a run of its own gates rather than from a cache
-  entry. The `check` job sets `timeout-minutes: 20`, so a hung build frees the
-  shared runner after 20 minutes. That allows for the three Docker builds
-  described above (the test phase, the lint phase, then the image), each held to
-  the 5-minute Docker build limit below, plus the bootstrap. A separate workflow
-  limited to `main` by a `branches` list under `on: push` cannot be checked by
-  review: to try a change to it, add the feature branch to that list and push,
-  then remove the branch from the list again before merging. Keep any job in it
-  that publishes behind `if: github.ref_name == 'main'`, so the run from the
-  feature branch publishes nothing.
 
 - Use platform-standard formatters: `black` for Python, `prettier` for
   JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -375,21 +189,14 @@ style conventions are in separate documents:
   module under test to verify it compiles/parses. There is no excuse for
   `make test` to be a no-op.
 
-- `make test` must complete in under 60 seconds. That is the hard cap, and a
+- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
-  suite that exceeds it fails. Under 20 seconds is the target. A suite between
+  Makefile.
-  20 and 60 seconds is still green, but the overage must be filed as an
-  improvement bug against that repo. Add a 90-second timeout to the test
-  invocation (`go test -timeout 90s`). The backstop deliberately sits above the
-  hard cap so that it catches a genuinely hung test rather than a merely slow
-  one.
 
-- **The test command should use the conditional verbose rerun pattern.** Run
+- **`make test` should use the conditional verbose rerun pattern.** Run tests
-  tests without `-v` (verbose) first. If tests fail, automatically rerun with
+  without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
-  `-v` to show full output. This keeps CI logs and `docker build` output clean
+  show full output. This keeps CI logs and `docker build` output clean on
-  on success (just package/suite summaries) while providing full diagnostic
+  success (just package/suite summaries) while providing full diagnostic detail
-  detail on failure (every test case, every assertion). The command lives in the
+  on failure (every test case, every assertion). The general shell pattern:
-  `test` phase of the `Dockerfile`, since `script/test` builds that phase; the
-  Makefile form below is the same pattern for any repo-local invocation:
 
     ```makefile
     test:
@@ -402,25 +209,11 @@ style conventions are in separate documents:
 
     ```makefile
     test:
-    	@go test -count=1 -timeout 90s -race -cover ./... || \
+    	@go test -timeout 30s -race -cover ./... || \
     		{ echo "--- Rerunning with -v for details ---"; \
-    		  go test -count=1 -timeout 90s -race -v ./...; exit 1; }
+    		  go test -timeout 30s -race -v ./...; exit 1; }
     ```
 
-    `-count=1` is required on both invocations: it defeats Go's test _result_
-    cache, so neither run can report a stored pass in place of running the
-    tests. It leaves the build cache alone, so it costs the runtime of the suite
-    and no recompilation.
-
-    That cache is Go's own, separate from Docker's layer cache. Go stores a
-    passing result in its cache directory (`GOCACHE`), and when the same tests
-    run again on unchanged code it prints that result, marked `(cached)`,
-    without running them. The directory lasts from one run to the next on a
-    developer's machine, where this target runs, and in the `test` phase of the
-    `Dockerfile`, which mounts it to keep compiled packages between check
-    builds, so the phase's `go test` runs carry `-count=1` too. A new
-    `CACHEBUST` value (above) is what makes that step run on an unchanged tree.
-
     Python example:
 
     ```makefile
@@ -446,89 +239,10 @@ style conventions are in separate documents:
   must be in `.gitignore`. No exceptions.
 
 - `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
-  editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
+  editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
-  `node_modules/`, and the repo's own build outputs. Fetch the standard
+  Fetch the standard `.gitignore` from
-  `.gitignore` from
   `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
-  a new repo. A repo's `.gitignore` is the standard file followed by the repo's
+  a new repo.
-  own entries, such as its binaries; a re-vendor replaces the standard part and
-  keeps those entries. These patterns are written to `.gitignore`'s own
-  semantics, in which an unanchored pattern already matches at every depth; they
-  are not a `.dockerignore` and must not be transplanted into one unmodified.
-
-- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
-  across unmodified leaves secrets in the build context.** Docker matches with
-  `moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
-  `*` does not cross `/` and a pattern without a leading `**/` is anchored at
-  the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
-  therefore excludes only the copies at the repository root, while `config/.env`
-  and `certs/server.key` still reach the context and can land in an image layer
-  — which is more dangerous than a short file with no secret patterns at all,
-  because it reads as solved and stops anyone looking. Give every
-  depth-independent pattern the `**/` prefix and leave only genuinely
-  root-anchored entries unprefixed: `.claude`, and the repo's own host-built
-  binary, written `/myapp` and never `**/myapp`, which would also match
-  `cmd/myapp/` and delete the package directory from the context. Matching is
-  case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
-  secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
-  and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
-  also catches something the build needs, re-include it with a negation
-  (`!docs/example.env`); deleting the pattern reopens the exposure for every
-  other file it covers. Fetch the standard `.dockerignore` from
-  `https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
-  it with the repo's own artifacts.
-
-- **In-repo agent scratch belongs in both files, written to each file's own
-  semantics.** `.claude/` holds one worktree per in-flight agent — an entire
-  additional checkout of the repo — so under `COPY . .` the build context
-  inflates by a multiple of the repo and another session's unreviewed work can
-  be copied into an image layer. In `.gitignore` the entry is `.claude/`,
-  unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
-  prefix, because the prefixed form would also delete any nested directory of
-  that name from the build. Anchoring carries a known gap that the canonical
-  `.dockerignore` states in its own comment, since consuming repos receive the
-  file and not the tracker: the directory is created in the agent's working
-  directory, so a repo running agents in subdirectories still ships
-  `services/api/.claude/` and must add its own anchored entry there.
-
-- **A plain `docker build .` of a clone stamps the version that
-  `git describe --tags --always` gives**, derived from the `.git` in the build
-  context as the canonical `Dockerfile` above shows. Without its failure check,
-  a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
-  and the build would still exit 0. `script/docker` and `script/cibuild` pass
-  the version they compute on the host; it takes precedence. They do this
-  byte-identically across repos:
-
-    ```sh
-    # The version and the tag each get their own line: a failing command
-    # substitution inside an argument does not trip `set -e`, so the inline
-    # form degrades to an empty constant.
-    version="$(git describe --tags --always --dirty 2>/dev/null || true)"
-    [ -n "$version" ] || version="unknown"
-    tag="$(script/projectname)"
-    docker build --build-arg CACHEBUST="$(date +%s)-$$" \
-        --build-arg VERSION="$version" \
-        -t "$tag" .
-    ```
-
-    `--always` makes an untagged repo yield an abbreviated commit hash rather
-    than failing, and the `[ -n "$version" ]` line is the single place the
-    fallback is applied — a live check that fires on a build from an export with
-    no `.git` and on a repository with no commits yet. Do not fold it into the
-    substitution as `|| echo unknown`, which makes the guard unreachable. The
-    Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
-    there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
-    Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
-    the scripts stay byte-identical. One consequence for CI: the standard
-    checkout action clones shallow and fetches no tags, so the canonical
-    `.gitea/workflows/check.yml` sets `fetch-depth: 0` on its checkout step.
-
-- **Verify `.dockerignore` by enumerating the image, not by reading the
-  patterns.** Plant files at the root _and_ at least two directories deep, build
-  a probe image that does `COPY . .`, and list what actually landed
-  (`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
-  size is not a substitute: a nested secret is a few bytes, and BuildKit
-  transfers only the delta from the previous build.
 
 - **No build artifacts in version control.** Code-derived data (compiled
   bundles, minified output, generated assets) must never be committed to the
@@ -544,61 +258,9 @@ style conventions are in separate documents:
 - Make all changes on a feature branch. You can do whatever you want on a
   feature branch.
 
-- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
+- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
-  _NEVER_ be modified by an agent: fetch it from
+  manually by the user. Fetch from
-  `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
+  `https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
-  byte-identical, so that no repo can quietly loosen its own linting. Linter
-  configuration changes are made to the canonical copy in the `prompts` repo and
-  reach consuming repos by re-vendoring; an agent may open a PR against
-  canonical, which only the user merges. One list is exempt from byte-identity,
-  because it cannot be written once for every repo: the `deny` list of the
-  `test-support` depguard rule, where a repo names its own test-support packages
-  by full import path. A repo adds entries there and changes nothing else, and a
-  re-vendor carries its entries forward. The canonical golangci-lint version is
-  v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
-  image
-  (`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
-  which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
-  directive must not name a newer Go minor version than the one golangci-lint
-  was built with, or golangci-lint refuses to lint it: this release lints
-  `go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
-  installs golangci-lint on the host. A repo sets the lint phase digest to the
-  one named here and re-vendors `.golangci.yml` in the same commit, whichever of
-  the two prompted the change: the canonical copy can name linters that an older
-  golangci-lint rejects, and a newer golangci-lint can add linters that
-  `default: all` switches on until the canonical copy disables them.
-
-- **`script/bootstrap` installs a pinned tool by comparing versions, never by
-  testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
-  `PATH` only, so on an already-provisioned machine the pin is inert and a
-  version bump is a silent no-op — while the Dockerfile, installing into a clean
-  image, gets the pinned version, so a local `make check` and `make docker` can
-  disagree about what the tool even is. The canonical form:
-    - compares the installed version against the pin over the **whole** version
-      token; a parser that stops at the first `-` reports `2.12.2` for a host
-      running `2.12.2-rc1` and skips the install;
-    - treats absent, non-zero, empty or unrecognised `--version` output as a
-      mismatch, so the failure direction is a redundant install and never a
-      skipped one;
-    - after installing, re-resolves the binary the way callers do — `hash -r`,
-      then through `PATH`, not through the directory the installer wrote to —
-      and fails naming the resolved path, since an install that a shadowing
-      binary hides succeeds while changing nothing any caller sees;
-    - is actually called, and prints the version on both success paths: a
-      function defined and never invoked has the same exit status and the same
-      empty output as one that worked.
-
-    Node is the exception to the whole-token comparison: the canonical
-    `script/bootstrap` compares only its major version, because the node 22
-    alpine image the `Dockerfile` stages start from is not the exact pin, and
-    nvm cannot install a prebuilt node on alpine.
-
-    Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
-
-    A Go tool a repo needs on the host is installed with `go install` pinned to
-    a commit hash (`go install <package>@<commit hash>`). It is never tracked as
-    a `go.mod` tool dependency or through a `tools.go` file, either of which
-    pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
 
 - When pinning images or packages by hash, add a comment above the reference
   with the version and date (YYYY-MM-DD).
@@ -709,21 +371,15 @@ style conventions are in separate documents:
       Never edit existing migrations after release.
 
 - All repos should have an `.editorconfig` enforcing the project's indentation
-  settings: the standard file from
+  settings.
-  `https://git.eeqj.de/sneak/prompts/raw/branch/main/.editorconfig`, which sets
-  tabs for `Makefile` and Go files, followed by the repo's own sections, such as
-  one for another language it uses. A re-vendor replaces the standard part and
-  keeps those sections.
 
 - Avoid putting files in the repo root unless necessary. Root should contain
-  only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
+  only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
-  `Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
+  `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
-  and language-specific config). Everything else goes in a subdirectory.
+  language-specific config). Everything else goes in a subdirectory. Canonical
-  Canonical subdirectory names:
+  subdirectory names:
     - `bin/` — executable scripts and tools
-    - `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
+    - `cmd/` — Go command entrypoints
-      body is a single call into `internal/` or `pkg/`, no project logic in
-      `cmd/`
     - `configs/` — configuration templates and examples
     - `deploy/` — deployment manifests (k8s, compose, terraform)
     - `docs/` — documentation and markdown (README.md stays in root)
@@ -750,7 +406,3 @@ style conventions are in separate documents:
     - Go: `go.mod`, `go.sum`, `.golangci.yml`
     - JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
     - Python: `pyproject.toml`
-
-- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
-  is never committed under a file or directory named after one agent tool, such
-  as `CLAUDE.md` or `.claude/`, and never split into separate memory files.

TODO.md

@@ -1,515 +1,417 @@
 # Workflow
 
-- take an issue from the `1.0.0` milestone on the tracker; work not yet on the
+- take an issue from the `1.0.0` milestone on the tracker; work not
-  tracker gets filed as an issue first
+  yet on the tracker gets filed as an issue first
-- branch from `next`
+- branch (from `main`)
 - do the work, with tests, in small focused commits
-- record it at the top of Completed Steps (`TODO.md` changes in the same commit
+- record it at the top of Completed Steps (`TODO.md` changes in the
-  as the work)
+  same commit as the work)
-- push the branch and open a PR against `next` whose title ends with
+- push the branch and open a PR whose title ends with
   ` (closes #N)`
-- an independent review gates each merge to `next`; every finding is addressed
+- an independent review gates the merge; every finding is addressed
   or explicitly rebutted on the PR
-- only the owner merges `next` to `main`
+- merge to `main` once the review passes
 
 # Status
 
 - pre-1.0
-- the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
+- the Gitea tracker is authoritative for the pre-1.0 backlog: the
-  under the `1.0.0` milestone are what remains before the tag, and this file
+  open issues under the `1.0.0` milestone are what remains before
-  records history and process, not the queue
+  the tag, and this file records history and process, not the queue
 
 # Next Step
 
 - take the next issue from the `1.0.0` milestone on the tracker:
-  https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
+  https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is
-  of truth for what is left before 1.0.0. Individual issues are deliberately not
+  the source of truth for what is left before 1.0.0. Individual
-  restated here; a copy in this file drifts out of date the moment the tracker
+  issues are deliberately not restated here; a copy in this file
-  moves
+  drifts out of date the moment the tracker moves
 
 # Completed Steps
 
-- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so
+- run all linting in Docker via `Dockerfile.lint` and `script/lint`
-  `go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has
+  (2026-08-10, branch `next`, closes
-  passed the build; README "Install" says so, says what `@main` and `v0.0.1`
+  https://git.eeqj.de/sneak/sfdupes/issues/46): per the owner ruling, the
-  give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the
+  linter runs inside a container invoked through the `script/`
-  early build without a database. When the linker set no version,
+  entrypoint and is never installed on a host. New root
-  `sfdupes --version` prints the main module's version that Go recorded in the
+  `Dockerfile.lint` COPYs the repo into the digest-pinned
-  binary, the tag or the pseudo-version of a `go install` build, and `dev` only
+  `golangci/golangci-lint:v2.12.2` image and runs
-  when neither is known (2026-10-17,
+  `golangci-lint config verify` and `golangci-lint run` as build
-  https://git.eeqj.de/sneak/sfdupes/issues/116)
+  steps, so a successful build IS a clean lint; `script/lint` is
+  reduced to building it. `script/bootstrap` loses the `go install`,
+  the pin constants, the version parser and `verify_golangci_lint`
+  outright rather than hardening them — with nothing linting on the
+  host, the `$GOPATH/bin` versus `PATH` problem that motivated them has
+  no subject — and now warns rather than fails when `docker` is absent.
+  Two traps handled. A lint build on an unchanged tree returns success
+  in well under a second having run no linter, which is
+  https://git.eeqj.de/sneak/sfdupes/issues/32 and
+  https://git.eeqj.de/sneak/sfdupes/issues/39 again, so
+  `Dockerfile.lint` carries `ARG CHECK_EPOCH` referenced
+  inside every gate `RUN` (BuildKit hashes the expanded command, not
+  the declaration) and `script/lint` passes `"$(date +%s)-$$"` — the
+  PID matters because two lint runs land inside the same second easily.
+  And nothing inside an image build may shell out to docker, so the
+  main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
+  instead of `make lint`, and its build stage runs `make test` and
+  `make fmt-check` instead of the `make check` aggregate (`make`, not
+  the scripts bare, because the Makefile's `export CGO_ENABLED = 0`
+  only reaches what it invokes). `COPY --from=lint`
+  `/usr/bin/golangci-lint` is replaced by
+  `COPY --from=lint /src/go.sum /dev/null`: the copied binary was the
+  only edge forcing BuildKit to finish linting before the build stage
+  starts, and dropping it without replacing the edge would have ended
+  fail-fast linting silently under a still-green build. That is
+  canonical `REPO_POLICIES.md:107`'s ordering edge, restored.
+  `ENV PATH=/home/builder/go/bin:$PATH` is gone with the `go install`
+  that justified it. `script/verify-linter-pin` is retired, deleted
+  along with its README entry, because both of its subjects ceased to
+  exist in the same change: it compared a linter binary against
+  `GOLANGCI_LINT_VERSION` in `script/bootstrap`, and there is now
+  neither a binary crossing between stages nor a version pin in
+  bootstrap. The drift it guarded has not gone away, it has moved — the
+  linter is still pinned twice, now as the `FROM` line of
+  `Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage,
+  with nothing syncing them, which is exactly what
+  https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
+  replacement is one new `script/verify-lint-image-pin`,
+  run as a gate in both files, which compares the two references to
+  each other and deliberately restates neither: a hardcoded expected
+  digest would be a third copy and the same drift one file further out.
+  `golangci-lint config verify` is included per the ruling, and the
+  concern about its unpinned live HTTPS schema fetch was measured
+  rather than assumed — under `--network none` the pinned binary both
+  passes a valid config and rejects an invalid one with the jsonschema
+  error, so it validates from an embedded schema and linting needs no
+  network; the README says so instead of claiming a requirement that
+  does not exist. Verified: `make lint` green with every `PATH`
+  directory containing a `golangci-lint` removed
+  (`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
+  `command -v golangci-lint` empty); two consecutive `script/lint` runs
+  on an untouched tree both executed the linter, 27.7s and 28.7s in the
+  lint step under distinct epochs with the `COPY . .` layer `CACHED`
+  above them, at 42.2s and 41.8s wall clock — the no-cache rule was not
+  weakened to shorten that. Negative control: a planted
+  `var unusedIssue46Sentinel = 1` failed `script/lint` with
+  `report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and
+  failed `make docker` at `[lint 9/9]` with the build stage stopped at
+  `[builder 3/12]` — `COPY --from=lint`, `script/bootstrap`, the test
+  gate and `make build` all zero occurrences — then reverted clean. The
+  drift guard fails on a tag-only disagreement, on a digest-only
+  disagreement, and on an unreadable reference, naming both sides.
+  `make docker` green in 5m35s with all six gates executing under one
+  epoch (lint 37.6s, test 25.2s reporting
+  `ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`).
+  The non-root quirk still holds: in the builder image with the Go test
+  cache off, `--user 0:0` fails `TestScanHardlinkRunFailsTogether`
+  (exit 1) where the unprivileged user passes (exit 0). Noted for
+  follow-up, not fixed here: `golangci-lint` warns that the
+  `gomodguard` linter is deprecated since v2.12.0 in favour of
+  `gomodguard_v2`.
 
-- `scan` creates the database's directory where the database path leads: the
+- install the Docker build stage's prerequisites by running
-  path with its last element cut off as text, never cleaned, so a `..` after a
+  `script/bootstrap` instead of `apk add --no-cache make` inline
-  symlinked directory is followed through the symlink, as when the file is
+  (2026-08-09, branch `dockerfile-bootstrap`, closes #42): canonical
-  opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
+  `REPO_POLICIES.md:97` requires it, and the inline install left the
-  symlink to `/data/x`, it creates `/data/new` and the database in it, where it
+  build stage maintaining its own notion of the toolchain — exactly
-  used to create `/srv/new` and then fail (2026-10-17,
+  the divergence #24 exists to close, one layer down. The stage now
-  https://git.eeqj.de/sneak/sfdupes/issues/119)
+  copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`,
-
+  which ends in `go mod download`, so the separate invocation of that
-- `scan` never records its own database: once the database and its `.lock` file
+  is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, and moves
-  are open, it notes the device and inode of the database file, its `-wal` and
+  above the bootstrap layer. It is the only edge making this stage
-  `-shm` files and its `.lock` file, and the walk passes over a file with one of
+  depend on the lint stage, so deleting it as redundant would end
-  them wherever the operands reach it, through a symlinked directory included,
+  fail-fast linting silently. Letting bootstrap install its own linter
-  printing no line and counting nothing. A record an earlier scan left for one
+  here would have reintroduced the second toolchain and paid for a
-  of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
+  from-source build of it. What makes the two stages provably one
-
+  toolchain rather than two that happen to agree is a new
-- `scan` never records or reads anything that is not on a persistent filesystem
+  `script/verify-linter-pin`, run in the build stage on the binary
-  (`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
+  that arrives from the lint stage, before bootstrap: it fails the
-  matched by the device ID of each such filesystem, listed as the scan starts
+  build naming both versions unless that binary is the version
-  from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
+  `script/bootstrap` pins. Bootstrap's own check could not serve that
-  however the walk or the content phase reaches it; no flag turns that off.
+  purpose — it reinstalls its pin from source and then verifies
-  Below an operand, its walk does not enter the default excludes either, so
+  whatever `PATH` resolves, so drift self-heals silently and a lint
-  `sfdupes scan /` skips the kernel's view of the running system and the
+  stage image bumped on its own would lint at the new version while
-  temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
+  `make check` ran at the old one, green. The linter version is pinned
-  `/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
+  in two independent places (the lint stage image digest and
-  `/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
+  `GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
-  Windows list, not used yet. An operand at or inside a default exclude is still
+  half-applied bump is now a build failure. The pin is read out of
-  walked. The scan prints a line for each path it leaves out, without counting
+  `script/bootstrap`, which stays the single source of truth; a pin
-  it as skipped, and the records there count as outside the scanned operands, as
+  that cannot be read is a hard failure, not a skip. The check needs
-  for `-x`. `--no-default-excludes` turns off the default excludes only, and
+  no `CHECK_EPOCH`: its only inputs are the copied binary and
-  `scan --help` lists them (2026-10-17,
+  `script/`, so Docker invalidates the layer exactly when a cached
-  https://git.eeqj.de/sneak/sfdupes/issues/124)
+  result would stop being true, and it is documented with the other
-
+  entrypoints in the README. `$GOPATH/bin` joins `PATH` because
-- the rate a progress display shows covers the items completed over the last 5
+  that is where bootstrap's `go install` lands and bootstrap verifies
-  seconds, or since the line before when stderr is not a terminal, so it falls
+  its installs against what `PATH` resolves — nothing in the image is
-  to 0 while nothing completes, and a burst after a wait no longer reads as a
+  shadowed by it, the directory does not exist until bootstrap runs.
-  steady rate of over 1,000 files a second; the ETA still comes from the phase's
+  Everything added sits above `ARG CHECK_EPOCH`, and the `chown` and
-  average. Once no item has completed for 10 seconds, the word `stalled` and
+  `USER builder` still precede `make check`. Verified: the guard fails
-  where the phase waits take the rate's place, as `stalled disk/db`: `disk`,
+  the build with both versions named when the lint stage's linter is
-  `db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
+  faked to a different version, and an unmodified build still passes
-  either, each phase giving its word where it is defined. A line too wide for
+  it; bootstrap runs clean under Alpine's `sh` and its `apk` branch,
-  the terminal loses its bar first, then its rate or those words, so the phase,
+  installing `git` and `make` and finding the copied
-  progress, elapsed time and ETA stay whole. On an 80-column terminal, at counts
+  linter already at the pin; a second build served the bootstrap and
-  like those of README's example, the content phase's checks and reads lose
+  dependency layers `CACHED` while both gates ran with a fresh epoch;
-  those words at most times in minutes, and every other phase keeps them
+  a planted `unused` finding failed the build at the lint gate in
-  whenever the elapsed time and the ETA are both in minutes. The content phase's
+  48.9s with the build stage's `make check` never starting; and the
-  last partial batch is now written under a display of its own (2026-10-16,
+  suite run in the image as `--user 0:0` fails
-  https://git.eeqj.de/sneak/sfdupes/issues/125)
+  `TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged
-
+  user is still load-bearing. That last check needs the Go test cache
-- new subcommand `info` prints the database's details on stdout, one per line as
+  disabled — the first attempt reported `ok ... (cached)` as root,
-  a name, a tab and the value: its path and where it came from, the sizes of the
+  reusing the result the build-time run had left in the shared cache,
-  database file and of its `-wal` file when there is one, when the database file
+  which would have read as a pass. Build wall time, on a shared host
-  was last changed, the schema version, the number of records, how many have
+  running many concurrent builds and so noisy: 2m13s on an unchanged
-  head and tail hashes and how many a content hash, and the sizes of all records
+  tree, 2m17s and 4m29s for two builds after a source change, 5m14s
-  added up. It opens the database as `report` does, and shows two phases on
+  cold. Only the cold one breaches the policy ceiling, and not because
-  stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come
+  of this change — `chown -R builder:builder /src /home/builder` walks
-  from one read-only transaction, the counts from one query (2026-10-16,
+  the module cache and re-runs on every source change, and it alone
-  https://git.eeqj.de/sneak/sfdupes/issues/117)
+  varied between 77s and 210s across those four builds, which is also
-
+  the whole spread in the totals. The same cold measurement against
-- `report` shows its work on stderr in two phases, `[1/2] open` and
+  `main` is 5m03s with a 209s `chown`. Filed as #43
-  `[2/2] read`, with the progress display `scan` and `trees` use, from the
+- bust the Docker layer cache for the gate steps, so `script/cibuild`
-  moment it starts until the summary line: `open` says it is opening the
+  and `script/docker` cannot report a green they did not earn
-  database, also while it waits on a busy one; `read` counts the records, then
+  (2026-08-09, branch `cibuild-cache-bust`, closes #32): both scripts
-  reads them against that count in signature order, which SQLite takes from the
+  were bare `docker build` invocations with no cache control, and the
-  `files_signature` index without sorting, so the first record comes at once.
+  `Dockerfile` copies the tree before running its gates, so on an
-  `report` forms the duplicate groups itself, holding only those of the size it
+  unchanged tree Docker served those layers from cache and the build
-  is reading, and writes them ordered by first path before it reads the next
+  exited 0 having executed nothing. That is not hypothetical here —
-  size. stdout and the summary line are unchanged, and SQLite no longer needs a
+  every merge this repo has done is a non-fast-forward merge of an
-  temporary file to sort in (2026-10-16,
+  undiverged branch, so each merge commit's tree is byte-identical to
-  https://git.eeqj.de/sneak/sfdupes/issues/114)
+  the branch head's and each merge CI run was almost certainly a full
-
+  cache hit; and PR #31's reviewer found `make docker` returning
-- `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`,
+  success as a 17-layer cache hit, catching it only by being
-  `[3/4] group` and `[4/4] write`, each with the progress display `scan` uses,
+  suspicious. The fix is `ARG CHECK_EPOCH` with the scripts passing
-  from the moment it starts until the summary line: `open` says it is opening
+  `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or break
-  the database, also while it waits on a busy one; `read` counts the records,
+  it. `ARG` is scoped per stage and this `Dockerfile` has three gates
-  then reads them against that count; `group` counts the directories it groups,
+  across two — `make fmt-check` and `make lint` in the lint stage,
-  and `write` the groups it writes. A display's rate names what its phase
+  `make check` in the build stage — so a single declaration would have
-  counts: files, records, directories or groups. stdout and the summary line are
+  left one stage silently cacheable; it is declared in both. And
-  unchanged. When stdout and stderr are the same file, as one terminal is, none
+  BuildKit hashes the expanded command, not the declaration, so a
-  of stdout's lines shares a line with the display; when they are not, the
+  declared-but-unreferenced `ARG` invalidates nothing: each gate `RUN`
-  display keeps being drawn while stdout waits on a slow reader (2026-10-16,
+  echoes the epoch, which also puts the value in the build log as
-  https://git.eeqj.de/sneak/sfdupes/issues/113)
+  evidence the layer really ran. Placement is below the dependency
-
+  layers on purpose — a build that goes cold every time would be a
-- each `scan` phase's progress display names the phase and its place among the
+  different bug, not a fix. Verified by running each script twice back
-  five, as `[2/5] walk`, from the moment it starts, with its progress, elapsed
+  to back on an unchanged tree under `BUILDKIT_PROGRESS=plain`: all
-  time and ETA: `load` counts the records first, the walk's total is estimated
+  three gates executed on all four runs, each with a fresh epoch in
-  from the records under the operands, and the content phase counts the records
+  the log (`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s
-  it checks from SQLite's grouping alone, then says it is searching the database
+  then 53.4s), and twelve steps were still served `CACHED` in the
-  until the first one is returned. On a terminal a phase with an exact total
+  steady state — both `go mod download`s, `apk add`, `adduser`, the
-  draws its line followed by the library's bar, and any other a spinner followed
+  `chown`, every `go.mod`/`go.sum` and source copy, the linter copy
-  by the line; every display is drawn again several times a second. No line
+  out of the lint stage, and the binary copy into the runtime stage.
-  reaches the terminal's last column, however narrow, also after the terminal is
+  The lint stage still gates the build stage: with a deliberate
-  made narrower during a phase: a line too wide leaves out its rate, then its
+  `unused` finding planted in the tree, the build failed at
-  bar, keeping the phase, progress, elapsed time and ETA whole, and is cut short
+  `make lint` in 36.1s and the build-stage `make check` never started.
-  only when those alone do not fit, as below about 70 columns late in a large
+  The build stage also still drops to the unprivileged `builder` user
-  scan. In plain mode the line is printed every 5 seconds whether or not an item
+  before `make check`, which the suite depends on rather than merely
-  completed (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/112)
+  prefers: forcing the same image to run the tests as root fails
-
+  `TestScanHardlinkRunFailsTogether`, because root reads straight
-- `scan`, `report` and `trees` first print a line on stderr naming the database
+  through the `chmod(0)` the test uses to prove hard links are read
-  they use, as an absolute path, and whether that path is the default or comes
+  once. This is the local fix only; propagating it to the canonical
-  from `SFDUPES_DATABASE`; `scan` says when the database is new. README
+  templates is `prompts` #26
-  "Database" says when `scan` writes each file (2026-10-16,
+- check the installed golangci-lint version in `script/bootstrap`
-  https://git.eeqj.de/sneak/sfdupes/issues/111)
+  instead of only its presence (2026-08-09, branch
-
+  `bootstrap-version-check`, closes #24): `missing golangci-lint` meant
-- re-vendor the canonical files and model scripts from `sneak/prompts` `next` at
+  any linter already on `PATH` satisfied the check, so the pin was never
-  `e3f79c4`: golangci-lint v2.14.0; lint and test are phases of the `Dockerfile`
+  consulted and the v2.12.2 bump from #3 was inert on every host that
-  that write no image, and `make test` runs the suite under the race detector,
+  already had one — this host ran v2.10.1 against a v2.12.2 pin,
-  so `Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race` are
+  `make check` went green, and `make docker` then rejected the same
-  gone; every `docker build` in `script/` passes a new `CACHEBUST` value, which
+  commit with findings the local gate never saw. The version now lives
-  every stage of the `Dockerfile` declares first, so every `RUN` step runs on
+  in one place, `GOLANGCI_LINT_VERSION`, with the `go install` module
-  every build, while the steps that compile keep Go's build cache and the tests
+  ref derived from it so a bump cannot half-apply; a
-  run with `-count=1`; prettier runs on the host, from the node and yarn
+  `golangci_lint_version` helper parses `golangci-lint --version`
-  `script/bootstrap` installs, so the `prettier` and `markdown` stages are gone
+  (taking the field after the word `version` and tolerating an optional
-  and the build stage installs `git` and `make` itself; a new push cancels the
+  leading `v`, which the module ref carries and the binary's output does
-  workflow's older run on the same branch, and a run stops after 20 minutes;
+  not), and any version that is not the pin — older, newer, absent or
-  `.claude/settings.json` is deleted (2026-10-15,
+  unparseable — is reinstalled. The install is then verified against the
-  https://git.eeqj.de/sneak/sfdupes/issues/95)
+  binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
-
+  `GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes
-- with `-x`, an operand under another operand is walked on its own instead of
+  first on `PATH`, so a wrong-version one sitting ahead of it — nix,
-  being dropped when a directory from it (from its directory, for a regular
+  apt, brew, apk, or the `/usr/local/bin` copy the `Dockerfile` builder
-  file) up to but not including that operand is on a different filesystem than
+  stage makes — would swallow the install and leave the local gate
-  that operand, whose walk then cannot reach it, and the records of files gone
+  disagreeing with CI under an affirmative `bootstrap complete`.
-  from below it are deleted (2026-10-14,
+  Bootstrap now re-reads the effective version after installing and, on
-  https://git.eeqj.de/sneak/sfdupes/issues/106)
+  a mismatch, prints both paths and both versions to stderr and exits
-
+  non-zero instead of claiming success; it does not reorder anyone's
-- with `-x`, `scan` keeps the records below a directory on another filesystem
+  `PATH` or delete their binary. The `--version` call keeps its stderr
-  under its operand, and counts their sizes in the size census, instead of
+  connected, so a present-but-broken binary says why rather than
-  deleting them (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
+  reinstalling forever in silence, and is bounded by `timeout(1)` where
-
+  that exists, so a wedged binary cannot hang bootstrap. `git`, `make`
-- the hash and content phases open each file without blocking and check that it
+  and `go` keep their presence-only checks and now say why in a
-  is still a regular file, so one replaced by a FIFO or device node after it was
+  comment: they are host package-manager tools the repo deliberately
-  statted is warned about and skipped instead of hanging the scan (2026-10-14,
+  does not pin, with `go.mod` governing the language version and the
-  https://git.eeqj.de/sneak/sfdupes/issues/100)
+  digest-pinned images covering reproducible builds. Verified on this
-
+  host by bootstrapping from v2.10.1 to v2.12.2 and running it again to
-- the scan lock file is named from the database's path with symlinks resolved,
+  a no-op, plus stub runs of the real script under `dash` covering a
-  after `scan` creates the database file if it is missing, so a scan through a
+  thirteen-input parse matrix (absent, older, newer, host-style,
-  symlink to the database fails while another scan holds the lock (2026-10-14,
+  image-style, leading-`v`, stderr-only, empty, non-zero exit, impostor
-  https://git.eeqj.de/sneak/sfdupes/issues/101)
+  binary, `(devel)`, trailing `version`), a shadowed install that must
-
+  exit non-zero, an install destination not on `PATH` at all, `GOBIN`
-- on a terminal, a warning for the last file of a phase no longer erases the
+  set, and a wedged binary that must hit the timeout; `make check` and
-  phase's finished bar: the bar is drawn again below the warning (2026-10-14,
+  `make lint` are clean at v2.12.2, so v2.10.1 was not hiding any
-  https://git.eeqj.de/sneak/sfdupes/issues/102)
+  findings on `main`
-
-- `scan` records mtime to the nanosecond, as whole seconds in `mtime` plus
-  `mtime_nsec`, and compares it at that resolution, so a same-size rewrite
-  within the same second is re-hashed (2026-10-07,
-  https://git.eeqj.de/sneak/sfdupes/issues/12)
-
-- cut the narration from `TODO.md` Completed Steps and from the comments in
-  `script/`, `Dockerfile` and `Dockerfile.lint` (gone since
-  https://git.eeqj.de/sneak/sfdupes/issues/95); §Workflow now branches from and
-  merges to `next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
-
-- `make test-race` ran the test suite under the race detector in a cgo-enabled
-  container, outside `make check` (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/18). Since
-  https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
-  under the race detector, in the `Dockerfile`'s Debian-based `test` phase, and
-  `make test-race` is gone
-
-- a bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
-  rather than serve the gates from cache (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/39). Since
-  https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
-  `REPO_POLICIES.md` requires, and may serve the gates from cache; the builds in
-  `script/` pass a new `CACHEBUST` value, so theirs always run
-
-- `make fmt` and `make fmt-check` run prettier over all Markdown, and CI checks
-  it; all Markdown reformatted (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/19)
-
-- `script/lint` writes no image, so a run no longer leaves an untagged one
-  behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
-
-- tests cover a missing database, `scan` keeping stdout empty, its skip warning,
-  the `report` and `trees` summary lines, and every subcommand going through
-  `runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
-
-- `.golangci.yml` replaced with the current canonical copy, which uses
-  `gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/26)
-
-- a test fails when either `hashWorker` cancellation check in `scan.go` is
-  removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)
-
-- a database path holding `?`, `#` or `%` opens exactly the file it names
-  (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
-
-- `scan` rejects `--workers` below 1 as a usage error instead of running
-  single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
-
-- a test fails when either walk cancellation check in `scan.go` is removed
-  (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
-
-- test that `scan` refuses a database with another schema version (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/64)
-
-- correct four inaccurate comments in `cancel_test.go` and rename
-  `walkCancelInFlightDirs` to `walkCancelInFlightFiles` (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/33)
-
-- test the `-x` filesystem-boundary rules in `subdirJob` (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/17)
-
-- `scan` creates the schema in one transaction; a version-0 database with a
-  `files` table is refused with a clear schema-version error (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/11)
-
-- README documents install, Docker, a daily cron scan and how to read and check
-  the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
-
-- the `Dockerfile` build stage kept the Go module cache out of `builder`'s home
-  and copied the sources with `--chown`, so no `chown -R` walked them
-  (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/43). Since
-  https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
-  nothing changes owner: the build stage only compiles, as root, and the tests
-  run as `nobody` in the `test` phase
-
-- `--version` prints `sfdupes VERSION` to stdout; README documents it and
-  `--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
-
-- `scan` stops cleanly on `SIGINT` or `SIGTERM`: commits what it has hashed,
-  deletes nothing more, exits 1 (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/5)
-
-- `report` and `trees` stream the records instead of holding them all in memory;
-  the schema gains the `files_signature` index (2026-10-04,
-  https://git.eeqj.de/sneak/sfdupes/issues/14)
-
-- progress prints at once on a non-terminal, uses a real terminal test, and
-  prints warnings through a spinner instead of racing its redraw (2026-10-03,
-  https://git.eeqj.de/sneak/sfdupes/issues/13)
-
-- warn about and skip symlink, socket, FIFO, device and `.zfs` operands, keeping
-  the records beneath them (2026-10-03,
-  https://git.eeqj.de/sneak/sfdupes/issues/9)
-
-- `scan` holds a lock on a lock file beside the database for its whole run, so a
-  second `scan` fails at once with exit 1 (2026-10-03,
-  https://git.eeqj.de/sneak/sfdupes/issues/53)
-
-- test stdout write failures in `report` and `trees`; README states that
-  `| head` ends sfdupes by `SIGPIPE` and `>&-` writes to `/dev/null`
-  (2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)
-
-- `report` and `trees` open the database read-only, and `scan` leaves it out of
-  WAL mode, so reading needs only read access (2026-10-03, closes
-  https://git.eeqj.de/sneak/sfdupes/issues/8)
-
-- escape tabs, newlines, carriage returns and backslashes in report, trees and
-  warning paths; the root directory's path is `/` (2026-10-03,
-  https://git.eeqj.de/sneak/sfdupes/issues/7)
-
-- stamp the git tag or short commit in a plain `docker build .` instead of `dev`
-  (2026-10-02, branch `next`, closes
-  https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` sends `.git`
-  without `.git/config`; the build stage stamps the `VERSION` build argument,
-  else `git describe --tags --always`, and fails if the context carries `.git`
-  and the version is still empty, `dev` or `unknown`. CI checks out the full
-  history (`fetch-depth: 0`) so it stamps the same value as `make build`.
-
-- replace the 1 KiB end-window sampling with the head/tail plus content-hash
-  ladder (2026-09-22, branch `next`, closes
-  https://git.eeqj.de/sneak/sfdupes/issues/61); README "Duplicate detection"
-  documents every rung. A file under 10 MiB is hashed in full, and its `head`,
-  `tail` and `content` all hold that hash. A larger file gets only its 64 KiB
-  `head` and `tail` in the hash phase; the content phase, after the update
-  phase, reads it for `content` (the whole file below 50 MiB, gigabyte-spaced 1
-  MiB samples at or above) only when its size, `head` and `tail` match another
-  record's from this scan or an earlier one, and never reads a file gone or
-  changed since its record was written. `report` and `trees` leave out any
-  record without a `content` hash. The `content` column is part of the version 1
-  schema.
-
-- remove the dead `files.dat` references from `Makefile`, `.gitignore` and
-  `.dockerignore` (2026-09-21, branch `next`, closes
-  https://git.eeqj.de/sneak/sfdupes/issues/22)
-
-- fix the lint-image pin comments and `FROM` form in `Dockerfile` and
-  `Dockerfile.lint` (2026-08-10, branch `next`, closes
-  https://git.eeqj.de/sneak/sfdupes/issues/25): both pins became the policy
-  `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
-  without the false `(Debian-based)` note or the tag. Since
-  https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
-  holds the only golangci-lint pin, in that form, so `Dockerfile.lint` and
-  `script/verify-lint-image-pin`, which compared the two pins, are gone.
-
-- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
-  branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
-  owner ruling the linter is never installed on a host, and
-  `golangci-lint config verify` runs before `golangci-lint run`.
-  `script/bootstrap` stopped installing or pinning the linter, and
-  `script/verify-linter-pin` was retired. Since
-  https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the
-  `Dockerfile`'s `lint` phase, which `script/lint` builds alone and the build
-  stage depends on through `COPY --from=lint /src/go.sum /dev/null`;
-  `Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside
-  an image build may run docker, so the phase calls `golangci-lint` directly.
-
-- install the Docker build stage's prerequisites by running `script/bootstrap`
-  instead of `apk add --no-cache make` inline (2026-08-09, branch
-  `dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42),
-  with `script/verify-linter-pin` failing the build unless the linter copied
-  from the lint stage was the version `script/bootstrap` pinned. Builds then
-  took up to 5m14s cold, mostly in a `chown -R` of the module cache, filed as
-  https://git.eeqj.de/sneak/sfdupes/issues/43. Since
-  https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
-  `make` with `apk add --no-cache` and only compiles; the `lint` and `test`
-  phases are the gates
-- bust the Docker layer cache for the gate steps, so `script/cibuild` and
-  `script/docker` cannot report a green they did not earn (2026-08-09, branch
-  `cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the
-  `Dockerfile` copies the tree before its gates, so on an unchanged tree Docker
-  served them from cache and the build exited 0 having run nothing. The fix was
-  a `CHECK_EPOCH` build argument; since
-  https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
-  passes a new `CACHEBUST` value instead, which every stage of the `Dockerfile`
-  declares first. Run as root, the tests fail
-  `TestScanHardlinkRunFailsTogether`, because root reads through the `chmod(0)`
-  the test relies on, so the `test` phase runs them as `nobody`
-- check the installed golangci-lint version in `script/bootstrap` instead of
-  only its presence (2026-08-09, branch `bootstrap-version-check`, closes
-  https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in
-  `GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it, and
-  any installed version that is not the pin — older, newer, absent or
-  unparseable — is reinstalled. `go install` writes into `GOBIN` (or
-  `GOPATH/bin`) while `make lint` runs the first `golangci-lint` on `PATH`, so
-  bootstrap re-reads the effective version after installing and, on a mismatch,
-  prints both paths and both versions and exits non-zero; it does not reorder
-  `PATH` or delete anyone's binary. The `--version` call keeps its stderr and is
-  bounded by `timeout(1)` where that exists. `git`, `make` and `go` keep
-  presence-only checks. Verified by bootstrapping this host from v2.10.1 to
-  v2.12.2 and again to a no-op, and by stub runs of the script under `dash`
-  covering a thirteen-input version-parse matrix, a shadowed install that must
-  exit non-zero, an install destination not on `PATH`, `GOBIN` set, and a wedged
-  binary that must hit the timeout; `make check` and `make lint` are clean at
-  v2.12.2, so v2.10.1 was not hiding any findings on `main`
 - unwind the hash worker pool on the error path (2026-08-09, branch
-  `hash-pool-cleanup`, closes https://git.eeqj.de/sneak/sfdupes/issues/6): the
+  `hash-pool-cleanup`, closes #6): `hashPhase` used to return the
-  pool is now an owned, context-aware `hashPool`: every blocking send in the
+  moment `recordRun` failed and abandon the pool — the feeder parked
-  feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path
+  forever on a full `jobs` channel and every worker on a full
-  out, and `hashPhase` defers `pool.stop()`, which cancels and then drains
+  `results` channel. That only stopped being invisible when #4 landed
-  `results` until the last goroutine has exited — draining is what frees a
+  and `runScan` began unwinding instead of calling `os.Exit`. The
-  worker already parked on a send. `ctx` is threaded from `cmd.Context()`
+  pool is now an owned, context-aware `hashPool`: every blocking send
-  through `runScan`, `syncScan`, both worker pools and the whole database layer,
+  in the feeder and the workers selects on `ctx.Done()`, `jobs` is
-  as the first parameter everywhere. The walk pool gets the same treatment plus
+  closed on every path out, and `hashPhase` defers `pool.stop()`,
-  a `ctx.Err()` guard after the walk: a cancelled walk yields a partial size
+  which cancels and then drains `results` until the last goroutine
-  census, and every file it never reached looks vanished to the update phase.
+  has exited — draining is what frees a worker already parked on a
-  That phase's own `BeginTx` also fails on the cancelled context before deleting
+  send. `ctx` is threaded from `cmd.Context()` through `runScan`,
-  anything, but the guard is the barrier that still holds once an interrupted
+  `syncScan`, both worker pools and the whole database layer (it is
-  scan may commit what it has. Tests drive `run(scan)` against a database whose
+  the first parameter everywhere), so #5 can hand this path a signal
-  insert trigger aborts and assert that the scan fails instead of hanging and
+  and needs to add nothing else. The walk pool never leaked, because
-  that `runtime.NumGoroutine()` polls back to its pre-scan baseline; others
+  `walkPhase` always drains its events to close, but it has the same
-  cancel a scan part-way through the walk, deterministically, by counting its
+  unbounded-send shape and #5 will give it an early return, so it
-  own consultations of `ctx.Done()`, and assert that it stops at the guard
+  gets the same treatment plus a `ctx.Err()` guard after the walk: a
-  holding a partial census and a still-populated record index, with every record
+  cancelled walk yields a partial size census, and every file it never
-  intact. Direct tests of `sendEvent`, the walk workers, `dispatchDirs`,
+  reached looks vanished to the update phase. That phase's own
-  `feedHashJobs`, `hashWorker` and `hashPhase` cover the remaining cancellation
+  `BeginTx` fails on the same cancelled context before deleting
-  branches of both pools
+  anything, so the guard is defence in depth rather than the only
-- guarantee the database is closed on every fatal exit path (2026-08-09, branch
+  barrier — but it is the one that survives #5 deciding an interrupted
-  `db-close-on-fatal`, closes https://git.eeqj.de/sneak/sfdupes/issues/4):
+  scan may commit what it has. Tests drive `run(scan)` against a
-  `fatalf` and its `os.Exit(1)` are gone, so the deferred `db.Close()` — and
+  database whose insert trigger aborts, and assert both that the scan
-  with it the SQLite WAL checkpoint — now actually runs when a subcommand fails;
+  fails instead of hanging and that `runtime.NumGoroutine()` polls
-  `runScan`, `runReport`, `runTrees`, `loadRecords` and `resolveRoots` return
+  back to its pre-scan baseline; a second set cancels a scan part-way
-  errors instead. The single exit point is `run` in `main.go`: it maps a
+  through the walk — deterministically, by counting the scan's own
-  `fatalError` (anything a subcommand returned) to exit 1 and cobra's own
+  consultations of `ctx.Done()` rather than racing a timer — and
-  argument and flag errors to exit 2, which keeps a runtime failure from being
+  asserts that it stops at the guard holding a partial census and a
-  reported as a usage error or printing the usage text. New `main_test.go`
+  still-populated record index, with every record intact. The
-  drives the CLI in-process and asserts the exit codes from README §Error
+  remaining cancellation branches of both pools are covered by direct
-  handling plus the stdout/stderr split, including that a fatal error raised
+  tests of `sendEvent`, the walk workers, `dispatchDirs`,
-  after the database is open leaves no `-wal`/`-shm` sidecar behind for `scan`,
+  `feedHashJobs`, `hashWorker` and `hashPhase`
-  `report` or `trees`
+- guarantee the database is closed on every fatal exit path
-- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
+  (2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and
-  `golangci-v2.12.2`, merged as `38a01bd`, closes
+  its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with
-  https://git.eeqj.de/sneak/sfdupes/issues/3): bumped the pinned linter in the
+  it the SQLite WAL checkpoint — now actually runs when a subcommand
-  `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, and
+  fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and
-  replaced `.golangci.yml` with the canonical file — the linter settings (`lll`,
+  `resolveRoots` return errors instead. The single exit point is `run`
-  `funlen`, `cyclop`, `dupl` thresholds) now live under `linters.settings` per
+  in `main.go`: it maps a `fatalError` (anything a subcommand
-  the v2 schema, so they are actually applied; no new lint findings surfaced
+  returned) to exit 1 and cobra's own argument and flag errors to exit
-- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
+  2, which keeps a runtime failure from being reported as a usage
-  like the other managed repos (2026-07-26, commit `3abeacf`, closes
+  error or printing the usage text. New `main_test.go` drives the CLI
-  https://git.eeqj.de/sneak/sfdupes/issues/1): all 12 `script/` entrypoints
+  in-process and asserts the exit codes from README §Error handling
-  exist (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
+  plus the stdout/stderr split, including that a fatal error raised
-  `fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`)
+  after the database is open leaves no `-wal`/`-shm` sidecar behind
-  and every Makefile target is now a thin shim over them
+  for `scan`, `report` or `trees`
+- update golangci-lint to v2.12.2 with the canonical config
+  (2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
+  closes #3): bumped the pinned linter in the `Dockerfile` lint
+  stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
+  `.golangci.yml` with the canonical file — the linter settings
+  (`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
+  `linters.settings` per the v2 schema, so they are actually
+  applied; no new lint findings surfaced
+- convert Makefile targets to scripts-to-rule-them-all `script/`
+  entrypoints like the other managed repos (2026-07-26, commit
+  `3abeacf`, closes #1): all 12 `script/` entrypoints exist
+  (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`,
+  `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
+  `install-precommit`) and every Makefile target is now a thin shim
+  over them, matching the other managed repos
 - make the binary the default Make target (2026-07-24, branch
-  `make-default-target`): plain `make` now builds `sfdupes` (previously it ran
+  `make-default-target`): plain `make` now builds `sfdupes`
-  `check` plus `build`); `make build` remains as an alias
+  (previously it ran `check` plus `build`); `make build` remains as
-- scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
+  an alias
-  `scan-wide-phases`): all operands seed the shared walk pool and every pass
+- scan-wide phases, concurrent operands, batched updates (2026-07-24,
-  runs once over the whole scan, so totals and ETAs are scan-global; the
+  branch `scan-wide-phases`): all operands seed the shared walk pool
-  per-operand walk/hash/update cycles and their stderr announcements are gone;
+  and every pass runs once over the whole scan, so totals and ETAs
-  the update pass commits in batched transactions — the filesystem is
+  are scan-global; the per-operand walk/hash/update cycles and their
-  authoritative and the database an eventually-consistent reflection, so
+  stderr announcements are gone; the update pass commits in batched
-  scan-level atomicity is not required
+  transactions — the filesystem is authoritative and the database an
+  eventually-consistent reflection, so scan-level atomicity is not
+  required
 - split the stat pass back out of the walk (2026-07-24, branch
-  `parallel-phases`): phases are strictly sequential again — walk, stat, hash,
+  `parallel-phases`): phases are strictly sequential again — walk,
-  update per operand — with parallelism only inside each phase; the walk
+  stat, hash, update per operand — with parallelism only inside each
-  enumerates paths with per-directory workers and the stat pass lstats them with
+  phase; the walk enumerates paths with per-directory workers and the
-  per-file workers, restoring the exact total/ETA stat bar
+  stat pass lstats them with per-file workers, restoring the exact
-- announce each operand on stderr before its passes (2026-07-24, branch
+  total/ETA stat bar
-  `scan-operand-progress`): with per-operand walk/hash/update cycles, a
+- announce each operand on stderr before its passes (2026-07-24,
-  multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
+  branch `scan-operand-progress`): with per-operand walk/hash/update
-  whole run's
+  cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
-- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
+  that looked like the whole run's — an operator watching operand 3 of
-  goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
+  14 hash 300k files concluded 20M files were being skipped
-  4h on a ZFS server); it is now a per-directory worker-pool traversal that
+- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
-  records size/mtime during the walk (folding away the separate stat pass,
+  was a single goroutine and took hours at ~20M files on a busy pool
-  halving metadata I/O), and each `PATH` operand commits in its own transaction
+  (observed: 22M files in 4h on a ZFS server); it is now a
-  so an interrupted scan keeps completed operands
+  per-directory worker-pool traversal that records size/mtime during
+  the walk (folding away the separate stat pass, halving metadata
+  I/O), and each `PATH` operand commits in its own transaction so an
+  interrupted scan keeps completed operands
 
-- persistent scan database (2026-07-24, branch `persistent-database`): `scan`
+- persistent scan database (2026-07-24, branch `persistent-database`):
-  now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
+  `scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
-  disabled) keyed by absolute path that survives between runs — a rescan hashes
+  Go, cgo stays disabled) keyed by absolute path that survives between
-  only new or changed files (by mtime/size), deletes records for files vanished
+  runs — a rescan hashes only new or changed files (by mtime/size),
-  from under the scanned operands, and leaves records outside them untouched, so
+  deletes records for files vanished from under the scanned operands,
-  `scan` can be cronned daily; `report` and `trees` read the database (no
+  and leaves records outside them untouched, so `scan` can be cronned
-  positional arguments) instead of a scan stream. Database at
+  daily; `report` and `trees` read the database (no positional
-  `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
+  arguments) instead of a scan stream. Database at
-  journaling plus a single-transaction update keep a report run during a scan
+  `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
-  safe
+  WAL journaling plus a single-transaction update keep a report run
-- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
+  during a scan safe
-  and push `main` plus tags (2026-07-23)
+- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
+  `v0.0.1`, and push `main` plus tags (2026-07-23)
 - `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
-  `PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
+  `PATH...` operands via cobra flags replacing the `/srv` `-root`
-  `-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
+  default; new `-x`/`--one-file-system` flag (GNU convention) to stop
-  boundaries, which are crossed by default
+  at filesystem boundaries, which are crossed by default
 - bring the repo into full policy compliance (2026-07-23, branch
   `repo-policy-compliance`; checklist below)
-- `git init` with README-only first commit; code baseline committed on `main`
+- `git init` with README-only first commit; code baseline committed on
-  (2026-07-22)
+  `main` (2026-07-22)
 - implement `scan`, `report`, and `trees` subcommands (pre-git history)
 
 # Future Steps
 
-- possible later features (explicitly out of scope per README): full-content
+- possible later features (explicitly out of scope per README):
-  verification of candidates, removal-script helpers
+  full-content verification of candidates, removal-script helpers
 
 # Repo Policy Compliance
 
-Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
+Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing
-checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
+repo checklist, and the Go styleguide. Code is already gofmt-clean, so no
-formatting commit is needed.
+standalone formatting commit is needed.
 
-- [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
+- [x] `.gitignore` missing — the compiled `sfdupes` binary and
-      untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
+      `files.dat` sit untracked in the tree; needs OS/editor/Go
+      artifacts plus secrets patterns
 - [x] `.editorconfig` missing
-- [x] `LICENSE` missing and README has no License section (MIT assumed from
+- [x] `LICENSE` missing and README has no License section (MIT assumed
-      house convention — user to confirm)
+      from house convention — user to confirm)
 - [x] `REPO_POLICIES.md` missing from repo root
-- [x] `.golangci.yml` missing (install canonical copy); code must then pass
+- [x] `.golangci.yml` missing (install canonical copy); code must then
-      `make lint` (150 findings fixed; `make lint` is clean)
+      pass `make lint` (150 findings fixed; `make lint` is clean)
-- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
+- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`,
-      `docker`, `hooks`; `check` currently depends on `build`, which writes the
+      `fmt-check`, `docker`, `hooks`; `check` currently depends on
-      binary (`make check` must not modify files)
+      `build`, which writes the binary (`make check` must not modify
-- [x] no tests — `go test ./...` has nothing to run; policy requires real tests
+      files)
-      with a 30-second timeout and the conditional `-v` rerun pattern (suite
+- [x] no tests — `go test ./...` has nothing to run; policy requires
-      covers parsing, grouping, digests, suppression, hashing, and the scan
+      real tests with a 30-second timeout and the conditional `-v`
-      pipeline; 64% coverage)
+      rerun pattern (suite covers parsing, grouping, digests,
-- [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
+      suppression, hashing, and the scan pipeline; 64% coverage)
-      lint stage, build stage running `make check`
+- [x] `Dockerfile` missing — Go multistage with hash-pinned images:
+      fail-fast lint stage, build stage running `make check`
 - [x] `.dockerignore` missing
-- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
+- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push,
-      action pinned by commit SHA)
+      checkout action pinned by commit SHA)
 - [x] README lacks required sections: Description first line
-      (name/purpose/category/license/author), Getting Started, Rationale, TODO,
+      (name/purpose/category/license/author), Getting Started,
-      License, Author
+      Rationale, TODO, License, Author
-- [x] README non-goal "no git repository setup and no CI" is stale now that the
+- [x] README non-goal "no git repository setup and no CI" is stale now
-      repo is under git with CI
+      that the repo is under git with CI
-- [x] pre-commit hook not installed (`make hooks` once the target exists)
+- [x] pre-commit hook not installed (`make hooks` once the target
+      exists)
 
 Accepted divergences (no action):
 
-- flat single-package layout with `.go` files in the repo root — fine for a
+- flat single-package layout with `.go` files in the repo root — fine
-  small single-binary tool per the Go styleguide; the tracker audit agrees
+  for a small single-binary tool per the Go styleguide; the tracker
+  audit agrees
+- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0`
+  (pure-Go builds) and the race detector requires cgo

cancel_test.go

@@ -4,35 +4,20 @@ import (
 	"context"
 	"database/sql"
 	"errors"
-	"fmt"
 	"os"
-	"os/signal"
 	"path/filepath"
-	"slices"
 	"strconv"
-	"strings"
 	"sync"
 	"sync/atomic"
-	"syscall"
 	"testing"
 	"time"
 )
 
-// This file gathers the tests for scan cancellation and worker-pool
+// poolUnwind bounds how long a goroutine is given to leave a pool
-// unwinding. Everything it exercises lives in scan.go, so by the repo's
+// after its context is cancelled. Only a failing run ever waits this
-// convention of one test file per source file it would belong in
+// long: a pool that ignored its cancellation parks forever, and this
-// scan_test.go. It is kept separate on purpose: cancellation behaviour
+// is what turns that into a failed assertion instead of a suite that
-// cuts across both the walk pool and the hash pool as a single concern,
+// hangs until the test binary's own timeout.
-// and scan_test.go is already over 1,600 lines. That is the deliberate
-// exception the convention otherwise expects to be stated.
-
-// poolUnwind bounds how long a test waits for a cancellation to take
-// effect: for a goroutine to return or a channel to close once its
-// context is cancelled, or for a signal to cancel the scan's context.
-// Only a failing run waits this long, and the bound is what makes that
-// failure an assertion instead of a hang. A call made without it, as
-// most of this file's scans are, has no bound: a regression that parks
-// it is caught only as the test binary's own timeout.
 const poolUnwind = 2 * time.Second
 
 // walkClock is a context whose cancellation is driven by the scan's
@@ -43,14 +28,9 @@ const poolUnwind = 2 * time.Second
 //
 // The accounting behind the n chosen by each test: every blocking
 // channel operation in the walk selects on Done, so the walk spends
-// one consultation per file event plus a couple per directory. The
+// one consultation per file event plus a couple per directory, while
-// index load that runs ahead of it also consults Done, but a bounded
+// the index load that runs ahead of it spends a small fixed number
-// number of times that does not grow with the record count. The tests
+// (three) whatever the record count.
-// depend on that property, not on the bound's exact value: each test
-// sets n from the consultations of the walk, plus those of the hash
-// phase when it cancels mid-hash, far from both ends of the phase it
-// interrupts, so the cancellation lands inside that phase whatever the
-// record count.
 type walkClock struct {
 	n    int64
 	seen atomic.Int64
@@ -110,10 +90,7 @@ const (
 	walkCancelFilesPerDir  = 20
 	walkCancelFiles        = walkCancelDirs * walkCancelFilesPerDir
 	walkCancelWorkers      = 4
-	// The most files the walkCancelWorkers directories already in
+	walkCancelInFlightDirs = walkCancelWorkers * walkCancelFilesPerDir
-	// flight when the scan is cancelled can still emit, at
-	// walkCancelFilesPerDir each. A file count, not a directory count.
-	walkCancelInFlightFiles = walkCancelWorkers * walkCancelFilesPerDir
 )
 
 // walkCancelAtDone is the consultation on which the fixture's context
@@ -173,13 +150,9 @@ func assertRecordsIntact(t *testing.T, db *sql.DB, before []string) {
 // Every one of those records would look vanished to the update phase.
 // The guard is what stops the scan there, and this test is what
 // notices if it stops doing so: deleting the guard, or making it
-// unreachable, makes the scan carry its truncated view into the update
+// unreachable, makes the scan carry its truncated view into a later
-// phase, which counts every record the walk never reached for removal.
+// phase and fail there instead, with a wrapped error rather than the
-//
+// bare cancellation.
-// The syncScan call here is not bounded by poolUnwind: a regression
-// that left a worker pool parked would hang it, and that regression is
-// caught only by the test binary's own timeout, not by a quick
-// assertion.
 //
 //nolint:paralleltest // counts goroutines: must not run beside others
 func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
@@ -196,7 +169,7 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
 	base := baselineGoroutines(t)
 
 	st, err := syncScan(newWalkClock(walkCancelAtDone), db,
-		[]string{dir}, walkCancelWorkers, false, walkRules{})
+		[]string{dir}, walkCancelWorkers, false)
 
 	assertWalkGuardAborted(t, st, err)
 	assertRecordsIntact(t, db, before)
@@ -209,10 +182,10 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
 
 // assertWalkGuardAborted checks that the scan stopped at the post-walk
 // guard: with a census that is neither empty (the walk really ran)
-// nor complete (it really was cut short), and with no record counted
+// nor complete (it really was cut short), and with the guard's own
-// for removal. A removal count means the partial census was carried
+// bare cancellation as the error. A wrapped error means the partial
-// past the guard into the update phase, which is the failure this test
+// census was carried past the guard into the hash or update phase,
-// exists to catch.
+// which is the failure this test exists to catch.
 func assertWalkGuardAborted(t *testing.T, st scanStats, err error) {
 	t.Helper()
 
@@ -221,6 +194,12 @@ func assertWalkGuardAborted(t *testing.T, st scanStats, err error) {
 			err, context.Canceled)
 	}
 
+	if errors.Unwrap(err) != nil {
+		t.Errorf("syncScan reported %q, want the guard's bare "+
+			"cancellation: a wrapped error means the truncated census "+
+			"reached a later phase", err)
+	}
+
 	if st.unchanged == 0 {
 		t.Fatalf("stats = %+v: the census is empty, so the walk never "+
 			"ran and the guard was reached for the wrong reason", st)
@@ -232,11 +211,10 @@ func assertWalkGuardAborted(t *testing.T, st scanStats, err error) {
 	}
 
 	// The workers drop every directory still queued once the scan is
-	// cancelled, so only the files in the directories already in flight
+	// cancelled, so only the directories already in flight can add to
-	// can add to the census after the fact. A census beyond that bound
+	// the census after the fact. A census beyond that bound would mean
-	// would mean the cancellation was not observed where it should have
+	// the cancellation was not observed where it should have been.
-	// been.
+	limit := walkCancelAtDone + walkCancelInFlightDirs
-	limit := walkCancelAtDone + walkCancelInFlightFiles
 	if st.unchanged > limit {
 		t.Errorf("census covers %d files, want at most %d: the walk kept "+
 			"taking directories off the queue after cancellation",
@@ -268,8 +246,7 @@ func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
 	ctx, cancel := context.WithCancel(t.Context())
 	cancel()
 
-	st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false,
+	st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false)
-		walkRules{})
 	if !errors.Is(err, context.Canceled) {
 		t.Fatalf("syncScan on a cancelled context = %v, want %v",
 			err, context.Canceled)
@@ -283,256 +260,6 @@ func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
 	assertRecordsIntact(t, db, before)
 }
 
-// hashCancelAtDone is the consultation on which the mid-hash test's
-// context cancels itself. The walk of buildWalkCancelTree spends about
-// one per file and three per directory, and the hash phase then one per
-// file hashed, so this lands about half way through the hash phase.
-const hashCancelAtDone = walkCancelFiles + 3*walkCancelDirs +
-	walkCancelFiles/2
-
-// TestSyncScanCancelledMidHashKeepsHashedRecords cancels a first scan
-// part-way through its hash phase. The fixture holds fewer files than a
-// batch, so every file hashed is still waiting to be committed: the scan
-// must commit them all before it returns, and the next scan must hash
-// only the rest.
-func TestSyncScanCancelledMidHashKeepsHashedRecords(t *testing.T) {
-	t.Parallel()
-
-	dir := buildWalkCancelTree(t)
-	db := openTestDB(t)
-
-	st, err := syncScan(newWalkClock(hashCancelAtDone), db,
-		[]string{dir}, walkCancelWorkers, false, walkRules{})
-	if !errors.Is(err, context.Canceled) {
-		t.Fatalf("syncScan cancelled mid-hash = %v, want %v",
-			err, context.Canceled)
-	}
-
-	if st.walked != walkCancelFiles || st.added == 0 ||
-		st.added >= walkCancelFiles {
-		t.Fatalf("stats = %+v: want the walk complete and the hash phase "+
-			"cut short", st)
-	}
-
-	if got := len(dbRecords(t, db)); got != st.added {
-		t.Errorf("%d records after the cancelled scan, want the %d it hashed",
-			got, st.added)
-	}
-
-	hashed := st.added
-
-	st = syncTree(t, db, dir)
-	if st.added != walkCancelFiles-hashed || st.unchanged != hashed {
-		t.Errorf("next scan stats = %+v, want %d added %d unchanged",
-			st, walkCancelFiles-hashed, hashed)
-	}
-}
-
-// storedPaths opens the database at path as report does, which fails
-// unless it is a valid database, and returns its records' paths.
-func storedPaths(t *testing.T, path string) []string {
-	t.Helper()
-
-	db, err := openReportDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = db.Close() }()
-
-	return recordPaths(dbRecords(t, db))
-}
-
-// TestRunScanInterrupted calls the scan entrypoint with a context that
-// is already cancelled, as when a signal arrives at once. It must return
-// errInterrupted promptly, with its line on stderr after the one naming
-// the database and nothing on stdout, leave the database valid and as
-// it was, and leave nothing in the way of the next scan, which must
-// bring the database up to date.
-func TestRunScanInterrupted(t *testing.T) {
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	stdout := captureStdout(t)
-	stderr := captureStderr(t)
-	dir := buildSmokeTree(t)
-
-	err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
-		false)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	before := storedPaths(t, path)
-
-	// A vanished file and a new one: the interrupted scan records
-	// neither.
-	gone := filepath.Join(dir, "a", "unique.bin")
-
-	err = os.Remove(gone)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	added := writeFile(t, dir, "a/new.bin", pattern(50, 10))
-	shown := len(stderr())
-	done := make(chan struct{})
-
-	go func() {
-		defer close(done)
-
-		err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
-			false, false)
-	}()
-
-	awaitReturn(t, done, "runScan")
-
-	if !errors.Is(err, errInterrupted) {
-		t.Fatalf("runScan on a cancelled context = %v, want %v",
-			err, errInterrupted)
-	}
-
-	want := "scan: database " + path + " (from " + databaseEnv + ")\n" +
-		"scan: interrupted after 0 files\n"
-	if got := stderr()[shown:]; got != want {
-		t.Errorf("stderr = %q, want %q", got, want)
-	}
-
-	if got := stdout(); got != "" {
-		t.Errorf("stdout = %q, want nothing (data only)", got)
-	}
-
-	assertNoSidecars(t, path)
-
-	if got := storedPaths(t, path); !slices.Equal(got, before) {
-		t.Errorf("records = %q after the interrupted scan, want %q",
-			got, before)
-	}
-
-	err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
-		false)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	got := storedPaths(t, path)
-	if slices.Contains(got, gone) || !slices.Contains(got, added) {
-		t.Errorf("records = %q after the next scan, want %q gone and %q "+
-			"added", got, gone, added)
-	}
-}
-
-// TestRunScanInterruptedMidHash interrupts the scan entrypoint part-way
-// through its hash phase, after the database is open. It must return
-// errInterrupted, release the lock, end stderr with its line counting
-// every file the walk reached, write nothing to stdout, close the
-// database out of WAL mode, and keep the records it hashed.
-func TestRunScanInterruptedMidHash(t *testing.T) {
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	stdout := captureStdout(t)
-	stderr := captureStderr(t)
-	dir := buildWalkCancelTree(t)
-
-	err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
-		walkCancelWorkers, false, false)
-	if !errors.Is(err, errInterrupted) {
-		t.Fatalf("runScan interrupted mid-hash = %v, want %v",
-			err, errInterrupted)
-	}
-
-	holdScanLock(t, path)
-
-	want := fmt.Sprintf("scan: interrupted after %d files\n", walkCancelFiles)
-	if got := stderr(); !strings.HasSuffix(got, want) {
-		t.Errorf("stderr = %q, want it to end with %q", got, want)
-	}
-
-	if got := stdout(); got != "" {
-		t.Errorf("stdout = %q, want nothing (data only)", got)
-	}
-
-	assertNoSidecars(t, path)
-
-	db, err := openReportDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = db.Close() }()
-
-	// A plain close also removes the sidecars, but leaves WAL mode on.
-	var mode string
-
-	err = db.QueryRowContext(t.Context(), "PRAGMA journal_mode").Scan(&mode)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if mode != "delete" {
-		t.Errorf("journal mode = %q after the interrupted scan, want %q",
-			mode, "delete")
-	}
-
-	kept := len(dbRecords(t, db))
-	if kept == 0 || kept >= walkCancelFiles {
-		t.Errorf("%d records after the interrupted scan, want those it "+
-			"hashed: some but not all of the %d files", kept, walkCancelFiles)
-	}
-}
-
-// TestInterruptContextCatchesSIGTERM sends SIGTERM to the test process
-// while the scan's handler is installed, and checks that it cancels the
-// scan's context.
-//
-//nolint:paralleltest // signals the whole process: must not run beside a scan
-func TestInterruptContextCatchesSIGTERM(t *testing.T) {
-	// Caught here as well, so that a handler that misses SIGTERM fails
-	// this test instead of ending the test process.
-	caught := make(chan os.Signal, 1)
-	signal.Notify(caught, syscall.SIGTERM)
-
-	defer signal.Stop(caught)
-
-	ctx, stop := interruptContext(t.Context())
-	defer stop()
-
-	err := syscall.Kill(os.Getpid(), syscall.SIGTERM)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	select {
-	case <-ctx.Done():
-	case <-time.After(poolUnwind):
-		t.Fatal("SIGTERM did not cancel the scan's context")
-	}
-}
-
-// TestCommitFullBatchKeepsFailedBatch checks that a full batch whose
-// commit fails, as it does once the scan is interrupted, stays in the
-// batch, so that syncScan's final commit saves it.
-func TestCommitFullBatchKeepsFailedBatch(t *testing.T) {
-	t.Parallel()
-
-	s := &scanState{db: openTestDB(t)}
-	for i := range updateBatchSize {
-		s.batch = append(s.batch, scanRec{path: "/f" + strconv.Itoa(i)})
-	}
-
-	err := s.commitFullBatch(cancelledContext(t))
-	if !errors.Is(err, context.Canceled) {
-		t.Fatalf("commitFullBatch on a cancelled context = %v, want %v",
-			err, context.Canceled)
-	}
-
-	if len(s.batch) != updateBatchSize {
-		t.Errorf("batch holds %d records after the failed commit, want %d",
-			len(s.batch), updateBatchSize)
-	}
-}
-
 // drainClosed counts the values received from ch until it closes,
 // failing the test if it does not close within poolUnwind. A pool that
 // ignored its cancellation leaves its channel open with its goroutines
@@ -601,51 +328,20 @@ func TestSendEventAbandonsBlockedSend(t *testing.T) {
 	awaitReturn(t, done, "sendEvent")
 }
 
-// TestWalkOneDirStopsWhenCancelled checks that a cancelled scan stops
-// reading a directory instead of going through the rest of its
-// entries. A walk that kept going would return the subdirectory below
-// to descend into. Unlike a file event, that return is not a send the
-// cancellation can abandon, so the test catches the regression every
-// time.
-func TestWalkOneDirStopsWhenCancelled(t *testing.T) {
-	t.Parallel()
-
-	dir := t.TempDir()
-
-	err := os.Mkdir(filepath.Join(dir, "sub"), 0o750)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// Unbuffered and unread: on a cancelled scan every send gives up.
-	events := make(chan walkEvent)
-
-	subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false,
-		walkRules{}, events)
-	if len(subs) != 0 {
-		t.Errorf("cancelled walkOneDir returned %+v to descend into, "+
-			"want none", subs)
-	}
-}
-
 // TestWalkWorkersDropQueuedDirs checks that cancelled walk workers keep
 // reading jobs and drop the directories rather than stopping their
 // read: the range over jobs has to run out for the pool to tear down
-// and close its event stream. The queued directory does not exist, so
+// and close its event stream.
-// a worker that walked it anyway would send a warning before
-// walkOneDir's own cancellation check could stop it. On a cancelled
-// scan that send delivers or gives up at random, so with 64 jobs
-// queued the regression has a one in 2^64 chance of passing.
 func TestWalkWorkersDropQueuedDirs(t *testing.T) {
 	t.Parallel()
 
-	missing := filepath.Join(t.TempDir(), "missing")
+	dir := t.TempDir()
+	writeEmptyFiles(t, dir, walkCancelFilesPerDir)
 
-	jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false,
+	jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false)
-		walkRules{})
 
-	for range 64 {
+	for range 4 {
-		jobs <- dirJob{path: missing}
+		jobs <- dirJob{path: dir}
 	}
 
 	close(jobs)
@@ -673,7 +369,7 @@ func TestWalkWorkerAbandonsSubdirHandoff(t *testing.T) {
 	ctx, cancel := context.WithCancel(t.Context())
 	defer cancel()
 
-	jobs, subdirs, events := startWalkWorkers(ctx, 1, false, walkRules{})
+	jobs, subdirs, events := startWalkWorkers(ctx, 1, false)
 
 	// Fill the hand-back channel to its capacity — one slot per worker
 	// — so the worker's own hand-back is certain to block.
@@ -716,11 +412,7 @@ func TestDispatchDirsClosesJobsWhenCancelled(t *testing.T) {
 
 // TestFeedHashJobsClosesJobsWhenCancelled checks that the hash feeder
 // abandons the runs it has not queued yet and still closes the job
-// channel, which is what lets the workers' range terminate. The
+// channel, which is what lets the workers' range terminate.
-// receive on jobs below is not bounded: a feeder that returned without
-// closing jobs would leave that receive with no sender and no close, so
-// this regression is caught by the test binary's timeout rather than by
-// a bounded assertion.
 func TestFeedHashJobsClosesJobsWhenCancelled(t *testing.T) {
 	t.Parallel()
 
@@ -746,84 +438,41 @@ func TestFeedHashJobsClosesJobsWhenCancelled(t *testing.T) {
 // TestHashWorkerDropsQueuedRuns checks that a cancelled hash worker
 // keeps reading jobs and drops the runs rather than reading files
 // nobody wants the hashes of — while still letting the range run out
-// so the pool tears down. The hash function records that it was
+// so the pool tears down. The queued run names a file that does not
-// called, so a worker that hashed the queued run anyway is caught
+// exist, so a worker that hashed it anyway would produce a result.
-// every time.
 func TestHashWorkerDropsQueuedRuns(t *testing.T) {
 	t.Parallel()
 
 	done := make(chan struct{})
 	jobs := make(chan []fileRec, 1)
-	results := make(chan hashResult)
+	results := make(chan hashResult, 1)
 
-	jobs <- []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}}
+	run := []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}}
+
+	jobs <- run
 
 	close(jobs)
 
-	var hashed atomic.Bool
-
-	hash := func(path string, size int64) (string, string, string, error) {
-		hashed.Store(true)
-
-		return hashSignature(path, size)
-	}
-
 	go func() {
 		defer close(done)
 
-		hashWorker(cancelledContext(t), jobs, results, hash)
+		hashWorker(cancelledContext(t), jobs, results)
 	}()
 
 	awaitReturn(t, done, "hashWorker")
 
-	if hashed.Load() {
+	select {
-		t.Error("cancelled hash worker hashed the queued run, want it dropped")
+	case r := <-results:
+		t.Errorf("cancelled hash worker produced %+v, want the run dropped",
+			r)
+	default:
 	}
 }
 
-// TestHashWorkerAbandonsBlockedSend checks that a hash worker with a
-// result to deliver and nobody to deliver it to leaves once the scan
-// is cancelled, instead of holding the pool open. The scan tests do
-// not catch this: stop drains results, which frees a parked worker
-// anyway.
-func TestHashWorkerAbandonsBlockedSend(t *testing.T) {
-	t.Parallel()
-
-	ctx, cancel := context.WithCancel(t.Context())
-	defer cancel()
-
-	done := make(chan struct{})
-	jobs := make(chan []fileRec, 1)
-	// Unbuffered and unread, with jobs left open: the worker's only way
-	// out is the cancellation case beside its send.
-	results := make(chan hashResult)
-
-	jobs <- []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}}
-
-	// The scan is cancelled while the worker hashes, so the worker has
-	// already passed the check that drops queued runs.
-	hash := func(path string, size int64) (string, string, string, error) {
-		cancel()
-
-		return hashSignature(path, size)
-	}
-
-	go func() {
-		defer close(done)
-
-		hashWorker(ctx, jobs, results, hash)
-	}()
-
-	awaitReturn(t, done, "hashWorker")
-}
-
 // TestHashPhaseCancelledReturnsContextError checks the result loop's
 // own exit: with the pool cancelled, no result will ever arrive, and
 // the loop must leave through the cancellation rather than wait for a
-// receive that cannot happen. This call is not bounded by poolUnwind: a
+// receive that cannot happen.
-// loop that dropped its cancellation case would block on that receive,
-// so the regression surfaces as the test binary's timeout rather than
-// as a bounded assertion.
 func TestHashPhaseCancelledReturnsContextError(t *testing.T) {
 	t.Parallel()
 

db.go

@@ -6,15 +6,11 @@ import (
 	"errors"
 	"fmt"
 	"io/fs"
-	"net/url"
 	"os"
 	"path/filepath"
 	"slices"
 	"strconv"
-	"strings"
-	"time"
 
-	"golang.org/x/sys/unix"
 	// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
 	// disabled.
 	_ "modernc.org/sqlite"
@@ -36,51 +32,29 @@ const schemaVersion = 1
 // scan.
 const dbDirPerm = 0o755
 
-// dbFilePerm is the mode for a database file created by scan, the mode
-// SQLite gives a database file it creates.
-const dbFilePerm = 0o644
-
-// lockFilePerm is the mode for the scan lock file. Anyone who can open
-// the file can hold the lock and keep every scan from running, so it
-// is open to its owner only.
-const lockFilePerm = 0o600
-
 // createTableSQL is the schema applied to a fresh database. Paths are
-// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime
+// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
-// holds whole Unix seconds and mtime_nsec the nanoseconds within that
-// second.
 const createTableSQL = `
 CREATE TABLE files (
     path  BLOB PRIMARY KEY,
     size  INTEGER NOT NULL,
     mtime INTEGER NOT NULL,
-    mtime_nsec INTEGER NOT NULL,
     head  TEXT NOT NULL,
-    tail       TEXT NOT NULL,
+    tail  TEXT NOT NULL
-    content    TEXT NOT NULL
 ) WITHOUT ROWID
 `
 
-// createIndexSQL indexes the records by signature, so report can read
-// them in signature order without sorting the table.
-const createIndexSQL = `
-CREATE INDEX files_signature ON files (size, head, tail, content)
-`
-
 // upsertSQL inserts one file record, replacing any existing record for
 // the same path.
 const upsertSQL = `
-INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content)
+INSERT INTO files (path, size, mtime, head, tail)
-VALUES (?, ?, ?, ?, ?, ?, ?)
+VALUES (?, ?, ?, ?, ?)
 ON CONFLICT (path) DO UPDATE SET
     size = excluded.size, mtime = excluded.mtime,
-    mtime_nsec = excluded.mtime_nsec,
+    head = excluded.head, tail = excluded.tail
-    head = excluded.head, tail = excluded.tail,
-    content = excluded.content
 `
 
-// errNoDatabase reports a missing database file for report, trees and
+// errNoDatabase reports a missing database file for report/trees.
-// info.
 var errNoDatabase = errors.New(
 	"no database (run \"sfdupes scan\" first, or set " + databaseEnv + ")")
 
@@ -88,89 +62,26 @@ var errNoDatabase = errors.New(
 // does not understand.
 var errSchemaVersion = errors.New("unsupported database schema version")
 
-// errScanRunning reports that another scan holds the lock on the
-// database.
-var errScanRunning = errors.New("another scan is running")
-
 // databasePath resolves the database location: SFDUPES_DATABASE when
-// set and non-empty, the compiled-in default otherwise. It returns the
+// set and non-empty, the compiled-in default otherwise.
-// path, with the working directory, its symlinks resolved, put in front
+func databasePath() string {
-// of it when it is relative, and where the path came from, "default" or
-// "from SFDUPES_DATABASE", for the line each subcommand prints first on
-// stderr.
-func databasePath() (string, string, error) {
-	path, source := defaultDatabasePath, "default"
 	if p := os.Getenv(databaseEnv); p != "" {
-		path, source = p, "from "+databaseEnv
+		return p
 	}
 
-	if filepath.IsAbs(path) {
+	return defaultDatabasePath
-		return path, source, nil
-	}
-
-	// The working directory as the kernel reports it, with no symlink
-	// in it, as README "Database" says. os.Getwd may return PWD
-	// instead, which can name it through a symlink.
-	wd, err := unix.Getwd()
-	if err != nil {
-		return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
-	}
-
-	// Not filepath.Abs or filepath.Join: they clean the path, which
-	// drops each ".." together with the directory name before it. When
-	// that directory is a symlink, the cleaned path names another file.
-	// The working directory ends in "/" only when it is the root.
-	return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
 }
 
-// createDatabaseDirectory creates the directory the database at path
+// openDB opens the SQLite database at path with WAL journaling and a
-// goes in, and any directory above it, if missing. path is absolute, as
+// busy timeout, so a report can run while a cron scan is in progress.
-// databasePath returns it. The directory is path with its last element
+// It does not create or verify the schema.
-// cut off as text, by filepath.Split. filepath.Dir would clean it,
+func openDB(path string) (*sql.DB, error) {
-// dropping each ".." together with the directory name before it; when
+	dsn := "file:" + path +
-// that name is a symlink, the cleaned path names another directory than
+		"?_pragma=busy_timeout(10000)" +
-// the one the system reaches when it opens path.
-func createDatabaseDirectory(path string) error {
-	dir, _ := filepath.Split(path)
-
-	err := os.MkdirAll(dir, dbDirPerm)
-	if err != nil {
-		return fmt.Errorf("create database directory: %w", err)
-	}
-
-	return nil
-}
-
-// scanParams are the connection parameters for scan: read-write, with
-// WAL journaling and a busy timeout, so a report can run while a cron
-// scan is in progress. closeScanDatabase leaves WAL mode again.
-const scanParams = "_pragma=busy_timeout(10000)" +
 		"&_pragma=journal_mode(WAL)" +
 		"&_pragma=synchronous(NORMAL)"
 
-// reportParams are the connection parameters for report, trees and
+	db, err := sql.Open("sqlite", dsn)
-// info: read-only, with the same busy timeout. They set no journal
-// mode, because setting one is a write.
-const reportParams = "mode=ro" +
-	"&_pragma=busy_timeout(10000)" +
-	"&_pragma=query_only(1)"
-
-// openDB opens the SQLite database at path with the connection
-// parameters params. It does not create or verify the schema.
-func openDB(path, params string) (*sql.DB, error) {
-	// The path is escaped into a file: URI, so ?, # and % in it stay
-	// part of the file name. SQLite reads what follows file:// up to
-	// the next / as a host name, so an absolute path goes after an
-	// empty host (file:///abs) and a relative path goes without one
-	// (file:rel).
-	uri := url.URL{
-		Scheme:   "file",
-		OmitHost: !filepath.IsAbs(path),
-		Path:     path,
-		RawQuery: params,
-	}
-
-	db, err := sql.Open("sqlite", uri.String())
 	if err != nil {
 		return nil, fmt.Errorf("open database %s: %w", path, err)
 	}
@@ -183,70 +94,15 @@ func openDB(path, params string) (*sql.DB, error) {
 	return db, nil
 }
 
-// lockScanDatabase takes the lock that keeps a second scan off the
-// database at path: an exclusive flock(2) on the file beside it named
-// by the database's path with symlinks resolved and ".lock" appended,
-// so every path to one database names the same lock file. If missing,
-// the database's parent directory and an empty database file are
-// created first, so the lock file is named the same way before the
-// database is set up as after. A lock held by another scan fails at
-// once instead of waiting. The lock lasts until the returned file is
-// closed or the process ends. The lock file is never deleted: a scan
-// that deleted it would let the next scan lock a new file while
-// another still holds the old one.
-func lockScanDatabase(path string) (*os.File, error) {
-	err := createDatabaseDirectory(path)
-	if err != nil {
-		return nil, err
-	}
-
-	// Like SQLite, this follows a symlink at path and creates its
-	// target. SQLite opens an empty file as a new database.
-	//nolint:gosec // the operator chooses the database path
-	dbFile, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, dbFilePerm)
-	if err != nil {
-		return nil, fmt.Errorf("create database: %w", err)
-	}
-
-	_ = dbFile.Close()
-
-	resolved, err := filepath.EvalSymlinks(path)
-	if err != nil {
-		return nil, fmt.Errorf("resolve database path: %w", err)
-	}
-
-	lockPath := resolved + ".lock"
-
-	//nolint:gosec // the operator chooses the database path
-	f, err := os.OpenFile(lockPath, os.O_RDWR|os.O_CREATE, lockFilePerm)
-	if err != nil {
-		return nil, err
-	}
-
-	err = unix.Flock(int(f.Fd()), unix.LOCK_EX|unix.LOCK_NB)
-	if err != nil {
-		_ = f.Close()
-
-		if errors.Is(err, unix.EWOULDBLOCK) {
-			return nil, fmt.Errorf("%w (lock held on %s)",
-				errScanRunning, lockPath)
-		}
-
-		return nil, fmt.Errorf("lock %s: %w", lockPath, err)
-	}
-
-	return f, nil
-}
-
 // openScanDatabase opens the database for the scan subcommand, creating
 // the file, its parent directory, and the schema as needed.
 func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
-	err := createDatabaseDirectory(path)
+	err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
 	if err != nil {
-		return nil, err
+		return nil, fmt.Errorf("create database directory: %w", err)
 	}
 
-	db, err := openDB(path, scanParams)
+	db, err := openDB(path)
 	if err != nil {
 		return nil, err
 	}
@@ -261,63 +117,9 @@ func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
 	return db, nil
 }
 
-// databaseFiles returns, by device and inode, the files of the database
+// openReportDatabase opens an existing database for the report and
-// at path that the scan keeps open: the database file, its -wal and
+// trees subcommands. A missing database file is an error directing the
-// -shm files and its .lock file, each named from the database's path
+// user to run scan first; the schema version must match exactly.
-// with symlinks resolved, as SQLite and lockScanDatabase name them. The
-// scan calls it once the database and the lock file are open, so that
-// its walk passes over them however it reaches them, through a
-// symlinked directory included. A file that does not exist then is left
-// out, since no walk can reach it. There is no -journal file while the
-// database is in WAL mode.
-func databaseFiles(path string) ([]fileRec, error) {
-	resolved, err := filepath.EvalSymlinks(path)
-	if err != nil {
-		return nil, fmt.Errorf("resolve database path: %w", err)
-	}
-
-	var files []fileRec
-
-	for _, p := range []string{
-		resolved, resolved + "-wal", resolved + "-shm", resolved + ".lock",
-	} {
-		fi, err := os.Lstat(p)
-		if errors.Is(err, fs.ErrNotExist) {
-			continue
-		}
-
-		if err != nil {
-			return nil, fmt.Errorf("database: %w", err)
-		}
-
-		dev, ino := inodeOfInfo(fi)
-		files = append(files, fileRec{path: p, dev: dev, ino: ino})
-	}
-
-	return files, nil
-}
-
-// closeScanDatabase switches the database at path from WAL back to
-// rollback-journal mode and closes it. Out of WAL mode the database
-// file alone holds the whole database, so a reader needs no -wal or
-// -shm file beside it, nor write access to create them. The switch
-// fails while a report has the database open; the database then stays
-// in WAL mode, still readable, until a later scan closes it.
-func closeScanDatabase(ctx context.Context, db *sql.DB, path string) {
-	// Runs on the way out of a cancelled scan too.
-	_, err := db.ExecContext(context.WithoutCancel(ctx),
-		"PRAGMA journal_mode = DELETE")
-	if err != nil {
-		fmt.Fprintf(os.Stderr, "scan: database %s left in WAL mode: %v\n",
-			path, err)
-	}
-
-	_ = db.Close()
-}
-
-// openReportDatabase opens an existing database for the report, trees
-// and info subcommands. A missing database file is an error directing
-// the user to run scan first; the schema version must match exactly.
 func openReportDatabase(ctx context.Context,
 	path string,
 ) (*sql.DB, error) {
@@ -330,18 +132,12 @@ func openReportDatabase(ctx context.Context,
 		return nil, fmt.Errorf("database: %w", err)
 	}
 
-	db, err := openDB(path, reportParams)
+	db, err := openDB(path)
 	if err != nil {
 		return nil, err
 	}
 
 	v, err := userVersion(ctx, db)
-	if err == nil && v == 0 {
-		// An empty database passes this check and fails the version
-		// check below.
-		err = checkUnversioned(ctx, db)
-	}
-
 	if err != nil {
 		_ = db.Close()
 
@@ -368,11 +164,6 @@ func initSchema(ctx context.Context, db *sql.DB) error {
 
 	switch v {
 	case 0:
-		err = checkUnversioned(ctx, db)
-		if err != nil {
-			return err
-		}
-
 		return createSchema(ctx, db)
 	case schemaVersion:
 		return nil
@@ -382,65 +173,20 @@ func initSchema(ctx context.Context, db *sql.DB) error {
 	}
 }
 
-// checkUnversioned checks a database at user_version 0 before it is
-// taken for an empty one. createSchema creates the files table and
-// sets the version together, so a files table at version 0 was made by
-// something else. Adopting it could corrupt unrelated data, so that is
-// a schema-version error telling the operator to remove the file and
-// rescan.
-func checkUnversioned(ctx context.Context, db *sql.DB) error {
-	var name string
-
-	err := db.QueryRowContext(ctx,
-		"SELECT name FROM sqlite_master "+
-			"WHERE type = 'table' AND name = 'files'").Scan(&name)
-
-	switch {
-	case err == nil:
-		return fmt.Errorf(
-			"has a files table but no schema version; "+
-				"remove the file and rescan: %w", errSchemaVersion)
-	case errors.Is(err, sql.ErrNoRows):
-		return nil
-	default:
-		return fmt.Errorf("check for files table: %w", err)
-	}
-}
-
 // createSchema applies the schema to a fresh database and stamps the
-// schema version in one transaction, so a creation stopped partway, by
+// schema version.
-// an interrupt or an error, leaves an empty database the next scan
-// sets up, never a files table at version 0, which checkUnversioned
-// refuses.
 func createSchema(ctx context.Context, db *sql.DB) error {
-	tx, err := db.BeginTx(ctx, nil)
+	_, err := db.ExecContext(ctx, createTableSQL)
 	if err != nil {
 		return fmt.Errorf("create schema: %w", err)
 	}
 
-	defer func() { _ = tx.Rollback() }()
+	_, err = db.ExecContext(ctx,
-
-	_, err = tx.ExecContext(ctx, createTableSQL)
-	if err != nil {
-		return fmt.Errorf("create schema: %w", err)
-	}
-
-	_, err = tx.ExecContext(ctx, createIndexSQL)
-	if err != nil {
-		return fmt.Errorf("create schema: %w", err)
-	}
-
-	_, err = tx.ExecContext(ctx,
 		"PRAGMA user_version = "+strconv.Itoa(schemaVersion))
 	if err != nil {
 		return fmt.Errorf("set schema version: %w", err)
 	}
 
-	err = tx.Commit()
-	if err != nil {
-		return fmt.Errorf("create schema: %w", err)
-	}
-
 	return nil
 }
 
@@ -456,108 +202,17 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
 	return v, nil
 }
 
-// loadFileRows gives total the number of records, then streams every
+// loadFileRows reads every record from the files table.
-// record to fn in path order: byte order, which is the order of the
+func loadFileRows(ctx context.Context, db *sql.DB) ([]scanRec, error) {
-// primary key, so SQLite does not sort. The count and the records are
+	rows, err := db.QueryContext(ctx,
-// read in one transaction, so they agree while a scan is committing.
+		"SELECT path, size, mtime, head, tail FROM files")
-func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
-	fn func(r scanRec),
-) error {
-	// Everything goes through tx: db has one connection, so a query on
-	// db would wait for tx forever.
-	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
 	if err != nil {
-		return fmt.Errorf("read records: %w", err)
+		return nil, fmt.Errorf("read records: %w", err)
-	}
-
-	defer func() { _ = tx.Rollback() }()
-
-	var n int64
-
-	err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
-	if err != nil {
-		return fmt.Errorf("count records: %w", err)
-	}
-
-	total(n)
-
-	rows, err := tx.QueryContext(ctx,
-		"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
-			"FROM files ORDER BY path")
-	if err != nil {
-		return fmt.Errorf("read records: %w", err)
 	}
 
 	defer func() { _ = rows.Close() }()
 
-	for rows.Next() {
+	var recs []scanRec
-		var (
-			path      []byte
-			sec, nsec int64
-			r         scanRec
-		)
-
-		err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail,
-			&r.content)
-		if err != nil {
-			return fmt.Errorf("read record: %w", err)
-		}
-
-		r.path = string(path)
-		r.mtime = time.Unix(sec, nsec)
-		fn(r)
-	}
-
-	err = rows.Err()
-	if err != nil {
-		return fmt.Errorf("read records: %w", err)
-	}
-
-	return nil
-}
-
-// signatureRowsSQL selects every record's signature and path in
-// signature order, largest size first. That is the order of the
-// files_signature index read backwards, so SQLite reads the index alone
-// and returns the first row at once, with nothing to sort.
-const signatureRowsSQL = `
-SELECT size, head, tail, content, path
-FROM files
-ORDER BY size DESC, head DESC, tail DESC, content DESC
-`
-
-// loadSignatureRows gives total the number of records, then streams
-// every record's signature and path to fn, without its mtime, in the
-// order of signatureRowsSQL. The count and the records are read in one
-// transaction, so they agree while a scan is committing. An error from
-// fn stops the reading and is returned as it is.
-func loadSignatureRows(ctx context.Context, db *sql.DB,
-	total func(n int64), fn func(r scanRec) error,
-) error {
-	// Everything goes through tx: the report connection is the only
-	// one, so a query on db would wait for tx forever.
-	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
-	if err != nil {
-		return fmt.Errorf("read records: %w", err)
-	}
-
-	defer func() { _ = tx.Rollback() }()
-
-	var n int64
-
-	err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
-	if err != nil {
-		return fmt.Errorf("count records: %w", err)
-	}
-
-	total(n)
-
-	rows, err := tx.QueryContext(ctx, signatureRowsSQL)
-	if err != nil {
-		return fmt.Errorf("read records: %w", err)
-	}
-
-	defer func() { _ = rows.Close() }()
 
 	for rows.Next() {
 		var (
@@ -565,91 +220,21 @@ func loadSignatureRows(ctx context.Context, db *sql.DB,
 			r    scanRec
 		)
 
-		err = rows.Scan(&r.size, &r.head, &r.tail, &r.content, &path)
+		err = rows.Scan(&path, &r.size, &r.mtime, &r.head, &r.tail)
 		if err != nil {
-			return fmt.Errorf("read record: %w", err)
+			return nil, fmt.Errorf("read record: %w", err)
 		}
 
 		r.path = string(path)
-
+		recs = append(recs, r)
-		err = fn(r)
-		if err != nil {
-			return err
-		}
 	}
 
 	err = rows.Err()
 	if err != nil {
-		return fmt.Errorf("read records: %w", err)
+		return nil, fmt.Errorf("read records: %w", err)
 	}
 
-	return nil
+	return recs, nil
-}
-
-// countRecords returns the number of records in the database.
-func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
-	var n int64
-
-	err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
-	if err != nil {
-		return 0, fmt.Errorf("count records: %w", err)
-	}
-
-	return n, nil
-}
-
-// databaseCountsSQL counts the records, those with head and tail
-// hashes and those with a content hash, and adds up their sizes, in one
-// pass over the table. A SUM over no rows is NULL, so on an empty table
-// each SUM falls back to 0.
-const databaseCountsSQL = `
-SELECT COUNT(*),
-       COALESCE(SUM(head <> '' AND tail <> ''), 0),
-       COALESCE(SUM(content <> ''), 0),
-       COALESCE(SUM(size), 0)
-FROM files
-`
-
-// databaseCounts is what info reads from the database: its schema
-// version and the counts of databaseCountsSQL.
-type databaseCounts struct {
-	schemaVersion int
-	records       int64 // every record
-	headTail      int64 // records with head and tail hashes
-	content       int64 // records with a content hash
-	size          int64 // the sizes of all records added up
-}
-
-// readDatabaseCounts reads the schema version, then the counts of
-// databaseCountsSQL, in one read-only transaction, so both describe the
-// database at one moment while a scan is committing.
-func readDatabaseCounts(ctx context.Context,
-	db *sql.DB,
-) (databaseCounts, error) {
-	var c databaseCounts
-
-	// Everything goes through tx: the report connection is the only
-	// one, so a query on db would wait for tx forever.
-	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
-	if err != nil {
-		return c, fmt.Errorf("count records: %w", err)
-	}
-
-	defer func() { _ = tx.Rollback() }()
-
-	err = tx.QueryRowContext(ctx, "PRAGMA user_version").
-		Scan(&c.schemaVersion)
-	if err != nil {
-		return c, fmt.Errorf("read schema version: %w", err)
-	}
-
-	err = tx.QueryRowContext(ctx, databaseCountsSQL).
-		Scan(&c.records, &c.headTail, &c.content, &c.size)
-	if err != nil {
-		return c, fmt.Errorf("count records: %w", err)
-	}
-
-	return c, nil
 }
 
 // loadFileMeta streams every record's path, size, mtime, and whether
@@ -657,10 +242,10 @@ func readDatabaseCounts(ctx context.Context,
 // values, and skipping the hash columns keeps the scan's in-memory
 // index small on multi-million-file databases.
 func loadFileMeta(ctx context.Context, db *sql.DB,
-	fn func(path string, size int64, mtime time.Time, hashed bool),
+	fn func(path string, size, mtime int64, hashed bool),
 ) error {
 	rows, err := db.QueryContext(ctx,
-		"SELECT path, size, mtime, mtime_nsec, head <> '' FROM files")
+		"SELECT path, size, mtime, head <> '' FROM files")
 	if err != nil {
 		return fmt.Errorf("read records: %w", err)
 	}
@@ -670,95 +255,16 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
 	for rows.Next() {
 		var (
 			path        []byte
-			size, sec, nsec int64
+			size, mtime int64
 			hashed      int64
 		)
 
-		err = rows.Scan(&path, &size, &sec, &nsec, &hashed)
+		err = rows.Scan(&path, &size, &mtime, &hashed)
 		if err != nil {
 			return fmt.Errorf("read record: %w", err)
 		}
 
-		fn(string(path), size, time.Unix(sec, nsec), hashed != 0)
+		fn(string(path), size, mtime, hashed != 0)
-	}
-
-	err = rows.Err()
-	if err != nil {
-		return fmt.Errorf("read records: %w", err)
-	}
-
-	return nil
-}
-
-// contentGroupsSQL selects each group (the records sharing a size,
-// head, and tail) of records of at least headTailMin bytes that has more
-// than one record and at least one record without a content hash, with
-// the number of its records.
-const contentGroupsSQL = `
-SELECT size, head, tail, COUNT(*) AS records
-FROM files
-WHERE size >= ? AND head <> ''
-GROUP BY size, head, tail
-HAVING COUNT(*) > 1 AND SUM(content = '') > 0
-`
-
-// contentCandidatesSQL selects every record of the groups
-// contentGroupsSQL selects, with whether each record has a content
-// hash. SQLite does the grouping, so no other record's hashes are
-// loaded into memory; the rows come ordered by size, head, and tail, so
-// each group's rows arrive together.
-const contentCandidatesSQL = `
-SELECT f.path, f.size, f.mtime, f.mtime_nsec, f.head, f.tail, f.content <> ''
-FROM files AS f
-JOIN (` + contentGroupsSQL + `) AS g USING (size, head, tail)
-ORDER BY size, head, tail
-`
-
-// countContentCandidates returns the number of rows of
-// contentCandidatesSQL, adding up the records of the groups
-// contentGroupsSQL selects, so it neither joins nor sorts.
-func countContentCandidates(ctx context.Context, db *sql.DB) (int64, error) {
-	var n int64
-
-	err := db.QueryRowContext(ctx,
-		"SELECT COALESCE(SUM(records), 0) FROM ("+contentGroupsSQL+")",
-		headTailMin).Scan(&n)
-	if err != nil {
-		return 0, fmt.Errorf("count content candidates: %w", err)
-	}
-
-	return n, nil
-}
-
-// loadContentCandidates streams the rows of contentCandidatesSQL to fn:
-// each record, without its content hash, and whether it has one.
-func loadContentCandidates(ctx context.Context, db *sql.DB,
-	fn func(r scanRec, hashed bool),
-) error {
-	rows, err := db.QueryContext(ctx, contentCandidatesSQL, headTailMin)
-	if err != nil {
-		return fmt.Errorf("read records: %w", err)
-	}
-
-	defer func() { _ = rows.Close() }()
-
-	for rows.Next() {
-		var (
-			path      []byte
-			sec, nsec int64
-			r         scanRec
-			hashed    int64
-		)
-
-		err = rows.Scan(&path, &r.size, &sec, &nsec, &r.head, &r.tail,
-			&hashed)
-		if err != nil {
-			return fmt.Errorf("read record: %w", err)
-		}
-
-		r.path = string(path)
-		r.mtime = time.Unix(sec, nsec)
-		fn(r, hashed != 0)
 	}
 
 	err = rows.Err()
@@ -841,8 +347,8 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
 	defer func() { _ = st.Close() }()
 
 	for _, r := range upserts {
-		_, err = st.ExecContext(ctx, []byte(r.path), r.size,
+		_, err = st.ExecContext(ctx,
-			r.mtime.Unix(), r.mtime.Nanosecond(), r.head, r.tail, r.content)
+			[]byte(r.path), r.size, r.mtime, r.head, r.tail)
 		if err != nil {
 			return fmt.Errorf("upsert %s: %w", r.path, err)
 		}

db_test.go

@@ -5,28 +5,17 @@ import (
 	"database/sql"
 	"errors"
 	"fmt"
-	"io/fs"
-	"os"
 	"path/filepath"
 	"slices"
 	"strings"
 	"testing"
-	"time"
 )
 
-// testDBPath returns a database path inside a fresh temp dir, with
+// testDBPath returns a database path inside a fresh temp dir.
-// symlinks resolved as the scan lock resolves them, so the lock file
-// is this path plus ".lock" even where the temp dir is reached through
-// a symlink.
 func testDBPath(t *testing.T) string {
 	t.Helper()
 
-	dir, err := filepath.EvalSymlinks(t.TempDir())
+	return filepath.Join(t.TempDir(), "db.sqlite")
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	return filepath.Join(dir, "db.sqlite")
 }
 
 // openTestDB creates a fresh scan database in a temp dir.
@@ -44,47 +33,16 @@ func openTestDB(t *testing.T) *sql.DB {
 }
 
 func TestDatabasePath(t *testing.T) {
-	// README §Database: SFDUPES_DATABASE when set and non-empty, the
+	t.Setenv(databaseEnv, "")
-	// default otherwise, with the working directory, its symlinks
+
-	// resolved, put in front of a relative path. Nothing is cleaned, so
+	if got := databasePath(); got != defaultDatabasePath {
-	// a ".." stays.
+		t.Errorf("databasePath() = %q, want %q", got, defaultDatabasePath)
-	dir, err := filepath.EvalSymlinks(t.TempDir())
-	if err != nil {
-		t.Fatal(err)
 	}
 
-	t.Chdir(dir)
+	t.Setenv(databaseEnv, "/custom/place.sqlite")
 
-	fromEnv := "from " + databaseEnv
+	if got := databasePath(); got != "/custom/place.sqlite" {
-	cases := []struct {
+		t.Errorf("databasePath() = %q, want the env override", got)
-		env, path, source string
-	}{
-		{"", defaultDatabasePath, "default"},
-		{"/custom/place.sqlite", "/custom/place.sqlite", fromEnv},
-		{"/custom/link/../place.sqlite", "/custom/link/../place.sqlite",
-			fromEnv},
-		{"sub/place.sqlite", dir + "/sub/place.sqlite", fromEnv},
-		{"link/../place.sqlite", dir + "/link/../place.sqlite", fromEnv},
-	}
-
-	for _, tc := range cases {
-		t.Setenv(databaseEnv, tc.env)
-
-		path, source, err := databasePath()
-		if err != nil || path != tc.path || source != tc.source {
-			t.Errorf("databasePath() with %s=%q = %q, %q, %v; want %q, %q",
-				databaseEnv, tc.env, path, source, err, tc.path, tc.source)
-		}
-	}
-
-	// In the root directory the path gets one slash in front, not two.
-	t.Chdir("/")
-	t.Setenv(databaseEnv, "place.sqlite")
-
-	path, _, err := databasePath()
-	if err != nil || path != "/place.sqlite" {
-		t.Errorf("databasePath() in / = %q, %v; want %q",
-			path, err, "/place.sqlite")
 	}
 }
 
@@ -114,188 +72,9 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
 
 	defer func() { _ = db.Close() }()
 
-	if recs := dbRecords(t, db); len(recs) != 0 {
+	recs, err := loadFileRows(t.Context(), db)
-		t.Fatalf("records = %v, want none", recs)
+	if err != nil || len(recs) != 0 {
-	}
+		t.Fatalf("loadFileRows = %v, %v; want empty, nil", recs, err)
-}
-
-// TestScanCreatesDatabaseDirectoryAfterSymlink has a database path go
-// through a symlink to a directory, then "..", to a missing directory.
-// With srv/link a symlink to data/x, srv/link/../new/db.sqlite leads to
-// data/new/db.sqlite. lockScanDatabase and openScanDatabase, the two
-// places scan creates the database's directory, must each create the
-// database there, and nothing at srv/new, where the path leads once
-// cleaned as text.
-func TestScanCreatesDatabaseDirectoryAfterSymlink(t *testing.T) {
-	t.Parallel()
-
-	creates := map[string]func(ctx context.Context, path string) error{
-		"lockScanDatabase": func(_ context.Context, path string) error {
-			lock, err := lockScanDatabase(path)
-			if err == nil {
-				_ = lock.Close()
-			}
-
-			return err
-		},
-		"openScanDatabase": func(ctx context.Context, path string) error {
-			db, err := openScanDatabase(ctx, path)
-			if err == nil {
-				_ = db.Close()
-			}
-
-			return err
-		},
-	}
-
-	for name, create := range creates {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			dir, err := filepath.EvalSymlinks(t.TempDir())
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			err = os.MkdirAll(filepath.Join(dir, "data", "x"), 0o750)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			err = os.Mkdir(filepath.Join(dir, "srv"), 0o750)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			err = os.Symlink(filepath.Join(dir, "data", "x"),
-				filepath.Join(dir, "srv", "link"))
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			err = create(t.Context(), dir+"/srv/link/../new/db.sqlite")
-			if err != nil {
-				t.Fatalf("%s: %v", name, err)
-			}
-
-			_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
-			if err != nil {
-				t.Errorf("database the path leads to: %v", err)
-			}
-
-			_, err = os.Lstat(filepath.Join(dir, "srv", "new"))
-			if !errors.Is(err, fs.ErrNotExist) {
-				t.Errorf("srv/new: stat error %v, want it missing", err)
-			}
-		})
-	}
-}
-
-// TestDatabaseFiles opens a database through a symlink to its file:
-// databaseFiles finds the database file and its -wal and -shm files
-// beside the symlink's target, where SQLite keeps them, and leaves out
-// the .lock file, which only a scan creates.
-func TestDatabaseFiles(t *testing.T) {
-	t.Parallel()
-
-	path := testDBPath(t)
-	link := filepath.Join(t.TempDir(), "link.sqlite")
-
-	err := os.Symlink(path, link)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	db, err := openScanDatabase(t.Context(), link)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = db.Close() }()
-
-	files, err := databaseFiles(link)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	got := make([]string, 0, len(files))
-	for _, f := range files {
-		got = append(got, f.path)
-	}
-
-	want := []string{path, path + "-wal", path + "-shm"}
-	if !slices.Equal(got, want) {
-		t.Errorf("databaseFiles(%s) = %q, want %q", link, got, want)
-	}
-}
-
-func TestOpenDatabaseUnversionedForeign(t *testing.T) {
-	t.Parallel()
-
-	// A database that has a files table but user_version 0, written by
-	// some other tool. report, trees and scan must refuse it with the
-	// schema-version error, not adopt it and not emit a raw SQLite
-	// "table files already exists".
-	path := testDBPath(t)
-
-	db, err := sql.Open("sqlite", path)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	_, err = db.ExecContext(t.Context(), "CREATE TABLE files (x INTEGER)")
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	_ = db.Close()
-
-	_, err = openReportDatabase(t.Context(), path)
-	if !errors.Is(err, errSchemaVersion) ||
-		!strings.Contains(err.Error(), "remove the file and rescan") {
-		t.Fatalf("report: err = %v, want errSchemaVersion telling the "+
-			"operator to remove the file and rescan", err)
-	}
-
-	_, err = openScanDatabase(t.Context(), path)
-	if !errors.Is(err, errSchemaVersion) ||
-		!strings.Contains(err.Error(), "remove the file and rescan") {
-		t.Fatalf("scan: err = %v, want errSchemaVersion telling the "+
-			"operator to remove the file and rescan", err)
-	}
-}
-
-func TestSchemaCreationStoppedPartway(t *testing.T) {
-	t.Parallel()
-
-	// A first scan stopped while creating the schema must leave a
-	// database the next scan accepts. max_page_count(2) leaves room for
-	// the files table but not its index, so schema creation fails right
-	// after CREATE TABLE, a point an interrupt could also stop it at.
-	path := testDBPath(t)
-
-	db, err := openDB(path, scanParams+"&_pragma=max_page_count(2)")
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	err = initSchema(t.Context(), db)
-	_ = db.Close()
-
-	if err == nil {
-		t.Fatal("initSchema with no room for the index succeeded")
-	}
-
-	db, err = openScanDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatalf("next scan: %v", err)
-	}
-
-	defer func() { _ = db.Close() }()
-
-	v, err := userVersion(t.Context(), db)
-	if err != nil || v != schemaVersion {
-		t.Fatalf("userVersion = %d, %v; want %d, nil", v, err, schemaVersion)
 	}
 }
 
@@ -308,11 +87,9 @@ func TestOpenReportDatabaseMissing(t *testing.T) {
 	}
 }
 
-func TestOpenDatabaseVersionMismatch(t *testing.T) {
+func TestOpenReportDatabaseVersionMismatch(t *testing.T) {
 	t.Parallel()
 
-	// A database stamped with a schema version other than 0 and
-	// schemaVersion. report, trees and scan must all refuse it.
 	path := testDBPath(t)
 
 	db, err := openScanDatabase(t.Context(), path)
@@ -329,12 +106,7 @@ func TestOpenDatabaseVersionMismatch(t *testing.T) {
 
 	_, err = openReportDatabase(t.Context(), path)
 	if !errors.Is(err, errSchemaVersion) {
-		t.Fatalf("report: err = %v, want errSchemaVersion", err)
+		t.Fatalf("err = %v, want errSchemaVersion", err)
-	}
-
-	_, err = openScanDatabase(t.Context(), path)
-	if !errors.Is(err, errSchemaVersion) {
-		t.Fatalf("scan: err = %v, want errSchemaVersion", err)
 	}
 }
 
@@ -358,212 +130,52 @@ func TestOpenReportDatabaseOK(t *testing.T) {
 	_ = db.Close()
 }
 
-func TestCloseScanDatabaseWhileReportOpen(t *testing.T) {
-	t.Parallel()
-
-	// A report holding the database open stops scan from taking it out
-	// of WAL mode. The -wal and -shm files must then stay beside it, so
-	// that a later report still needs only read access.
-	path := testDBPath(t)
-
-	scanDB, err := openScanDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	reportDB, err := openReportDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	closeScanDatabase(t.Context(), scanDB, path)
-
-	_ = reportDB.Close()
-
-	_, err = os.Stat(path + "-wal")
-	if err != nil {
-		t.Fatalf("no -wal left: the switch out of WAL mode was not "+
-			"stopped: %v", err)
-	}
-
-	makeReadOnly(t, path)
-
-	reportDB, err = openReportDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatalf("openReportDatabase: %v", err)
-	}
-
-	defer func() { _ = reportDB.Close() }()
-
-	err = loadFileRows(t.Context(), reportDB, func(int64) {},
-		func(scanRec) {})
-	if err != nil {
-		t.Fatalf("loadFileRows: %v", err)
-	}
-}
-
-// TestLoadRowsCountAgreeDuringScan has a scan commit records after
-// loadFileRows, as trees calls it, or loadSignatureRows, as report
-// calls it, has counted the records and before it reads them, as when
-// trees or report runs during a scan: each must read exactly as many
-// records as it counted.
-func TestLoadRowsCountAgreeDuringScan(t *testing.T) {
-	t.Parallel()
-
-	loads := map[string]func(context.Context, *sql.DB, func(int64),
-		func(scanRec)) error{
-		"loadFileRows": loadFileRows,
-		"loadSignatureRows": func(ctx context.Context, db *sql.DB,
-			total func(int64), fn func(scanRec),
-		) error {
-			return loadSignatureRows(ctx, db, total, func(r scanRec) error {
-				fn(r)
-
-				return nil
-			})
-		},
-	}
-
-	for name, load := range loads {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			path := testDBPath(t)
-
-			scanDB, err := openScanDatabase(t.Context(), path)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			defer func() { _ = scanDB.Close() }()
-
-			recs := smokeTreeRecs()
-
-			err = applyChanges(t.Context(), scanDB, recs[:3], nil, nil)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			reportDB, err := openReportDatabase(t.Context(), path)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			defer func() { _ = reportDB.Close() }()
-
-			var counted, read int64
-
-			err = load(t.Context(), reportDB, func(n int64) {
-				counted = n
-
-				err := applyChanges(t.Context(), scanDB, recs[3:], nil, nil)
-				if err != nil {
-					t.Error(err)
-				}
-			}, func(scanRec) { read++ })
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			if counted != 3 || read != counted {
-				t.Errorf("counted %d records and read %d, want 3 and 3",
-					counted, read)
-			}
-		})
-	}
-}
-
-// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
-// from the files_signature index alone, with nothing to sort: the
-// first record comes at once, so report's read phase counts each
-// record as it comes rather than waiting on SQLite for all of them.
-func TestSignatureRowsNeedNoSort(t *testing.T) {
-	t.Parallel()
-
-	db := openTestDB(t)
-
-	rows, err := db.QueryContext(t.Context(),
-		"EXPLAIN QUERY PLAN "+signatureRowsSQL)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = rows.Close() }()
-
-	var plan []string
-
-	for rows.Next() {
-		var (
-			id, parent, unused int
-			detail             string
-		)
-
-		err = rows.Scan(&id, &parent, &unused, &detail)
-		if err != nil {
-			t.Fatal(err)
-		}
-
-		plan = append(plan, detail)
-	}
-
-	err = rows.Err()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	want := []string{"SCAN files USING COVERING INDEX files_signature"}
-	if !slices.Equal(plan, want) {
-		t.Errorf("query plan = %q, want %q", plan, want)
-	}
-}
-
 func TestApplyChangesRoundTrip(t *testing.T) {
 	t.Parallel()
 
 	db := openTestDB(t)
 
 	// Paths may contain tabs and newlines; the database must store
-	// them byte-exactly. Every hash, content included, comes back as
+	// them byte-exactly.
-	// written.
 	recs := []scanRec{
-		{
+		{size: 2, mtime: 20, head: "h2", tail: "t2", path: "/a/tab\tnew\nline"},
-			size: 2, mtime: time.Unix(20, 999_999_999), head: "h2", tail: "t2",
+		{size: 1, mtime: 10, head: "h1", tail: "t1", path: "/a/x"},
-			content: "c2", path: "/a/tab\tnew\nline",
-		},
-		{
-			size: 1, mtime: time.Unix(10, 0), head: "h1", tail: "t1",
-			content: "c1", path: "/a/x",
-		},
 	}
 
-	prog := newProgress(progressParams{label: "update", total: 4})
+	err := applyChanges(t.Context(), db, recs, nil,
-	defer prog.finish()
+		newProgress("update", 2))
-
-	err := applyChanges(t.Context(), db, recs, nil, prog)
 	if err != nil {
 		t.Fatalf("applyChanges: %v", err)
 	}
 
-	// The records come back in path order, which is the order of recs.
+	got, err := loadFileRows(t.Context(), db)
-	got := dbRecords(t, db)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	slices.SortFunc(got, func(a, b scanRec) int {
+		return strings.Compare(a.path, b.path)
+	})
+
 	if !slices.Equal(got, recs) {
 		t.Fatalf("rows = %+v, want %+v", got, recs)
 	}
 
 	// An upsert for an existing path updates in place; a delete
 	// removes exactly its path.
-	upd := scanRec{
+	upd := scanRec{size: 3, mtime: 30, head: "h3", tail: "t3", path: "/a/x"}
-		size: 3, mtime: time.Unix(30, 0), head: "h3", tail: "t3", content: "c3",
-		path: "/a/x",
-	}
 
 	err = applyChanges(t.Context(), db, []scanRec{upd},
-		[]string{"/a/tab\tnew\nline"}, prog)
+		[]string{"/a/tab\tnew\nline"}, newProgress("update", 2))
 	if err != nil {
 		t.Fatalf("applyChanges: %v", err)
 	}
 
-	got = dbRecords(t, db)
+	got, err = loadFileRows(t.Context(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+
 	if len(got) != 1 || got[0] != upd {
 		t.Fatalf("rows = %+v, want just %+v", got, upd)
 	}
@@ -581,21 +193,20 @@ func TestApplyChangesBatching(t *testing.T) {
 	recs := make([]scanRec, 0, n)
 	for i := range n {
 		recs = append(recs, scanRec{
-			size: int64(i), mtime: time.Unix(1, 0), head: "h", tail: "t",
+			size: int64(i), mtime: 1, head: "h", tail: "t",
 			path: fmt.Sprintf("/batch/%07d", i),
 		})
 	}
 
-	prog := newProgress(progressParams{label: "update", total: int64(2 * n)})
+	err := applyChanges(t.Context(), db, recs, nil,
-	defer prog.finish()
+		newProgress("update", int64(n)))
-
-	err := applyChanges(t.Context(), db, recs, nil, prog)
 	if err != nil {
 		t.Fatalf("applyChanges: %v", err)
 	}
 
-	if got := dbRecords(t, db); len(got) != n {
+	got, err := loadFileRows(t.Context(), db)
-		t.Fatalf("records = %d, want %d", len(got), n)
+	if err != nil || len(got) != n {
+		t.Fatalf("loadFileRows = %d rows, %v; want %d", len(got), err, n)
 	}
 
 	deletes := make([]string, 0, n)
@@ -603,71 +214,14 @@ func TestApplyChangesBatching(t *testing.T) {
 		deletes = append(deletes, r.path)
 	}
 
-	err = applyChanges(t.Context(), db, nil, deletes, prog)
+	err = applyChanges(t.Context(), db, nil, deletes,
+		newProgress("update", int64(n)))
 	if err != nil {
 		t.Fatalf("applyChanges deletes: %v", err)
 	}
 
-	if got := dbRecords(t, db); len(got) != 0 {
+	got, err = loadFileRows(t.Context(), db)
-		t.Fatalf("records = %d, want 0", len(got))
+	if err != nil || len(got) != 0 {
-	}
+		t.Fatalf("loadFileRows = %d rows, %v; want 0", len(got), err)
-}
-
-// TestCountContentCandidates checks that countContentCandidates counts
-// the records the content phase's search returns: those of each group
-// of two or more records of headTailMin or more with a head, where at
-// least one has no content hash.
-func TestCountContentCandidates(t *testing.T) {
-	t.Parallel()
-
-	db := openTestDB(t)
-
-	// Each group's records: size, head, and content, one per record.
-	groups := []struct {
-		size     int64
-		head     string
-		contents []string
-	}{
-		{headTailMin, "h", []string{"", "c", "c"}}, // returned
-		{headTailMin + 1, "h", []string{"", ""}},   // returned
-		{headTailMin + 2, "h", []string{"c", "c"}}, // all hashed
-		{headTailMin + 3, "h", []string{""}},       // one record
-		{headTailMin - 1, "h", []string{"", ""}},   // too small
-		{headTailMin + 4, "", []string{"", ""}},    // no head
-	}
-
-	var recs []scanRec
-
-	for _, g := range groups {
-		for _, content := range g.contents {
-			recs = append(recs, scanRec{
-				size: g.size, mtime: time.Unix(1, 0), head: g.head,
-				tail: "t", content: content,
-				path: fmt.Sprintf("/r/%d", len(recs)),
-			})
-		}
-	}
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatalf("applyChanges: %v", err)
-	}
-
-	got, err := countContentCandidates(t.Context(), db)
-	if err != nil {
-		t.Fatalf("countContentCandidates: %v", err)
-	}
-
-	var returned int64
-
-	err = loadContentCandidates(t.Context(), db,
-		func(scanRec, bool) { returned++ })
-	if err != nil {
-		t.Fatalf("loadContentCandidates: %v", err)
-	}
-
-	if got != 5 || returned != 5 {
-		t.Errorf("count = %d, records returned = %d, want 5 and 5",
-			got, returned)
 	}
 }

exclude.go

@@ -1,195 +0,0 @@
-package main
-
-// This file holds two rules for the walk (README "Rules for the walk"):
-// the default excludes, the paths a scan's walk does not enter below an
-// operand unless scan is given --no-default-excludes, and the
-// filesystems that are not persistent, on which the scan never records
-// or reads anything.
-
-import (
-	"bufio"
-	"errors"
-	"fmt"
-	"io"
-	"io/fs"
-	"path/filepath"
-	"slices"
-	"strings"
-
-	"golang.org/x/sys/unix"
-)
-
-// The systems that have default excludes, as runtime.GOOS names them.
-const (
-	goosLinux   = "linux"
-	goosDarwin  = "darwin"
-	goosWindows = "windows"
-)
-
-// The fields of a line of mountInfoPath that readMountInfo reads,
-// counted from 0 (see proc(5)): the device ID, as major:minor, is the
-// third field, and after the sixth come optional fields, then a field
-// holding "-", then the filesystem type.
-const (
-	deviceField         = 2
-	optionalFieldsStart = 6
-)
-
-// typesPerHelpLine is how many filesystem types the scan command's help
-// lists on each line.
-const typesPerHelpLine = 6
-
-// errMountInfoLine reports a line of mountInfoPath without the fields
-// readMountInfo reads.
-var errMountInfoLine = errors.New("unexpected line")
-
-// walkRules holds the rules for the walk that a scan sets up as it
-// starts. excludes are the default excludes, or none with
-// --no-default-excludes. nonPersistent maps the device ID of every
-// mounted filesystem that is not persistent to its type; the scan
-// records and reads nothing on one of those, whatever the flags.
-// databaseFiles are the scan's own database files, which the walk
-// passes over (see databaseFiles in db.go).
-type walkRules struct {
-	excludes      []string
-	nonPersistent map[uint64]string
-	databaseFiles []fileRec
-}
-
-// defaultExcludes returns the default excludes of the system goos
-// names, as runtime.GOOS does: a fixed list on Linux, macOS and
-// Windows, and none on any other system.
-func defaultExcludes(goos string) []string {
-	switch goos {
-	case goosLinux:
-		return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
-	case goosDarwin:
-		// A walk of / reaches everything in /System/Volumes/Data through
-		// the firmlinks, so walking it as well would record each of
-		// those files twice.
-		return []string{
-			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
-			"/System/Volumes/VM", "/System/Volumes/Data",
-		}
-	case goosWindows:
-		// Relative to the root of the drive an operand lies on. The
-		// first three are the page, swap and hibernation files.
-		return []string{
-			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
-			"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
-			`Users\*\AppData\Local\Temp`,
-		}
-	default:
-		return nil
-	}
-}
-
-// isDefaultExclude reports whether path is one of excludes, in which a
-// * matches one path component.
-func isDefaultExclude(path string, excludes []string) bool {
-	for _, pattern := range excludes {
-		// The patterns are fixed, so none is malformed.
-		if ok, _ := filepath.Match(pattern, path); ok {
-			return true
-		}
-	}
-
-	return false
-}
-
-// nonPersistentFilesystemTypes returns the filesystem types that are
-// not persistent: filesystems that the kernel makes up or that are kept
-// in memory. devfs is the macOS filesystem of /dev; the others are
-// Linux names.
-func nonPersistentFilesystemTypes() []string {
-	return []string{
-		"proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "ramfs",
-		"cgroup", "cgroup2", "debugfs", "tracefs", "securityfs", "pstore",
-		"bpf", "configfs", "fusectl", "mqueue", "hugetlbfs", "efivarfs",
-		"binfmt_misc", "autofs", "nsfs", "rpc_pipefs", "selinuxfs", "devfs",
-	}
-}
-
-// nonPersistentLine returns the line the scan prints for path when fi,
-// its lstat, puts it on a filesystem in rules.nonPersistent, and ""
-// when it is on any other.
-func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
-	dev, ok := deviceOfInfo(fi)
-	if !ok {
-		return ""
-	}
-
-	fsType, found := rules.nonPersistent[dev]
-	if !found {
-		return ""
-	}
-
-	return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
-		path, fsType)
-}
-
-// readMountInfo reads mountinfo, laid out as mountInfoPath is, and
-// returns the device ID of every mounted filesystem that is not
-// persistent, with its type. The device ID is the one each line gives,
-// not that of the file now at its mount point, so a mount that a later
-// mount hides never adds the device of another filesystem.
-func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
-	types := nonPersistentFilesystemTypes()
-	devices := make(map[uint64]string)
-	lines := bufio.NewScanner(mountinfo)
-
-	for lines.Scan() {
-		fields := strings.Fields(lines.Text())
-
-		sep := slices.Index(fields, "-")
-		if sep < optionalFieldsStart || sep+1 >= len(fields) {
-			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
-		}
-
-		var major, minor uint32
-
-		_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
-		if err != nil {
-			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
-		}
-
-		fsType := fields[sep+1]
-		if slices.Contains(types, fsType) {
-			devices[unix.Mkdev(major, minor)] = fsType
-		}
-	}
-
-	return devices, lines.Err()
-}
-
-// defaultExcludesHelp describes, for the scan command's help, the
-// default excludes of the system goos names and, where the scan lists
-// the mounted filesystems, the types it never records anything on.
-func defaultExcludesHelp(goos string) string {
-	var help strings.Builder
-
-	dirs := defaultExcludes(goos)
-	if len(dirs) == 0 {
-		help.WriteString("This system has no default excludes.\n")
-	} else {
-		help.WriteString("Below an operand, the walk does not enter these " +
-			"default excludes, unless\n--no-default-excludes is given:\n\n")
-
-		for _, dir := range dirs {
-			fmt.Fprintf(&help, "  %s\n", dir)
-		}
-	}
-
-	if goos == goosLinux || goos == goosDarwin {
-		help.WriteString("\nWith or without --no-default-excludes, the scan " +
-			"records nothing on a\nfilesystem that is not persistent, " +
-			"one of these types:\n\n")
-
-		types := nonPersistentFilesystemTypes()
-		for line := range slices.Chunk(types, typesPerHelpLine) {
-			fmt.Fprintf(&help, "  %s\n", strings.Join(line, ", "))
-		}
-	}
-
-	return help.String()
-}

exclude_darwin.go

@@ -1,52 +0,0 @@
-//go:build darwin
-
-package main
-
-import (
-	"slices"
-
-	"golang.org/x/sys/unix"
-)
-
-// nonPersistentFilesystems returns the device ID of every mounted
-// filesystem that is not persistent, with its type, from the system's
-// list of mounted filesystems. The device ID is read with lstat on the
-// mount point. Where one filesystem is mounted over another on the same
-// mount point, the later one in the list counts. A mount point that
-// lstat cannot reach is left out.
-func nonPersistentFilesystems() (map[uint64]string, error) {
-	// The first call counts the filesystems, the second lists them.
-	n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
-	if err != nil {
-		return nil, err
-	}
-
-	list := make([]unix.Statfs_t, n)
-
-	n, err = unix.Getfsstat(list, unix.MNT_NOWAIT)
-	if err != nil {
-		return nil, err
-	}
-
-	mounts := make(map[string]string, n)
-
-	for _, st := range list[:n] {
-		mounts[unix.ByteSliceToString(st.Mntonname[:])] =
-			unix.ByteSliceToString(st.Fstypename[:])
-	}
-
-	types := nonPersistentFilesystemTypes()
-	devices := make(map[uint64]string)
-
-	for mountPoint, fsType := range mounts {
-		if !slices.Contains(types, fsType) {
-			continue
-		}
-
-		if dev, ok := deviceOfPath(mountPoint); ok {
-			devices[dev] = fsType
-		}
-	}
-
-	return devices, nil
-}

exclude_other.go

@@ -1,36 +0,0 @@
-//go:build !darwin
-
-package main
-
-import (
-	"fmt"
-	"os"
-	"runtime"
-)
-
-// mountInfoPath is the file in which Linux lists the mounts the reading
-// process sees.
-const mountInfoPath = "/proc/self/mountinfo"
-
-// nonPersistentFilesystems returns the device ID of every mounted
-// filesystem that is not persistent, with its type: on Linux read from
-// mountInfoPath (see readMountInfo), and on any other system none.
-func nonPersistentFilesystems() (map[uint64]string, error) {
-	if runtime.GOOS != goosLinux {
-		return map[uint64]string{}, nil
-	}
-
-	f, err := os.Open(mountInfoPath)
-	if err != nil {
-		return nil, err
-	}
-
-	defer func() { _ = f.Close() }()
-
-	devices, err := readMountInfo(f)
-	if err != nil {
-		return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
-	}
-
-	return devices, nil
-}

exclude_test.go

@@ -1,583 +0,0 @@
-package main
-
-import (
-	"bytes"
-	"errors"
-	"maps"
-	"os"
-	"path/filepath"
-	"runtime"
-	"slices"
-	"strings"
-	"testing"
-
-	"golang.org/x/sys/unix"
-)
-
-func TestDefaultExcludes(t *testing.T) {
-	t.Parallel()
-
-	// README "Rules for the walk": each system's list.
-	cases := map[string][]string{
-		goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
-		goosDarwin: {
-			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
-			"/System/Volumes/VM", "/System/Volumes/Data",
-		},
-		goosWindows: {
-			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
-			"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
-			`Users\*\AppData\Local\Temp`,
-		},
-		"freebsd": nil,
-	}
-
-	for goos, want := range cases {
-		if got := defaultExcludes(goos); !slices.Equal(got, want) {
-			t.Errorf("defaultExcludes(%q) = %q, want %q", goos, got, want)
-		}
-	}
-}
-
-func TestIsDefaultExclude(t *testing.T) {
-	t.Parallel()
-
-	// A * matches one path component, no more and no fewer.
-	excludes := []string{"/scratch", "/home/*/cache"}
-
-	cases := map[string]bool{
-		"/scratch":        true,
-		"/scratch/x":      false,
-		"/home/u/cache":   true,
-		"/home/u/v/cache": false,
-		"/home/cache":     false,
-	}
-
-	for path, want := range cases {
-		if got := isDefaultExclude(path, excludes); got != want {
-			t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
-		}
-	}
-}
-
-func TestReadMountInfo(t *testing.T) {
-	t.Parallel()
-
-	// The second-to-last line has no optional fields. The tmpfs on
-	// /data/a/b is hidden by the root filesystem's /srv, mounted over
-	// /data/a after it: the tmpfs still gives its own device, and the
-	// root filesystem's device is not listed.
-	mountinfo := strings.Join([]string{
-		"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
-		"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
-		"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
-		"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw",
-		"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw",
-		"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
-	}, "\n")
-
-	got, err := readMountInfo(strings.NewReader(mountinfo))
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	want := map[uint64]string{
-		unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs",
-		unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs",
-	}
-	if !maps.Equal(got, want) {
-		t.Errorf("readMountInfo = %v, want %v", got, want)
-	}
-
-	for _, line := range []string{
-		"22 1 8:1 / / rw,relatime\n",
-		"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n",
-	} {
-		_, err = readMountInfo(strings.NewReader(line))
-		if !errors.Is(err, errMountInfoLine) {
-			t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
-				errMountInfoLine)
-		}
-	}
-}
-
-func TestNonPersistentFilesystems(t *testing.T) {
-	t.Parallel()
-
-	if runtime.GOOS != goosLinux {
-		t.Skip("the test knows a mount only on Linux: /proc")
-	}
-
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	dev, ok := deviceOfPath("/proc")
-	if !ok {
-		t.Fatal("no device for /proc")
-	}
-
-	if got := nonPersistent[dev]; got != "proc" {
-		t.Errorf("filesystem of /proc = %q, want %q", got, "proc")
-	}
-}
-
-func TestDefaultExcludesHelp(t *testing.T) {
-	t.Parallel()
-
-	for _, goos := range []string{goosLinux, goosDarwin, goosWindows} {
-		help := defaultExcludesHelp(goos)
-
-		for _, dir := range defaultExcludes(goos) {
-			if !strings.Contains(help, "\n  "+dir+"\n") {
-				t.Errorf("help for %s does not list %s:\n%s", goos, dir, help)
-			}
-		}
-
-		// Only Linux and macOS list their mounted filesystems.
-		for _, fsType := range nonPersistentFilesystemTypes() {
-			listed := strings.Contains(help, " "+fsType+",") ||
-				strings.Contains(help, " "+fsType+"\n")
-			if listed != (goos != goosWindows) {
-				t.Errorf("help for %s lists %s: %v", goos, fsType, listed)
-			}
-		}
-	}
-
-	want := "This system has no default excludes.\n"
-	if got := defaultExcludesHelp("freebsd"); got != want {
-		t.Errorf("help for freebsd = %q, want %q", got, want)
-	}
-
-	// scan's help shows those of the system it runs on.
-	var stdout, stderr bytes.Buffer
-
-	run([]string{cmdScan, "-h"}, &stdout, &stderr)
-
-	help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
-	if !strings.Contains(stderr.String(), help) {
-		t.Errorf("scan --help = %q, want it to hold %q", stderr.String(), help)
-	}
-}
-
-// TestScanDoesNotEnterDefaultExclude checks that the walk does not
-// enter a default exclude below an operand. It says so on a line of its
-// own and does not count it as skipped. The records below it count as
-// outside the operands: the record of a file gone from it is kept, and
-// its size joins the size census, while a file added to it is not
-// recorded.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
-	stderr := captureStderr(t)
-	dir := t.TempDir()
-	db := openTestDB(t)
-	excluded := filepath.Join(dir, "tmp")
-
-	// A scan with no default excludes records the file in excluded.
-	gone := writeFile(t, excluded, "gone", pattern(1, 100))
-	syncTree(t, db, dir)
-
-	err := os.Remove(gone)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	unseen := writeFile(t, excluded, "new", pattern(2, 10))
-
-	// The same size as gone, so it is hashed only if the census holds the
-	// size of gone's record.
-	added := writeFile(t, dir, "added", pattern(3, 100))
-
-	st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
-		walkRules{excludes: []string{excluded}})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.skipped != 0 {
-		t.Errorf("skipped = %d, want 0", st.skipped)
-	}
-
-	recs := dbRecords(t, db)
-	recordByPath(t, recs, gone)
-
-	if recordByPath(t, recs, added).head == "" {
-		t.Errorf("%s not hashed: the census lacks the size of %s",
-			added, gone)
-	}
-
-	if slices.Contains(recordPaths(recs), unseen) {
-		t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
-	}
-
-	line := "\nscan: skipped " + excluded + " (default exclude)\n"
-	if !strings.Contains(stderr(), line) {
-		t.Errorf("stderr = %q, want %q", stderr(), line)
-	}
-}
-
-// TestScanWalksOperandInDefaultExclude checks that an operand at or
-// inside a default exclude is walked, also when it lies under another
-// operand, whose walk does not enter the default exclude.
-func TestScanWalksOperandInDefaultExclude(t *testing.T) {
-	t.Parallel()
-
-	dir := t.TempDir()
-	excluded := filepath.Join(dir, "tmp")
-	f := writeFile(t, excluded, "sub/f", pattern(1, 10))
-	sub := filepath.Dir(f)
-
-	cases := map[string][]string{
-		"at it":                      {excluded},
-		"inside it":                  {sub},
-		"inside it under an operand": {dir, sub},
-	}
-
-	for name, roots := range cases {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			db := openTestDB(t)
-
-			st, err := syncScan(t.Context(), db, roots, 1, false,
-				walkRules{excludes: []string{excluded}})
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			if st.added != 1 {
-				t.Errorf("added = %d, want %s recorded once", st.added, f)
-			}
-
-			recordByPath(t, dbRecords(t, db), f)
-		})
-	}
-}
-
-// TestScanNamesNoDefaultExcludeThatIsAnOperand checks that a default
-// exclude given as an operand under another operand, which the walk of
-// that other operand does not enter, gets no line saying it was
-// skipped: it is walked as an operand of its own.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
-	stderr := captureStderr(t)
-	dir := t.TempDir()
-	db := openTestDB(t)
-	excluded := filepath.Join(dir, "tmp")
-	f := writeFile(t, excluded, "f", pattern(1, 10))
-
-	st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
-		walkRules{excludes: []string{excluded}})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.added != 1 {
-		t.Errorf("added = %d, want %s recorded once", st.added, f)
-	}
-
-	recordByPath(t, dbRecords(t, db), f)
-
-	if strings.Contains(stderr(), "scan: skipped") {
-		t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
-	}
-}
-
-// TestRunScanTempDirOperand scans a tree in the tests' temporary
-// directory, which on Linux lies inside the default exclude /tmp unless
-// TMPDIR names another place, with the system's own default excludes:
-// the scan walks the whole tree and reports no default exclude.
-func TestRunScanTempDirOperand(t *testing.T) {
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	dir := t.TempDir()
-	if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
-		t.Skip(dir + " is not inside a default exclude")
-	}
-
-	f := writeFile(t, dir, "sub/f", pattern(1, 10))
-
-	stderr := scanOK(t, dir)
-	if strings.Contains(stderr, "scan: skipped") {
-		t.Errorf("stderr = %q, want no line saying a path was skipped",
-			stderr)
-	}
-
-	assertRecorded(t, path, f, true)
-}
-
-// TestRunScanNoDefaultExcludes scans /var through the command. Its
-// /var/tmp, on a persistent filesystem, is a default exclude on Linux:
-// without --no-default-excludes the walk does not enter it and says so,
-// and with that flag the walk records a file the test puts there.
-func TestRunScanNoDefaultExcludes(t *testing.T) {
-	if runtime.GOOS != goosLinux {
-		t.Skip("only Linux has /var/tmp among its default excludes")
-	}
-
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	rules := walkRules{nonPersistent: nonPersistent}
-
-	for _, dir := range []string{"/var", "/var/tmp"} {
-		fi, lstatErr := os.Lstat(dir)
-		if lstatErr != nil || !fi.IsDir() ||
-			rules.nonPersistentLine(dir, fi) != "" {
-			t.Skip(dir + " is not a directory on a persistent filesystem here")
-		}
-	}
-
-	//nolint:usetesting // the directory must be inside /var/tmp
-	tmp, err := os.MkdirTemp("/var/tmp", "sfdupes-test-")
-	if err != nil {
-		t.Skip("cannot write to /var/tmp here: " + err.Error())
-	}
-
-	t.Cleanup(func() { _ = os.RemoveAll(tmp) })
-
-	f := writeFile(t, tmp, "f", pattern(1, 10))
-
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	line := "scan: skipped /var/tmp (default exclude)\n"
-
-	if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
-		t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
-	}
-
-	assertRecorded(t, path, f, false)
-
-	stderr := scanOK(t, "--no-default-excludes", "/var")
-	if strings.Contains(stderr, line) {
-		t.Errorf("scan --no-default-excludes /var: stderr = %q, want no %q",
-			stderr, line)
-	}
-
-	assertRecorded(t, path, f, true)
-}
-
-// TestRunScanSkipsNonPersistentOperand scans, through the command, a
-// directory on a filesystem that is not persistent, with and without
-// --no-default-excludes: either way it is skipped, with a line saying
-// so, and nothing in it is recorded.
-func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
-	shm, fsType, _ := shmTempDir(t)
-	writeFile(t, shm, "f", pattern(1, 10))
-
-	t.Setenv(databaseEnv, testDBPath(t))
-
-	line := "scan: skipped " + shm + " (" + fsType +
-		", not a persistent filesystem)\n"
-	summary := "scan: 0 files seen"
-
-	for _, flags := range [][]string{nil, {"--no-default-excludes"}} {
-		stderr := scanOK(t, append(flags, shm)...)
-		if !strings.Contains(stderr, line) ||
-			!strings.Contains(stderr, summary) {
-			t.Errorf("scan %q: stderr = %q, want %q and %q", flags, stderr,
-				line, summary)
-		}
-	}
-}
-
-// TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem
-// that is not persistent beside one on the tests' temporary directory.
-// The records an earlier scan left in it count as outside the operands:
-// the record of a file gone from it is kept, and its size joins the size
-// census, while a file added to it is not recorded, and its size does
-// not join the census.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
-	shm, fsType, rules := shmTempDir(t)
-	stderr := captureStderr(t)
-	dir := t.TempDir()
-	db := openTestDB(t)
-
-	// A scan that, without the list of filesystems, records gone.
-	gone := writeFile(t, shm, "gone", pattern(1, 100))
-	syncTree(t, db, shm)
-
-	err := os.Remove(gone)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	unseen := writeFile(t, shm, "new", pattern(2, 10))
-
-	// added has the size of gone, and lonely that of unseen: each is
-	// hashed only if the census holds that size.
-	added := writeFile(t, dir, "added", pattern(3, 100))
-	lonely := writeFile(t, dir, "lonely", pattern(4, 10))
-
-	st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.skipped != 0 {
-		t.Errorf("skipped = %d, want 0", st.skipped)
-	}
-
-	recs := dbRecords(t, db)
-	recordByPath(t, recs, gone)
-
-	if recordByPath(t, recs, added).head == "" {
-		t.Errorf("%s not hashed: the census lacks the size of %s",
-			added, gone)
-	}
-
-	if recordByPath(t, recs, lonely).head != "" {
-		t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen)
-	}
-
-	if slices.Contains(recordPaths(recs), unseen) {
-		t.Errorf("%s recorded", unseen)
-	}
-
-	line := "scan: skipped " + shm + " (" + fsType +
-		", not a persistent filesystem)\n"
-	if !strings.Contains(stderr(), line) {
-		t.Errorf("stderr = %q, want %q", stderr(), line)
-	}
-}
-
-// TestScanContentSkipsNonPersistent checks that the content phase does
-// not read a file on a filesystem that is not persistent, though an
-// earlier scan left a record for it: it is treated as a changed file,
-// so its record is left as it is and its identical copy in another
-// tree, having no other match, is not read either.
-func TestScanContentSkipsNonPersistent(t *testing.T) {
-	t.Parallel()
-
-	shm, _, rules := shmTempDir(t)
-	db := openTestDB(t)
-
-	// A scan that, without the list of filesystems, records a with a
-	// head and tail but no content hash.
-	a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
-
-	syncTree(t, db, shm)
-
-	before := recordByPath(t, dbRecords(t, db), a)
-	if before.head == "" || before.content != "" {
-		t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
-	}
-
-	b := sparseFile(t, t.TempDir(), "b", headTailMin)
-
-	st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
-		rules)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.skipped != 0 {
-		t.Errorf("skipped = %d, want 0", st.skipped)
-	}
-
-	recs := dbRecords(t, db)
-	if r := recordByPath(t, recs, a); r != before {
-		t.Errorf("record = %+v, want it left as %+v", r, before)
-	}
-
-	if r := recordByPath(t, recs, b); r.content != "" {
-		t.Errorf("%s: content = %q, want none: its only match is on %s",
-			b, r.content, shm)
-	}
-}
-
-// TestWalkLeavesOutNonPersistent starts the walk itself in a directory
-// on a filesystem that is not persistent, which a scan never does: it
-// drops such an operand first. Below a directory on a persistent
-// filesystem, one on another takes a mount, which a test cannot make.
-// The walk enters no directory and records no file there, and names
-// each.
-func TestWalkLeavesOutNonPersistent(t *testing.T) {
-	t.Parallel()
-
-	shm, fsType, rules := shmTempDir(t)
-	f := writeFile(t, shm, "f", pattern(1, 10))
-	sub := filepath.Dir(writeFile(t, shm, "sub/g", pattern(1, 10)))
-
-	var leftOut []string
-
-	for ev := range startWalk(t.Context(), []string{shm}, false, rules, 1) {
-		if ev.rec.path != "" {
-			t.Errorf("walk recorded %s", ev.rec.path)
-		}
-
-		if ev.leftOut == "" {
-			continue
-		}
-
-		leftOut = append(leftOut, ev.leftOut)
-
-		want := "scan: skipped " + ev.leftOut + " (" + fsType +
-			", not a persistent filesystem)"
-		if ev.warn != want {
-			t.Errorf("line = %q, want %q", ev.warn, want)
-		}
-	}
-
-	slices.Sort(leftOut)
-
-	if want := []string{f, sub}; !slices.Equal(leftOut, want) {
-		t.Errorf("walk left out %q, want %q", leftOut, want)
-	}
-}
-
-// shmTempDir returns a new directory in /dev/shm, removed when the test
-// ends, the type of its filesystem, and the rules holding the real
-// filesystems that are not persistent. It skips the test where /dev/shm
-// is on a persistent filesystem, or cannot be written to.
-func shmTempDir(t *testing.T) (string, string, walkRules) {
-	t.Helper()
-
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	dev, ok := deviceOfPath("/dev/shm")
-
-	fsType, found := nonPersistent[dev]
-	if !ok || !found {
-		t.Skip("/dev/shm is not a memory-backed filesystem here")
-	}
-
-	//nolint:usetesting // the directory must be inside /dev/shm
-	dir, err := os.MkdirTemp("/dev/shm", "sfdupes-test-")
-	if err != nil {
-		t.Skip("cannot write to /dev/shm here: " + err.Error())
-	}
-
-	t.Cleanup(func() { _ = os.RemoveAll(dir) })
-
-	return dir, fsType, walkRules{nonPersistent: nonPersistent}
-}
-
-// assertRecorded checks that the database at dbPath holds a record for
-// path when want is set, and none when it is not.
-func assertRecorded(t *testing.T, dbPath, path string, want bool) {
-	t.Helper()
-
-	db, err := openDB(dbPath, reportParams)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = db.Close() }()
-
-	if got := slices.Contains(recordPaths(dbRecords(t, db)), path); got != want {
-		t.Errorf("%s recorded: %v, want %v", path, got, want)
-	}
-}

go.mod

@@ -5,8 +5,6 @@ go 1.25.7
 require (
 	github.com/schollz/progressbar/v3 v3.19.1
 	github.com/spf13/cobra v1.10.2
-	golang.org/x/sys v0.46.0
-	golang.org/x/term v0.44.0
 	modernc.org/sqlite v1.54.0
 )
 
@@ -20,9 +18,9 @@ require (
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
 	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
+	golang.org/x/sys v0.46.0 // indirect
+	golang.org/x/term v0.44.0 // indirect
 	modernc.org/libc v1.74.1 // indirect
 	modernc.org/mathutil v1.7.1 // indirect
 	modernc.org/memory v1.11.0 // indirect
 )
-
-retract v0.0.1 // early build without a database; does not work as the README describes

info.go

@@ -1,128 +0,0 @@
-package main
-
-import (
-	"context"
-	"errors"
-	"fmt"
-	"io"
-	"io/fs"
-	"os"
-	"path/filepath"
-	"strings"
-	"time"
-)
-
-// infoPhase names one phase of info with its place among all of them,
-// as "[2/2] count", for the phase's progress display. The list here is
-// the one place the phases are counted.
-func infoPhase(name string) string {
-	return phaseLabel([]string{openPhase, "count"}, name)
-}
-
-// runInfo implements the info subcommand: it prints the database's
-// details on stdout (see writeInfo). It opens the database as report
-// does, read-only and without the scan lock, and never touches the
-// scanned filesystem. It names the database on stderr before it opens
-// it. Any database problem, including a missing database, is fatal.
-//
-// The work runs in two phases, each under a progress display of its
-// own on stderr: open (open the database) and count (count the records,
-// with one query). Each display ends before the next phase starts, and
-// before an error is returned to be printed. The details are written
-// once the last display has ended, so no display is drawn while they
-// are written.
-func runInfo(ctx context.Context, stdout io.Writer) error {
-	dbPath, source, err := databasePath()
-	if err != nil {
-		return err
-	}
-
-	fmt.Fprintf(os.Stderr, "info: database %s (%s)\n",
-		escapePath(dbPath), source)
-
-	// Opening can wait on a busy database for as long as the busy
-	// timeout.
-	prog := newProgress(progressParams{
-		label: infoPhase(openPhase), total: -1,
-		waitingFor: openingDatabase,
-	})
-	db, err := openReportDatabase(ctx, dbPath)
-
-	prog.finish()
-
-	if err != nil {
-		return err
-	}
-
-	defer func() { _ = db.Close() }()
-
-	// One query counts everything, so the phase never has a total.
-	prog = newProgress(progressParams{
-		label: infoPhase("count"), total: -1,
-		waitingFor: countingRecords,
-	})
-	counts, err := readDatabaseCounts(ctx, db)
-
-	prog.finish()
-
-	if err != nil {
-		return fmt.Errorf("database %s: %w", dbPath, err)
-	}
-
-	return writeInfo(stdout, dbPath, source, counts)
-}
-
-// writeInfo writes the details of the database at dbPath to stdout,
-// one per line, as a name, a tab and the value (README "`info` mode"):
-// the path and source as databasePath returns them, the sizes of the
-// database file and of its -wal file when there is one, when the
-// database file was last changed, and the schema version and record
-// counts that readDatabaseCounts read.
-func writeInfo(stdout io.Writer, dbPath, source string,
-	counts databaseCounts,
-) error {
-	// SQLite keeps the -wal file beside the database file, named from
-	// its path with symlinks resolved.
-	file, err := filepath.EvalSymlinks(dbPath)
-	if err != nil {
-		return fmt.Errorf("database: %w", err)
-	}
-
-	dbFile, err := os.Stat(file)
-	if err != nil {
-		return fmt.Errorf("database: %w", err)
-	}
-
-	walFile, err := os.Stat(file + "-wal")
-	if err != nil && !errors.Is(err, fs.ErrNotExist) {
-		return fmt.Errorf("database: %w", err)
-	}
-
-	var b strings.Builder
-
-	fmt.Fprintf(&b, "database\t%s (%s)\n", escapePath(dbPath), source)
-	fmt.Fprintf(&b, "database file size\t%d (%s)\n",
-		dbFile.Size(), humanBytes(dbFile.Size()))
-
-	if walFile != nil {
-		fmt.Fprintf(&b, "-wal file size\t%d (%s)\n",
-			walFile.Size(), humanBytes(walFile.Size()))
-	}
-
-	fmt.Fprintf(&b, "modified\t%s\n",
-		dbFile.ModTime().UTC().Format(time.RFC3339))
-	fmt.Fprintf(&b, "schema version\t%d\n", counts.schemaVersion)
-	fmt.Fprintf(&b, "records\t%d\n", counts.records)
-	fmt.Fprintf(&b, "records with head and tail hashes\t%d\n",
-		counts.headTail)
-	fmt.Fprintf(&b, "records with content hash\t%d\n", counts.content)
-	fmt.Fprintf(&b, "size of recorded files\t%d (%s)\n",
-		counts.size, humanBytes(counts.size))
-
-	_, err = io.WriteString(stdout, b.String())
-	if err != nil {
-		return fmt.Errorf("write stdout: %w", err)
-	}
-
-	return nil
-}

info_test.go

@@ -1,168 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"strings"
-	"testing"
-	"time"
-)
-
-// infoOpenLine is the plain line of info's open phase when it opens the
-// database at once; it prints it as the phase starts and as it ends.
-const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
-	"eta unknown"
-
-// infoCountLine is the plain line of info's count phase when the count
-// takes under a second; it prints it as the phase starts and as it
-// ends.
-const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " +
-	"eta unknown"
-
-// infoRecs are four records: a duplicate pair with every hash, a file
-// of 20 MiB with head and tail hashes but no content hash yet, and a
-// file never hashed.
-func infoRecs() []scanRec {
-	return []scanRec{
-		{size: 5, head: "h", tail: "t", content: "c", path: "/d/a"},
-		{size: 5, head: "h", tail: "t", content: "c", path: "/d/b"},
-		{size: 20 << 20, head: "bh", tail: "bt", path: "/d/big"},
-		{size: 7, path: "/d/unique"},
-	}
-}
-
-// TestRunInfo runs info on an empty and on a filled database, with
-// stdout and stderr the same file that is not a terminal, and checks
-// everything written there: the line naming the database, each phase's
-// line as it starts and as it ends, and then the details, each on a
-// line of its own. No -wal file is beside a database scan has closed.
-func TestRunInfo(t *testing.T) {
-	cases := map[string]struct {
-		recs []scanRec
-		want string // the details after the database file's time
-	}{
-		"empty": {
-			want: "schema version\t1\n" +
-				"records\t0\n" +
-				"records with head and tail hashes\t0\n" +
-				"records with content hash\t0\n" +
-				"size of recorded files\t0 (0 B)\n",
-		},
-		"filled": {
-			recs: infoRecs(),
-			want: "schema version\t1\n" +
-				"records\t4\n" +
-				"records with head and tail hashes\t3\n" +
-				"records with content hash\t2\n" +
-				"size of recorded files\t20971537 (20.0 MiB)\n",
-		},
-	}
-
-	for name, tc := range cases {
-		t.Run(name, func(t *testing.T) {
-			path := seedDatabase(t, tc.recs)
-			t.Setenv(databaseEnv, path)
-
-			file, err := os.Stat(path)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			stderr := captureStderr(t)
-
-			code := run([]string{cmdInfo}, os.Stderr, os.Stderr)
-			if code != exitOK {
-				t.Fatalf("run(info) = %d, want %d; stderr: %s",
-					code, exitOK, stderr())
-			}
-
-			named := path + " (from " + databaseEnv + ")\n"
-			want := "info: database " + named +
-				infoOpenLine + "\n" + infoOpenLine + "\n" +
-				infoCountLine + "\n" + infoCountLine + "\n" +
-				"database\t" + named +
-				fmt.Sprintf("database file size\t%d (%s)\n",
-					file.Size(), humanBytes(file.Size())) +
-				"modified\t" + file.ModTime().UTC().Format(time.RFC3339) +
-				"\n" + tc.want
-
-			if got := stderr(); got != want {
-				t.Errorf("stdout and stderr = %q, want %q", got, want)
-			}
-		})
-	}
-}
-
-// TestRunInfoDuringScan runs info while a scan holds the lock and has
-// the database open in WAL mode, with records committed to the -wal
-// file only. info must count those records and show the -wal file's
-// size, also when the database path is a symlink: SQLite keeps the
-// -wal file beside the symlink's target.
-func TestRunInfoDuringScan(t *testing.T) {
-	path := testDBPath(t)
-	link := filepath.Join(t.TempDir(), "link.sqlite")
-
-	err := os.Symlink(path, link)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	db, err := openScanDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	t.Cleanup(func() { _ = db.Close() })
-
-	err = applyChanges(t.Context(), db, infoRecs(), nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	holdScanLock(t, path)
-
-	wal, err := os.Stat(path + "-wal")
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	for _, named := range []string{path, link} {
-		t.Run(filepath.Base(named), func(t *testing.T) {
-			t.Setenv(databaseEnv, named)
-
-			stdout := runStdout(t, cmdInfo)
-
-			for _, want := range []string{
-				"database\t" + named + " (from " + databaseEnv + ")\n",
-				fmt.Sprintf("\n-wal file size\t%d (%s)\n",
-					wal.Size(), humanBytes(wal.Size())),
-				"\nrecords\t4\n",
-			} {
-				if !strings.Contains(stdout, want) {
-					t.Errorf("stdout = %q, want it to hold %q", stdout, want)
-				}
-			}
-		})
-	}
-}
-
-func TestRunInfoNeedsOnlyReadAccess(t *testing.T) {
-	// README "`info` mode": like report, info needs only read access to
-	// the database file and none to its directory.
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	scanFixture(t)
-	makeReadOnly(t, path)
-
-	stdout := runStdout(t, cmdInfo)
-
-	if want := "\nrecords\t2\n"; !strings.Contains(stdout, want) {
-		t.Errorf("stdout = %q, want it to hold %q", stdout, want)
-	}
-
-	if strings.Contains(stdout, "-wal") {
-		t.Errorf("stdout = %q, want no -wal file between scans", stdout)
-	}
-}

main.go

@@ -1,22 +1,16 @@
 // Command sfdupes quickly identifies candidate duplicate files across
-// very large filesystems without reading every byte of every file.
+// very large filesystems without reading full file contents. Files are
-// Files are considered duplicates when their sizes are equal and they
+// considered duplicates when they have identical size, identical SHA-256
-// agree on a short ladder of SHA-256 hashes. A file under 10 MiB is
+// of their first 1024 bytes, and identical SHA-256 of their last 1024
-// hashed in full. A larger file is compared on the hashes of its first
+// bytes. scan maintains a persistent SQLite database of file signatures
-// and last 64 KiB, and only when those match another file's is its
+// (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) that the
-// content hash computed and compared: of the whole file when it is
+// reporting subcommands read.
-// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
-// or larger. scan maintains a persistent SQLite database of file
-// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
-// that the other subcommands read.
 //
 // Usage:
 //
-//	sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
+//	sfdupes scan [--workers N] [-x] PATH...
 //	sfdupes report > dupes.tsv
 //	sfdupes trees > dupetrees.tsv
-//	sfdupes info
-//	sfdupes --version
 //
 // See README.md for the complete specification.
 package main
@@ -28,7 +22,6 @@ import (
 	"io"
 	"os"
 	"runtime"
-	"runtime/debug"
 
 	"github.com/spf13/cobra"
 )
@@ -46,7 +39,6 @@ const (
 	cmdScan   = "scan"
 	cmdReport = "report"
 	cmdTrees  = "trees"
-	cmdInfo   = "info"
 )
 
 // errNoSubcommand is returned by the root command when it is invoked
@@ -54,60 +46,29 @@ const (
 // cobra prints for it is the whole message.
 var errNoSubcommand = errors.New("no subcommand")
 
-// errWorkersBelowOne is the usage error for a scan --workers value
-// below 1.
-var errWorkersBelowOne = errors.New("--workers must be at least 1")
-
 // Version is the build version, injected at link time via -ldflags
-// (see the Makefile); empty when the linker set none, as for a binary
+// (see the Makefile); "dev" for a plain go build.
-// built by go install. buildVersion turns it into what --version prints.
 //
 //nolint:gochecknoglobals // written only by the linker
-var Version string
+var Version = "dev"
-
-// buildVersion returns the version that sfdupes --version prints
-// (README "Install"): the version the linker set, when it set one;
-// otherwise the main module's version that Go recorded in the binary,
-// a tag such as v0.0.2 or the pseudo-version of an @main build; and
-// "dev" when neither is known. info is what debug.ReadBuildInfo
-// returns, nil when the binary carries none. Go records "(devel)" for
-// a build that has no version, such as one inside a copy of the source
-// without git.
-func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
-	if linkerVersion != "" {
-		return linkerVersion
-	}
-
-	if info == nil || info.Main.Version == "" ||
-		info.Main.Version == "(devel)" {
-		return "dev"
-	}
-
-	return info.Main.Version
-}
 
 func main() {
-	// Once the reader of a stdout pipe has gone, as in "sfdupes report |
+	os.Exit(run(os.Args[1:], os.Stderr))
-	// head", the Go runtime ends the process with SIGPIPE on the next
-	// write instead of returning an error (README "Error handling").
-	// Registering for SIGPIPE with os/signal would change that.
-	os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))
 }
 
 // run executes args against the command tree and returns the process
 // exit code. It is the program's single exit point: the subcommands
 // return their errors instead of exiting, so every deferred cleanup —
 // above all closing the database, which checkpoints the SQLite WAL —
-// runs before the process ends. The report, trees and info subcommands
+// runs before the process ends.
-// write their data to stdout.
+func run(args []string, stderr io.Writer) int {
-func run(args []string, stdout, stderr io.Writer) int {
 	// A nil slice makes cobra fall back to os.Args, which would let a
 	// test binary's own flags reach the command tree.
 	if args == nil {
 		args = []string{}
 	}
 
-	root := newRootCommand(stdout, stderr)
+	root := newRootCommand(stderr)
 	root.SetArgs(args)
 
 	err := root.Execute()
@@ -117,9 +78,6 @@ func run(args []string, stdout, stderr io.Writer) int {
 	switch {
 	case err == nil:
 		return exitOK
-	case errors.Is(err, errInterrupted):
-		// The interrupted scan has printed its own line.
-		return exitFatal
 	case errors.As(err, &fatal):
 		// The command ran and failed: a runtime error, reported
 		// without the usage text that a usage error gets.
@@ -127,38 +85,22 @@ func run(args []string, stdout, stderr io.Writer) int {
 
 		return exitFatal
 	default:
-		// A usage error, which cobra has already reported on stderr.
+		// A usage error: cobra has already printed the message and
+		// the usage text.
 		return exitUsage
 	}
 }
 
 // newRootCommand builds the command tree. Everything on stdout is
-// machine-readable data, the version line included; all human-facing
+// machine-readable data; all human-facing output (help, usage, errors)
-// output (help, usage, errors) goes to stderr.
+// goes to stderr.
-func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
+func newRootCommand(stderr io.Writer) *cobra.Command {
-	var showVersion bool
-
-	printVersion := runE(func(context.Context, []string) error {
-		info, _ := debug.ReadBuildInfo()
-
-		_, err := fmt.Fprintf(stdout, "sfdupes %s\n",
-			buildVersion(Version, info))
-		if err != nil {
-			return fmt.Errorf("write stdout: %w", err)
-		}
-
-		return nil
-	})
-
 	root := &cobra.Command{
 		Use:     "sfdupes",
-		Short: "Find candidate duplicate files by size and head/tail/content SHA-256",
+		Short:   "Find candidate duplicate files by size and head/tail SHA-256",
+		Version: Version,
 		Args:    cobra.NoArgs,
-		RunE: func(cmd *cobra.Command, args []string) error {
+		RunE: func(cmd *cobra.Command, _ []string) error {
-			if showVersion {
-				return printVersion(cmd, args)
-			}
-
 			// A missing subcommand prints usage and exits 2: cobra
 			// prints the usage text for the returned error, and run
 			// maps everything that is not a fatal error to exit 2.
@@ -171,17 +113,30 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 	root.SetErr(stderr)
 	root.CompletionOptions.DisableDefaultCmd = true
 
-	// Cobra's built-in version flag prints through the help writer,
+	var (
-	// stderr; this one prints to stdout.
+		scanWorkers int
-	root.Flags().BoolVarP(&showVersion, "version", "v", false,
+		scanOneFS   bool
-		"print the version to stdout")
+	)
+
+	scanCmd := &cobra.Command{
+		Use:   cmdScan + " [--workers N] [-x] PATH...",
+		Short: "Walk trees and synchronize the scan database",
+		Args:  cobra.MinimumNArgs(1),
+		RunE: runE(func(ctx context.Context, args []string) error {
+			return runScan(ctx, args, scanWorkers, scanOneFS)
+		}),
+	}
+	scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
+		"concurrent workers for the walk and hash phases")
+	scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
+		"do not cross filesystem boundaries")
 
 	reportCmd := &cobra.Command{
 		Use:   cmdReport,
 		Short: "Read the scan database and print the file-level duplicates report",
 		Args:  cobra.NoArgs,
 		RunE: runE(func(ctx context.Context, _ []string) error {
-			return runReport(ctx, stdout)
+			return runReport(ctx)
 		}),
 	}
 
@@ -190,83 +145,22 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 		Short: "Read the scan database and print the duplicate-tree report",
 		Args:  cobra.NoArgs,
 		RunE: runE(func(ctx context.Context, _ []string) error {
-			return runTrees(ctx, stdout)
+			return runTrees(ctx)
 		}),
 	}
 
-	infoCmd := &cobra.Command{
+	root.AddCommand(scanCmd, reportCmd, treesCmd)
-		Use:   cmdInfo,
-		Short: "Read the scan database and print its details",
-		Args:  cobra.NoArgs,
-		RunE: runE(func(ctx context.Context, _ []string) error {
-			return runInfo(ctx, stdout)
-		}),
-	}
-
-	root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
 
 	return root
 }
 
-// newScanCommand builds the scan subcommand, with its --workers, -x and
+// runE adapts a subcommand implementation to cobra's RunE. Cobra
-// --no-default-excludes flags. Its help lists the default excludes of
+// prints the error and the command's usage text for every error RunE
-// the system it runs on, and the filesystem types it never records
+// returns, but a subcommand that ran and failed has no usage problem
-// anything on.
+// to report: both are silenced here, and the error is marked fatal so
-func newScanCommand() *cobra.Command {
+// that run reports it on stderr and exits 1 rather than 2. The command's
-	var (
+// context is handed to the implementation: cancelling it unwinds the
-		scanWorkers           int
+// scan's worker pools.
-		scanOneFS             bool
-		scanNoDefaultExcludes bool
-	)
-
-	short := "Walk trees and synchronize the scan database"
-
-	scanCmd := &cobra.Command{
-		Use:   cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
-		Short: short,
-		Long:  short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
-		Args:  cobra.MinimumNArgs(1),
-		PreRunE: func(cmd *cobra.Command, _ []string) error {
-			return checkScanWorkers(cmd, scanWorkers)
-		},
-		RunE: runE(func(ctx context.Context, args []string) error {
-			ctx, stop := interruptContext(ctx)
-			defer stop()
-
-			return runScan(ctx, args, scanWorkers, scanOneFS,
-				scanNoDefaultExcludes)
-		}),
-	}
-	scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
-		"concurrent workers for the walk, hash, and content phases")
-	scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
-		"do not cross filesystem boundaries")
-	scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
-		false, "enter the default excludes, the paths listed above")
-
-	return scanCmd
-}
-
-// checkScanWorkers rejects a scan --workers value below 1. That is a
-// usage error reported in one line: cobra prints the returned message
-// without the usage text, and run exits 2.
-func checkScanWorkers(cmd *cobra.Command, workers int) error {
-	if workers >= 1 {
-		return nil
-	}
-
-	cmd.SilenceUsage = true
-
-	return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
-}
-
-// runE adapts a subcommand implementation, or the version print, to
-// cobra's RunE. Cobra prints the error and the command's usage text for
-// every error RunE returns, but a subcommand that ran and failed has no
-// usage problem to report: both are silenced here, and the error is
-// marked fatal so that run reports it on stderr and exits 1 rather than
-// 2. The command's context is handed to the implementation: cancelling
-// it unwinds the scan's worker pools.
 func runE(
 	fn func(ctx context.Context, args []string) error,
 ) func(*cobra.Command, []string) error {

package.json

@@ -1,5 +0,0 @@
-{
-  "devDependencies": {
-    "prettier": "3.8.1"
-  }
-}

progress.go

@@ -2,520 +2,145 @@ package main
 
 import (
 	"fmt"
-	"io"
-	"math"
 	"os"
-	"slices"
-	"strings"
-	"sync"
 	"time"
 
 	"github.com/schollz/progressbar/v3"
-	"golang.org/x/term"
 )
 
-// plainInterval is the time between progress lines while a phase runs,
+// plainInterval is the minimum time between progress lines when stderr
-// when stderr is not a TTY.
+// is not a TTY.
 const plainInterval = 5 * time.Second
 
-// redrawInterval is the time between redraws of the progress display on
+// barThrottle caps how often the TTY progress bar redraws.
-// a TTY.
+const barThrottle = 100 * time.Millisecond
-const redrawInterval = 100 * time.Millisecond
 
-// rateWindow is the time the rate a display shows covers: the items
+// walkSpinnerType selects the progressbar library's spinner style used
-// completed over the last rateWindow, which is the time since the line
+// for the walk pass, which has no known total.
-// before when stderr is not a TTY.
+const walkSpinnerType = 14
-const rateWindow = 5 * time.Second
-
-// stallTime is how long a phase goes with no item completed before its
-// line says it has stalled, and where it waits, in place of the rate.
-const stallTime = 10 * time.Second
-
-// spinnerType selects the progressbar library's spinner style.
-const spinnerType = 14
-
-// barWidth is the width of the progressbar library's bar between its
-// two ends, as in the example line in README "Progress".
-const barWidth = 10
 
 // percentScale converts a fraction to a percentage.
 const percentScale = 100
 
-// fallbackWidth is the width taken for a terminal whose width cannot be
-// read, as the progressbar library takes it.
-const fallbackWidth = 80
-
-// spinnerMargin is the columns of a spinner's line besides its text:
-// the spinner and the three spaces the library draws around the text,
-// and the terminal's last column, which every line stays clear of so
-// that no terminal wraps it.
-const spinnerMargin = 5
-
-// barMargin is the columns of a bar's line besides its text: the
-// percent and the bar the library draws after the text, the spaces
-// around them, and the terminal's last column.
-const barMargin = barWidth + 11
-
-// countingRecords is what the display of a phase that reads the records
-// says while it counts them, before it has its total.
-const countingRecords = "counting the records"
-
-// openPhase is the name of the phase in which report, trees and info
-// open the database, and openingDatabase what its display says while
-// it does.
-const (
-	openPhase       = "open"
-	openingDatabase = "opening the database"
-)
-
 // stderrIsTTY reports whether stderr is attached to a terminal.
 func stderrIsTTY() bool {
-	return term.IsTerminal(int(os.Stderr.Fd()))
+	fi, err := os.Stderr.Stat()
-}
+	if err != nil {
-
+		return false
-// terminalWidth returns the width of the terminal on stderr, or
-// fallbackWidth, as the progressbar library does, when it cannot read
-// one.
-func terminalWidth() int {
-	width, _, err := term.GetSize(int(os.Stderr.Fd()))
-	if err != nil || width <= 0 {
-		return fallbackWidth
 	}
 
-	return width
+	return fi.Mode()&os.ModeCharDevice != 0
 }
 
-// phaseLabel names phase with its place among phases, as "[2/5] walk".
+// progress renders one scan pass's progress on stderr. On a TTY it
-func phaseLabel(phases []string, phase string) string {
+// delegates to the progressbar library (spinner style when the total is
-	return fmt.Sprintf("[%d/%d] %s",
+// unknown, full bar with count/percent/rate/elapsed/ETA otherwise). When
-		slices.Index(phases, phase)+1, len(phases), phase)
+// stderr is not a TTY it emits no ANSI redraws: it prints a plain
-}
+// one-line update no more often than every plainInterval.
-
-// progressParams describe one phase's progress display as it starts.
-type progressParams struct {
-	label      string // the phase and its place, from phaseLabel
-	unit       string // what the phase counts, as "files"
-	total      int64  // the items the phase will complete; -1 when not known
-	estimate   bool   // total is an estimate, which the count may pass
-	waitingFor string // what the phase waits for until setTotal is called
-	stalledOn  string // where the phase waits, as "disk/db"; see stallTime
-}
-
-// drawnCount is the count of items completed as of one draw, and when
-// that draw was.
-type drawnCount struct {
-	at    time.Time
-	count int64
-}
-
-// progress renders one phase's progress on stderr. On a TTY the
-// progressbar library draws line, fitted to the terminal's width: after
-// the library's spinner, or, for a phase with an exact total, before the
-// library's bar when the terminal has room for the bar beside it. It is
-// drawn again every redrawInterval, so its elapsed time keeps moving
-// while the phase waits for its next item. When stderr is not a TTY it
-// emits no ANSI redraws: it prints line as the phase starts, every
-// plainInterval while it runs, and as it ends.
 //
-// The draws come from a goroutine of their own, so mu guards the fields
+// All methods must be called from the main goroutine only. A nil
-// below it. Every method must be called from the main goroutine. A nil
+// *progress is a valid no-display receiver: every method is a no-op,
-// *progress is a valid no-display receiver: every method is a no-op, so
+// so batched database flushes during the streaming pass can reuse the
-// batched database flushes during the hash phase can reuse the update
+// update-pass helpers without rendering anything.
-// phase's helpers without rendering anything, and its writer writes
-// with no display to clear.
 type progress struct {
-	now   func() time.Time // reads the time
-	width func() int       // reads the terminal's width; nil when not a TTY
-	stop  chan struct{}    // closed by finish to end the draws
-	done  chan struct{}    // closed once the draws have ended
-
-	mu         sync.Mutex
-	bar        *progressbar.ProgressBar // what the library draws; nil until then
-	withBar    bool                     // bar is the library's bar, not its spinner
-	widest     int                      // the widest line bar has drawn, in columns
 	label string
-	unit       string
+	total int64 // -1 when unknown (walk pass)
-	total      int64
+	bar   *progressbar.ProgressBar
-	estimate   bool
-	waitingFor string
-	stalledOn  string
 	count int64
-	start      time.Time // when the phase started, for the elapsed time
+	start time.Time
-	rateStart  time.Time // when the count started, for the ETA
+	last  time.Time
-	lastItem   time.Time // when the last item was completed, or the count started
-	// drawn holds the count at each draw over the last rateWindow, oldest
-	// first; the rate counts from the first. It keeps the draw before the
-	// latest even when that is older, as the line before is when lines
-	// are plainInterval apart and a tick comes late.
-	drawn []drawnCount
 }
 
-// newProgress starts the display of one phase on stderr.
+func newProgress(label string, total int64) *progress {
-func newProgress(params progressParams) *progress {
+	p := &progress{label: label, total: total, start: time.Now()}
-	if stderrIsTTY() {
+	if !stderrIsTTY() {
-		return startProgress(params, terminalWidth, redrawInterval, time.Now)
+		return p
 	}
 
-	return startProgress(params, nil, plainInterval, time.Now)
+	opts := []progressbar.Option{
-}
+		progressbar.OptionSetWriter(os.Stderr),
-
+		progressbar.OptionSetDescription(label),
-// startProgress shows the phase params describes at once, and again
+		progressbar.OptionShowCount(),
-// every interval until finish: with the progressbar library on a
+		progressbar.OptionShowIts(),
-// terminal, whose width each draw reads by calling width, or as a plain
+		progressbar.OptionSetItsString("files"),
-// line when width is nil. The display reads the time by calling now.
+		progressbar.OptionSetElapsedTime(true),
-func startProgress(params progressParams, width func() int,
+		progressbar.OptionThrottle(barThrottle),
-	interval time.Duration, now func() time.Time,
+		// Render at zero immediately: a phase must be visible the
-) *progress {
+		// moment it starts, even before its first item completes.
-	start := now()
+		progressbar.OptionSetRenderBlankState(true),
-	p := &progress{
+	}
-		now: now, width: width,
+	if total >= 0 {
-		stop: make(chan struct{}), done: make(chan struct{}),
+		opts = append(opts,
-		label: params.label, unit: params.unit, total: params.total,
+			progressbar.OptionSetPredictTime(true),
-		estimate: params.estimate, waitingFor: params.waitingFor,
+			progressbar.OptionShowElapsedTimeOnFinish(),
-		stalledOn: params.stalledOn, start: start, rateStart: start,
+		)
-		lastItem: start, drawn: []drawnCount{{at: start}},
+	} else {
+		opts = append(opts,
+			progressbar.OptionSetPredictTime(false),
+			progressbar.OptionSpinnerType(walkSpinnerType),
+		)
 	}
 
-	// Show the zero state at once: the first item may take minutes,
+	p.bar = progressbar.NewOptions64(total, opts...)
-	// and a phase must never look hung.
-	p.draw()
-
-	go func() {
-		defer close(p.done)
-
-		ticker := time.NewTicker(interval)
-		defer ticker.Stop()
-
-		for {
-			select {
-			case <-p.stop:
-				return
-			case <-ticker.C:
-				p.draw()
-			}
-		}
-	}()
 
 	return p
 }
 
-// draw shows the phase's progress: on a TTY in place of the line
+// increment records one completed item and refreshes the display.
-// before, and otherwise as a plain line. It first adds the count to
-// drawn, then drops from drawn the draws older than rateWindow, except
-// the one before this draw.
-func (p *progress) draw() {
-	p.mu.Lock()
-	defer p.mu.Unlock()
-
-	now := p.now()
-
-	p.drawn = append(p.drawn, drawnCount{at: now, count: p.count})
-	for len(p.drawn) > 2 && now.Sub(p.drawn[0].at) > rateWindow {
-		p.drawn = p.drawn[1:]
-	}
-
-	if p.width == nil {
-		fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true))
-
-		return
-	}
-
-	width := p.width()
-
-	// A phase with an exact total gets the library's bar when its whole
-	// line, without the percent the bar shows, fits beside the bar: the
-	// bar gives way before any part of the line. A total of 0 leaves the
-	// bar nothing to fill.
-	text, withBar := "", false
-	if p.total > 0 && !p.estimate {
-		text = p.line(now, math.MaxInt, false)
-		withBar = len(text) <= width-barMargin
-	}
-
-	margin := barMargin
-	if !withBar {
-		margin = spinnerMargin
-		text = p.line(now, width-margin, true)
-	}
-
-	// The library makes a bar or a spinner, and before each draw erases
-	// its line with as many spaces as the widest line it has drawn. A
-	// display gets a new one when it turns from one to the other, as its
-	// line grows or the terminal is resized, and when the terminal has
-	// been made too narrow for that erase, which would wrap and leave a
-	// line behind at every draw.
-	if p.bar == nil || p.withBar != withBar || p.widest >= width {
-		p.newBar(withBar, width)
-	}
-
-	if withBar {
-		_ = p.bar.Set64(p.count)
-	}
-
-	p.bar.Describe(text)
-
-	// The library drew the text and its margin, less the terminal's last
-	// column, which the margin keeps clear.
-	p.widest = max(p.widest, len(text)+margin-1)
-}
-
-// newBar has the library make what draw draws: a bar for the phase's
-// total when withBar is set, and a spinner when not. Either shows the
-// text draw gives it, and nothing of its own besides the spinner, or
-// the bar and its percent. p.mu must be held.
-func (p *progress) newBar(withBar bool, width int) {
-	if p.bar != nil {
-		// Erase what was drawn before, short of the last column of a
-		// terminal that may have been made narrower since: the library's
-		// own erase is as wide as the widest line it drew.
-		blank := strings.Repeat(" ", min(p.widest, width-1))
-		fmt.Fprint(os.Stderr, "\r"+blank+"\r")
-	}
-
-	total := int64(-1) // the library's spinner
-	if withBar {
-		total = p.total
-	}
-
-	p.bar = progressbar.NewOptions64(total,
-		progressbar.OptionSetWriter(os.Stderr),
-		progressbar.OptionSetWidth(barWidth),
-		progressbar.OptionSpinnerType(spinnerType),
-		// The spinner turns at each draw, not on a timer of the
-		// library's own.
-		progressbar.OptionSetSpinnerChangeInterval(0),
-		// The text carries the elapsed time and the ETA.
-		progressbar.OptionSetElapsedTime(false),
-		progressbar.OptionSetPredictTime(false),
-	)
-	p.withBar, p.widest = withBar, 0
-}
-
-// setTotal gives a phase that started without a total its total, once
-// it is known. The rate, the ETA and the time with no item completed
-// count from then on: the time spent waiting for the total says nothing
-// about how fast items go.
-func (p *progress) setTotal(total int64) {
-	if p == nil {
-		return
-	}
-
-	now := p.now()
-
-	p.mu.Lock()
-	defer p.mu.Unlock()
-
-	p.total = total
-	p.rateStart, p.lastItem = now, now
-	p.drawn = []drawnCount{{at: now, count: p.count}}
-}
-
-// increment records one completed item, which the display shows at its
-// next draw.
 func (p *progress) increment() {
 	if p == nil {
 		return
 	}
 
-	now := p.now()
-
-	p.mu.Lock()
-	defer p.mu.Unlock()
-
 	p.count++
-	p.lastItem = now
+	if p.bar != nil {
+		_ = p.bar.Add(1)
+
+		return
+	}
+
+	if time.Since(p.last) >= plainInterval {
+		p.last = time.Now()
+		fmt.Fprintln(os.Stderr, p.plainLine())
+	}
 }
 
-// warnf prints a one-line warning to stderr without corrupting the
+// warnf prints a one-line warning to stderr without corrupting the bar.
-// display. The whole message is escaped like a report's path columns,
-// so a path holding a newline cannot split the warning.
 func (p *progress) warnf(format string, args ...any) {
 	if p == nil {
 		return
 	}
 
-	msg := escapePath(fmt.Sprintf(format, args...))
-
-	p.mu.Lock()
-	defer p.mu.Unlock()
-
 	if p.bar != nil {
-		// The warning takes the display's line; the next draw shows
-		// the display again, on the line below.
 		_ = p.bar.Clear()
 	}
 
-	fmt.Fprintln(os.Stderr, msg)
+	fmt.Fprintf(os.Stderr, format+"\n", args...)
 }
 
-// progressWriter writes a phase's output to out, which may be the file
+// finish terminates the pass's display.
-// the phase's display is drawn on, as stdout is in a plain "sfdupes
-// trees" on a terminal. It holds what is written to it and writes that
-// to out in pieces of about ioBufSize bytes. When out is the file
-// stderr is, each piece is written under the display's lock with the
-// display cleared first, as warnf does for a warning: no draw lands in
-// the middle of a piece, and the next draw shows the display again, on
-// the line below. Every write to it must end with a newline, as a row
-// written with one fmt.Fprintf does, so that each piece ends a line.
-// Flush writes what it holds once the last row is written.
-type progressWriter struct {
-	// p is the display to clear, nil unless out is the file stderr is.
-	p   *progress
-	out io.Writer
-	buf []byte
-}
-
-// writer returns a progressWriter writing to out around p's display.
-// It clears the display and takes its lock only when out is the file
-// stderr is: the same device and inode, as os.SameFile compares them,
-// which is one terminal, or one pipe or file that both are sent to.
-// Otherwise it writes to out without the lock, so the display keeps
-// being drawn while a write to out waits for a slow reader.
-func (p *progress) writer(out io.Writer) *progressWriter {
-	w := &progressWriter{out: out}
-
-	if f, ok := out.(*os.File); ok {
-		outInfo, outErr := f.Stat()
-		errInfo, errErr := os.Stderr.Stat()
-
-		if outErr == nil && errErr == nil && os.SameFile(outInfo, errInfo) {
-			w.p = p
-		}
-	}
-
-	return w
-}
-
-// Write holds b, and writes what it holds to out once that comes to
-// ioBufSize bytes.
-func (w *progressWriter) Write(b []byte) (int, error) {
-	w.buf = append(w.buf, b...)
-	if len(w.buf) < ioBufSize {
-		return len(b), nil
-	}
-
-	return len(b), w.Flush()
-}
-
-// Flush writes what w holds to out. When out is the file stderr is, the
-// display is cleared first and not drawn again until it is written.
-func (w *progressWriter) Flush() error {
-	if w.p != nil {
-		w.p.mu.Lock()
-		defer w.p.mu.Unlock()
-
-		if w.p.bar != nil {
-			_ = w.p.bar.Clear()
-		}
-	}
-
-	_, err := w.out.Write(w.buf)
-	w.buf = w.buf[:0]
-
-	return err
-}
-
-// finish ends the phase's display, drawn once more as the phase ends.
-// A phase stopped short of its total, as an interrupted one is, shows
-// the count it stopped at rather than a full bar.
 func (p *progress) finish() {
 	if p == nil {
 		return
 	}
 
-	close(p.stop)
+	if p.bar != nil {
-	<-p.done
+		_ = p.bar.Finish()
 
-	p.draw()
-
-	if p.width != nil {
-		// The display stays on the TTY as drawn; what follows starts
-		// on a line of its own.
 		fmt.Fprintln(os.Stderr)
+
+		return
 	}
+
+	fmt.Fprintln(os.Stderr, p.plainLine())
 }
 
-// line formats the phase's progress as of now in at most room columns,
+// plainLine formats a non-TTY progress line.
-// as
+func (p *progress) plainLine() string {
-//
+	elapsed := time.Since(p.start)
-//	[3/5] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s
-//
-// where "files" is the phase's unit, without the percent when percent
-// is false, as before the library's bar, which shows its own. A phase
-// with no total shows the items done instead, or what it waits for. The
-// rate is that since the first draw in drawn, and the ETA comes from
-// the rate since the count started. Once no item has been completed for
-// stallTime, the rate, which would read 0, gives way to the word
-// "stalled" and where the phase waits, as "stalled disk/db".
-//
-// The phase and its place, the progress, the elapsed time and the ETA
-// are kept whole. A line wider than room leaves out the rate, the words
-// of a stalled phase, or what the phase waits for; only if it is still
-// wider is it cut at room. Every part is ASCII, so its length is its
-// width in columns. p.mu must be held.
-func (p *progress) line(now time.Time, room int, percent bool) string {
-	elapsed := now.Sub(p.start).Round(time.Second)
-
-	// head and tail hold the parts kept whole; extra, between them, is
-	// the part left out.
-	var head, extra, tail string
-
-	if p.total < 0 && p.waitingFor != "" {
-		head, extra = p.label+": ", p.waitingFor+", "
-		tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed)
-	} else {
-		since := p.drawn[0]
-		rate := perSecond(p.count-since.count, now.Sub(since.at))
-		average := perSecond(p.count, now.Sub(p.rateStart))
-
-		head = p.label + ": " + p.itemsDone(percent) + ", "
-		extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit)
-		tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average))
-
-		if now.Sub(p.lastItem) >= stallTime {
-			extra = "stalled " + p.stalledOn + ", "
-		}
-	}
-
-	full, kept := head+extra+tail, head+tail
-
-	switch {
-	case len(full) <= room:
-		return full
-	case len(kept) <= room:
-		return kept
-	}
-
-	return kept[:max(room, 0)]
-}
-
-// perSecond returns count items over d as items a second, or 0 when no
-// time has passed.
-func perSecond(count int64, d time.Duration) float64 {
-	if d <= 0 {
-		return 0
-	}
-
-	return float64(count) / d.Seconds()
-}
-
-// itemsDone formats the items done, as "[1234/5000] 25%", with "~"
-// before a total that is an estimate and without the percent when
-// percent is false, or as "1234 files", in the phase's unit, when the
-// phase has no total. p.mu must be held.
-func (p *progress) itemsDone(percent bool) string {
 	if p.total < 0 {
-		return fmt.Sprintf("%d %s", p.count, p.unit)
+		return fmt.Sprintf("%s: %d files, elapsed %s",
-	}
+			p.label, p.count, elapsed.Round(time.Second))
-
-	approx := ""
-	if p.estimate {
-		approx = "~"
-	}
-
-	done := fmt.Sprintf("[%d/%s%d]", p.count, approx, p.total)
-	if !percent {
-		return done
 	}
 
 	pct := float64(percentScale)
@@ -523,24 +148,19 @@ func (p *progress) itemsDone(percent bool) string {
 		pct = percentScale * float64(p.count) / float64(p.total)
 	}
 
-	return fmt.Sprintf("%s %.0f%%", done, pct)
+	rate := 0.0
-}
+	if s := elapsed.Seconds(); s > 0 {
-
+		rate = float64(p.count) / s
-// eta returns the time the phase has left at rate items a second, or
-// "unknown": with no total, before the first item, and once the count
-// passes an estimate. p.mu must be held.
-func (p *progress) eta(rate float64) string {
-	switch {
-	case p.total < 0, p.estimate && p.count > p.total:
-		return "unknown"
-	case p.count >= p.total:
-		return "0s"
-	case rate <= 0:
-		return "unknown"
 	}
 
-	remaining := time.Duration(
+	eta := "?"
-		float64(p.total-p.count) / rate * float64(time.Second))
 
-	return remaining.Round(time.Second).String()
+	if rate > 0 && p.count <= p.total {
+		remaining := time.Duration(float64(p.total-p.count) / rate * float64(time.Second))
+		eta = remaining.Round(time.Second).String()
+	}
+
+	return fmt.Sprintf("%s: [%d/%d] %.0f%% %.0f files/s elapsed %s eta %s",
+		p.label, p.count, p.total, pct, rate,
+		elapsed.Round(time.Second), eta)
 }

progress_test.go

@@ -1,989 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"io"
-	"math"
-	"os"
-	"path/filepath"
-	"slices"
-	"strings"
-	"sync"
-	"testing"
-	"time"
-	"unicode/utf8"
-)
-
-// testRedraw is the redraw interval the tests give a display, so that
-// many redraws fit in a few milliseconds.
-const testRedraw = time.Millisecond
-
-// testDeadline bounds how long a test waits for a display to draw what
-// it expects.
-const testDeadline = 5 * time.Second
-
-// hashStalledOn is where the hash phase says it waits once it has
-// stalled, as readRuns gives it.
-const hashStalledOn = "disk/db"
-
-// testClock is a clock a test moves by hand, for a display that reads
-// it from its own goroutine too.
-type testClock struct {
-	mu  sync.Mutex
-	now time.Time
-}
-
-// Now returns the clock's time.
-func (c *testClock) Now() time.Time {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-
-	return c.now
-}
-
-// Add moves the clock on by d.
-func (c *testClock) Add(d time.Duration) {
-	c.mu.Lock()
-	defer c.mu.Unlock()
-
-	c.now = c.now.Add(d)
-}
-
-//nolint:paralleltest // replaces the process-wide os.Stderr
-func TestStderrIsTTYFalseForNonTerminals(t *testing.T) {
-	r, pipe, err := os.Pipe()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	regular, err := os.Create(filepath.Join(t.TempDir(), "stderr"))
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	devNull, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	saved := os.Stderr
-
-	t.Cleanup(func() {
-		os.Stderr = saved
-
-		for _, f := range []*os.File{r, pipe, regular, devNull} {
-			_ = f.Close()
-		}
-	})
-
-	cases := map[string]*os.File{
-		"a pipe":         pipe,
-		"a regular file": regular,
-		os.DevNull:       devNull,
-	}
-
-	for name, f := range cases {
-		os.Stderr = f
-
-		if stderrIsTTY() {
-			t.Errorf("stderrIsTTY() = true with stderr on %s", name)
-		}
-	}
-}
-
-func TestPhaseLabel(t *testing.T) {
-	t.Parallel()
-
-	phases := []string{"load", "walk", "hash"}
-	if got := phaseLabel(phases, "walk"); got != "[2/3] walk" {
-		t.Errorf("phaseLabel = %q, want %q", got, "[2/3] walk")
-	}
-}
-
-// TestProgressLine checks the line a display shows in each state, ten
-// seconds into the phase, with no draw since the first and the last
-// item just completed.
-func TestProgressLine(t *testing.T) {
-	t.Parallel()
-
-	now := time.Now()
-	ago := now.Add(-10 * time.Second)
-
-	cases := []struct {
-		p    *progress
-		want string
-	}{{
-		p: &progress{
-			label: scanPhase("load"), total: -1,
-			waitingFor: countingRecords,
-		},
-		want: "[1/5] load: counting the records, elapsed 10s, eta unknown",
-	}, {
-		p: &progress{
-			label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
-		},
-		want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
-	}, {
-		p:    &progress{label: scanPhase("hash"), unit: scanUnit, total: 100},
-		want: "[3/5] hash: [0/100] 0%, 0 files/s, elapsed 10s, eta unknown",
-	}, {
-		p:    &progress{label: scanPhase("update"), unit: scanUnit, total: 0},
-		want: "[4/5] update: [0/0] 100%, 0 files/s, elapsed 10s, eta 0s",
-	}, {
-		p: &progress{
-			label: scanPhase("walk"), unit: scanUnit, total: 100,
-			estimate: true, count: 20,
-		},
-		want: "[2/5] walk: [20/~100] 20%, 2 files/s, elapsed 10s, eta 40s",
-	}, {
-		// Past the estimate the walk cannot tell how long it has left.
-		p: &progress{
-			label: scanPhase("walk"), unit: scanUnit, total: 100,
-			estimate: true, count: 120,
-		},
-		want: "[2/5] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " +
-			"eta unknown",
-	}, {
-		// A first scan's walk has no estimate.
-		p: &progress{
-			label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20,
-		},
-		want: "[2/5] walk: 20 files, 2 files/s, elapsed 10s, eta unknown",
-	}, {
-		// The rate is in what the phase counts.
-		p: &progress{
-			label: treesPhase("group"), unit: "directories", total: 100,
-			count: 20,
-		},
-		want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " +
-			"eta 40s",
-	}}
-
-	for _, tc := range cases {
-		tc.p.start, tc.p.rateStart, tc.p.lastItem = ago, ago, now
-		tc.p.drawn = []drawnCount{{at: ago}}
-
-		if got := tc.p.line(now, math.MaxInt, true); got != tc.want {
-			t.Errorf("line = %q, want %q", got, tc.want)
-		}
-	}
-}
-
-// TestLineFitsRoom checks how a line is fitted to the room it has, ten
-// seconds into the phase, and fifty into a phase with no file read for
-// the last 42: it leaves out the rate, what the phase waits for, or the
-// words of a stalled phase, and is cut only when what is left is still
-// too wide.
-func TestLineFitsRoom(t *testing.T) {
-	t.Parallel()
-
-	now := time.Now()
-	ago := now.Add(-10 * time.Second)
-	hash := &progress{
-		label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
-		start: ago, rateStart: ago, lastItem: now,
-		drawn: []drawnCount{{at: ago}},
-	}
-	content := &progress{
-		label: scanPhase("content"), total: -1,
-		waitingFor: "searching the database", start: ago, rateStart: ago,
-	}
-
-	started := now.Add(-50 * time.Second)
-	stalled := &progress{
-		label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
-		stalledOn: hashStalledOn, start: started, rateStart: started,
-		lastItem: now.Add(-42 * time.Second),
-		drawn:    []drawnCount{{at: now.Add(-rateWindow), count: 20}},
-	}
-
-	cases := []struct {
-		p       *progress
-		room    int
-		percent bool
-		want    string
-	}{{
-		p: hash, room: 57, percent: true,
-		want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
-	}, {
-		p: hash, room: 56, percent: true,
-		want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40s",
-	}, {
-		p: hash, room: 45, percent: true,
-		want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40",
-	}, {
-		// Before the library's bar, which shows the percent.
-		p: hash, room: 57, percent: false,
-		want: "[3/5] hash: [20/100], 2 files/s, elapsed 10s, eta 40s",
-	}, {
-		p: content, room: 40, percent: true,
-		want: "[5/5] content: elapsed 10s, eta unknown",
-	}, {
-		p: stalled, room: 65, percent: true,
-		want: "[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " +
-			"eta 3m20s",
-	}, {
-		p: stalled, room: 64, percent: true,
-		want: "[3/5] hash: [20/100] 20%, elapsed 50s, eta 3m20s",
-	}}
-
-	for _, tc := range cases {
-		if got := tc.p.line(now, tc.room, tc.percent); got != tc.want {
-			t.Errorf("line(%d, %t) = %q, want %q",
-				tc.room, tc.percent, got, tc.want)
-		}
-	}
-}
-
-// TestSetTotalStartsRate checks that setTotal starts a phase's rate,
-// its ETA and its time with no record read from when the total becomes
-// known, not from when the phase started waiting for it: after 20
-// seconds of waiting for the total, the phase has not stalled 5 seconds
-// after it, and has 10 seconds after it with no record read.
-func TestSetTotalStartsRate(t *testing.T) {
-	t.Parallel()
-
-	clock := &testClock{now: time.Now()}
-	waited := clock.Now().Add(-20 * time.Second)
-	p := &progress{
-		now: clock.Now, label: scanPhase("load"), unit: scanUnit, total: -1,
-		stalledOn: "db", start: waited, rateStart: waited,
-		lastItem: waited, drawn: []drawnCount{{at: waited}},
-	}
-
-	p.setTotal(100)
-
-	clock.Add(5 * time.Second)
-
-	want := "[1/5] load: [0/100] 0%, 0 files/s, elapsed 25s, eta unknown"
-	if got := p.line(clock.Now(), math.MaxInt, true); got != want {
-		t.Errorf("line = %q, want %q", got, want)
-	}
-
-	clock.Add(5 * time.Second)
-
-	want = "[1/5] load: [0/100] 0%, stalled db, elapsed 30s, eta unknown"
-	if got := p.line(clock.Now(), math.MaxInt, true); got != want {
-		t.Errorf("line = %q, want %q", got, want)
-	}
-
-	// Then 20 records are read.
-	for range 20 {
-		p.increment()
-	}
-
-	want = "[1/5] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s"
-	if got := p.line(clock.Now(), math.MaxInt, true); got != want {
-		t.Errorf("line = %q, want %q", got, want)
-	}
-}
-
-// awaitOutput waits until stderr holds want at least times times, and
-// fails the test if it never does.
-func awaitOutput(t *testing.T, stderr func() string, want string,
-	times int,
-) {
-	t.Helper()
-
-	deadline := time.Now().Add(testDeadline)
-	for strings.Count(stderr(), want) < times {
-		if time.Now().After(deadline) {
-			t.Fatalf("stderr = %q, want %q at least %d times",
-				stderr(), want, times)
-		}
-
-		time.Sleep(testRedraw)
-	}
-}
-
-// TestPlainProgressPrintsAtStartAndEnd checks that when stderr is not a
-// terminal a phase prints its line the moment it starts, before its
-// first item, nothing for the items it completes before plainInterval
-// has passed, and its line again as it ends.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestPlainProgressPrintsAtStartAndEnd(t *testing.T) {
-	stderr := captureStderr(t)
-	p := newProgress(progressParams{
-		label: scanPhase("hash"), unit: scanUnit, total: 1000,
-	})
-
-	for range 1000 {
-		p.increment()
-	}
-
-	p.finish()
-
-	lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
-
-	want := "[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown"
-	if len(lines) != 2 || lines[0] != want ||
-		!strings.HasPrefix(lines[1], "[3/5] hash: [1000/1000] 100%, ") {
-		t.Errorf("stderr lines = %q, want %q, then the line at 1000",
-			lines, want)
-	}
-}
-
-// TestPlainProgressPrintsWhileWaiting checks that a phase waiting with
-// no completed items still prints a line every interval.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestPlainProgressPrintsWhileWaiting(t *testing.T) {
-	stderr := captureStderr(t)
-	p := startProgress(progressParams{
-		label: scanPhase("load"), total: -1,
-		waitingFor: countingRecords,
-	}, nil, testRedraw, time.Now)
-
-	awaitOutput(t, stderr,
-		"[1/5] load: counting the records, elapsed 0s, eta unknown\n", 3)
-	p.finish()
-}
-
-// TestPlainProgressStalledProducer checks that a phase whose files stop
-// coming, as when one large file is being read or a disk is slow, goes
-// on printing its line every interval: the rate falls to 0 once no file
-// has been read for rateWindow, and once none has for stallTime the
-// line says the phase has stalled, and where it waits, in place of the
-// rate. The test moves the display's clock by hand.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestPlainProgressStalledProducer(t *testing.T) {
-	stderr := captureStderr(t)
-	clock := &testClock{now: time.Now()}
-	p := startProgress(progressParams{
-		label: scanPhase("hash"), unit: scanUnit, total: 100,
-		stalledOn: hashStalledOn,
-	}, nil, testRedraw, clock.Now)
-
-	// Pass or fail, the display ends before stderr is put back.
-	t.Cleanup(p.finish)
-
-	// 20 files read as the phase starts, then none.
-	for range 20 {
-		p.increment()
-	}
-
-	clock.Add(5 * time.Second)
-	awaitOutput(t, stderr,
-		"[3/5] hash: [20/100] 20%, 4 files/s, elapsed 5s, eta 20s\n", 3)
-
-	clock.Add(time.Second)
-	awaitOutput(t, stderr,
-		"[3/5] hash: [20/100] 20%, 0 files/s, elapsed 6s, eta 24s\n", 3)
-
-	clock.Add(4 * time.Second)
-	awaitOutput(t, stderr,
-		"[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 10s, eta 40s\n",
-		3)
-}
-
-// TestPlainProgressRateAfterStall draws a phase's plain line by hand, a
-// little over plainInterval after the line before, as a late tick
-// does, through a stall and a burst of files after it: the rate covers
-// the files read since the line before, while the ETA comes from the
-// phase's average.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestPlainProgressRateAfterStall(t *testing.T) {
-	stderr := captureStderr(t)
-	clock := &testClock{now: time.Now()}
-
-	// No timed draws: the test draws.
-	p := startProgress(progressParams{
-		label: scanPhase("hash"), unit: scanUnit, total: 1000,
-		stalledOn: hashStalledOn,
-	}, nil, time.Hour, clock.Now)
-
-	late := plainInterval + time.Millisecond
-
-	clock.Add(late)
-	p.draw()
-	clock.Add(late)
-	p.draw()
-
-	for range 100 {
-		p.increment()
-	}
-
-	clock.Add(late)
-	p.finish()
-
-	want := []string{
-		"[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown",
-		"[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 5s, eta unknown",
-		"[3/5] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, eta unknown",
-		"[3/5] hash: [100/1000] 10%, 20 files/s, elapsed 15s, eta 2m15s",
-	}
-
-	lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
-	if !slices.Equal(lines, want) {
-		t.Errorf("stderr lines = %q, want %q", lines, want)
-	}
-}
-
-// TestProgressRateAfterStallRedrawn draws a phase by hand every
-// redrawInterval, as a terminal does, through a stall of stallTime and
-// a burst of files after it: the rate covers the files read over the
-// last rateWindow, and falls to 0 once the burst is older than that,
-// while the ETA comes from the phase's average. Plain lines show each
-// draw.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestProgressRateAfterStallRedrawn(t *testing.T) {
-	stderr := captureStderr(t)
-	clock := &testClock{now: time.Now()}
-
-	// No timed draws: the test draws.
-	p := startProgress(progressParams{
-		label: scanPhase("hash"), unit: scanUnit, total: 1000,
-		stalledOn: hashStalledOn,
-	}, nil, time.Hour, clock.Now)
-
-	// redraw draws the display every redrawInterval for d.
-	redraw := func(d time.Duration) {
-		for range d / redrawInterval {
-			clock.Add(redrawInterval)
-			p.draw()
-		}
-	}
-
-	redraw(stallTime)
-
-	for range 50 {
-		p.increment()
-	}
-
-	redraw(rateWindow + redrawInterval)
-	p.finish()
-
-	want := "[3/5] hash: [50/1000] 5%, 10 files/s, elapsed 15s, eta 4m45s\n" +
-		"[3/5] hash: [50/1000] 5%, 0 files/s, elapsed 15s, eta 4m47s\n"
-	if !strings.Contains(stderr(), want) {
-		t.Errorf("stderr = %q, want the lines %q", stderr(), want)
-	}
-}
-
-// lastFrame returns what a terminal shows of the frames a display drew:
-// the last one. The library starts each frame with a carriage return
-// and erases the previous one with spaces first.
-func lastFrame(out string) string {
-	var shown string
-
-	for frame := range strings.SplitSeq(out, "\r") {
-		if strings.TrimSpace(frame) != "" {
-			shown = frame
-		}
-	}
-
-	return shown
-}
-
-// TestTerminalRedrawsWhileWaiting checks that the terminal display of a
-// phase, spinner or bar, keeps being drawn while the phase waits for its
-// next item, with the elapsed time as of now, and shows every item
-// completed before the wait. stderr is not a terminal here, so the
-// displays take it to be fallbackWidth wide.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestTerminalRedrawsWhileWaiting(t *testing.T) {
-	stderr := captureStderr(t)
-
-	cases := []struct {
-		params progressParams
-		want   string // how the line shows a count of 50
-	}{
-		{
-			progressParams{label: scanPhase("walk"), unit: scanUnit, total: -1},
-			"50 files,",
-		},
-		{
-			progressParams{label: scanPhase("hash"), unit: scanUnit, total: 100},
-			"[50/100],",
-		},
-	}
-
-	for _, tc := range cases {
-		before := len(stderr())
-		since := func() string { return stderr()[before:] }
-		p := startProgress(tc.params, terminalWidth, testRedraw, time.Now)
-
-		for range 50 {
-			p.increment()
-		}
-
-		// An hour into the phase, without waiting an hour.
-		p.mu.Lock()
-		p.start = p.start.Add(-time.Hour)
-		p.mu.Unlock()
-
-		awaitOutput(t, since, "elapsed 1h0m0s", 1)
-		p.finish()
-
-		if shown := lastFrame(since()); !strings.Contains(shown, tc.want) {
-			t.Errorf("terminal shows %q, want %q", shown, tc.want)
-		}
-	}
-}
-
-// drawLate draws the display params describes on a terminal width
-// columns wide as it starts and once more as it ends, with count items
-// done two and a half hours into the phase, the last just now, and
-// returns what it wrote to stderr.
-func drawLate(stderr func() string, params progressParams, width int,
-	count int64,
-) string {
-	before := len(stderr())
-	clock := &testClock{now: time.Now()}
-
-	// No timed draws come between the two.
-	p := startProgress(params, func() int { return width }, time.Hour,
-		clock.Now)
-
-	// Without counting the items one by one or waiting the hours.
-	clock.Add(150 * time.Minute)
-
-	p.mu.Lock()
-	p.count, p.lastItem = count, clock.Now()
-	p.mu.Unlock()
-
-	p.finish()
-
-	return stderr()[before:]
-}
-
-// TestTerminalLinesKeepETA checks the line each display shows on an
-// 80-column terminal at a large scan's counts and times, and the line
-// of trees' read phase at the same counts: the rate is left out, so
-// that the elapsed time and the ETA stay whole.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestTerminalLinesKeepETA(t *testing.T) {
-	stderr := captureStderr(t)
-
-	cases := []struct {
-		params progressParams
-		want   string
-	}{{
-		// Past its estimate, the walk cannot tell how long it has left.
-		params: progressParams{
-			label: scanPhase("walk"), unit: scanUnit, total: 1200000,
-			estimate: true,
-		},
-		want: "[2/5] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " +
-			"eta unknown",
-	}, {
-		params: progressParams{
-			label: scanPhase("hash"), unit: scanUnit, total: 9876543,
-		},
-		want: "[3/5] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " +
-			"eta 17h30m0s",
-	}, {
-		params: progressParams{
-			label: scanPhase("content"), unit: scanUnit, total: 9876543,
-		},
-		want: "[5/5] content: [1234567/9876543] 12%, elapsed 2h30m0s, " +
-			"eta 17h30m0s",
-	}, {
-		params: progressParams{
-			label: treesPhase("read"), unit: "records", total: 9876543,
-		},
-		want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " +
-			"eta 17h30m0s",
-	}}
-
-	for _, tc := range cases {
-		out := drawLate(stderr, tc.params, 80, 1234567)
-		if shown := lastFrame(out); !strings.Contains(shown, tc.want) {
-			t.Errorf("terminal shows %q, want %q", shown, tc.want)
-		}
-	}
-}
-
-// TestTerminalStalledLines draws every phase's line on an 80-column
-// terminal once the phase has stalled, its last item done 42 seconds
-// before. At an ordinary scan's counts and times, 12345 of 45678 items
-// done four and a half minutes in, the line shows where the phase waits
-// whole, beside the phase, progress, elapsed time and ETA. At a large
-// scan's, 1234567 of 9876543 two and a half hours in, those four still
-// show whole, as the words give way first where they do not fit.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestTerminalStalledLines(t *testing.T) {
-	stderr := captureStderr(t)
-
-	// drawStalled draws the display params describes as it starts and
-	// once more as it ends, with count of total items done elapsed into
-	// the phase, and returns what the terminal shows.
-	drawStalled := func(params progressParams, total, count int64,
-		elapsed time.Duration,
-	) string {
-		before := len(stderr())
-		clock := &testClock{now: time.Now()}
-		params.total = total
-
-		// No timed draws come between the two.
-		p := startProgress(params, func() int { return 80 }, time.Hour,
-			clock.Now)
-
-		clock.Add(elapsed - 42*time.Second)
-
-		p.mu.Lock()
-		p.count, p.lastItem = count, clock.Now()
-		p.mu.Unlock()
-
-		clock.Add(42 * time.Second)
-		p.finish()
-
-		return lastFrame(stderr()[before:])
-	}
-
-	phases := []progressParams{
-		{label: scanPhase("load"), stalledOn: "db"},
-		{label: scanPhase("walk"), estimate: true, stalledOn: "disk"},
-		{label: scanPhase("hash"), stalledOn: hashStalledOn},
-		{label: scanPhase("update"), stalledOn: "db"},
-		// The content phase's checks and reads, then its last write.
-		{label: scanPhase("content"), stalledOn: "disk/db"},
-		{label: scanPhase("content"), stalledOn: "db"},
-		{label: treesPhase("read"), stalledOn: "db"},
-		{label: treesPhase("group"), stalledOn: "cpu"},
-		{label: treesPhase("write"), stalledOn: "out"},
-		{label: reportPhase("read"), stalledOn: "db/out"},
-	}
-
-	for _, params := range phases {
-		approx := ""
-		if params.estimate {
-			approx = "~"
-		}
-
-		shown := drawStalled(params, 45678, 12345,
-			4*time.Minute+30*time.Second)
-		want := params.label + ": [12345/" + approx + "45678] 27%, stalled " +
-			params.stalledOn + ", elapsed 4m30s, eta 12m9s"
-
-		if !strings.Contains(shown, want) {
-			t.Errorf("terminal shows %q, want %q", shown, want)
-		}
-
-		shown = drawStalled(params, 9876543, 1234567, 150*time.Minute)
-		for _, want := range []string{
-			params.label + ": [1234567/" + approx + "9876543] 12%, ",
-			", elapsed 2h30m0s, eta 17h30m0s",
-		} {
-			if !strings.Contains(shown, want) {
-				t.Errorf("terminal shows %q, want %q", shown, want)
-			}
-		}
-	}
-}
-
-// TestTerminalLinesFitWidth draws a large scan's bar display and a small
-// scan's, two and a half hours into the phase, on terminals from 120
-// columns down to 40, and checks that every line stays clear of the
-// terminal's last column. A line that reaches it wraps, and every
-// redraw then leaves a line behind. The bar is drawn where it has room,
-// as at 120 columns, and left out where it has none.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestTerminalLinesFitWidth(t *testing.T) {
-	stderr := captureStderr(t)
-
-	for _, width := range []int{120, 100, 80, 72, 60, 50, 40} {
-		for _, total := range []int64{9876543, 20000} {
-			params := progressParams{
-				label: scanPhase("content"), unit: scanUnit, total: total,
-			}
-			out := drawLate(stderr, params, width, total/8)
-
-			if width == 120 && !strings.Contains(out, "% |") {
-				t.Errorf("no bar on a terminal %d wide: %q", width, out)
-			}
-
-			lines := strings.FieldsFuncSeq(out, func(r rune) bool {
-				return r == '\r' || r == '\n'
-			})
-
-			for line := range lines {
-				if n := utf8.RuneCountInString(line); n >= width {
-					t.Errorf("a line of %d columns on a terminal %d wide: %q",
-						n, width, line)
-				}
-			}
-		}
-	}
-}
-
-// TestTerminalNarrowedRedrawsInPlace draws a large scan's display, two
-// and a half hours into the phase, on a terminal 120 columns wide, then
-// makes the terminal narrower than that line and draws it again a few
-// times. Nothing written after that, the erase before each redraw
-// included, may reach the narrower terminal's last column: it would
-// wrap, and every redraw would leave a line behind. At these widths the
-// spinner stays a spinner and the bar gives way to the spinner.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
-	stderr := captureStderr(t)
-
-	cases := []struct {
-		params progressParams
-		narrow int
-	}{
-		{
-			progressParams{label: scanPhase("walk"), unit: scanUnit, total: -1},
-			60,
-		},
-		{
-			progressParams{
-				label: scanPhase("hash"), unit: scanUnit, total: 9876543,
-			},
-			90,
-		},
-	}
-
-	for _, tc := range cases {
-		width := 120
-		clock := &testClock{now: time.Now()}
-
-		// No timed draws: the test draws.
-		p := startProgress(tc.params, func() int { return width }, time.Hour,
-			clock.Now)
-
-		clock.Add(150 * time.Minute)
-
-		p.mu.Lock()
-		p.count, p.lastItem = 1234567, clock.Now()
-		p.mu.Unlock()
-
-		p.draw()
-
-		width = tc.narrow
-		before := len(stderr())
-
-		p.draw()
-		p.draw()
-		p.finish()
-
-		lines := strings.FieldsFuncSeq(stderr()[before:], func(r rune) bool {
-			return r == '\r' || r == '\n'
-		})
-
-		for line := range lines {
-			if n := utf8.RuneCountInString(line); n >= width {
-				t.Errorf("%d columns written on a terminal made %d wide: %q",
-					n, width, line)
-			}
-		}
-	}
-}
-
-// TestProgressWarningsOnOwnLines drives the terminal displays, spinner
-// and bar, through a run of warnings with no items between them, as when
-// the walk meets many unreadable paths, for many redraws: every warning
-// must land on a line of its own, never inside a redraw.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestProgressWarningsOnOwnLines(t *testing.T) {
-	stderr := captureStderr(t)
-
-	cases := []progressParams{
-		{label: scanPhase("walk"), total: -1},
-		{label: scanPhase("hash"), total: 10},
-	}
-
-	issued := 0
-
-	for _, params := range cases {
-		p := startProgress(params, terminalWidth, testRedraw, time.Now)
-
-		// No pause between warnings: one written straight to stderr is
-		// garbled only if a redraw lands while it is being written.
-		for start := time.Now(); time.Since(start) < 50*testRedraw; issued++ {
-			p.warnf("warning")
-		}
-
-		p.finish()
-	}
-
-	// A terminal shows each line as the text after its last carriage
-	// return.
-	shown := 0
-
-	for line := range strings.SplitSeq(stderr(), "\n") {
-		if !strings.Contains(line, "warning") {
-			continue
-		}
-
-		shown++
-
-		if text := line[strings.LastIndex(line, "\r")+1:]; text != "warning" {
-			t.Errorf("terminal shows %q, want %q", text, "warning")
-		}
-	}
-
-	if shown != issued {
-		t.Errorf("%d warning lines, want %d", shown, issued)
-	}
-}
-
-// TestProgressWriterRowsOnOwnLines drives the terminal displays,
-// spinner and bar, through a run of rows written through the display's
-// writer to the same terminal, as stdout is in a plain "sfdupes trees",
-// each written out at once, for many redraws: the terminal must show
-// every row whole, in order, on a line of its own.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestProgressWriterRowsOnOwnLines(t *testing.T) {
-	stderr := captureStderr(t)
-
-	cases := []progressParams{
-		{
-			label: treesPhase("open"), total: -1,
-			waitingFor: "opening the database",
-		},
-		{label: treesPhase("write"), unit: "groups", total: 10},
-	}
-
-	var want []string
-
-	for _, params := range cases {
-		p := startProgress(params, terminalWidth, testRedraw, time.Now)
-		out := p.writer(os.Stderr)
-
-		// No pause between rows: one is garbled only if a redraw lands
-		// on its line.
-		for start := time.Now(); time.Since(start) < 50*testRedraw; {
-			row := fmt.Sprintf("row %d", len(want))
-			want = append(want, row)
-
-			_, err := fmt.Fprintln(out, row)
-			if err == nil {
-				err = out.Flush()
-			}
-
-			if err != nil {
-				t.Fatal(err)
-			}
-		}
-
-		p.finish()
-	}
-
-	// A terminal shows each line as the text after its last carriage
-	// return.
-	var shown []string
-
-	for line := range strings.SplitSeq(stderr(), "\n") {
-		if strings.Contains(line, "row ") {
-			shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
-		}
-	}
-
-	if len(shown) != len(want) {
-		t.Fatalf("terminal shows %d rows, want %d", len(shown), len(want))
-	}
-
-	for i := range want {
-		if shown[i] != want[i] {
-			t.Fatalf("terminal shows %q, want %q", shown[i], want[i])
-		}
-	}
-}
-
-// TestProgressWriterStalledStdoutKeepsLines writes a row larger than a
-// pipe holds through the display's writer to a pipe nobody reads yet,
-// as stdout is when its reader has stopped while stderr goes to a file:
-// while the write waits, the display must go on printing its line.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestProgressWriterStalledStdoutKeepsLines(t *testing.T) {
-	stderr := captureStderr(t)
-
-	r, w, err := os.Pipe()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	t.Cleanup(func() {
-		_ = w.Close()
-		_ = r.Close()
-	})
-
-	p := startProgress(progressParams{label: treesPhase("write"), total: 1},
-		nil, testRedraw, time.Now)
-	out := p.writer(w)
-	written := make(chan error, 1)
-
-	go func() {
-		_, writeErr := fmt.Fprintln(out, strings.Repeat("x", ioBufSize))
-		written <- writeErr
-	}()
-
-	// Pass or fail, the pipe is then read so that the write ends, and
-	// the display ends before stderr is put back.
-	t.Cleanup(func() {
-		go func() {
-			_, _ = io.Copy(io.Discard, r)
-		}()
-
-		writeErr := <-written
-		if writeErr != nil {
-			t.Error(writeErr)
-		}
-
-		p.finish()
-	})
-
-	// The first byte read shows that the write has started; the rest
-	// of the row waits until the pipe is read.
-	_, err = r.Read(make([]byte, 1))
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	line := "[4/4] write: "
-	awaitOutput(t, stderr, line, strings.Count(stderr(), line)+3)
-}
-
-// TestTerminalStoppedShortKeepsCount checks that the terminal display
-// of a phase that stops before its total, as an interrupted one does,
-// is left showing the count it stopped at rather than a full bar.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestTerminalStoppedShortKeepsCount(t *testing.T) {
-	stderr := captureStderr(t)
-	p := startProgress(progressParams{label: scanPhase("hash"), total: 10},
-		terminalWidth, redrawInterval, time.Now)
-
-	p.increment()
-	p.increment()
-	p.finish()
-
-	if shown := lastFrame(stderr()); !strings.Contains(shown, "[2/10],") {
-		t.Errorf("terminal shows %q, want a count of 2 of 10", shown)
-	}
-}
-
-// TestWarningAfterLastItemKeepsBar checks that a warning printed after
-// a phase's last item, as for a failed read of the last file, leaves the
-// phase's full bar on the terminal below the warning.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestWarningAfterLastItemKeepsBar(t *testing.T) {
-	stderr := captureStderr(t)
-	p := startProgress(progressParams{label: scanPhase("hash"), total: 2},
-		terminalWidth, testRedraw, time.Now)
-
-	p.increment()
-	p.increment()
-
-	awaitOutput(t, stderr, "[2/2],", 1)
-	p.warnf("warning")
-	p.finish()
-
-	if shown := lastFrame(stderr()); !strings.Contains(shown, "[2/2],") {
-		t.Errorf("terminal shows %q, want a count of 2 of 2", shown)
-	}
-}

report.go

@@ -1,14 +1,12 @@
 package main
 
 import (
+	"bufio"
 	"context"
-	"database/sql"
 	"fmt"
-	"io"
 	"os"
 	"slices"
 	"strings"
-	"time"
 )
 
 // ioBufSize is the buffer size for the buffered stdout writers.
@@ -19,248 +17,140 @@ const ioBufSize = 1 << 20
 const minGroupSize = 2
 
 // scanRec is one file record from the database. The signature (size,
-// head, tail, content) is the duplicate key; mtime is informational
+// head, tail) is the duplicate key; mtime is informational only and
-// only and used by scan for change detection.
+// used by scan for change detection.
 type scanRec struct {
 	size  int64
-	mtime   time.Time
+	mtime int64
 	head  string
 	tail  string
-	content string
 	path  string
 }
 
-// reportPhase names one phase of report with its place among all of
+// loadRecords opens the database and reads every file record for the
-// them, as "[2/2] read", for the phase's progress display. The list
+// report and trees subcommands. Any database problem — including a
-// here is the one place the phases are counted.
+// missing database — is fatal. The error is returned rather than
-func reportPhase(name string) string {
+// exiting, so that the deferred close — which checkpoints the SQLite
-	return phaseLabel([]string{openPhase, "read"}, name)
+// WAL — always runs; the database is closed before the caller formats
-}
+// its output, so it stays closed even if that output fails.
+func loadRecords(ctx context.Context) ([]scanRec, error) {
+	dbPath := databasePath()
 
-// runReport implements the report subcommand: it prints the file-level
-// duplicates report as TSV on stdout. It never touches the scanned
-// filesystem; its only I/O is the database, stdout, and stderr. It
-// names the database on stderr before it opens it. Any database
-// problem, including a missing database, is fatal.
-//
-// The work runs in two phases, each under a progress display of its
-// own on stderr: open (open the database) and read (count the records,
-// then read them in signature order, writing each size's duplicate
-// groups to stdout once that size is read; see readReport). Each
-// display ends before the next phase starts, and before an error is
-// returned to be printed.
-func runReport(ctx context.Context, stdout io.Writer) error {
-	dbPath, source, err := databasePath()
-	if err != nil {
-		return err
-	}
-
-	fmt.Fprintf(os.Stderr, "report: database %s (%s)\n",
-		escapePath(dbPath), source)
-
-	// Opening can wait on a busy database for as long as the busy
-	// timeout.
-	prog := newProgress(progressParams{
-		label: reportPhase(openPhase), total: -1,
-		waitingFor: openingDatabase,
-	})
 	db, err := openReportDatabase(ctx, dbPath)
-
-	prog.finish()
-
 	if err != nil {
-		return err
+		return nil, err
 	}
 
 	defer func() { _ = db.Close() }()
 
-	// The read phase writes the report to stdout while it reads the
+	recs, err := loadFileRows(ctx, db)
-	// records, so it may wait on either.
+	if err != nil {
-	prog = newProgress(progressParams{
+		return nil, fmt.Errorf("database %s: %w", dbPath, err)
-		label: reportPhase("read"), unit: "records", total: -1,
+	}
-		waitingFor: countingRecords, stalledOn: "db/out",
-	})
-	report, err := readReport(ctx, db, dbPath, stdout, prog)
 
-	prog.finish()
+	return recs, nil
+}
 
+// dupeGroup is one set of candidate-duplicate files: identical size,
+// head hash, and tail hash. paths is sorted lexicographically; the
+// first entry is the group's "first", the rest are dupes.
+type dupeGroup struct {
+	size  int64
+	paths []string
+}
+
+// runReport implements the report subcommand: it reads every record
+// from the database and prints the file-level duplicates report as TSV
+// on stdout. It never touches the scanned filesystem; its only I/O is
+// the database, stdout, and stderr.
+func runReport(ctx context.Context) error {
+	recs, err := loadRecords(ctx)
 	if err != nil {
 		return err
 	}
 
+	dupes := collectDupeGroups(recs)
+
+	out := bufio.NewWriterSize(os.Stdout, ioBufSize)
+
+	_, err = fmt.Fprintln(out, "first\tdupe\tsize")
+	if err != nil {
+		return fmt.Errorf("write stdout: %w", err)
+	}
+
+	dupeFiles := 0
+
+	var reclaimable int64
+
+	for _, g := range dupes {
+		for _, p := range g.paths[1:] {
+			_, err = fmt.Fprintf(out, "%s\t%s\t%d\n",
+				g.paths[0], p, g.size)
+			if err != nil {
+				return fmt.Errorf("write stdout: %w", err)
+			}
+
+			dupeFiles++
+			reclaimable += g.size
+		}
+	}
+
+	err = out.Flush()
+	if err != nil {
+		return fmt.Errorf("write stdout: %w", err)
+	}
+
 	fmt.Fprintf(os.Stderr,
 		"report: %d records read, %d duplicate groups, %d dupe files, "+
 			"%s reclaimable\n",
-		report.records, report.dupeGroups, report.dupeFiles,
+		len(recs), len(dupes), dupeFiles, humanBytes(reclaimable))
-		humanBytes(report.reclaimable))
 
 	return nil
 }
 
-// readReport gives prog the number of records in db, the database at
+// collectDupeGroups groups records by signature and returns every group
-// dbPath, as its total, then reads every record in signature order,
+// with two or more paths, each group's paths sorted lexicographically,
-// counting each on prog, and writes the report to stdout: the header
+// groups ordered by size descending then by first path ascending.
-// line, then the rows of each size's duplicate groups once that size is
+func collectDupeGroups(recs []scanRec) []dupeGroup {
-// read. stdout may be the terminal prog is drawn on, so the lines go
+	groups := make(map[fileSig][]string)
-// through prog's writer. It returns what it wrote, with the totals for
-// the summary line.
-func readReport(ctx context.Context, db *sql.DB, dbPath string,
-	stdout io.Writer, prog *progress,
-) (*reportWriter, error) {
-	out := prog.writer(stdout)
-	report := &reportWriter{out: out}
 
-	_, err := fmt.Fprintln(out, "first\tdupe\tsize")
+	for _, r := range recs {
-	if err != nil {
+		// A record without hashes (its size was unique when last
-		return nil, fmt.Errorf("write stdout: %w", err)
+		// scanned) has unknown content and is never reported as a
+		// duplicate.
+		if r.head == "" {
+			continue
 		}
 
-	var writeErr error
+		k := fileSig{size: r.size, head: r.head, tail: r.tail}
+		groups[k] = append(groups[k], r.path)
+	}
 
-	err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
+	var dupes []dupeGroup
-		prog.increment()
 
-		writeErr = report.add(r)
+	for k, paths := range groups {
+		if len(paths) < minGroupSize {
+			continue
+		}
 
-		return writeErr
+		slices.Sort(paths)
+		dupes = append(dupes, dupeGroup{size: k.size, paths: paths})
+	}
+
+	// Biggest reclaimable space first; ties broken by first path.
+	slices.SortFunc(dupes, func(a, b dupeGroup) int {
+		if a.size != b.size {
+			if a.size > b.size {
+				return -1
+			}
+
+			return 1
+		}
+
+		return strings.Compare(a.paths[0], b.paths[0])
 	})
 
-	if writeErr != nil {
+	return dupes
-		return nil, fmt.Errorf("write stdout: %w", writeErr)
-	}
-
-	if err != nil {
-		return nil, fmt.Errorf("database %s: %w", dbPath, err)
-	}
-
-	err = report.finish()
-	if err == nil {
-		err = out.Flush()
-	}
-
-	if err != nil {
-		return nil, fmt.Errorf("write stdout: %w", err)
-	}
-
-	return report, nil
-}
-
-// reportWriter writes the report's rows to out from the records, added
-// in signature order, largest size first, as loadSignatureRows reads
-// them. The records sharing a signature come together, and so do the
-// records of one size, so it holds only the duplicate groups of the
-// size being read, and writes them, ordered by first path, once a
-// record of a smaller size comes. It counts what the summary line
-// reports.
-type reportWriter struct {
-	out io.Writer
-
-	sig    scanRec    // the signature being read; its path is not used
-	paths  []string   // the paths read so far with sig
-	groups [][]string // the duplicate groups of sig.size, each sorted
-
-	records, dupeGroups, dupeFiles int
-	reclaimable                    int64
-}
-
-// add takes the next record. One with the signature being read joins
-// its paths; any other ends them, and one of a smaller size first
-// writes the groups of the size before.
-func (w *reportWriter) add(r scanRec) error {
-	w.records++
-
-	// A record without a content hash has unknown content (README
-	// "Database"), so it is never a duplicate.
-	if r.content == "" {
-		return nil
-	}
-
-	if r.size == w.sig.size && r.head == w.sig.head &&
-		r.tail == w.sig.tail && r.content == w.sig.content {
-		w.paths = append(w.paths, r.path)
-
-		return nil
-	}
-
-	w.endGroup()
-
-	if r.size != w.sig.size {
-		err := w.writeGroups()
-		if err != nil {
-			return err
-		}
-	}
-
-	w.sig = r
-	w.paths = append(w.paths, r.path)
-
-	return nil
-}
-
-// endGroup ends the paths read with w.sig: two or more are a duplicate
-// group, held with its paths sorted until writeGroups writes it.
-func (w *reportWriter) endGroup() {
-	if len(w.paths) < minGroupSize {
-		// A path with no duplicate is dropped, and its slice reused.
-		w.paths = w.paths[:0]
-
-		return
-	}
-
-	slices.Sort(w.paths)
-	w.groups = append(w.groups, w.paths)
-	w.paths = nil
-}
-
-// writeGroups writes the duplicate groups held, all of size w.sig.size,
-// ordered by first path: for each group, a row for every path but the
-// first. No two groups share a first path, since no two records share
-// a path.
-func (w *reportWriter) writeGroups() error {
-	slices.SortFunc(w.groups, func(a, b []string) int {
-		return strings.Compare(a[0], b[0])
-	})
-
-	for _, g := range w.groups {
-		for _, path := range g[1:] {
-			_, err := fmt.Fprintf(w.out, "%s\t%s\t%d\n",
-				escapePath(g[0]), escapePath(path), w.sig.size)
-			if err != nil {
-				return err
-			}
-
-			w.dupeFiles++
-			w.reclaimable += w.sig.size
-		}
-
-		w.dupeGroups++
-	}
-
-	w.groups = nil
-
-	return nil
-}
-
-// finish writes the groups of the last size read.
-func (w *reportWriter) finish() error {
-	w.endGroup()
-
-	return w.writeGroups()
-}
-
-// escapePath returns a path as it is written in a report column (README
-// "Report output format"): a backslash, tab, newline or carriage return
-// becomes \\, \t, \n or \r, and every other byte is kept as it is.
-// Grouping and sorting use the raw path, never this form.
-func escapePath(p string) string {
-	// Most paths need no escaping; skip building a replacer for them.
-	if !strings.ContainsAny(p, "\\\t\n\r") {
-		return p
-	}
-
-	return strings.NewReplacer(
-		`\`, `\\`, "\t", `\t`, "\n", `\n`, "\r", `\r`,
-	).Replace(p)
 }
 
 // humanBytes formats a byte count in human units (binary prefixes).

report_test.go

@@ -1,404 +1,26 @@
 package main
 
 import (
-	"bytes"
-	"database/sql"
-	"errors"
-	"fmt"
-	"io"
-	"os"
-	"path/filepath"
 	"slices"
-	"strconv"
-	"strings"
 	"testing"
-	"time"
 )
 
-// awkwardDir is a directory name holding every byte the reports escape.
+func TestCollectDupeGroups(t *testing.T) {
-const awkwardDir = "/d/\tone\ntwo\rthree\\four"
-
-// awkwardPairRecs is a duplicate pair in sibling directories /d/A and
-// awkwardDir. A raw tab sorts before "A" but its escaped form `\t`
-// sorts after it, so awkwardDir coming first shows that sorting uses
-// the raw path.
-func awkwardPairRecs() []scanRec {
-	return []scanRec{
-		{size: 5, head: "h", tail: "t", content: "c", path: "/d/A/f"},
-		{size: 5, head: "h", tail: "t", content: "c", path: awkwardDir + "/f"},
-	}
-}
-
-// seedDatabase writes recs into a fresh database, closes it as scan
-// does, out of WAL mode, and returns its path.
-func seedDatabase(t *testing.T, recs []scanRec) string {
-	t.Helper()
-
-	path := testDBPath(t)
-
-	db, err := openScanDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	err = applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	closeScanDatabase(t.Context(), db, path)
-
-	return path
-}
-
-// dupeGroup is one duplicate group as report reads it: the size, and
-// the paths in report order, first path first.
-type dupeGroup struct {
-	size  int64
-	paths []string
-}
-
-// dupeGroups returns the duplicate groups report reads from db, in
-// report order, from the rows it writes: a group's rows come together,
-// each naming the group's first path.
-func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
-	t.Helper()
-
-	var out bytes.Buffer
-
-	_, err := readReport(t.Context(), db, "", &out, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	var groups []dupeGroup
-
-	rows := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n")
-	for _, row := range rows[1:] { // after the header
-		fields := strings.Split(row, "\t")
-		if len(fields) != 3 {
-			t.Fatalf("row %q, want first, dupe and size", row)
-		}
-
-		size, err := strconv.ParseInt(fields[2], 10, 64)
-		if err != nil {
-			t.Fatal(err)
-		}
-
-		if len(groups) == 0 || groups[len(groups)-1].paths[0] != fields[0] {
-			groups = append(groups,
-				dupeGroup{size: size, paths: []string{fields[0]}})
-		}
-
-		g := &groups[len(groups)-1]
-		g.paths = append(g.paths, fields[1])
-	}
-
-	return groups
-}
-
-// dupeGroupsOf writes recs into a fresh database and returns the
-// duplicate groups report reads from it.
-func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
-	t.Helper()
-
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	return dupeGroups(t, db)
-}
-
-func TestRunReportEscapesPaths(t *testing.T) {
-	t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
-
-	var stdout, stderr bytes.Buffer
-
-	code := run([]string{cmdReport}, &stdout, &stderr)
-	if code != exitOK {
-		t.Fatalf("run(report) = %d, want %d; stderr: %s",
-			code, exitOK, stderr.String())
-	}
-
-	want := "first\tdupe\tsize\n" +
-		`/d/\tone\ntwo\rthree\\four/f` + "\t/d/A/f\t5\n"
-	if got := stdout.String(); got != want {
-		t.Errorf("stdout = %q, want %q", got, want)
-	}
-}
-
-func TestReportStdoutFailsWhileReading(t *testing.T) {
-	t.Parallel()
-
-	// Every two records are a duplicate pair of a size of their own, and
-	// a pair's row is written once a smaller size is read. Each row
-	// holds two paths longer than dir, so the rows come to more than
-	// twice the stdout buffer, and stdout fails while records are still
-	// being read, not once every record is.
-	dir := "/" + strings.Repeat("d", 4096)
-
-	recs := make([]scanRec, 2*ioBufSize/len(dir))
-	for i := range recs {
-		recs[i] = scanRec{
-			size: int64(i/2) + 1, head: "h", tail: "t", content: "c",
-			path: fmt.Sprintf("%s/%d", dir, i),
-		}
-	}
-
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// Counts the records read, and draws nothing.
-	prog := &progress{now: time.Now}
-
-	_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
-	if !errors.Is(err, errWriteFailed) ||
-		!strings.HasPrefix(err.Error(), "write stdout: ") {
-		t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
-	}
-
-	if prog.count >= int64(len(recs)) {
-		t.Errorf("stdout failed after all %d records were read, want it "+
-			"to stop the read", prog.count)
-	}
-}
-
-// reportOpenLine is the plain line of report's open phase when it opens
-// the database at once; it prints it as the phase starts and as it ends.
-const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
-	"eta unknown"
-
-// smokeReport is report's stdout for the smoke-test records.
-const smokeReport = "first\tdupe\tsize\n" +
-	"/d/t1/f1\t/d/t2/f1\t3000\n" +
-	"/d/t1/f1\t/d/t3/f1\t3000\n" +
-	"/d/t1/sub/f2\t/d/t2/sub/f2\t100\n" +
-	"/d/t1/sub/f2\t/d/t3/sub/f2renamed\t100\n"
-
-// TestRunReportProgressNamesEachPhase runs report on the smoke-test
-// records with stderr not a terminal, and checks every line it prints
-// there. After the line naming the database, each phase prints a line
-// naming it with its place among the two as it starts and another as
-// it ends; read counts the 6 records before it reads them. The summary
-// comes last, and stdout holds the report alone.
-func TestRunReportProgressNamesEachPhase(t *testing.T) {
-	path := seedDatabase(t, smokeTreeRecs())
-	t.Setenv(databaseEnv, path)
-
-	stderr := captureStderr(t)
-
-	var stdout bytes.Buffer
-
-	code := run([]string{cmdReport}, &stdout, os.Stderr)
-	if code != exitOK {
-		t.Fatalf("run(report) = %d, want %d; stderr: %s",
-			code, exitOK, stderr())
-	}
-
-	if got := stdout.String(); got != smokeReport {
-		t.Errorf("stdout = %q, want %q", got, smokeReport)
-	}
-
-	// How each line starts; the rate at the end of a phase varies.
-	want := []string{
-		"report: database " + path + " (from " + databaseEnv + ")",
-		reportOpenLine,
-		reportOpenLine,
-		"[2/2] read: counting the records, elapsed 0s, eta unknown",
-		"[2/2] read: [6/6] 100%, ",
-		"report: 6 records read, 2 duplicate groups, 4 dupe files, " +
-			"6.1 KiB reclaimable",
-	}
-
-	lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
-	if len(lines) != len(want) {
-		t.Fatalf("stderr lines = %q, want %d lines starting %q",
-			lines, len(want), want)
-	}
-
-	for i := range want {
-		if !strings.HasPrefix(lines[i], want[i]) {
-			t.Errorf("stderr line %d = %q, want it to start %q",
-				i+1, lines[i], want[i])
-		}
-	}
-}
-
-// TestReadReportOnTerminal writes the smoke-test report to the terminal
-// its read phase's display is drawn on, as a plain "sfdupes report"
-// does: the terminal must show the header and every row whole, each on
-// a line of its own.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestReadReportOnTerminal(t *testing.T) {
-	stderr := captureStderr(t)
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, smokeTreeRecs(), nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// No timed draws: the display is drawn as it starts and as it ends.
-	p := startProgress(progressParams{
-		label: reportPhase("read"), unit: "records", total: -1,
-		waitingFor: countingRecords,
-	}, terminalWidth, time.Hour, time.Now)
-
-	_, err = readReport(t.Context(), db, "", os.Stderr, p)
-
-	p.finish()
-
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// A terminal shows each line as the text after its last carriage
-	// return.
-	var shown []string
-
-	for line := range strings.SplitSeq(stderr(), "\n") {
-		shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
-	}
-
-	want := strings.Split(strings.TrimSuffix(smokeReport, "\n"), "\n")
-	if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
-		t.Errorf("terminal shows lines %q, want %q first", shown, want)
-	}
-}
-
-func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
-	// README §Constraints: identical database contents give identical
-	// output, whatever order the records were inserted in.
-	recs := append(smokeTreeRecs(), awkwardPairRecs()...)
-	recs = append(recs,
-		scanRec{size: 50, head: "b", tail: "b", content: "b", path: "/y/2"},
-		scanRec{size: 50, head: "b", tail: "b", content: "b", path: "/y/1"},
-		scanRec{size: 50, head: "a", tail: "a", content: "a", path: "/x/2"},
-		scanRec{size: 50, head: "a", tail: "a", content: "a", path: "/x/1"},
-		scanRec{size: 50, path: "/x/unhashed"},
-	)
-
-	reversed := slices.Clone(recs)
-	slices.Reverse(reversed)
-
-	for _, name := range []string{cmdReport, cmdTrees} {
-		t.Run(name, func(t *testing.T) {
-			t.Setenv(databaseEnv, seedDatabase(t, recs))
-
-			forward := runStdout(t, name)
-
-			t.Setenv(databaseEnv, seedDatabase(t, reversed))
-
-			backward := runStdout(t, name)
-
-			if strings.Count(forward, "\n") < 3 {
-				t.Errorf("stdout = %q, want at least two rows", forward)
-			}
-
-			if forward != backward {
-				t.Errorf("stdout depends on insertion order: %q vs %q",
-					forward, backward)
-			}
-		})
-	}
-}
-
-// runStdout runs the subcommand name and returns its stdout, failing
-// the test unless it succeeds.
-func runStdout(t *testing.T, name string) string {
-	t.Helper()
-
-	var stdout, stderr bytes.Buffer
-
-	code := run([]string{name}, &stdout, &stderr)
-	if code != exitOK {
-		t.Fatalf("run(%s) = %d, want %d; stderr: %s",
-			name, code, exitOK, stderr.String())
-	}
-
-	return stdout.String()
-}
-
-func TestEscapePath(t *testing.T) {
-	t.Parallel()
-
-	cases := map[string]string{
-		"/srv/plain":    "/srv/plain",
-		"/a\tb":         `/a\tb`,
-		"/a\nb":         `/a\nb`,
-		"/a\rb":         `/a\rb`,
-		`/a\b`:          `/a\\b`,
-		`/a\tb`:         `/a\\tb`,
-		"/not-utf8\xff": "/not-utf8\xff",
-	}
-	for in, want := range cases {
-		if got := escapePath(in); got != want {
-			t.Errorf("escapePath(%q) = %q, want %q", in, got, want)
-		}
-	}
-}
-
-// TestWarnfEscapes checks that a warning naming a path that holds a
-// newline is still one line.
-//
-//nolint:paralleltest // replaces the process-wide os.Stderr
-func TestWarnfEscapes(t *testing.T) {
-	f, err := os.Create(filepath.Join(t.TempDir(), "stderr"))
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	saved := os.Stderr
-	os.Stderr = f
-
-	t.Cleanup(func() {
-		os.Stderr = saved
-
-		_ = f.Close()
-	})
-
-	(&progress{}).warnf("stat %s: %s", "/d/a\nb", "gone")
-
-	_, err = f.Seek(0, io.SeekStart)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	got, err := io.ReadAll(f)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	want := `stat /d/a\nb: gone` + "\n"
-	if string(got) != want {
-		t.Errorf("warning = %q, want %q", got, want)
-	}
-}
-
-func TestDupeGroups(t *testing.T) {
 	t.Parallel()
 
 	recs := []scanRec{
-		{size: 100, head: "h", tail: "t", content: "c", path: "/z/b"},
+		{size: 100, head: "h", tail: "t", path: "/z/b"},
-		{size: 100, head: "h", tail: "t", content: "c", path: "/z/a"},
+		{size: 100, head: "h", tail: "t", path: "/z/a"},
-		{size: 100, head: "h", tail: "t", content: "c", path: "/z/c"},
+		{size: 100, head: "h", tail: "t", path: "/z/c"},
-		{size: 4000, head: "H", tail: "T", content: "C", path: "/big/2"},
+		{size: 4000, head: "H", tail: "T", path: "/big/2"},
-		{size: 4000, head: "H", tail: "T", content: "C", path: "/big/1"},
+		{size: 4000, head: "H", tail: "T", path: "/big/1"},
 		// Same size as the /z group but a different head hash.
-		{size: 100, head: "other", tail: "t", content: "c", path: "/z/d"},
+		{size: 100, head: "other", tail: "t", path: "/z/d"},
 		// A singleton signature must not form a group.
-		{size: 7, head: "u", tail: "u", content: "u", path: "/lonely"},
+		{size: 7, head: "u", tail: "u", path: "/lonely"},
 	}
 
-	groups := dupeGroupsOf(t, recs)
+	groups := collectDupeGroups(recs)
 	if len(groups) != 2 {
 		t.Fatalf("len(groups) = %d, want 2", len(groups))
 	}
@@ -416,67 +38,33 @@ func TestDupeGroups(t *testing.T) {
 	}
 }
 
-func TestDupeGroupsContentSeparates(t *testing.T) {
+func TestCollectDupeGroupsMtimeExcluded(t *testing.T) {
-	t.Parallel()
-
-	// Same size, head, and tail, but different content hashes: the final
-	// rung keeps them apart, so no group forms. Matching content groups.
-	// Records without a content hash never group, not even with each
-	// other.
-	recs := []scanRec{
-		{size: 100, head: "h", tail: "t", content: "c1", path: "/a"},
-		{size: 100, head: "h", tail: "t", content: "c2", path: "/b"},
-		{size: 100, head: "h", tail: "t", content: "c1", path: "/c"},
-		{size: 100, head: "h", tail: "t", path: "/d"},
-		{size: 100, head: "h", tail: "t", path: "/e"},
-	}
-
-	groups := dupeGroupsOf(t, recs)
-	if len(groups) != 1 {
-		t.Fatalf("len(groups) = %d, want 1 (only the matching content)",
-			len(groups))
-	}
-
-	if !slices.Equal(groups[0].paths, []string{"/a", "/c"}) {
-		t.Errorf("group paths = %q, want /a /c", groups[0].paths)
-	}
-}
-
-func TestDupeGroupsMtimeExcluded(t *testing.T) {
 	t.Parallel()
 
 	// mtime is informational only; records differing only in mtime
 	// still group together.
 	recs := []scanRec{
-		{
+		{size: 9, mtime: 100, head: "h", tail: "t", path: "/m/1"},
-			size: 9, mtime: time.Unix(100, 0), head: "h", tail: "t",
+		{size: 9, mtime: 200, head: "h", tail: "t", path: "/m/2"},
-			content: "c", path: "/m/1",
-		},
-		{
-			size: 9, mtime: time.Unix(200, 0), head: "h", tail: "t",
-			content: "c", path: "/m/2",
-		},
 	}
 
-	groups := dupeGroupsOf(t, recs)
+	groups := collectDupeGroups(recs)
 	if len(groups) != 1 {
 		t.Fatalf("len(groups) = %d, want 1", len(groups))
 	}
 }
 
-func TestDupeGroupsTieBreak(t *testing.T) {
+func TestCollectDupeGroupsTieBreak(t *testing.T) {
 	t.Parallel()
 
-	// The hashes sort opposite to the first paths, so ordering the
-	// groups by hash instead of by first path fails this test.
 	recs := []scanRec{
-		{size: 50, head: "a", tail: "a", content: "a", path: "/beta/2"},
+		{size: 50, head: "b", tail: "b", path: "/beta/2"},
-		{size: 50, head: "a", tail: "a", content: "a", path: "/beta/1"},
+		{size: 50, head: "b", tail: "b", path: "/beta/1"},
-		{size: 50, head: "b", tail: "b", content: "b", path: "/alpha/2"},
+		{size: 50, head: "a", tail: "a", path: "/alpha/2"},
-		{size: 50, head: "b", tail: "b", content: "b", path: "/alpha/1"},
+		{size: 50, head: "a", tail: "a", path: "/alpha/1"},
 	}
 
-	groups := dupeGroupsOf(t, recs)
+	groups := collectDupeGroups(recs)
 	if len(groups) != 2 {
 		t.Fatalf("len(groups) = %d, want 2", len(groups))
 	}
@@ -488,53 +76,22 @@ func TestDupeGroupsTieBreak(t *testing.T) {
 	}
 }
 
-func TestDupeGroupsOrderedByFirstPath(t *testing.T) {
+func TestCollectDupeGroupsDeterministic(t *testing.T) {
-	t.Parallel()
-
-	// report reads the groups of size 50 in signature order, here /a,
-	// /z, /k, which is neither the order of their first paths nor its
-	// reverse, and must write them ordered by first path. The group of
-	// size 7 comes after them all, though its first path sorts first.
-	recs := []scanRec{
-		{size: 50, head: "a", tail: "a", content: "a", path: "/k/2"},
-		{size: 50, head: "a", tail: "a", content: "a", path: "/k/1"},
-		{size: 50, head: "b", tail: "b", content: "b", path: "/z/1"},
-		{size: 50, head: "b", tail: "b", content: "b", path: "/z/2"},
-		{size: 50, head: "c", tail: "c", content: "c", path: "/a/2"},
-		{size: 50, head: "c", tail: "c", content: "c", path: "/a/1"},
-		{size: 7, head: "d", tail: "d", content: "d", path: "/0/2"},
-		{size: 7, head: "d", tail: "d", content: "d", path: "/0/1"},
-	}
-
-	groups := dupeGroupsOf(t, recs)
-
-	got := make([]string, 0, len(groups))
-	for _, g := range groups {
-		got = append(got, g.paths[0])
-	}
-
-	want := []string{"/a/1", "/k/1", "/z/1", "/0/1"}
-	if !slices.Equal(got, want) {
-		t.Errorf("groups' first paths = %q, want %q", got, want)
-	}
-}
-
-func TestDupeGroupsDeterministic(t *testing.T) {
 	t.Parallel()
 
 	recs := []scanRec{
-		{size: 1, head: "a", tail: "a", content: "a", path: "/p/1"},
+		{size: 1, head: "a", tail: "a", path: "/p/1"},
-		{size: 1, head: "a", tail: "a", content: "a", path: "/p/2"},
+		{size: 1, head: "a", tail: "a", path: "/p/2"},
-		{size: 2, head: "b", tail: "b", content: "b", path: "/q/1"},
+		{size: 2, head: "b", tail: "b", path: "/q/1"},
-		{size: 2, head: "b", tail: "b", content: "b", path: "/q/2"},
+		{size: 2, head: "b", tail: "b", path: "/q/2"},
 	}
 
-	forward := dupeGroupsOf(t, recs)
+	forward := collectDupeGroups(recs)
 
 	reversed := slices.Clone(recs)
 	slices.Reverse(reversed)
 
-	backward := dupeGroupsOf(t, reversed)
+	backward := collectDupeGroups(reversed)
 	if !slices.EqualFunc(forward, backward, func(a, b dupeGroup) bool {
 		return a.size == b.size && slices.Equal(a.paths, b.paths)
 	}) {

script/bootstrap

@@ -2,23 +2,15 @@
 # script/bootstrap: install all dependencies needed to build and develop
 # this repo. Idempotent: every install is guarded by a check so already
 # installed tools are skipped. Base tooling comes from nix, apt, brew,
-# or apk (detected in that order); assumes nothing is present. The
+# or apk (detected in that order); assumes nothing is present (not git,
-# installed node is used only when its major version is the pinned one;
+# make, or go). The linter is NOT installed: golangci-lint runs via
-# otherwise (node missing, or another major) the pinned version is
+# docker only (script/lint), pinned by image digest, so the only lint
-# installed via nvm (installing nvm itself first, from a hash-verified
+# prerequisite is a working docker — which is warned about, not
-# release archive, never curl | sh), and yarn and the packages are
+# installed, because everything except linting works without it.
-# installed under it.
 set -eu
 
 ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
 
-# Pinned versions, 2026-07-06
-NODE_VERSION="22.17.0"
-NVM_VERSION="0.40.3"
-# sha256 of https://github.com/nvm-sh/nvm/archive/refs/tags/v0.40.3.tar.gz
-NVM_SHA256="5f4d6aaa04a177dc93c985e31dbc411ab6b8c6e1e21d8015dbc1372625fcd1d0"
-YARN_VERSION="1.22.22"
-
 PKGMGR=""
 SUDO=""
 APT_UPDATED=""
@@ -51,7 +43,6 @@ pkg_install() {
     case "$PKGMGR" in
         nix) nix-env -iA "nixpkgs.$1" ;;
         apt)
-            # Package lists may be empty (fresh images); refresh once per run.
             if [ -z "$APT_UPDATED" ]; then
                 $SUDO env DEBIAN_FRONTEND=noninteractive apt-get update
                 APT_UPDATED=1
@@ -67,103 +58,30 @@ missing() {
     ! command -v "$1" >/dev/null 2>&1
 }
 
-# verify_sha256 <file> <expected-hash>
-verify_sha256() {
-    if command -v sha256sum >/dev/null 2>&1; then
-        actual="$(sha256sum "$1" | cut -d' ' -f1)"
-    else
-        actual="$(shasum -a 256 "$1" | cut -d' ' -f1)"
-    fi
-    if [ "$actual" != "$2" ]; then
-        echo "bootstrap: sha256 mismatch for $1" >&2
-        echo "  expected: $2" >&2
-        echo "  actual:   $actual" >&2
-        exit 1
-    fi
-}
-
-# nvm is a bash script; run a command in a bash with nvm loaded
-nvm_sh() {
-    bash -c ". \"\$HOME/.nvm/nvm.sh\" && $*"
-}
-
-ensure_nvm() {
-    [ -s "$HOME/.nvm/nvm.sh" ] && return 0
-    # nvm prerequisites; nvm itself requires bash
-    if missing bash; then pkg_install bash bash bash bash; fi
-    if missing curl; then pkg_install curl curl curl curl; fi
-    if missing git; then pkg_install git git git git; fi
-    tmp="$(mktemp -d)"
-    curl -fsSL -o "$tmp/nvm.tar.gz" \
-        "https://github.com/nvm-sh/nvm/archive/refs/tags/v${NVM_VERSION}.tar.gz"
-    verify_sha256 "$tmp/nvm.tar.gz" "$NVM_SHA256"
-    mkdir -p "$HOME/.nvm"
-    tar -xzf "$tmp/nvm.tar.gz" -C "$HOME/.nvm" --strip-components=1
-    rm -rf "$tmp"
-}
-
-# True when the node on PATH has the pinned major version. Only the
-# major is compared: the Dockerfile stages start from a node 22 alpine
-# image whose exact version is not the pin, and nvm cannot install a
-# prebuilt node on alpine. Another major is not used: the pinned yarn 1
-# prints a deprecation warning under node 24. script/fmt and
-# script/fmt-check carry this function unchanged, to run the yarn
-# installed here.
-node_is_pinned_major() {
-    if ! command -v node >/dev/null 2>&1; then return 1; fi
-    installed="$(node --version)"
-    installed="${installed#v}"
-    [ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
-}
-
-ensure_node() {
-    if node_is_pinned_major; then
-        echo "bootstrap: using node $(node --version)"
-        return 0
-    fi
-    ensure_nvm
-    nvm_sh "nvm install $NODE_VERSION"
-    nvm_sh "nvm use $NODE_VERSION >/dev/null && \
-        echo \"bootstrap: using node \$(node --version)\""
-}
-
-ensure_yarn() {
-    if ! node_is_pinned_major; then
-        nvm_sh "nvm use $NODE_VERSION >/dev/null && corepack enable && \
-            corepack prepare yarn@$YARN_VERSION --activate"
-        return 0
-    fi
-    if ! missing yarn; then return 0; fi
-    if ! missing corepack; then
-        corepack enable
-        corepack prepare "yarn@$YARN_VERSION" --activate
-    else
-        npm install -g "yarn@$YARN_VERSION"
-    fi
-}
-
-install_js_deps() {
-    if node_is_pinned_major; then
-        yarn install --frozen-lockfile
-    else
-        nvm_sh "nvm use $NODE_VERSION >/dev/null && cd \"$ROOT\" && \
-            yarn install --frozen-lockfile"
-    fi
-}
-
 main() {
     cd "$ROOT"
 
-    if missing make; then pkg_install gnumake make make make; fi
+    # System tooling, deliberately unpinned: these come from the host
+    # package manager and whatever version it ships is what the host
+    # gets, so a presence check is the right check. The repo pins no
+    # system toolchain versions — the Go language version is governed by
+    # go.mod, and builds that must be reproducible run in the Docker
+    # image, whose base images are pinned by digest.
     if missing git; then pkg_install git git git git; fi
-    # Go builds the binary and runs gofmt. Presence is the whole check:
+    if missing make; then pkg_install gnumake make make make; fi
-    # go.mod names the Go version, and the tests and the linter run in
-    # digest-pinned images.
     if missing go; then pkg_install go golang go go; fi
 
-    ensure_node
+    # Linting runs via docker only (script/lint), so docker is a lint
-    ensure_yarn
+    # prerequisite rather than something bootstrap installs. Warn, do
-    install_js_deps
+    # not fail: everything except `make lint` — and, through it,
+    # `make check`, `make docker` and the pre-commit hook — works
+    # without it.
+    if missing docker; then
+        echo "bootstrap: WARNING: docker not found; make lint, make check" >&2
+        echo "bootstrap: and make docker require it. Install docker to" >&2
+        echo "bootstrap: run the linter." >&2
+    fi
+
     go mod download
 
     echo "bootstrap complete"

script/check

@@ -1,8 +1,6 @@
 #!/bin/sh
 # script/check: run all checks (test, lint, fmt-check). Our own
-# extension to scripts-to-rule-them-all. test and lint are Docker
+# extension to scripts-to-rule-them-all. Must not modify any files.
-# phases; fmt-check is native, because a formatter writes the working
-# tree. Must not modify any files.
 set -eu
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"

script/cibuild

@@ -1,32 +1,23 @@
 #!/bin/sh
-# script/cibuild: run the CI build. It bootstraps first: a CI runner
+# script/cibuild: run the CI build. The Dockerfile runs make fmt-check,
-# checks out and runs this and nothing else, and script/fmt-check runs
+# make lint and make check (via the script/ entrypoints) as build steps,
-# the formatter on the host, which a pristine checkout cannot do.
+# so a successful build implies all checks pass. The Gitea workflow runs
-# A new CACHEBUST value reruns every RUN step, since a cached gate phase
+# this on push.
-# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
+#
-# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
+# That implication holds only because of CHECK_EPOCH. A COPY layer is
-# first; a stage without that line is served from cache on an unchanged
+# invalidated by changed content, and a merge commit's tree is
-# tree, with no warning.
+# byte-identical to the branch head it merges, so without a fresh value
+# here Docker serves the gate layers from cache and the build reports a
+# green it never earned. Passing the current epoch invalidates the gate
+# layers on every run while leaving the pinned base images and
+# go mod download cached; see the Dockerfile for the placement.
 set -eu
 
-SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
-ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
 
 main() {
     cd "$ROOT"
-    "$SCRIPT_DIR/bootstrap"
+    docker build --build-arg CHECK_EPOCH="$(date +%s)" .
-    "$SCRIPT_DIR/check"
-    # The version and the tag each get their own line: a failing
-    # command substitution inside an argument does not trip `set -e`,
-    # so the inline form degrades silently to an empty constant. The
-    # VERSION build argument takes precedence over the version a build
-    # stage derives from the .git in the context.
-    version="$(git describe --tags --always --dirty 2>/dev/null || true)"
-    [ -n "$version" ] || version="unknown"
-    tag="$("$SCRIPT_DIR/projectname")"
-    docker build --build-arg CACHEBUST="$(date +%s)-$$" \
-        --build-arg VERSION="$version" \
-        -t "$tag" .
 }
 
 main "$@"

script/docker

@@ -1,11 +1,14 @@
 #!/bin/sh
 # script/docker: build the Docker image tagged with the project name.
-# Identical in all repos; the tag comes from script/projectname.
+# The tag comes from script/projectname.
-# A new CACHEBUST value reruns every RUN step, since a cached gate phase
+#
-# is a check that did not run; unlike --no-cache, it keeps Go's build cache.
+# CHECK_EPOCH is passed for the same reason script/cibuild passes it:
-# This relies on every stage of the Dockerfile declaring ARG CACHEBUST
+# without it Docker serves the Dockerfile's gate layers from cache on an
-# first; a stage without that line is served from cache on an unchanged
+# unchanged tree and this exits 0 having run neither the lint stage nor
-# tree, with no warning.
+# the builder stage's make check. This is the gate a developer or
+# reviewer runs by hand, so a cached pass here is the most misleading
+# result the repo can produce. Dependency layers sit above the ARG and
+# stay cached.
 set -eu
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -13,17 +16,10 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
 
 main() {
     cd "$ROOT"
-    # The version and the tag each get their own line: a failing
+    docker build \
-    # command substitution inside an argument does not trip `set -e`,
+        --build-arg CHECK_EPOCH="$(date +%s)" \
-    # so the inline form degrades silently to an empty constant. The
+        -t "$("$SCRIPT_DIR/projectname")" \
-    # VERSION build argument takes precedence over the version a build
+        .
-    # stage derives from the .git in the context.
-    version="$(git describe --tags --always --dirty 2>/dev/null || true)"
-    [ -n "$version" ] || version="unknown"
-    tag="$("$SCRIPT_DIR/projectname")"
-    docker build --build-arg CACHEBUST="$(date +%s)-$$" \
-        --build-arg VERSION="$version" \
-        -t "$tag" .
 }
 
 main "$@"

script/fmt

@@ -4,39 +4,9 @@ set -eu
 
 ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
 
-# Must match the pin in script/bootstrap.
-NODE_VERSION="22.17.0"
-
-# True when the node on PATH has the pinned major version: the same
-# function as in script/bootstrap, which uses it to decide where it
-# installs yarn.
-node_is_pinned_major() {
-    if ! command -v node >/dev/null 2>&1; then return 1; fi
-    installed="$(node --version)"
-    installed="${installed#v}"
-    [ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
-}
-
-# Run the yarn script/bootstrap installed: the yarn on PATH when the
-# node on PATH has the pinned major version, and otherwise yarn under
-# the pinned node in nvm, which bootstrap leaves off the PATH of the
-# shell that called it. nvm is a bash script, hence the subshell.
-run_yarn() {
-    if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
-        exec yarn "$@"
-    fi
-    if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
-        echo "fmt: no yarn; run script/bootstrap first" >&2
-        exit 1
-    fi
-    exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
-        shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
-}
-
 main() {
     cd "$ROOT"
     gofmt -s -w .
-    run_yarn run prettier --write '**/*.md' --tab-width 4 --prose-wrap always
 }
 
 main "$@"

script/fmt-check

@@ -1,60 +1,18 @@
 #!/bin/sh
-# script/fmt-check: check formatting (read-only).
+# script/fmt-check: check formatting (read-only). Same scope as
+# script/fmt, but fails instead of writing.
 set -eu
 
 ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
 
-# Must match the pin in script/bootstrap.
-NODE_VERSION="22.17.0"
-
-# True when the node on PATH has the pinned major version: the same
-# function as in script/bootstrap, which uses it to decide where it
-# installs yarn.
-node_is_pinned_major() {
-    if ! command -v node >/dev/null 2>&1; then return 1; fi
-    installed="$(node --version)"
-    installed="${installed#v}"
-    [ "${installed%%.*}" = "${NODE_VERSION%%.*}" ]
-}
-
-# Run the yarn script/bootstrap installed: the yarn on PATH when the
-# node on PATH has the pinned major version, and otherwise yarn under
-# the pinned node in nvm, which bootstrap leaves off the PATH of the
-# shell that called it. nvm is a bash script, hence the subshell.
-run_yarn() {
-    if node_is_pinned_major && command -v yarn >/dev/null 2>&1; then
-        exec yarn "$@"
-    fi
-    if node_is_pinned_major || [ ! -s "$HOME/.nvm/nvm.sh" ]; then
-        echo "fmt-check: no yarn; run script/bootstrap first" >&2
-        exit 1
-    fi
-    exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
-        shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
-}
-
 main() {
     cd "$ROOT"
-    status=0
+    files="$(gofmt -s -l .)"
-
-    # gofmt and prettier both run every time, so the output names each
-    # one that fails. Under set -e a bare assignment would end the
-    # script when gofmt fails (a Go file it cannot parse).
-    if ! files="$(gofmt -s -l .)"; then
-        echo "gofmt: failed; see its errors above" >&2
-        status=1
-    fi
     if [ -n "$files" ]; then
         echo "gofmt: files not formatted:" >&2
         echo "$files" >&2
-        status=1
+        exit 1
     fi
-
-    # run_yarn ends in exec; the subshell returns here afterwards.
-    (run_yarn run prettier --check '**/*.md' --tab-width 4 --prose-wrap always) ||
-        status=1
-
-    exit "$status"
 }
 
 main "$@"

script/install-precommit

@@ -1,15 +1,19 @@
 #!/bin/sh
 # script/install-precommit: install the git pre-commit hook that runs
 # script/precommit. Our own extension to scripts-to-rule-them-all.
+# Hooks are shared between the main checkout and all worktrees, so
+# resolve the common git dir instead of assuming .git is a directory.
 set -eu
 
 ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
 
 main() {
     cd "$ROOT"
-    hook=".git/hooks/pre-commit"
+    hooks_dir="$(git rev-parse --git-common-dir)/hooks"
-    printf '#!/bin/sh\nset -e\nscript/precommit\n' > .git/hooks/pre-commit
+    mkdir -p "$hooks_dir"
-    chmod +x .git/hooks/pre-commit
+    hook="$hooks_dir/pre-commit"
+    printf '#!/bin/sh\nset -e\nscript/precommit\n' > "$hook"
+    chmod +x "$hook"
     echo "pre-commit hook installed: runs script/precommit"
 }
 

script/lint

@@ -1,26 +1,26 @@
 #!/bin/sh
-# script/lint: run the linter. Linting is a phase of the Dockerfile and
+# script/lint: run the linter. golangci-lint is never installed on a
-# this builds that phase alone; the linter is never installed or run on
+# host: it runs via docker only, one way, everywhere — this builds
-# a developer host, where a shared result cache and a host-global lock
+# Dockerfile.lint, which COPYs the repo into the digest-pinned
-# make its answer untrustworthy.
+# golangci-lint image and lints as a build step, so a successful build
+# is a clean lint. The only prerequisite is a working docker; once the
+# pinned image is present nothing here reaches the network.
 #
-# The phase is not the last stage in the file, so it is built only when
+# CHECK_EPOCH is what makes the result mean anything. Without it docker
-# --target names it. A new CACHEBUST value reruns every RUN step, since
+# serves the gate layers from cache on an unchanged tree and this exits
-# a cached lint layer is a lint that did not run; unlike --no-cache, it
+# 0 in well under a second having run no linter. The PID is in the value
-# keeps Go's build cache. This relies on every stage of the Dockerfile
+# as well as the epoch because two lint runs land inside the same second
-# declaring ARG CACHEBUST first; a stage without that line is served
+# easily, and `date +%s` alone would cache the second one.
-# from cache on an unchanged tree, with no warning.
-# --output type=cacheonly writes no image, since nothing uses one.
 set -eu
 
-SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
-ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
 
 main() {
     cd "$ROOT"
-    docker build --build-arg CACHEBUST="$(date +%s)-$$" \
+    docker build \
-        --target lint \
+        --build-arg CHECK_EPOCH="$(date +%s)-$$" \
-        --output type=cacheonly .
+        -f Dockerfile.lint \
+        .
 }
 
 main "$@"

script/precommit

@@ -1,13 +1,13 @@
 #!/bin/sh
 # script/precommit: run by the git pre-commit hook; fails the commit if
-# checks fail. Our own extension to scripts-to-rule-them-all.
+# checks fail. Our own extension to scripts-to-rule-them-all. Go extra:
+# go mod tidy must be a no-op before the checks run.
 set -eu
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
 ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
 
 main() {
-    # Go extra: go mod tidy must be a no-op before the checks run.
     cd "$ROOT"
     go mod tidy
     if ! git diff --exit-code -- go.mod go.sum; then

script/setup

@@ -1,6 +1,6 @@
 #!/bin/sh
 # script/setup: set up the repo for development after a fresh clone:
-# installs dependencies and the git pre-commit hook.
+# installs dependencies (script/bootstrap) and the git pre-commit hook.
 set -eu
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"

script/test

@@ -1,23 +1,17 @@
 #!/bin/sh
-# script/test: run the test suite. Testing is a phase of the Dockerfile
+# script/test: run the test suite. Reruns verbosely on failure so CI
-# and this builds that phase alone, on the same terms as script/lint:
+# logs show which test failed.
-# --target because a phase that is not the last stage is built only when
-# named, and a new CACHEBUST value because a cached test layer is a test
-# that did not run; unlike --no-cache, it keeps Go's build cache. This
-# relies on every stage of the Dockerfile declaring ARG CACHEBUST first;
-# a stage without that line is served from cache on an unchanged tree,
-# with no warning. --output type=cacheonly writes no image, since
-# nothing uses one.
 set -eu
 
-SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
-ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
 
 main() {
     cd "$ROOT"
-    docker build --build-arg CACHEBUST="$(date +%s)-$$" \
+    go test -timeout 30s -cover ./... || {
-        --target test \
+        echo "--- Rerunning with -v for details ---"
-        --output type=cacheonly .
+        go test -timeout 30s -v ./...
+        exit 1
+    }
 }
 
 main "$@"

script/verify-lint-image-pin

@@ -0,0 +1,84 @@
+#!/bin/sh
+# script/verify-lint-image-pin: fail unless the golangci-lint image
+# referenced by Dockerfile.lint and the one referenced by the main
+# Dockerfile's lint stage are the same image at the same digest. Our own
+# extension to scripts-to-rule-them-all, not one of its entrypoints.
+#
+# The linter version is pinned in two independent files. That is the
+# shape #42 turned into a build failure rather than tolerate: nothing
+# else keeps the two in sync, and a bump applied to one file alone would
+# leave `make lint` and the fail-fast lint stage of `make docker`
+# linting the same tree against different rulesets, both green. This is
+# the single guard that stops it, run as a gate in both files.
+#
+# It deliberately restates neither pin. A hardcoded expected digest here
+# would be a third copy — one more thing to bump, and the same drift one
+# file further out. It compares the two files to each other and knows
+# nothing about which version is correct.
+#
+# A reference that cannot be read is a hard failure, not a skip: a
+# comparison of two empty strings succeeds, which would turn this guard
+# into exactly the unearned green it exists to prevent.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+LINT_DOCKERFILE="Dockerfile.lint"
+MAIN_DOCKERFILE="Dockerfile"
+
+# Echo the single golangci-lint image reference in the named Dockerfile.
+# Scans every argument of every FROM instruction rather than assuming a
+# field position, so `FROM --platform=... img AS stage` reads correctly.
+# Exits non-zero, with a diagnosis, unless there is exactly one.
+lint_image_ref() {
+    file="$1"
+
+    if [ ! -f "$file" ]; then
+        echo "verify-lint-image-pin: $file: not found" >&2
+        return 1
+    fi
+
+    refs="$(
+        awk '
+            toupper($1) == "FROM" {
+                for (i = 2; i <= NF; i++) {
+                    if ($i ~ /^golangci\/golangci-lint[:@]/) {
+                        print $i
+                    }
+                }
+            }
+        ' "$file"
+    )"
+
+    count="$(printf '%s' "$refs" | grep -c . || true)"
+    if [ "$count" -ne 1 ]; then
+        echo "verify-lint-image-pin: $file: expected exactly one" \
+            "golangci/golangci-lint FROM reference, found $count" >&2
+        return 1
+    fi
+
+    printf '%s\n' "$refs"
+}
+
+main() {
+    cd "$ROOT"
+
+    lint_ref="$(lint_image_ref "$LINT_DOCKERFILE")"
+    main_ref="$(lint_image_ref "$MAIN_DOCKERFILE")"
+
+    if [ "$lint_ref" != "$main_ref" ]; then
+        echo "verify-lint-image-pin: the linter image is pinned twice and" \
+            "the two pins disagree:" >&2
+        echo "verify-lint-image-pin:   $LINT_DOCKERFILE: $lint_ref" >&2
+        echo "verify-lint-image-pin:   $MAIN_DOCKERFILE: $main_ref" >&2
+        echo "verify-lint-image-pin: bump both FROM lines together, tag and" \
+            "digest, so script/lint and the Dockerfile lint stage keep" \
+            "running the same linter" >&2
+        exit 1
+    fi
+
+    echo "verify-lint-image-pin: $LINT_DOCKERFILE and $MAIN_DOCKERFILE" \
+        "agree on $lint_ref"
+}
+
+main "$@"

trees.go

@@ -1,155 +1,56 @@
 package main
 
 import (
+	"bufio"
 	"context"
 	"crypto/sha256"
-	"database/sql"
 	"fmt"
-	"io"
 	"os"
 	"slices"
 	"strconv"
 	"strings"
 )
 
-// treeNode is one directory reconstructed from the record paths.
+// fileSig is a file's duplicate signature; mtime is excluded.
+type fileSig struct {
+	size int64
+	head string
+	tail string
+}
+
+// treeNode is one directory reconstructed from the scan stream.
 type treeNode struct {
 	path      string
 	parent    *treeNode
-	// entries holds the serialized child entries until the digest is
+	dirs      map[string]*treeNode
-	// computed from them, and is then dropped.
+	files     map[string]fileSig
-	entries   []string
 	digest    [sha256.Size]byte
 	fileCount int64
 	totalSize int64
 }
 
-// treesPhase names one phase of trees with its place among all of them,
-// as "[2/4] read", for the phase's progress display. The list here is
-// the one place the phases are counted.
-func treesPhase(name string) string {
-	return phaseLabel([]string{openPhase, "read", "group", "write"}, name)
-}
-
 // runTrees implements the trees subcommand: it reads every record from
-// the database in path order, reconstructs the directory hierarchy from
+// the database, reconstructs the directory hierarchy from the record
-// the record paths, computes a Merkle-style digest per directory, and
+// paths, computes a Merkle-style digest per directory, and prints
-// prints maximal duplicate-tree groups as TSV on stdout. It never
+// maximal duplicate-tree groups as TSV on stdout. It never touches the
-// touches the scanned filesystem; its only I/O is the database, stdout,
+// scanned filesystem; its only I/O is the database, stdout, and
-// and stderr. It names the database on stderr before it opens it. Any
+// stderr.
-// database problem, including a missing database, is fatal.
+func runTrees(ctx context.Context) error {
-//
+	recs, err := loadRecords(ctx)
-// The work runs in four phases, each under a progress display of its
-// own on stderr: open (open the database), read (count the records,
-// then read them into the hierarchy), group (group the directories by
-// digest) and write (write the groups to stdout). Each display ends
-// before the next phase starts, and before an error is returned to be
-// printed.
-func runTrees(ctx context.Context, stdout io.Writer) error {
-	dbPath, source, err := databasePath()
 	if err != nil {
 		return err
 	}
 
-	fmt.Fprintf(os.Stderr, "trees: database %s (%s)\n",
+	super, allDirs := buildHierarchy(recs)
-		escapePath(dbPath), source)
+	super.compute()
 
-	// Opening can wait on a busy database for as long as the busy
+	dupes := collectTreeGroups(allDirs, super)
-	// timeout.
-	prog := newProgress(progressParams{
-		label: treesPhase(openPhase), total: -1,
-		waitingFor: openingDatabase,
-	})
-	db, err := openReportDatabase(ctx, dbPath)
 
-	prog.finish()
+	out := bufio.NewWriterSize(os.Stdout, ioBufSize)
 
+	_, err = fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
 	if err != nil {
-		return err
+		return fmt.Errorf("write stdout: %w", err)
-	}
-
-	defer func() { _ = db.Close() }()
-
-	prog = newProgress(progressParams{
-		label: treesPhase("read"), unit: "records", total: -1,
-		waitingFor: countingRecords, stalledOn: "db",
-	})
-	records, super, allDirs, err := readTree(ctx, db, prog)
-
-	prog.finish()
-
-	if err != nil {
-		return fmt.Errorf("database %s: %w", dbPath, err)
-	}
-
-	// Grouping works in memory, so it waits on nothing but the
-	// processor.
-	prog = newProgress(progressParams{
-		label: treesPhase("group"), unit: "directories",
-		total: int64(len(allDirs)), stalledOn: "cpu",
-	})
-	dupes := collectTreeGroups(allDirs, super, prog)
-
-	prog.finish()
-
-	prog = newProgress(progressParams{
-		label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
-		stalledOn: "out",
-	})
-	dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
-
-	prog.finish()
-
-	if err != nil {
-		return err
-	}
-
-	fmt.Fprintf(os.Stderr,
-		"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
-			"%s reclaimable\n",
-		records, len(dupes), dupeTrees, humanBytes(reclaimable))
-
-	return nil
-}
-
-// readTree gives prog the number of records in db as its total, then
-// reads every record in path order into the directory hierarchy,
-// counting each on prog. It returns the number of records read, the
-// super-root and every directory.
-func readTree(ctx context.Context, db *sql.DB,
-	prog *progress,
-) (int, *treeNode, []*treeNode, error) {
-	records := 0
-	tree := newTreeBuilder()
-
-	err := loadFileRows(ctx, db, prog.setTotal, func(r scanRec) {
-		records++
-
-		prog.increment()
-		tree.add(r)
-	})
-	if err != nil {
-		return 0, nil, nil, err
-	}
-
-	super, allDirs := tree.finish()
-
-	return records, super, allDirs, nil
-}
-
-// writeTreeGroups writes dupes to stdout as TSV, a header line and then
-// a row for every member of each group but its first, counting each
-// group on prog once its rows are written. stdout may be the terminal
-// prog is drawn on, so the lines go through prog's writer. It returns
-// the number of rows (dupe trees) and the bytes they could reclaim.
-func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
-	prog *progress,
-) (int, int64, error) {
-	out := prog.writer(stdout)
-
-	_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
-	if err != nil {
-		return 0, 0, fmt.Errorf("write stdout: %w", err)
 	}
 
 	dupeTrees := 0
@@ -160,158 +61,91 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
 		first := g[0]
 		for _, n := range g[1:] {
 			_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\n",
-				escapePath(first.path), escapePath(n.path),
+				first.path, n.path, first.fileCount, first.totalSize)
-				first.fileCount, first.totalSize)
 			if err != nil {
-				return 0, 0, fmt.Errorf("write stdout: %w", err)
+				return fmt.Errorf("write stdout: %w", err)
 			}
 
 			dupeTrees++
 			reclaimable += first.totalSize
 		}
-
-		prog.increment()
 	}
 
 	err = out.Flush()
 	if err != nil {
-		return 0, 0, fmt.Errorf("write stdout: %w", err)
+		return fmt.Errorf("write stdout: %w", err)
 	}
 
-	return dupeTrees, reclaimable, nil
+	fmt.Fprintf(os.Stderr,
+		"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
+			"%s reclaimable\n",
+		len(recs), len(dupes), dupeTrees, humanBytes(reclaimable))
+
+	return nil
 }
 
-// treeBuilder reconstructs the directory hierarchy from records added
+// buildHierarchy reconstructs the directory hierarchy from the record
-// in path order, under a synthetic super-root. Paths are split on "/";
+// paths under a synthetic super-root. Paths are split on "/"; for
-// for absolute paths the first component is empty, which becomes the
+// absolute paths the first component is empty, which simply becomes a
-// top-level directory with path "/". In path order all the paths under
+// top-level node representing "/". It returns the super-root and every
-// one directory come together, so a directory is complete once a path
+// directory node created.
-// outside it is added: its digest is computed then and its entries are
+func buildHierarchy(recs []scanRec) (*treeNode, []*treeNode) {
-// dropped. Only the directories holding the latest path keep entries.
-type treeBuilder struct {
-	super *treeNode
-	// open lists the directories holding the latest path, outermost
-	// first, starting with the super-root; names[i] is open[i]'s name.
-	open  []*treeNode
-	names []string
-	// dirs lists every completed directory.
-	dirs []*treeNode
-}
-
-func newTreeBuilder() *treeBuilder {
 	super := &treeNode{}
 
-	return &treeBuilder{
+	var allDirs []*treeNode
-		super: super,
-		open:  []*treeNode{super},
-		names: []string{""},
-	}
-}
 
-// add adds one record. Each record must come after the previous one in
+	for _, r := range recs {
-// path order (byte order); otherwise a completed directory would be
-// started again as a second directory with the same path.
-func (b *treeBuilder) add(r scanRec) {
 		comps := strings.Split(r.path, "/")
-	dirNames, name := comps[:len(comps)-1], comps[len(comps)-1]
 
-	// Keep the open directories that hold this path; complete the rest.
+		node := super
-	depth := 1
+		for _, c := range comps[:len(comps)-1] {
-	for depth < len(b.open) && depth <= len(dirNames) &&
+			child := node.dirs[c]
-		b.names[depth] == dirNames[depth-1] {
+			if child == nil {
-		depth++
+				childPath := c
+				if node != super {
+					childPath = node.path + "/" + c
 				}
 
-	b.closeTo(depth)
+				child = &treeNode{path: childPath, parent: node}
-
+				if node.dirs == nil {
-	for _, c := range dirNames[depth-1:] {
+					node.dirs = make(map[string]*treeNode)
-		b.openDir(c)
 				}
 
-	dir := b.open[len(b.open)-1]
+				node.dirs[c] = child
-	dir.entries = append(dir.entries, fileEntry(name, r))
+				allDirs = append(allDirs, child)
-	dir.fileCount++
-	dir.totalSize += r.size
-}
-
-// openDir starts the directory called name inside the innermost open
-// one.
-func (b *treeBuilder) openDir(name string) {
-	parent := b.open[len(b.open)-1]
-	path := parent.path + "/" + name
-
-	// The root directory's path is "/", not empty, and its children's
-	// paths start with one slash, not two.
-	switch {
-	case parent == b.super && name == "":
-		path = "/"
-	case parent == b.super:
-		path = name
-	case parent.path == "/":
-		path = "/" + name
 			}
 
-	b.open = append(b.open, &treeNode{path: path, parent: parent})
+			node = child
-	b.names = append(b.names, name)
-}
-
-// closeTo completes the open directories after the first n, innermost
-// first: each one's digest is computed and entered in its parent along
-// with its totals.
-func (b *treeBuilder) closeTo(n int) {
-	for len(b.open) > n {
-		last := len(b.open) - 1
-		dir, name := b.open[last], b.names[last]
-		b.open, b.names = b.open[:last], b.names[:last]
-
-		dir.computeDigest()
-
-		dir.parent.entries = append(dir.parent.entries,
-			"d\x00"+name+"\x00"+string(dir.digest[:]))
-		dir.parent.fileCount += dir.fileCount
-		dir.parent.totalSize += dir.totalSize
-		b.dirs = append(b.dirs, dir)
-	}
-}
-
-// finish completes every open directory and returns the super-root and
-// every directory.
-func (b *treeBuilder) finish() (*treeNode, []*treeNode) {
-	b.closeTo(1)
-
-	return b.super, b.dirs
-}
-
-// fileEntry serializes a file child for its directory's digest: its
-// name and its signature (size, head, tail, content); mtime is
-// excluded.
-func fileEntry(name string, r scanRec) string {
-	content := r.content
-
-	// A record without a content hash has unknown content (README
-	// "Database"): give it a signature no other file can share, so
-	// trees containing it never compare equal. Real hashes are hex, so
-	// the NUL-prefixed form cannot collide.
-	if content == "" {
-		content = "unhashed\x00" + r.path
 		}
 
-	return "f\x00" + name + "\x00" + strconv.FormatInt(r.size, 10) +
+		if node.files == nil {
-		"\x00" + r.head + "\x00" + r.tail + "\x00" + content
+			node.files = make(map[string]fileSig)
+		}
+
+		sig := fileSig{size: r.size, head: r.head, tail: r.tail}
+
+		// An unhashed record (its size was unique when last scanned)
+		// has unknown content: give it a signature no other file can
+		// share, so trees containing it never compare equal. Real
+		// heads are hex, so the NUL-prefixed form cannot collide.
+		if sig.head == "" {
+			sig.head = "unhashed\x00" + r.path
+		}
+
+		node.files[comps[len(comps)-1]] = sig
+	}
+
+	return super, allDirs
 }
 
-// collectTreeGroups groups directories by digest, counting each one on
+// collectTreeGroups groups directories by digest and returns every
-// prog as it is grouped, and returns every maximal group with two or
+// maximal group with two or more members, each group's members sorted
-// more members, each group's members sorted by path, groups ordered by
+// by path, groups ordered by tree size descending then by first path
-// tree size descending then by first path ascending.
+// ascending.
-func collectTreeGroups(allDirs []*treeNode, super *treeNode,
+func collectTreeGroups(allDirs []*treeNode, super *treeNode) [][]*treeNode {
-	prog *progress,
-) [][]*treeNode {
 	groups := make(map[[sha256.Size]byte][]*treeNode)
 	for _, d := range allDirs {
 		groups[d.digest] = append(groups[d.digest], d)
-
-		prog.increment()
 	}
 
 	var dupes [][]*treeNode
@@ -343,22 +177,38 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
 	return dupes
 }
 
-// computeDigest sets n's digest and drops its entries. A directory's
+// compute fills in digest, fileCount, and totalSize for n and all of
-// digest is the SHA-256 of its child entries — files serialized with
+// its descendants. A directory's digest is the SHA-256 of its child
-// name and signature, subdirectories with name and recursive digest —
+// entries — files serialized with name and signature, subdirectories
-// sorted byte-lexicographically. Filenames cannot contain NUL or "/",
+// with name and recursive digest — sorted byte-lexicographically.
-// so NUL delimiters are unambiguous.
+// Filenames cannot contain NUL or "/", so NUL delimiters are
-func (n *treeNode) computeDigest() {
+// unambiguous.
-	slices.Sort(n.entries)
+func (n *treeNode) compute() {
+	entries := make([]string, 0, len(n.dirs)+len(n.files))
+	for name, sig := range n.files {
+		entries = append(entries,
+			"f\x00"+name+"\x00"+strconv.FormatInt(sig.size, 10)+
+				"\x00"+sig.head+"\x00"+sig.tail)
+		n.fileCount++
+		n.totalSize += sig.size
+	}
+
+	for name, child := range n.dirs {
+		child.compute()
+		entries = append(entries, "d\x00"+name+"\x00"+string(child.digest[:]))
+		n.fileCount += child.fileCount
+		n.totalSize += child.totalSize
+	}
+
+	slices.Sort(entries)
 
 	h := sha256.New()
-	for _, e := range n.entries {
+	for _, e := range entries {
 		h.Write([]byte(e))
 		h.Write([]byte{0})
 	}
 
 	copy(n.digest[:], h.Sum(nil))
-	n.entries = nil
 }
 
 // suppressed reports whether a duplicate-tree group is non-maximal: its

trees_test.go

@@ -1,74 +1,31 @@
 package main
 
 import (
-	"bytes"
-	"database/sql"
-	"os"
 	"slices"
-	"strings"
 	"testing"
-	"time"
 )
 
 // Signature hashes shared by the smoke-test records.
 const (
 	f1Head = "f1h"
 	f1Tail = "f1t"
-	f1Content = "f1c"
 	f2Head = "f2h"
 	f2Tail = "f2t"
-	f2Content = "f2c"
 )
 
-// treesOpenLine is the plain line of trees' open phase when it opens
-// the database at once; it prints it as the phase starts and as it ends.
-const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
-	"eta unknown"
-
 // smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
 // /d/t2 are identical, /d/t3 differs from them only by one filename.
 func smokeTreeRecs() []scanRec {
 	return []scanRec{
-		{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t1/f1"},
+		{size: 3000, head: f1Head, tail: f1Tail, path: "/d/t1/f1"},
-		{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t1/sub/f2"},
+		{size: 100, head: f2Head, tail: f2Tail, path: "/d/t1/sub/f2"},
-		{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t2/f1"},
+		{size: 3000, head: f1Head, tail: f1Tail, path: "/d/t2/f1"},
-		{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t2/sub/f2"},
+		{size: 100, head: f2Head, tail: f2Tail, path: "/d/t2/sub/f2"},
-		{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t3/f1"},
+		{size: 3000, head: f1Head, tail: f1Tail, path: "/d/t3/f1"},
-		{size: 100, head: f2Head, tail: f2Tail, content: f2Content,
+		{size: 100, head: f2Head, tail: f2Tail, path: "/d/t3/sub/f2renamed"},
-			path: "/d/t3/sub/f2renamed"},
 	}
 }
 
-// dbTree builds the directory hierarchy from the records in db the way
-// trees does, and returns the super-root and every directory.
-func dbTree(t *testing.T, db *sql.DB) (*treeNode, []*treeNode) {
-	t.Helper()
-
-	tree := newTreeBuilder()
-
-	err := loadFileRows(t.Context(), db, func(int64) {}, tree.add)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	return tree.finish()
-}
-
-// treeOf writes recs into a fresh database and builds the directory
-// hierarchy from it the way trees does.
-func treeOf(t *testing.T, recs []scanRec) (*treeNode, []*treeNode) {
-	t.Helper()
-
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	return dbTree(t, db)
-}
-
 // nodeByPath finds the directory node with the given path.
 func nodeByPath(t *testing.T, dirs []*treeNode, path string) *treeNode {
 	t.Helper()
@@ -99,10 +56,11 @@ func groupPaths(groups [][]*treeNode) [][]string {
 	return out
 }
 
-func TestTreeCounts(t *testing.T) {
+func TestBuildHierarchyCounts(t *testing.T) {
 	t.Parallel()
 
-	_, dirs := treeOf(t, smokeTreeRecs())
+	super, dirs := buildHierarchy(smokeTreeRecs())
+	super.compute()
 
 	d := nodeByPath(t, dirs, "/d")
 	if d.fileCount != 6 || d.totalSize != 9300 {
@@ -123,190 +81,11 @@ func TestTreeCounts(t *testing.T) {
 	}
 }
 
-func TestTreeRootPath(t *testing.T) {
-	t.Parallel()
-
-	// The root directory's path is "/", never empty, and its
-	// children's paths start with a single slash.
-	_, dirs := treeOf(t, []scanRec{{path: "/f"}, {path: "/srv/g"}})
-
-	got := make([]string, 0, len(dirs))
-	for _, d := range dirs {
-		got = append(got, d.path)
-	}
-
-	slices.Sort(got)
-
-	want := []string{"/", "/srv"}
-	if !slices.Equal(got, want) {
-		t.Fatalf("directory paths = %q, want %q", got, want)
-	}
-}
-
-func TestTreeNamesSortingBeforeSlash(t *testing.T) {
-	t.Parallel()
-
-	// In path order "/a/b-x/f" and "/a/b.txt" come between the file
-	// "/a/b" and "/a/b/f", because "-" and "." sort before "/". Each
-	// directory must still be built once, whole, so /a matches /c.
-	recs := make([]scanRec, 0, 8)
-
-	for _, top := range []string{"/a", "/c"} {
-		for _, p := range []string{"/b", "/b-x/f", "/b.txt", "/b/f"} {
-			content := "c"
-			if p == "/b-x/f" {
-				content = "other"
-			}
-
-			recs = append(recs, scanRec{
-				size: 1, head: "h", tail: "t", content: content, path: top + p,
-			})
-		}
-	}
-
-	super, dirs := treeOf(t, recs)
-
-	got := make([]string, 0, len(dirs))
-	for _, d := range dirs {
-		got = append(got, d.path)
-	}
-
-	slices.Sort(got)
-
-	want := []string{"/", "/a", "/a/b", "/a/b-x", "/c", "/c/b", "/c/b-x"}
-	if !slices.Equal(got, want) {
-		t.Fatalf("directory paths = %q, want %q", got, want)
-	}
-
-	groups := collectTreeGroups(dirs, super, nil)
-
-	gotGroups := groupPaths(groups)
-	wantGroups := [][]string{{"/a", "/c"}}
-
-	if !slices.EqualFunc(gotGroups, wantGroups, slices.Equal) {
-		t.Fatalf("groups = %v, want %v", gotGroups, wantGroups)
-	}
-
-	if groups[0][0].fileCount != 4 || groups[0][0].totalSize != 4 {
-		t.Errorf("group totals: %d files %d bytes, want 4 4",
-			groups[0][0].fileCount, groups[0][0].totalSize)
-	}
-}
-
-func TestRunTreesEscapesPaths(t *testing.T) {
-	t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
-
-	var stdout, stderr bytes.Buffer
-
-	code := run([]string{cmdTrees}, &stdout, &stderr)
-	if code != exitOK {
-		t.Fatalf("run(trees) = %d, want %d; stderr: %s",
-			code, exitOK, stderr.String())
-	}
-
-	want := "first\tdupe\tfiles\tsize\n" +
-		`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\n"
-	if got := stdout.String(); got != want {
-		t.Errorf("stdout = %q, want %q", got, want)
-	}
-}
-
-// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
-// records with stderr not a terminal, and checks every line it prints
-// there. After the line naming the database, each phase prints a line
-// naming it with its place among the four as it starts and another as
-// it ends. read counts the 6 records before it reads them, group counts
-// the 8 directories, and write the 1 group. The summary comes last, and
-// stdout holds the report alone.
-func TestRunTreesProgressNamesEachPhase(t *testing.T) {
-	path := seedDatabase(t, smokeTreeRecs())
-	t.Setenv(databaseEnv, path)
-
-	stderr := captureStderr(t)
-
-	var stdout bytes.Buffer
-
-	code := run([]string{cmdTrees}, &stdout, os.Stderr)
-	if code != exitOK {
-		t.Fatalf("run(trees) = %d, want %d; stderr: %s",
-			code, exitOK, stderr())
-	}
-
-	report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
-	if got := stdout.String(); got != report {
-		t.Errorf("stdout = %q, want %q", got, report)
-	}
-
-	// How each line starts; the rate at the end of a phase varies.
-	want := []string{
-		"trees: database " + path + " (from " + databaseEnv + ")",
-		treesOpenLine,
-		treesOpenLine,
-		"[2/4] read: counting the records, elapsed 0s, eta unknown",
-		"[2/4] read: [6/6] 100%, ",
-		"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown",
-		"[3/4] group: [8/8] 100%, ",
-		"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown",
-		"[4/4] write: [1/1] 100%, ",
-		"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
-	}
-
-	lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
-	if len(lines) != len(want) {
-		t.Fatalf("stderr lines = %q, want %d lines starting %q",
-			lines, len(want), want)
-	}
-
-	for i := range want {
-		if !strings.HasPrefix(lines[i], want[i]) {
-			t.Errorf("stderr line %d = %q, want it to start %q",
-				i+1, lines[i], want[i])
-		}
-	}
-}
-
-// TestWriteTreeGroupsOnTerminal writes the smoke-test report to the
-// terminal its write phase's display is drawn on, as a plain "sfdupes
-// trees" does: the terminal must show the header and the row whole,
-// each on a line of its own.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestWriteTreeGroupsOnTerminal(t *testing.T) {
-	stderr := captureStderr(t)
-	super, dirs := treeOf(t, smokeTreeRecs())
-	dupes := collectTreeGroups(dirs, super, nil)
-
-	// No timed draws: the display is drawn as it starts and as it ends.
-	p := startProgress(progressParams{
-		label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
-	}, terminalWidth, time.Hour, time.Now)
-
-	_, _, err := writeTreeGroups(os.Stderr, dupes, p)
-
-	p.finish()
-
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// A terminal shows each line as the text after its last carriage
-	// return.
-	var shown []string
-
-	for line := range strings.SplitSeq(stderr(), "\n") {
-		shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
-	}
-
-	want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
-	if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
-		t.Errorf("terminal shows lines %q, want %q first", shown, want)
-	}
-}
-
 func TestTreeDigests(t *testing.T) {
 	t.Parallel()
 
-	_, dirs := treeOf(t, smokeTreeRecs())
+	super, dirs := buildHierarchy(smokeTreeRecs())
+	super.compute()
 
 	t1 := nodeByPath(t, dirs, "/d/t1")
 	t2 := nodeByPath(t, dirs, "/d/t2")
@@ -335,11 +114,12 @@ func TestTreeDigestContentSensitivity(t *testing.T) {
 	const sharedTail = "same"
 
 	recs := []scanRec{
-		{size: 10, head: sharedTail, tail: sharedTail, content: "c", path: "/r/a/f"},
+		{size: 10, head: sharedTail, tail: sharedTail, path: "/r/a/f"},
-		{size: 10, head: "DIFF", tail: sharedTail, content: "c", path: "/r/b/f"},
+		{size: 10, head: "DIFF", tail: sharedTail, path: "/r/b/f"},
 	}
 
-	_, dirs := treeOf(t, recs)
+	super, dirs := buildHierarchy(recs)
+	super.compute()
 
 	a := nodeByPath(t, dirs, "/r/a")
 	b := nodeByPath(t, dirs, "/r/b")
@@ -352,9 +132,10 @@ func TestTreeDigestContentSensitivity(t *testing.T) {
 func TestCollectTreeGroupsMaximal(t *testing.T) {
 	t.Parallel()
 
-	super, dirs := treeOf(t, smokeTreeRecs())
+	super, dirs := buildHierarchy(smokeTreeRecs())
+	super.compute()
 
-	groups := collectTreeGroups(dirs, super, nil)
+	groups := collectTreeGroups(dirs, super)
 
 	want := [][]string{{"/d/t1", "/d/t2"}}
 	if got := groupPaths(groups); !slices.EqualFunc(got, want,
@@ -376,16 +157,18 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
 
 	recs := smokeTreeRecs()
 
-	super, dirs := treeOf(t, recs)
+	super, dirs := buildHierarchy(recs)
+	super.compute()
 
-	forward := groupPaths(collectTreeGroups(dirs, super, nil))
+	forward := groupPaths(collectTreeGroups(dirs, super))
 
 	reversed := slices.Clone(recs)
 	slices.Reverse(reversed)
 
-	superR, dirsR := treeOf(t, reversed)
+	superR, dirsR := buildHierarchy(reversed)
+	superR.compute()
 
-	backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
+	backward := groupPaths(collectTreeGroups(dirsR, superR))
 	if !slices.EqualFunc(forward, backward, slices.Equal) {
 		t.Fatalf("output depends on record order: %v vs %v",
 			forward, backward)
@@ -398,13 +181,14 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
 	// Identical sibling dirs share a parent, so their group cannot be
 	// implied by a parent group and must be reported.
 	recs := []scanRec{
-		{size: 10, head: "h", tail: "t", content: "c", path: "/p/x1/f"},
+		{size: 10, head: "h", tail: "t", path: "/p/x1/f"},
-		{size: 10, head: "h", tail: "t", content: "c", path: "/p/x2/f"},
+		{size: 10, head: "h", tail: "t", path: "/p/x2/f"},
 	}
 
-	super, dirs := treeOf(t, recs)
+	super, dirs := buildHierarchy(recs)
+	super.compute()
 
-	got := groupPaths(collectTreeGroups(dirs, super, nil))
+	got := groupPaths(collectTreeGroups(dirs, super))
 
 	want := [][]string{{"/p/x1", "/p/x2"}}
 	if !slices.EqualFunc(got, want, slices.Equal) {
@@ -419,14 +203,15 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
 	// extra file, so the parents' digests differ and the x group must
 	// be reported.
 	recs := []scanRec{
-		{size: 10, head: "h", tail: "t", content: "c", path: "/p/a/x/f"},
+		{size: 10, head: "h", tail: "t", path: "/p/a/x/f"},
-		{size: 99, head: "e", tail: "e", content: "e", path: "/p/a/extra"},
+		{size: 99, head: "e", tail: "e", path: "/p/a/extra"},
-		{size: 10, head: "h", tail: "t", content: "c", path: "/q/b/x/f"},
+		{size: 10, head: "h", tail: "t", path: "/q/b/x/f"},
 	}
 
-	super, dirs := treeOf(t, recs)
+	super, dirs := buildHierarchy(recs)
+	super.compute()
 
-	got := groupPaths(collectTreeGroups(dirs, super, nil))
+	got := groupPaths(collectTreeGroups(dirs, super))
 
 	want := [][]string{{"/p/a/x", "/q/b/x"}}
 	if !slices.EqualFunc(got, want, slices.Equal) {

yarn.lock

@@ -1,8 +0,0 @@
-# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
-# yarn lockfile v1
-
-
-prettier@3.8.1:
-  version "3.8.1"
-  resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
-  integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==