1 Commits
Author SHA1 Message Date
sneak 56fd779f7a Format Markdown with prettier in make fmt and make fmt-check (closes #19)
check / check (push) Failing after 2s
script/fmt and script/fmt-check run prettier over every Markdown file
again, next to gofmt. prettier is pinned by hash through package.json
and yarn.lock, copied from the prompts repo with .prettierrc and
.prettierignore, and is never installed on a host: a new prettier stage
of the Dockerfile installs it into a digest-pinned node image, and both
scripts build that stage and run it with the repository mounted. CI
checks the Markdown in a markdown stage that the build stage waits on.
Because make fmt-check now runs docker, the Dockerfile runs gofmt
directly in its lint stage instead. All Markdown is reformatted.

Model: opus-5-5
2026-10-04 12:55:03 +00:00
16 changed files with 530 additions and 740 deletions
+2 -66
View File
@@ -10,20 +10,14 @@ run:
linters: linters:
default: all default: all
enable:
# Successor to the deprecated gomodguard. Named explicitly, rather than
# left to `default: all`, because it carries the module policy below.
- gomodguard_v2
disable: disable:
# Genuinely incompatible with project patterns # Genuinely incompatible with project patterns
- exhaustruct # Requires all struct fields - exhaustruct # Requires all struct fields
- depguard # Dependency allow/block lists
- godot # Requires comments to end with periods - godot # Requires comments to end with periods
- wsl # Deprecated, replaced by wsl_v5
- wrapcheck # Too verbose for internal packages - wrapcheck # Too verbose for internal packages
- varnamelen # Short names like db, id are idiomatic Go - varnamelen # Short names like db, id are idiomatic Go
# Deprecated: the warning is attached to the old name, so it is
# silenced by disabling that name, not by enabling the successor.
- wsl # Deprecated, replaced by wsl_v5
- gomodguard # Deprecated, replaced by gomodguard_v2
settings: settings:
lll: lll:
line-length: 88 line-length: 88
@@ -34,64 +28,6 @@ linters:
max-complexity: 15 max-complexity: 15
dupl: dupl:
threshold: 100 threshold: 100
depguard:
# Test-support code must not be compiled into the shipped binary. A
# test-support package exists to hand a test privileges the program
# itself must never have, so a file that is not a test must not import
# one. Test files, and the files inside a package whose directory name
# ends in `test`, are where that code belongs, and are exempt.
#
# The deny list below is the one part of this file a repository is
# expected to extend, and the only part it may. depguard matches an
# import path against a list of prefixes, so it cannot be told "any path
# whose last segment ends in test"; a repository's own test-support
# packages have to be named here one at a time, by full import path,
# under a module path that differs from repository to repository. Add
# them; change nothing else.
rules:
test-support:
list-mode: lax
files:
- "$all"
- "!$test"
- "!**/*test/**"
deny:
- pkg: net/http/httptest
desc: >-
Test-support code belongs in test files and in packages whose
directory name ends in test, not in the shipped binary.
# Only decisions already recorded in the Go package defaults are
# listed here. Every entry matches the module path exactly.
gomodguard_v2:
blocked:
- module: github.com/rs/zerolog
recommendations:
- log/slog
reason: "Structured logging is stdlib log/slog."
# One entry per pre-fork module path, because the later releases
# are separate paths. A prefix match would be shorter but would
# also reach github.com/go-redis/redismock, the test double for
# the successor these entries recommend.
- module: github.com/go-redis/redis
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v7
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/go-redis/redis/v8
recommendations:
- github.com/redis/go-redis/v9
reason: "Pre-fork module; use the maintained go-redis v9."
- module: github.com/sergi/go-diff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "No unified diff output; use go-udiff."
- module: github.com/hexops/gotextdiff
recommendations:
- github.com/aymanbagabas/go-udiff
reason: "Unmaintained fork; use go-udiff."
issues: issues:
max-issues-per-linter: 0 max-issues-per-linter: 0
+64 -52
View File
@@ -6,36 +6,41 @@ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
COPY . . COPY . .
# Cache-buster for the gate layers, and only for them: on an unchanged # Cache-buster for the gate layers, and only for them. Docker
# tree Docker would serve the gates below from cache and the build would # invalidates COPY only when the copied content changes, so on an
# exit 0 having run nothing. script/cibuild and script/docker pass a # unchanged tree the gates below would be served from cache and the
# fresh CHECK_EPOCH; a build without one, such as a bare # build would exit 0 having run nothing. script/cibuild and
# `docker build .`, fails at the check right after the ARG. # script/docker pass a fresh CHECK_EPOCH on every invocation.
# #
# ARG is per-stage, so the markdown and build stages declare it again. # Two properties this depends on. ARG is per-stage, so the markdown and
# Each gate RUN must reference the value: BuildKit hashes the expanded # build stages below declare it again; one declaration here would leave
# command, so a declared but unreferenced ARG invalidates nothing. Keep # their gates cacheable. And each gate RUN must reference the value,
# it below the dependency layers so they stay cached. # because BuildKit hashes the expanded command: a declared but
# unreferenced ARG invalidates nothing.
#
# It sits below the dependency layers deliberately. Everything above it
# (the pinned base image, go mod download) keeps its cache; only the
# gates go cold.
ARG CHECK_EPOCH ARG CHECK_EPOCH
RUN if [ -z "${CHECK_EPOCH}" ]; then \
echo "CHECK_EPOCH is unset; build via script/cibuild or script/docker" >&2; \
exit 1; \
fi
# These gates call the tools directly, not through `make lint` or # The linter is invoked directly here, not through `make lint`. That
# `make fmt-check`: both run docker, which cannot run inside a docker # target now runs `docker build -f Dockerfile.lint`, and a docker build
# build. This step is the gofmt half of `make fmt-check`; the markdown # cannot run a docker build: routing the gate through make would mean
# stage is its prettier half. gofmt's output is assigned to a variable # nesting docker inside this image. Same reason `make check` is gone
# first so that its own exit status, as when it cannot parse a file, # from the build stage below, and `make fmt-check` from both stages: it
# still fails the step. # runs prettier through docker too. Its gofmt half is the step below,
# its Markdown half the markdown stage further down. gofmt's output is
# assigned to a variable first so that its own exit status, as when it
# cannot parse a file, still fails the step.
RUN echo "gate gofmt, epoch ${CHECK_EPOCH}" && \ RUN echo "gate gofmt, epoch ${CHECK_EPOCH}" && \
files="$(gofmt -s -l .)" && \ files="$(gofmt -s -l .)" && \
if [ -n "$files" ]; then \ if [ -n "$files" ]; then \
echo "gofmt: files not formatted:" >&2; echo "$files" >&2; exit 1; \ echo "gofmt: files not formatted:" >&2; echo "$files" >&2; exit 1; \
fi fi
# Fails the build when the FROM above and the one in Dockerfile.lint pin # The FROM above and the one in Dockerfile.lint pin the same linter
# different linter images. # twice, and nothing else keeps them in sync; this fails the build when
# they disagree. See the script for why it restates neither pin.
RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \ RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
script/verify-lint-image-pin script/verify-lint-image-pin
@@ -66,13 +71,9 @@ WORKDIR /src
# Markdown stage: the Markdown half of `make fmt-check`, as a gate. # Markdown stage: the Markdown half of `make fmt-check`, as a gate.
FROM prettier AS markdown FROM prettier AS markdown
COPY . . COPY . .
# Second per-stage declaration of the gate cache-buster and its check; # Second per-stage declaration of the gate cache-buster; see the lint
# see the lint stage above. # stage above.
ARG CHECK_EPOCH ARG CHECK_EPOCH
RUN if [ -z "${CHECK_EPOCH}" ]; then \
echo "CHECK_EPOCH is unset; build via script/cibuild or script/docker" >&2; \
exit 1; \
fi
RUN echo "gate prettier, epoch ${CHECK_EPOCH}" && \ RUN echo "gate prettier, epoch ${CHECK_EPOCH}" && \
prettier --check '**/*.md' --tab-width 4 --prose-wrap always prettier --check '**/*.md' --tab-width 4 --prose-wrap always
@@ -82,13 +83,16 @@ FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c
# We never build or run as root. Create an unprivileged user and point # We never build or run as root. Create an unprivileged user and point
# HOME and the build cache at its home so go build and go test can write # HOME and the build cache at its home so go build and go test can write
# it when we drop to it below. # it when we drop to it below. $GOPATH/bin is deliberately not on PATH:
# script/bootstrap no longer `go install`s anything (the linter runs
# from a pinned image, never from a host install), so nothing lands
# there and adding it would only widen what this image resolves.
# #
# The module cache stays at the base image's default /go/pkg/mod and # The module cache is kept outside that home, at the base image's
# belongs to root: script/bootstrap fills it as root. Do not move it # default /go/pkg/mod, and belongs to root: script/bootstrap fills it as
# into the home and hand it over with `chown -R`: that walks every file # root. Do not move it into the home and hand it over with `chown -R`:
# in it, which took from about 80 s to over ten minutes on a shared # that walks every file in it, which took from about 80 s to over ten
# host, depending on load. # minutes on a shared host, depending on load.
RUN adduser -D -u 1000 builder RUN adduser -D -u 1000 builder
ENV HOME=/home/builder ENV HOME=/home/builder
ENV GOPATH=/home/builder/go ENV GOPATH=/home/builder/go
@@ -97,16 +101,24 @@ ENV GOCACHE=/home/builder/.cache/go-build
WORKDIR /src WORKDIR /src
# No-op file copies whose only purpose is the build-graph edge: they # No-op file copies whose only purpose is the build-graph edge: they are
# make this stage depend on the lint and markdown stages, so BuildKit # what make this stage depend on the lint and markdown stages, and so
# finishes those gates before compilation and tests start. Remove one # what forces BuildKit to finish gofmt, the pin guard, lint and prettier
# and the build silently stops gating on that stage and still exits 0. # before compilation and tests start. Remove one and the fail-fast
# design dies silently — the build stops gating on that stage and still
# exits 0. The first replaces a copy of the linter binary itself, which
# is no longer wanted here: nothing in this stage runs the linter,
# because `make lint` is now a docker build and a docker build cannot
# run inside one.
COPY --from=lint /src/go.sum /dev/null COPY --from=lint /src/go.sum /dev/null
COPY --from=markdown /src/go.sum /dev/null COPY --from=markdown /src/go.sum /dev/null
# Install development prerequisites the same way a developer does. Only # Install development prerequisites the same way a developer does,
# script/ and the dependency manifests are copied first, so this layer # rather than duplicating the installs inline. Only script/ and the
# stays cached until they change. Bootstrap ends in `go mod download`. # dependency manifests are copied first, nothing else, so this layer
# stays cached until the scripts or the dependencies change — bootstrap
# ends in `go mod download`, which is why there is no separate
# invocation of it here.
COPY script/ script/ COPY script/ script/
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN script/bootstrap RUN script/bootstrap
@@ -129,22 +141,22 @@ RUN chown builder:builder /src /go/pkg/mod/cache/download && \
COPY --chown=builder:builder . . COPY --chown=builder:builder . .
USER builder USER builder
# Fail the build unless the branch is green. Runs as non-root: root # Fail the build unless the branch is green. Runs as non-root so the
# would bypass the chmod(0) the permission-denied tests rely on. # permission-denied test paths are exercised legitimately (root would
# bypass the chmod(0) the tests rely on).
# #
# The gate is `make test`, not `make check`, which runs docker; lint and # The gate is `make test`, not `make check`: that aggregate runs
# the format checks ran in the lint and markdown stages above. `make`, # `script/lint` and `script/fmt-check`, which both run docker, and
# not the script directly, because the Makefile's # nothing inside an image build may shell out to docker. Lint and the
# format checks are not skipped by this — they ran in the lint and
# markdown stages above, which this stage's COPY --from lines make
# prerequisites. `make`, not the script directly, because the Makefile's
# `export CGO_ENABLED = 0` applies only to what it invokes. # `export CGO_ENABLED = 0` applies only to what it invokes.
# #
# Third per-stage declaration of the gate cache-buster and its check; # Third per-stage declaration of the gate cache-buster; see the lint
# see the lint stage above. It is placed after USER so the drop to the # stage above for why one is not enough. It is placed after USER so the
# unprivileged user still happens before the checks run. # drop to the unprivileged user still happens before the checks run.
ARG CHECK_EPOCH ARG CHECK_EPOCH
RUN if [ -z "${CHECK_EPOCH}" ]; then \
echo "CHECK_EPOCH is unset; build via script/cibuild or script/docker" >&2; \
exit 1; \
fi
RUN echo "gate test, epoch ${CHECK_EPOCH}" && make test RUN echo "gate test, epoch ${CHECK_EPOCH}" && make test
# The version stamped into the binary: the VERSION build argument when # The version stamped into the binary: the VERSION build argument when
+35 -21
View File
@@ -1,12 +1,14 @@
# Lint-only image, built by script/lint: the repo is copied into the # Lint-only image: this is how the linter runs, everywhere. The repo is
# pinned golangci-lint image and the linter runs as a build step, so a # COPYed into the pinned golangci-lint image and the linter runs as a
# successful build is a clean lint. No bind mount, so it works when the # build step, so a successful build IS a clean lint. golangci-lint is
# docker daemon is remote. # never installed on a host — one toolchain, pinned by digest, identical
# on a laptop and in CI — and this works even when the docker daemon is
# remote and bind mounts are impossible.
# #
# It is separate from the main Dockerfile's lint stage because # script/lint builds this file. It is a separate image from the lint
# script/lint must not depend on the rest of that build; the two FROM # stage of the main Dockerfile because script/lint must not depend on
# lines are kept identical by script/verify-lint-image-pin, run as a # the rest of that build; the two FROM lines are kept identical by
# gate below. # script/verify-lint-image-pin, run as a gate below.
# golangci/golangci-lint:v2.12.2, 2026-08-07 # golangci/golangci-lint:v2.12.2, 2026-08-07
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240
@@ -18,26 +20,38 @@ RUN go mod download
COPY . . COPY . .
# Cache-buster for the gate layers, and only for them; caching of the # Cache-buster for the gate layers, and only for them. Caching of the
# lint run is waived by ruling. On an unchanged tree the gates below # lint run is waived by ruling: COPY is invalidated only by changed
# would be served from cache and this build would exit 0 in under a # content, so on an unchanged tree the gates below would be served from
# second having run no linter. script/lint passes a fresh value on every # cache and this build would exit 0 in under a second having run no
# linter at all. That exact false green has bitten this repo twice
# already (#32, #39). script/lint passes a fresh value on every
# invocation. # invocation.
# #
# Each gate RUN must reference the value: BuildKit hashes the expanded # Each gate RUN must reference the value, because BuildKit hashes the
# command, so a declared but unreferenced ARG invalidates nothing. Keep # expanded command and not the ARG declaration: a declared but
# it below the dependency layers so they stay cached. # unreferenced ARG invalidates nothing. The ARG sits below the
# dependency layers deliberately — everything above it keeps its cache,
# only the gates go cold.
ARG CHECK_EPOCH ARG CHECK_EPOCH
# Fails the build when the FROM above and the main Dockerfile's lint # The linter version is pinned in two places, here and in the main
# stage pin different linter images. # Dockerfile's lint stage. Nothing else keeps them in sync, so a
# half-applied bump is a build failure; see the script.
RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \ RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
script/verify-lint-image-pin script/verify-lint-image-pin
# Validates .golangci.yml against golangci-lint's JSON schema, which the # Validates .golangci.yml against golangci-lint's JSON schema. The
# pinned binary embeds: measured under `--network none`, it passes a # concern about this step was that it fetches that schema over a live,
# valid config and rejects an invalid one. No gate step makes a network # unpinned HTTPS call; measured on the pinned image, it does not. The
# call, but `go mod download` above needs the network on a cold cache. # binary carries the schema for its own version, so under
# `--network none` this both passes on a valid config and still rejects
# an invalid one with the jsonschema error. That holds for the gate
# steps generally — none of them makes a network call — but not for
# this build as a whole: `go mod download` above needs the network on a
# cold cache, and under `--network none` a first build fails there
# before reaching any gate. That layer stays cached, so only a warm
# cache lints offline, until go.mod or go.sum changes.
RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \ RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
golangci-lint config verify --config .golangci.yml golangci-lint config verify --config .golangci.yml
+1 -4
View File
@@ -6,7 +6,7 @@ BINARY := sfdupes
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev) VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
LDFLAGS := -X main.Version=$(VERSION) LDFLAGS := -X main.Version=$(VERSION)
.PHONY: sfdupes build bootstrap setup test test-race lint fmt fmt-check check docker hooks clean .PHONY: sfdupes build bootstrap setup test lint fmt fmt-check check docker hooks clean
# Standard targets are thin shims; the implementations live in script/ # Standard targets are thin shims; the implementations live in script/
# per the scripts-to-rule-them-all pattern. # per the scripts-to-rule-them-all pattern.
@@ -27,9 +27,6 @@ setup:
test: test:
@script/test @script/test
test-race:
@script/test-race
lint: lint:
@script/lint @script/lint
+11 -29
View File
@@ -263,9 +263,7 @@ All three subcommands operate on a single SQLite database file:
- Location: the value of the `SFDUPES_DATABASE` environment variable when set - Location: the value of the `SFDUPES_DATABASE` environment variable when set
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
command-line flag. The path names the file exactly, whatever characters it command-line flag.
holds (`?`, `#` and `%` included); a relative path is relative to the working
directory.
- `scan` creates the database (and its parent directory) on first use. `report` - `scan` creates the database (and its parent directory) on first use. `report`
and `trees` require an existing database; a missing database file is a fatal and `trees` require an existing database; a missing database file is a fatal
error (exit 1) telling the user to run `scan` first. error (exit 1) telling the user to run `scan` first.
@@ -742,33 +740,22 @@ entrypoints are:
installed: they run in Docker (see `script/lint` and `script/fmt`) and never installed: they run in Docker (see `script/lint` and `script/fmt`) and never
from a host install, so there is no host copy to drift from the pin. A missing from a host install, so there is no host copy to drift from the pin. A missing
`docker` is warned about rather than installed or treated as fatal — `docker` is warned about rather than installed or treated as fatal —
everything except linting, formatting and `make test-race` works without it. everything except linting and formatting works without it. Ends with
Ends with `go mod download`. `go mod download`.
- `script/setup` — make a fresh clone ready for development: runs - `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`. `script/bootstrap`, then `script/install-precommit`.
- `script/projectname` — print this project's name (`sfdupes`). Scripts that - `script/projectname` — print this project's name (`sfdupes`). Scripts that
need the name call it, so they stay identical across repositories. need the name call it, so they stay identical across repositories.
- `script/test` — run the test suite with a 30-second timeout and coverage - `script/test` — run the test suite with a 30-second timeout and coverage
enabled, rerunning verbosely on failure so the logs show which test failed. enabled, rerunning verbosely on failure so the logs show which test failed.
- `script/test-race` — run the test suite under the race detector with a
60-second timeout. The detector needs cgo and a C compiler, which the build
never uses, so the tests run in a digest-pinned Debian `golang` image that has
`gcc`, with the checkout mounted read-only; the container is removed when it
exits. They run as the calling user, or as `nobody` when that is root, because
several tests make a file unreadable and root reads it anyway; only then must
the checkout be readable by other users. Not part of `script/check`. Every run
starts with empty caches, so it needs the network and takes minutes, and the
mount needs a local docker daemon.
- `script/lint` — run the linter. It builds `Dockerfile.lint`, which copies the - `script/lint` — run the linter. It builds `Dockerfile.lint`, which copies the
repository into the digest-pinned `golangci/golangci-lint` image and runs repository into the digest-pinned `golangci/golangci-lint` image and runs
`golangci-lint config verify` and `golangci-lint run` as build steps, so a `golangci-lint config verify` and `golangci-lint run` as build steps, so a
successful build is a clean lint. That exit status is all it produces, so it successful build is a clean lint. The linter is never run on the host, which
runs with `--output=type=cacheonly` and writes no image; a run leaves only makes a working `docker` the one prerequisite for linting — and therefore for
build cache. The linter is never run on the host, which makes a working `make check` and the pre-commit hook. Offline machines: the gate steps
`docker` the one prerequisite for linting — and therefore for `make check` and themselves make no network calls. `golangci-lint run` does not, and neither
the pre-commit hook. Offline machines: the gate steps themselves make no does `golangci-lint config verify` — it validates against a schema the pinned
network calls. `golangci-lint run` does not, and neither does
`golangci-lint config verify` — it validates against a schema the pinned
binary embeds, measured under `--network none` to both pass a valid config and binary embeds, measured under `--network none` to both pass a valid config and
reject an invalid one. The build around them does. `Dockerfile.lint` runs reject an invalid one. The build around them does. `Dockerfile.lint` runs
`go mod download` before the gates and this module has external dependencies, `go mod download` before the gates and this module has external dependencies,
@@ -830,12 +817,9 @@ from binary-versus-pin to pin-versus-pin, which is what
gate layers from cache and the build exits 0 having executed no tests and no gate layers from cache and the build exits 0 having executed no tests and no
lint — a green it never earned, and one this repository has produced twice. lint — a green it never earned, and one this repository has produced twice.
`CHECK_EPOCH` invalidates the gate layers on every run while leaving the pinned `CHECK_EPOCH` invalidates the gate layers on every run while leaving the pinned
base images and the dependency layers cached. Each script's value carries the base images and the dependency layers cached. `script/lint`'s value carries the
process id as well as the epoch, because two runs land inside the same second process id as well as the epoch, because two lint runs land inside the same
easily and a bare epoch would cache the second one. Each `Dockerfile` stage with second easily and a bare epoch would cache the second one.
gates fails when the value is empty, so a bare `docker build .` stops with
`CHECK_EPOCH is unset; build via script/cibuild or script/docker` instead of
serving the gates from cache.
## Build ## Build
@@ -849,8 +833,6 @@ compile recipe:
- `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook. - `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook.
- `make test` — run the test suite (30-second timeout; reruns with `-v` on - `make test` — run the test suite (30-second timeout; reruns with `-v` on
failure). failure).
- `make test-race` — run the test suite under the race detector, in Docker (see
`script/test-race`); requires `docker`. Not part of `make check`.
- `make lint` — run `golangci-lint` with the repo config, in Docker (see - `make lint` — run `golangci-lint` with the repo config, in Docker (see
`script/lint`); requires `docker`. `script/lint`); requires `docker`.
- `make fmt` / `make fmt-check` — format the Go sources and the Markdown / - `make fmt` / `make fmt-check` — format the Go sources and the Markdown /
+273 -196
View File
@@ -2,15 +2,14 @@
- take an issue from the `1.0.0` milestone on the tracker; work not yet on the - take an issue from the `1.0.0` milestone on the tracker; work not yet on the
tracker gets filed as an issue first tracker gets filed as an issue first
- branch from `next` - branch (from `main`)
- do the work, with tests, in small focused commits - do the work, with tests, in small focused commits
- record it at the top of Completed Steps (`TODO.md` changes in the same commit - record it at the top of Completed Steps (`TODO.md` changes in the same commit
as the work) as the work)
- push the branch and open a PR against `next` whose title ends with - push the branch and open a PR whose title ends with ` (closes #N)`
` (closes #N)` - an independent review gates the merge; every finding is addressed or
- an independent review gates each merge to `next`; every finding is addressed explicitly rebutted on the PR
or explicitly rebutted on the PR - merge to `main` once the review passes
- only the owner merges `next` to `main`
# Status # Status
@@ -29,39 +28,10 @@
# Completed Steps # Completed Steps
- cut the narration from `TODO.md` Completed Steps and from the comments in
`script/` and both Dockerfiles; §Workflow now branches from and merges to
`next` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/49)
- `make test-race` runs the test suite under the race detector in a cgo-enabled
container, outside `make check` (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/18)
- a bare `docker build .` fails with a message naming `script/cibuild` and
`script/docker` instead of serving the gates from cache (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/39)
- `make fmt` and `make fmt-check` run prettier over all Markdown, in Docker, and - `make fmt` and `make fmt-check` run prettier over all Markdown, in Docker, and
CI checks it; all Markdown reformatted (2026-10-04, CI checks it; all Markdown reformatted (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/19) https://git.eeqj.de/sneak/sfdupes/issues/19)
- `script/lint` writes no image, so a run no longer leaves an untagged one
behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
- tests cover a missing database, `scan` keeping stdout empty, its skip warning,
the `report` and `trees` summary lines, and every subcommand going through
`runE` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
- `.golangci.yml` replaced with the current canonical copy, which uses
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/26)
- a test fails when either `hashWorker` cancellation check in `scan.go` is
removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)
- a database path holding `?`, `#` or `%` opens exactly the file it names
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
- `scan` rejects `--workers` below 1 as a usage error instead of running - `scan` rejects `--workers` below 1 as a usage error instead of running
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10) single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
@@ -126,24 +96,29 @@
- stamp the git tag or short commit in a plain `docker build .` instead of `dev` - stamp the git tag or short commit in a plain `docker build .` instead of `dev`
(2026-10-02, branch `next`, closes (2026-10-02, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` sends `.git` https://git.eeqj.de/sneak/sfdupes/issues/67): `.dockerignore` now sends
without `.git/config`; the build stage stamps the `VERSION` build argument, `.git`, without `.git/config`, and the `Dockerfile` build stage takes the
else `git describe --tags --always`, and fails if the context carries `.git` `VERSION` build argument when one is given, otherwise
and the version is still empty, `dev` or `unknown`. CI checks out the full `git describe --tags --always` of that `.git`. The build fails if the context
history (`fetch-depth: 0`) so it stamps the same value as `make build`. carries `.git` and the version still comes out empty, `dev` or `unknown`. The
CI checkout step fetches the full history (`fetch-depth: 0`) so CI sees the
tag and stamps the same value as `make build`.
- replace the 1 KiB end-window sampling with the head/tail plus content-hash - replace the 1 KiB end-window sampling with the head/tail plus content-hash
ladder (2026-09-22, branch `next`, closes ladder (2026-09-22, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/61); README "Duplicate detection" https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is hashed in
documents every rung. A file under 10 MiB is hashed in full, and its `head`, full and compared directly, with no end-window step — its `head`, `tail`, and
`tail` and `content` all hold that hash. A larger file gets only its 64 KiB `content` all hold the whole-file hash. A file at 10 MiB or above gets only
`head` and `tail` in the hash phase; the content phase, after the update the 64 KiB `head` and `tail` in the hash phase; a new content phase, after the
phase, reads it for `content` (the whole file below 50 MiB, gigabyte-spaced 1 update phase, reads it for its `content` hash — the whole file below 50 MiB,
MiB samples at or above) only when its size, `head` and `tail` match another gigabyte-spaced 1 MiB samples at or above — only when its size, `head`, and
record's from this scan or an earlier one, and never reads a file gone or `tail` match another record's, from the same scan or stored by an earlier one,
changed since its record was written. `report` and `trees` leave out any so a stored file gains its content hash when it gains a match. A file that is
record without a `content` hash. The `content` column is part of the version 1 gone or has changed since its record was written is not read. The `content`
schema. column is part of the version 1 schema. `report` and `trees` group by the
extended signature and leave out any record without a `content` hash, so the
ladder is applied across the whole database. README "Duplicate detection"
documents every rung including the probabilistic large-file path.
- remove the dead `files.dat` references from `Makefile`, `.gitignore` and - remove the dead `files.dat` references from `Makefile`, `.gitignore` and
`.dockerignore` (2026-09-21, branch `next`, closes `.dockerignore` (2026-09-21, branch `next`, closes
@@ -151,162 +126,264 @@
- fix the lint-image pin comments and `FROM` form in `Dockerfile` and - fix the lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` (2026-08-10, branch `next`, closes `Dockerfile.lint` (2026-08-10, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/25): both pins are now the policy https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`, `(Debian-based)` parenthetical (v2.12.1 was Debian too) and the redundant tag,
without the false `(Debian-based)` note or the tag; digest unchanged. so both pins are the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare
`script/verify-lint-image-pin` still matches the tagless form, and a tag on `FROM image@sha256:...`. Digest unchanged. `script/verify-lint-image-pin`
one side only is caught as a plain mismatch. parses those `FROM` lines and still matches the tagless form; its advice line
lost the now meaningless "tag and digest". With no tag in either reference, a
tag-only disagreement no longer exists — a one-sided tag is caught as a plain
mismatch.
- run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10, - run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
owner ruling the linter is never installed on a host. `Dockerfile.lint` copies owner ruling, the linter runs inside a container invoked through the `script/`
entrypoint and is never installed on a host. New root `Dockerfile.lint` COPYs
the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
runs `golangci-lint config verify` and `golangci-lint run` as build steps; runs `golangci-lint config verify` and `golangci-lint run` as build steps, so
`script/lint` builds it. `script/bootstrap` no longer installs or pins the a successful build IS a clean lint; `script/lint` is reduced to building it.
linter, and warns rather than fails when `docker` is absent; `script/bootstrap` loses the `go install`, the pin constants, the version
`ENV PATH=/home/builder/go/bin:$PATH` went with its `go install`. parser and `verify_golangci_lint` outright rather than hardening them — with
`script/verify-linter-pin` is retired; `script/verify-lint-image-pin`, a gate nothing linting on the host, the `$GOPATH/bin` versus `PATH` problem that
in both files, compares their two `FROM` lines and restates neither pin. motivated them has no subject — and now warns rather than fails when `docker`
Traps: an unchanged tree lets a lint build pass in under a second having run is absent. Two traps handled. A lint build on an unchanged tree returns
no linter, so every gate `RUN` references `ARG CHECK_EPOCH` (BuildKit hashes success in well under a second having run no linter, which is
the expanded command) and `script/lint` passes `"$(date +%s)-$$"`, the PID https://git.eeqj.de/sneak/sfdupes/issues/32 and
because two runs land in the same second easily. Nothing inside an image build https://git.eeqj.de/sneak/sfdupes/issues/39 again, so `Dockerfile.lint`
may shell out to docker, so the `Dockerfile` lint stage calls `golangci-lint` carries `ARG CHECK_EPOCH` referenced inside every gate `RUN` (BuildKit hashes
directly and the build stage runs `make test` and `make fmt-check` instead of the expanded command, not the declaration) and `script/lint` passes
`make check`, through `make` because the Makefile's `export CGO_ENABLED = 0` `"$(date +%s)-$$"` — the PID matters because two lint runs land inside the
only reaches what it invokes. `COPY --from=lint /src/go.sum /dev/null` same second easily. And nothing inside an image build may shell out to docker,
replaces the copied linter binary as the only edge making the build stage wait so the main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
for lint; dropping it would end fail-fast linting under a still-green build. instead of `make lint`, and its build stage runs `make test` and
`golangci-lint config verify`, included per the ruling, validates from an `make fmt-check` instead of the `make check` aggregate (`make`, not the
embedded schema with no network call, but `go mod download` above the gates scripts bare, because the Makefile's `export CGO_ENABLED = 0` only reaches
still needs the network on a cold cache. Verified: `make lint` green with no what it invokes). `COPY --from=lint` `/usr/bin/golangci-lint` is replaced by
`golangci-lint` on `PATH`; two back-to-back `script/lint` runs on an untouched `COPY --from=lint /src/go.sum /dev/null`: the copied binary was the only edge
tree both ran the linter (27.7s and 28.7s in the lint step, `COPY . .` forcing BuildKit to finish linting before the build stage starts, and dropping
`CACHED` above); a planted unused variable failed `script/lint`, and failed it without replacing the edge would have ended fail-fast linting silently
`make docker` at `[lint 9/9]` with the build stage stopped at under a still-green build. That is canonical `REPO_POLICIES.md:107`'s ordering
`[builder 3/12]`; the drift guard fails on a tag-only, a digest-only and an edge, restored. `ENV PATH=/home/builder/go/bin:$PATH` is gone with the
unreadable reference, naming both sides; under `--network none` config verify `go install` that justified it. `script/verify-linter-pin` is retired, deleted
passes a valid config and rejects an invalid one; `make docker` green in 5m35s along with its README entry, because both of its subjects ceased to exist in
with all six gates run under one epoch (lint 37.6s, test 25.2s reporting the same change: it compared a linter binary against `GOLANGCI_LINT_VERSION`
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`); in the in `script/bootstrap`, and there is now neither a binary crossing between
builder image with the Go test cache off, `--user 0:0` still fails stages nor a version pin in bootstrap. The drift it guarded has not gone away,
`TestScanHardlinkRunFailsTogether` where the unprivileged user passes. Noted it has moved — the linter is still pinned twice, now as the `FROM` line of
for follow-up, not fixed here: `golangci-lint` warns that `gomodguard` is `Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage, with
deprecated since v2.12.0 in favour of `gomodguard_v2`. nothing syncing them, which is exactly what
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
replacement is one new `script/verify-lint-image-pin`, run as a gate in both
files, which compares the two references to each other and deliberately
restates neither: a hardcoded expected digest would be a third copy and the
same drift one file further out. `golangci-lint config verify` is included per
the ruling, and the concern about its unpinned live HTTPS schema fetch was
measured rather than assumed — under `--network none` the pinned binary both
passes a valid config and rejects an invalid one with the jsonschema error, so
it validates from an embedded schema and makes no network call of its own. The
README scopes that to the gate steps rather than to linting as a whole:
`Dockerfile.lint` runs `go mod download` above them, so a cold cache still
needs the network and only a warm one lints offline. Verified: `make lint`
green with every `PATH` directory containing a `golangci-lint` removed
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
`command -v golangci-lint` empty); two consecutive `script/lint` runs on an
untouched tree both executed the linter, 27.7s and 28.7s in the lint step
under distinct epochs with the `COPY . .` layer `CACHED` above them, at 42.2s
and 41.8s wall clock — the no-cache rule was not weakened to shorten that.
Negative control: a planted `var unusedIssue46Sentinel = 1` failed
`script/lint` with
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and failed
`make docker` at `[lint 9/9]` with the build stage stopped at `[builder 3/12]`
— `COPY --from=lint`, `script/bootstrap`, the test gate and `make build` all
zero occurrences — then reverted clean. The drift guard fails on a tag-only
disagreement, on a digest-only disagreement, and on an unreadable reference,
naming both sides. `make docker` green in 5m35s with all six gates executing
under one epoch (lint 37.6s, test 25.2s reporting
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). The
non-root quirk still holds: in the builder image with the Go test cache off,
`--user 0:0` fails `TestScanHardlinkRunFailsTogether` (exit 1) where the
unprivileged user passes (exit 0). Noted for follow-up, not fixed here:
`golangci-lint` warns that the `gomodguard` linter is deprecated since v2.12.0
in favour of `gomodguard_v2`.
- install the Docker build stage's prerequisites by running `script/bootstrap` - install the Docker build stage's prerequisites by running `script/bootstrap`
instead of `apk add --no-cache make` inline (2026-08-09, branch instead of `apk add --no-cache make` inline (2026-08-09, branch
`dockerfile-bootstrap`, closes https://git.eeqj.de/sneak/sfdupes/issues/42): `dockerfile-bootstrap`, closes #42): canonical `REPO_POLICIES.md:97` requires
the stage copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`, it, and the inline install left the build stage maintaining its own notion of
which ends in `go mod download`, so the separate call to it is gone. the toolchain — exactly the divergence #24 exists to close, one layer down.
`COPY --from=lint /usr/bin/golangci-lint` stays and moves above the bootstrap The stage now copies `script/` plus `go.mod`/`go.sum` and runs
layer: it is the only edge making this stage depend on the lint stage, so `script/bootstrap`, which ends in `go mod download`, so the separate
deleting it would end fail-fast linting silently. A new invocation of that is gone. `COPY --from=lint /usr/bin/golangci-lint` stays,
`script/verify-linter-pin`, run in the build stage before bootstrap, fails the and moves above the bootstrap layer. It is the only edge making this stage
build naming both versions unless that copied binary is the version depend on the lint stage, so deleting it as redundant would end fail-fast
`script/bootstrap` pins; a pin it cannot read is a hard failure, not a skip. linting silently. Letting bootstrap install its own linter here would have
`$GOPATH/bin` joins `PATH`, where bootstrap's `go install` lands. Everything reintroduced the second toolchain and paid for a from-source build of it. What
added sits above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still makes the two stages provably one toolchain rather than two that happen to
precede `make check`. Verified: the guard fails the build with both versions agree is a new `script/verify-linter-pin`, run in the build stage on the
named when the lint stage's linter is faked to another version, and passes an binary that arrives from the lint stage, before bootstrap: it fails the build
unmodified build; bootstrap runs clean under Alpine's `sh` and `apk`, finding naming both versions unless that binary is the version `script/bootstrap`
the copied linter already at the pin; a second build served the bootstrap and pins. Bootstrap's own check could not serve that purpose — it reinstalls its
dependency layers `CACHED` while both gates ran with a fresh epoch; a planted pin from source and then verifies whatever `PATH` resolves, so drift
`unused` finding failed the build at the lint gate in 48.9s with the build self-heals silently and a lint stage image bumped on its own would lint at the
stage's `make check` never starting; and the suite run in the image as new version while `make check` ran at the old one, green. The linter version
`--user 0:0` fails `TestScanHardlinkRunFailsTogether`, so the drop to the is pinned in two independent places (the lint stage image digest and
unprivileged user is still needed. That last check needs the Go test cache `GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
off: as root it first reported `ok ... (cached)`, reusing the build-time half-applied bump is now a build failure. The pin is read out of
result. Build times on a noisy shared host: 2m13s on an unchanged tree, 2m17s `script/bootstrap`, which stays the single source of truth; a pin that cannot
and 4m29s after a source change, 5m14s cold, which breaches the policy be read is a hard failure, not a skip. The check needs no `CHECK_EPOCH`: its
ceiling; `chown -R builder:builder /src /home/builder` walks the module cache only inputs are the copied binary and `script/`, so Docker invalidates the
and alone varied from 77s to 210s across those builds, and `main` measured layer exactly when a cached result would stop being true, and it is documented
5m03s cold with a 209s `chown`. Filed as with the other entrypoints in the README. `$GOPATH/bin` joins `PATH` because
https://git.eeqj.de/sneak/sfdupes/issues/43 that is where bootstrap's `go install` lands and bootstrap verifies its
installs against what `PATH` resolves — nothing in the image is shadowed by
it, the directory does not exist until bootstrap runs. Everything added sits
above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still precede
`make check`. Verified: the guard fails the build with both versions named
when the lint stage's linter is faked to a different version, and an
unmodified build still passes it; bootstrap runs clean under Alpine's `sh` and
its `apk` branch, installing `git` and `make` and finding the copied linter
already at the pin; a second build served the bootstrap and dependency layers
`CACHED` while both gates ran with a fresh epoch; a planted `unused` finding
failed the build at the lint gate in 48.9s with the build stage's `make check`
never starting; and the suite run in the image as `--user 0:0` fails
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged user is
still load-bearing. That last check needs the Go test cache disabled — the
first attempt reported `ok ... (cached)` as root, reusing the result the
build-time run had left in the shared cache, which would have read as a pass.
Build wall time, on a shared host running many concurrent builds and so noisy:
2m13s on an unchanged tree, 2m17s and 4m29s for two builds after a source
change, 5m14s cold. Only the cold one breaches the policy ceiling, and not
because of this change — `chown -R builder:builder /src /home/builder` walks
the module cache and re-runs on every source change, and it alone varied
between 77s and 210s across those four builds, which is also the whole spread
in the totals. The same cold measurement against `main` is 5m03s with a 209s
`chown`. Filed as #43
- bust the Docker layer cache for the gate steps, so `script/cibuild` and - bust the Docker layer cache for the gate steps, so `script/cibuild` and
`script/docker` cannot report a green they did not earn (2026-08-09, branch `script/docker` cannot report a green they did not earn (2026-08-09, branch
`cibuild-cache-bust`, closes https://git.eeqj.de/sneak/sfdupes/issues/32): the `cibuild-cache-bust`, closes #32): both scripts were bare `docker build`
`Dockerfile` copies the tree before its gates, so on an unchanged tree Docker invocations with no cache control, and the `Dockerfile` copies the tree before
served them from cache and the build exited 0 having run nothing. Both scripts running its gates, so on an unchanged tree Docker served those layers from
now pass `--build-arg CHECK_EPOCH="$(date +%s)"`. `ARG` is per stage and the cache and the build exited 0 having executed nothing. That is not hypothetical
gates span two stages, so it is declared in both; BuildKit hashes the expanded here — every merge this repo has done is a non-fast-forward merge of an
command, so each gate `RUN` echoes the epoch, which also logs it as evidence undiverged branch, so each merge commit's tree is byte-identical to the branch
the layer ran. It sits below the dependency layers so they stay cached. head's and each merge CI run was almost certainly a full cache hit; and PR
Verified under `BUILDKIT_PROGRESS=plain`, each script run twice back to back #31's reviewer found `make docker` returning success as a 17-layer cache hit,
on an unchanged tree: all three gates ran on all four runs with a fresh epoch catching it only by being suspicious. The fix is `ARG CHECK_EPOCH` with the
(`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s then 53.4s), and scripts passing `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or
thirteen steps were still served `CACHED`. With a planted `unused` finding the break it. `ARG` is scoped per stage and this `Dockerfile` has three gates
build failed at `make lint` in 36.1s and the build-stage `make check` never across two — `make fmt-check` and `make lint` in the lint stage, `make check`
started. Run as root, the same image fails `TestScanHardlinkRunFailsTogether`, in the build stage — so a single declaration would have left one stage
because root reads through the `chmod(0)` the test relies on, so the build silently cacheable; it is declared in both. And BuildKit hashes the expanded
stage must drop to the unprivileged `builder` user. Local fix only; command, not the declaration, so a declared-but-unreferenced `ARG` invalidates
propagating it to the canonical templates is nothing: each gate `RUN` echoes the epoch, which also puts the value in the
https://git.eeqj.de/sneak/prompts/issues/26 build log as evidence the layer really ran. Placement is below the dependency
layers on purpose — a build that goes cold every time would be a different
bug, not a fix. Verified by running each script twice back to back on an
unchanged tree under `BUILDKIT_PROGRESS=plain`: all three gates executed on
all four runs, each with a fresh epoch in the log (`script/cibuild` 78.8s then
61.1s; `script/docker` 61.1s then 53.4s), and twelve steps were still served
`CACHED` in the steady state — both `go mod download`s, `apk add`, `adduser`,
the `chown`, every `go.mod`/`go.sum` and source copy, the linter copy out of
the lint stage, and the binary copy into the runtime stage. The lint stage
still gates the build stage: with a deliberate `unused` finding planted in the
tree, the build failed at `make lint` in 36.1s and the build-stage
`make check` never started. The build stage also still drops to the
unprivileged `builder` user before `make check`, which the suite depends on
rather than merely prefers: forcing the same image to run the tests as root
fails `TestScanHardlinkRunFailsTogether`, because root reads straight through
the `chmod(0)` the test uses to prove hard links are read once. This is the
local fix only; propagating it to the canonical templates is `prompts` #26
- check the installed golangci-lint version in `script/bootstrap` instead of - check the installed golangci-lint version in `script/bootstrap` instead of
only its presence (2026-08-09, branch `bootstrap-version-check`, closes only its presence (2026-08-09, branch `bootstrap-version-check`, closes #24):
https://git.eeqj.de/sneak/sfdupes/issues/24): the version lives only in `missing golangci-lint` meant any linter already on `PATH` satisfied the
`GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it, and check, so the pin was never consulted and the v2.12.2 bump from #3 was inert
any installed version that is not the pin — older, newer, absent or on every host that already had one — this host ran v2.10.1 against a v2.12.2
unparseable — is reinstalled. `go install` writes into `GOBIN` (or pin, `make check` went green, and `make docker` then rejected the same commit
`GOPATH/bin`) while `make lint` runs the first `golangci-lint` on `PATH`, so with findings the local gate never saw. The version now lives in one place,
bootstrap re-reads the effective version after installing and, on a mismatch, `GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it so a
prints both paths and both versions and exits non-zero; it does not reorder bump cannot half-apply; a `golangci_lint_version` helper parses
`PATH` or delete anyone's binary. The `--version` call keeps its stderr and is `golangci-lint --version` (taking the field after the word `version` and
bounded by `timeout(1)` where that exists. `git`, `make` and `go` keep tolerating an optional leading `v`, which the module ref carries and the
presence-only checks. Verified by bootstrapping this host from v2.10.1 to binary's output does not), and any version that is not the pin — older, newer,
v2.12.2 and again to a no-op, and by stub runs of the script under `dash` absent or unparseable — is reinstalled. The install is then verified against
covering a thirteen-input version-parse matrix, a shadowed install that must the binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
exit non-zero, an install destination not on `PATH`, `GOBIN` set, and a wedged `GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes first on
binary that must hit the timeout; `make check` and `make lint` are clean at `PATH`, so a wrong-version one sitting ahead of it — nix, apt, brew, apk, or
v2.12.2, so v2.10.1 was not hiding any findings on `main` the `/usr/local/bin` copy the `Dockerfile` builder stage makes — would swallow
the install and leave the local gate disagreeing with CI under an affirmative
`bootstrap complete`. Bootstrap now re-reads the effective version after
installing and, on a mismatch, prints both paths and both versions to stderr
and exits non-zero instead of claiming success; it does not reorder anyone's
`PATH` or delete their binary. The `--version` call keeps its stderr
connected, so a present-but-broken binary says why rather than reinstalling
forever in silence, and is bounded by `timeout(1)` where that exists, so a
wedged binary cannot hang bootstrap. `git`, `make` and `go` keep their
presence-only checks and now say why in a comment: they are host
package-manager tools the repo deliberately does not pin, with `go.mod`
governing the language version and the digest-pinned images covering
reproducible builds. Verified on this host by bootstrapping from v2.10.1 to
v2.12.2 and running it again to a no-op, plus stub runs of the real script
under `dash` covering a thirteen-input parse matrix (absent, older, newer,
host-style, image-style, leading-`v`, stderr-only, empty, non-zero exit,
impostor binary, `(devel)`, trailing `version`), a shadowed install that must
exit non-zero, an install destination not on `PATH` at all, `GOBIN` set, and a
wedged binary that must hit the timeout; `make check` and `make lint` are
clean at v2.12.2, so v2.10.1 was not hiding any findings on `main`
- unwind the hash worker pool on the error path (2026-08-09, branch - unwind the hash worker pool on the error path (2026-08-09, branch
`hash-pool-cleanup`, closes https://git.eeqj.de/sneak/sfdupes/issues/6): the `hash-pool-cleanup`, closes #6): `hashPhase` used to return the moment
pool is now an owned, context-aware `hashPool`: every blocking send in the `recordRun` failed and abandon the pool — the feeder parked forever on a full
feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path `jobs` channel and every worker on a full `results` channel. That only stopped
out, and `hashPhase` defers `pool.stop()`, which cancels and then drains being invisible when #4 landed and `runScan` began unwinding instead of
`results` until the last goroutine has exited — draining is what frees a calling `os.Exit`. The pool is now an owned, context-aware `hashPool`: every
worker already parked on a send. `ctx` is threaded from `cmd.Context()` blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is
through `runScan`, `syncScan`, both worker pools and the whole database layer, closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels
as the first parameter everywhere. The walk pool gets the same treatment plus and then drains `results` until the last goroutine has exited — draining is
a `ctx.Err()` guard after the walk: a cancelled walk yields a partial size what frees a worker already parked on a send. `ctx` is threaded from
census, and every file it never reached looks vanished to the update phase. `cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole
That phase's own `BeginTx` also fails on the cancelled context before deleting database layer (it is the first parameter everywhere), so #5 can hand this
anything, but the guard is the barrier that still holds once an interrupted path a signal and needs to add nothing else. The walk pool never leaked,
scan may commit what it has. Tests drive `run(scan)` against a database whose because `walkPhase` always drains its events to close, but it has the same
insert trigger aborts and assert that the scan fails instead of hanging and unbounded-send shape and #5 will give it an early return, so it gets the same
that `runtime.NumGoroutine()` polls back to its pre-scan baseline; others treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a
cancel a scan part-way through the walk, deterministically, by counting its partial size census, and every file it never reached looks vanished to the
own consultations of `ctx.Done()`, and assert that it stops at the guard update phase. That phase's own `BeginTx` fails on the same cancelled context
holding a partial census and a still-populated record index, with every record before deleting anything, so the guard is defence in depth rather than the
intact. Direct tests of `sendEvent`, the walk workers, `dispatchDirs`, only barrier — but it is the one that survives #5 deciding an interrupted scan
`feedHashJobs`, `hashWorker` and `hashPhase` cover the remaining cancellation may commit what it has. Tests drive `run(scan)` against a database whose
branches of both pools insert trigger aborts, and assert both that the scan fails instead of hanging
and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; a
second set cancels a scan part-way through the walk — deterministically, by
counting the scan's own consultations of `ctx.Done()` rather than racing a
timer — and asserts that it stops at the guard holding a partial census and a
still-populated record index, with every record intact. The remaining
cancellation branches of both pools are covered by direct tests of
`sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
and `hashPhase`
- guarantee the database is closed on every fatal exit path (2026-08-09, branch - guarantee the database is closed on every fatal exit path (2026-08-09, branch
`db-close-on-fatal`, closes https://git.eeqj.de/sneak/sfdupes/issues/4): `db-close-on-fatal`, closes #4): `fatalf` and its `os.Exit(1)` are gone, so
`fatalf` and its `os.Exit(1)` are gone, so the deferred `db.Close()` — and the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now
with it the SQLite WAL checkpoint — now actually runs when a subcommand fails; actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`,
`runScan`, `runReport`, `runTrees`, `loadRecords` and `resolveRoots` return `loadRecords` and `resolveRoots` return errors instead. The single exit point
errors instead. The single exit point is `run` in `main.go`: it maps a is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned)
`fatalError` (anything a subcommand returned) to exit 1 and cobra's own to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a
argument and flag errors to exit 2, which keeps a runtime failure from being runtime failure from being reported as a usage error or printing the usage
reported as a usage error or printing the usage text. New `main_test.go` text. New `main_test.go` drives the CLI in-process and asserts the exit codes
drives the CLI in-process and asserts the exit codes from README §Error from README §Error handling plus the stdout/stderr split, including that a
handling plus the stdout/stderr split, including that a fatal error raised fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar
after the database is open leaves no `-wal`/`-shm` sidecar behind for `scan`, behind for `scan`, `report` or `trees`
`report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch - update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
`golangci-v2.12.2`, merged as `38a01bd`, closes `golangci-v2.12.2`, merged as `38a01bd`, closes #3): bumped the pinned linter
https://git.eeqj.de/sneak/sfdupes/issues/3): bumped the pinned linter in the in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2,
`Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2, and and replaced `.golangci.yml` with the canonical file — the linter settings
replaced `.golangci.yml` with the canonical file — the linter settings (`lll`, (`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
`funlen`, `cyclop`, `dupl` thresholds) now live under `linters.settings` per `linters.settings` per the v2 schema, so they are actually applied; no new
the v2 schema, so they are actually applied; no new lint findings surfaced lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints - convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
like the other managed repos (2026-07-26, commit `3abeacf`, closes like the other managed repos (2026-07-26, commit `3abeacf`, closes #1): all 12
https://git.eeqj.de/sneak/sfdupes/issues/1): all 12 `script/` entrypoints `script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`,
exist (`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`, `lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`) `install-precommit`) and every Makefile target is now a thin shim over them,
and every Makefile target is now a thin shim over them matching the other managed repos
- make the binary the default Make target (2026-07-24, branch - make the binary the default Make target (2026-07-24, branch
`make-default-target`): plain `make` now builds `sfdupes` (previously it ran `make-default-target`): plain `make` now builds `sfdupes` (previously it ran
`check` plus `build`); `make build` remains as an alias `check` plus `build`); `make build` remains as an alias
@@ -325,7 +402,8 @@
- announce each operand on stderr before its passes (2026-07-24, branch - announce each operand on stderr before its passes (2026-07-24, branch
`scan-operand-progress`): with per-operand walk/hash/update cycles, a `scan-operand-progress`): with per-operand walk/hash/update cycles, a
multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
whole run's whole run's — an operator watching operand 3 of 14 hash 300k files concluded
20M files were being skipped
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single - parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
goroutine and took hours at ~20M files on a busy pool (observed: 22M files in goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
4h on a ZFS server); it is now a per-directory worker-pool traversal that 4h on a ZFS server); it is now a per-directory worker-pool traversal that
@@ -397,6 +475,5 @@ Accepted divergences (no action):
- flat single-package layout with `.go` files in the repo root — fine for a - flat single-package layout with `.go` files in the repo root — fine for a
small single-binary tool per the Go styleguide; the tracker audit agrees small single-binary tool per the Go styleguide; the tracker audit agrees
- `make test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go - `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go
builds) and the race detector requires cgo, so the detector runs in a separate builds) and the race detector requires cgo
cgo-enabled container, `make test-race`, which is not part of `make check`
+27 -67
View File
@@ -344,14 +344,13 @@ func storedPaths(t *testing.T, path string) []string {
// TestRunScanInterrupted calls the scan entrypoint with a context that // TestRunScanInterrupted calls the scan entrypoint with a context that
// is already cancelled, as when a signal arrives at once. It must return // is already cancelled, as when a signal arrives at once. It must return
// errInterrupted promptly with its one line on stderr and nothing on // errInterrupted promptly with its one line on stderr, leave the
// stdout, leave the database valid and as it was, and leave nothing in // database valid and as it was, and leave nothing in the way of the
// the way of the next scan, which must bring the database up to date. // next scan, which must bring the database up to date.
func TestRunScanInterrupted(t *testing.T) { func TestRunScanInterrupted(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
stdout := captureStdout(t)
stderr := captureStderr(t) stderr := captureStderr(t)
dir := buildSmokeTree(t) dir := buildSmokeTree(t)
@@ -394,10 +393,6 @@ func TestRunScanInterrupted(t *testing.T) {
t.Errorf("stderr = %q, want %q", got, want) t.Errorf("stderr = %q, want %q", got, want)
} }
if got := stdout(); got != "" {
t.Errorf("stdout = %q, want nothing (data only)", got)
}
assertNoSidecars(t, path) assertNoSidecars(t, path)
if got := storedPaths(t, path); !slices.Equal(got, before) { if got := storedPaths(t, path); !slices.Equal(got, before) {
@@ -420,13 +415,12 @@ func TestRunScanInterrupted(t *testing.T) {
// TestRunScanInterruptedMidHash interrupts the scan entrypoint part-way // TestRunScanInterruptedMidHash interrupts the scan entrypoint part-way
// through its hash phase, after the database is open. It must return // through its hash phase, after the database is open. It must return
// errInterrupted, release the lock, end stderr with its line counting // errInterrupted, release the lock, end stderr with its line counting
// every file the walk reached, write nothing to stdout, close the // every file the walk reached, close the database out of WAL mode, and
// database out of WAL mode, and keep the records it hashed. // keep the records it hashed.
func TestRunScanInterruptedMidHash(t *testing.T) { func TestRunScanInterruptedMidHash(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
stdout := captureStdout(t)
stderr := captureStderr(t) stderr := captureStderr(t)
dir := buildWalkCancelTree(t) dir := buildWalkCancelTree(t)
@@ -444,10 +438,6 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
t.Errorf("stderr = %q, want it to end with %q", got, want) t.Errorf("stderr = %q, want it to end with %q", got, want)
} }
if got := stdout(); got != "" {
t.Errorf("stdout = %q, want nothing (data only)", got)
}
assertNoSidecars(t, path) assertNoSidecars(t, path)
db, err := openReportDatabase(t.Context(), path) db, err := openReportDatabase(t.Context(), path)
@@ -739,77 +729,47 @@ func TestFeedHashJobsClosesJobsWhenCancelled(t *testing.T) {
// TestHashWorkerDropsQueuedRuns checks that a cancelled hash worker // TestHashWorkerDropsQueuedRuns checks that a cancelled hash worker
// keeps reading jobs and drops the runs rather than reading files // keeps reading jobs and drops the runs rather than reading files
// nobody wants the hashes of — while still letting the range run out // nobody wants the hashes of — while still letting the range run out
// so the pool tears down. The hash function records that it was // so the pool tears down. The queued run names a file that does not
// called, so a worker that hashed the queued run anyway is caught // exist, so a worker that hashed it anyway would produce a result.
// every time. //
// hashWorker's other cancellation exit, abandoning the send of a
// result, is reachable from the scan: stop cancels the pool before it
// drains results, so a worker waiting on that send can leave through
// it. The tests that stop a scan mid-hash, among them
// TestScanHashWriteFailureUnwindsPool, reach it in some runs only,
// depending on timing, and no test fails without it, since stop's
// drain frees a waiting worker anyway. This test, for its part, catches
// a removed drop check in some runs only: a worker that hashes the run
// anyway then picks at random between sending the result and leaving.
func TestHashWorkerDropsQueuedRuns(t *testing.T) { func TestHashWorkerDropsQueuedRuns(t *testing.T) {
t.Parallel() t.Parallel()
done := make(chan struct{}) done := make(chan struct{})
jobs := make(chan []fileRec, 1) jobs := make(chan []fileRec, 1)
results := make(chan hashResult) results := make(chan hashResult, 1)
jobs <- []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}} run := []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}}
jobs <- run
close(jobs) close(jobs)
var hashed atomic.Bool
hash := func(path string, size int64) (string, string, string, error) {
hashed.Store(true)
return hashSignature(path, size)
}
go func() { go func() {
defer close(done) defer close(done)
hashWorker(cancelledContext(t), jobs, results, hash) hashWorker(cancelledContext(t), jobs, results, hashSignature)
}() }()
awaitReturn(t, done, "hashWorker") awaitReturn(t, done, "hashWorker")
if hashed.Load() { select {
t.Error("cancelled hash worker hashed the queued run, want it dropped") case r := <-results:
t.Errorf("cancelled hash worker produced %+v, want the run dropped",
r)
default:
} }
} }
// TestHashWorkerAbandonsBlockedSend checks that a hash worker with a
// result to deliver and nobody to deliver it to leaves once the scan
// is cancelled, instead of holding the pool open. The scan tests do
// not catch this: stop drains results, which frees a parked worker
// anyway.
func TestHashWorkerAbandonsBlockedSend(t *testing.T) {
t.Parallel()
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
done := make(chan struct{})
jobs := make(chan []fileRec, 1)
// Unbuffered and unread, with jobs left open: the worker's only way
// out is the cancellation case beside its send.
results := make(chan hashResult)
jobs <- []fileRec{{path: filepath.Join(t.TempDir(), "missing"), size: 1}}
// The scan is cancelled while the worker hashes, so the worker has
// already passed the check that drops queued runs.
hash := func(path string, size int64) (string, string, string, error) {
cancel()
return hashSignature(path, size)
}
go func() {
defer close(done)
hashWorker(ctx, jobs, results, hash)
}()
awaitReturn(t, done, "hashWorker")
}
// TestHashPhaseCancelledReturnsContextError checks the result loop's // TestHashPhaseCancelledReturnsContextError checks the result loop's
// own exit: with the pool cancelled, no result will ever arrive, and // own exit: with the pool cancelled, no result will ever arrive, and
// the loop must leave through the cancellation rather than wait for a // the loop must leave through the cancellation rather than wait for a
+1 -14
View File
@@ -6,7 +6,6 @@ import (
"errors" "errors"
"fmt" "fmt"
"io/fs" "io/fs"
"net/url"
"os" "os"
"path/filepath" "path/filepath"
"slices" "slices"
@@ -108,19 +107,7 @@ const reportParams = "mode=ro" +
// openDB opens the SQLite database at path with the connection // openDB opens the SQLite database at path with the connection
// parameters params. It does not create or verify the schema. // parameters params. It does not create or verify the schema.
func openDB(path, params string) (*sql.DB, error) { func openDB(path, params string) (*sql.DB, error) {
// The path is escaped into a file: URI, so ?, # and % in it stay db, err := sql.Open("sqlite", "file:"+path+"?"+params)
// part of the file name. SQLite reads what follows file:// up to
// the next / as a host name, so an absolute path goes after an
// empty host (file:///abs) and a relative path goes without one
// (file:rel).
uri := url.URL{
Scheme: "file",
OmitHost: !filepath.IsAbs(path),
Path: path,
RawQuery: params,
}
db, err := sql.Open("sqlite", uri.String())
if err != nil { if err != nil {
return nil, fmt.Errorf("open database %s: %w", path, err) return nil, fmt.Errorf("open database %s: %w", path, err)
} }
+33 -202
View File
@@ -8,7 +8,6 @@ import (
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"slices"
"strconv" "strconv"
"strings" "strings"
"testing" "testing"
@@ -85,41 +84,21 @@ func makeReadOnly(t *testing.T, path string) {
// captureStderr redirects os.Stderr to a file for the rest of the test // captureStderr redirects os.Stderr to a file for the rest of the test
// and returns a function reading back everything written to it. scan // and returns a function reading back everything written to it. scan
// writes its warnings and summary, and report and trees their // writes its warnings and summary straight to os.Stderr, not to the
// summaries, straight to os.Stderr, not to the stderr writer run is // stderr writer run is given.
// given.
func captureStderr(t *testing.T) func() string { func captureStderr(t *testing.T) func() string {
t.Helper() t.Helper()
return capture(t, &os.Stderr) f, err := os.Create(filepath.Join(t.TempDir(), "stderr"))
}
// captureStdout does for os.Stdout what captureStderr does for
// os.Stderr. scan is never given run's stdout writer, so anything it
// printed would go straight to os.Stdout. The scan tests pass os.Stdout
// as run's stdout too, so the one capture sees both.
func captureStdout(t *testing.T) func() string {
t.Helper()
return capture(t, &os.Stdout)
}
// capture redirects *std, which is os.Stdout or os.Stderr, to a file
// for the rest of the test and returns a function reading back
// everything written to it.
func capture(t *testing.T, std **os.File) func() string {
t.Helper()
f, err := os.Create(filepath.Join(t.TempDir(), "output"))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
saved := *std saved := os.Stderr
*std = f os.Stderr = f
t.Cleanup(func() { t.Cleanup(func() {
*std = saved os.Stderr = saved
_ = f.Close() _ = f.Close()
}) })
@@ -204,33 +183,30 @@ func TestRunFatalAfterOpenClosesDatabase(t *testing.T) {
// sidecar check is evidence of the close only for scan: report and // sidecar check is evidence of the close only for scan: report and
// trees only read a database that is out of WAL mode, which leaves // trees only read a database that is out of WAL mode, which leaves
// nothing on disk whether they close it or not. // nothing on disk whether they close it or not.
// cases := map[string][]string{
// The subcommands come from the command tree, so a new one is cmdScan: {cmdScan},
// checked too: one wired with a bare RunE instead of runE reports cmdReport: {cmdReport},
// its failure as a usage error, exit 2 with the usage text. cmdTrees: {cmdTrees},
for _, cmd := range newRootCommand(io.Discard, io.Discard).Commands() { }
name := cmd.Name()
for name, args := range cases {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
path := brokenDatabase(t) path := brokenDatabase(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
args := []string{name}
if name == cmdScan { if name == cmdScan {
args = append(args, t.TempDir()) args = append(args, t.TempDir())
} }
stdout := captureStdout(t) var stdout, stderr bytes.Buffer
var stderr bytes.Buffer code := run(args, &stdout, &stderr)
code := run(args, os.Stdout, &stderr)
if code != exitFatal { if code != exitFatal {
t.Errorf("run(%v) = %d, want %d", args, code, exitFatal) t.Errorf("run(%v) = %d, want %d", args, code, exitFatal)
} }
assertNoSidecars(t, path) assertNoSidecars(t, path)
assertFatalOutput(t, stderr.String(), stdout()) assertFatalOutput(t, stderr.String(), stdout.String())
// Proof that the failure happened after the open: only a // Proof that the failure happened after the open: only a
// query against the opened database can report this. // query against the opened database can report this.
@@ -242,24 +218,22 @@ func TestRunFatalAfterOpenClosesDatabase(t *testing.T) {
} }
} }
func TestRunNonexistentPathIsFatalNotUsage(t *testing.T) { func TestRunMissingOperandIsFatalNotUsage(t *testing.T) {
// README §Error handling: a PATH operand that does not exist is a // README §Error handling: a PATH operand that does not exist is a
// fatal error (1), not a usage error (2) — and a runtime failure // fatal error (1), not a usage error (2) — and a runtime failure
// must not dump the usage text. // must not dump the usage text.
t.Setenv(databaseEnv, testDBPath(t)) t.Setenv(databaseEnv, testDBPath(t))
stdout := captureStdout(t) var stdout, stderr bytes.Buffer
var stderr bytes.Buffer
missing := filepath.Join(t.TempDir(), "nope") missing := filepath.Join(t.TempDir(), "nope")
code := run([]string{cmdScan, missing}, os.Stdout, &stderr) code := run([]string{cmdScan, missing}, &stdout, &stderr)
if code != exitFatal { if code != exitFatal {
t.Errorf("run(scan %s) = %d, want %d", missing, code, exitFatal) t.Errorf("run(scan %s) = %d, want %d", missing, code, exitFatal)
} }
assertFatalOutput(t, stderr.String(), stdout()) assertFatalOutput(t, stderr.String(), stdout.String())
} }
// assertFatalOutput checks that a fatal error was reported the way // assertFatalOutput checks that a fatal error was reported the way
@@ -283,34 +257,6 @@ func assertFatalOutput(t *testing.T, stderr, stdout string) {
} }
} }
func TestRunMissingDatabaseIsFatal(t *testing.T) {
// README §Database: report and trees need an existing database; a
// missing one exits 1 with a message telling the user to run scan.
for _, name := range []string{cmdReport, cmdTrees} {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
var stdout, stderr bytes.Buffer
code := run([]string{name}, &stdout, &stderr)
if code != exitFatal {
t.Errorf("run(%s) = %d, want %d", name, code, exitFatal)
}
want := "sfdupes: " + path + ": no database (run \"sfdupes " +
"scan\" first, or set " + databaseEnv + ")\n"
if got := stderr.String(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
}
if got := stdout.String(); got != "" {
t.Errorf("stdout = %q, want nothing (data only)", got)
}
})
}
}
func TestRunUsageErrors(t *testing.T) { func TestRunUsageErrors(t *testing.T) {
// Usage errors keep exiting 2 with cobra's own report on stderr. // Usage errors keep exiting 2 with cobra's own report on stderr.
cases := map[string]struct { cases := map[string]struct {
@@ -488,16 +434,17 @@ func scanFixture(t *testing.T) []string {
func scanOK(t *testing.T, operands ...string) string { func scanOK(t *testing.T, operands ...string) string {
t.Helper() t.Helper()
stdout := captureStdout(t) var stdout bytes.Buffer
stderr := captureStderr(t) stderr := captureStderr(t)
code := run(append([]string{cmdScan}, operands...), os.Stdout, os.Stderr) code := run(append([]string{cmdScan}, operands...), &stdout, os.Stderr)
if code != exitOK { if code != exitOK {
t.Fatalf("run(scan %q) = %d, want %d; stderr: %s", t.Fatalf("run(scan %q) = %d, want %d; stderr: %s",
operands, code, exitOK, stderr()) operands, code, exitOK, stderr())
} }
if got := stdout(); got != "" { if got := stdout.String(); got != "" {
t.Errorf("scan stdout = %q, want nothing (data only)", got) t.Errorf("scan stdout = %q, want nothing (data only)", got)
} }
@@ -505,41 +452,10 @@ func scanOK(t *testing.T, operands ...string) string {
} }
func TestRunScanSucceedsDespiteWarnings(t *testing.T) { func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
// README §Error handling: a scan that skips a file it cannot read
// warns, counts the skip in its summary, and still exits 0, which
// scanOK checks along with the empty stdout.
if os.Geteuid() == 0 {
t.Skip("root ignores file permissions")
}
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
dir := t.TempDir() scanFixture(t)
writeFile(t, dir, "a.bin", pattern(1, 300))
// Same size as a.bin, so the scan reads it, and the read fails.
unreadable := writeFile(t, dir, "unreadable.bin", pattern(2, 300))
err := os.Chmod(unreadable, 0)
if err != nil {
t.Fatal(err)
}
stderr := scanOK(t, dir)
warning := "hash " + unreadable + ": open " + unreadable +
": permission denied\n"
if !strings.Contains(stderr, warning) {
t.Errorf("stderr = %q, want %q", stderr, warning)
}
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
assertNoSidecars(t, path) assertNoSidecars(t, path)
} }
@@ -646,14 +562,12 @@ func TestRunReportSucceeds(t *testing.T) {
dupes := scanFixture(t) dupes := scanFixture(t)
var stdout bytes.Buffer var stdout, stderr bytes.Buffer
stderr := captureStderr(t) code := run([]string{cmdReport}, &stdout, &stderr)
code := run([]string{cmdReport}, &stdout, os.Stderr)
if code != exitOK { if code != exitOK {
t.Fatalf("run(report) = %d, want %d; stderr: %s", t.Fatalf("run(report) = %d, want %d; stderr: %s",
code, exitOK, stderr()) code, exitOK, stderr.String())
} }
want := "first\tdupe\tsize\n" + dupes[0] + "\t" + dupes[1] + "\t300\n" want := "first\tdupe\tsize\n" + dupes[0] + "\t" + dupes[1] + "\t300\n"
@@ -661,12 +575,6 @@ func TestRunReportSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want) t.Errorf("stdout = %q, want %q", got, want)
} }
want = "report: 2 records read, 1 duplicate groups, 1 dupe files, " +
"300 B reclaimable\n"
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
}
assertNoSidecars(t, path) assertNoSidecars(t, path)
} }
@@ -676,14 +584,12 @@ func TestRunTreesSucceeds(t *testing.T) {
dupes := scanFixture(t) dupes := scanFixture(t)
var stdout bytes.Buffer var stdout, stderr bytes.Buffer
stderr := captureStderr(t) code := run([]string{cmdTrees}, &stdout, &stderr)
code := run([]string{cmdTrees}, &stdout, os.Stderr)
if code != exitOK { if code != exitOK {
t.Fatalf("run(trees) = %d, want %d; stderr: %s", t.Fatalf("run(trees) = %d, want %d; stderr: %s",
code, exitOK, stderr()) code, exitOK, stderr.String())
} }
// The two directories holding the duplicate pair are duplicate // The two directories holding the duplicate pair are duplicate
@@ -694,12 +600,6 @@ func TestRunTreesSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want) t.Errorf("stdout = %q, want %q", got, want)
} }
want = "trees: 2 records read, 1 duplicate tree groups, 1 dupe trees, " +
"300 B reclaimable\n"
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
}
assertNoSidecars(t, path) assertNoSidecars(t, path)
} }
@@ -739,73 +639,6 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
} }
} }
// assertRunsUseDatabase runs scan, then report and trees, against the
// database that SFDUPES_DATABASE names, the file name in dir. It fails
// unless the reports find the duplicate pair the scan recorded and dir
// then holds only that file and its lock file: nothing was created
// under a shortened name.
func assertRunsUseDatabase(t *testing.T, dir, name string) {
t.Helper()
dupes := scanFixture(t)
want := "first\tdupe\tsize\n" + dupes[0] + "\t" + dupes[1] + "\t300\n"
if got := runStdout(t, cmdReport); got != want {
t.Errorf("report stdout = %q, want %q", got, want)
}
want = "first\tdupe\tfiles\tsize\n" +
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
if got := runStdout(t, cmdTrees); got != want {
t.Errorf("trees stdout = %q, want %q", got, want)
}
entries, err := os.ReadDir(dir)
if err != nil {
t.Fatal(err)
}
got := make([]string, 0, len(entries))
for _, e := range entries {
got = append(got, e.Name())
}
if wantFiles := []string{name, name + ".lock"}; !slices.Equal(got, wantFiles) {
t.Errorf("%s holds %q, want %q", dir, got, wantFiles)
}
}
func TestRunDatabasePathUsedAsGiven(t *testing.T) {
// README §Database: the path names the database file exactly. In
// SQLite's connection string an unescaped ? or # would end the file
// name and % would start an escape, and a path starting with //
// could be read as a host name. %25 is a valid escape, so unescaped
// this name opens a file named a without any error.
const name = "a?b#c%25d e.sqlite"
t.Run("absolute", func(t *testing.T) {
dir := t.TempDir()
t.Setenv(databaseEnv, filepath.Join(dir, name))
assertRunsUseDatabase(t, dir, name)
})
t.Run("leading double slash", func(t *testing.T) {
dir := t.TempDir()
t.Setenv(databaseEnv, "/"+filepath.Join(dir, name))
assertRunsUseDatabase(t, dir, name)
})
t.Run("relative", func(t *testing.T) {
dir := t.TempDir()
t.Chdir(dir)
t.Setenv(databaseEnv, name)
assertRunsUseDatabase(t, dir, name)
})
}
// holdScanLock takes the lock on the database at path, as a running // holdScanLock takes the lock on the database at path, as a running
// scan does, and holds it until the test ends. It fails the test when // scan does, and holds it until the test ends. It fails the test when
// the lock is already held. // the lock is already held.
@@ -829,11 +662,9 @@ func TestRunSecondScanFails(t *testing.T) {
holdScanLock(t, path) holdScanLock(t, path)
stdout := captureStdout(t) var stdout, stderr bytes.Buffer
var stderr bytes.Buffer code := run([]string{cmdScan, t.TempDir()}, &stdout, &stderr)
code := run([]string{cmdScan, t.TempDir()}, os.Stdout, &stderr)
if code != exitFatal { if code != exitFatal {
t.Errorf("run(scan) = %d, want %d", code, exitFatal) t.Errorf("run(scan) = %d, want %d", code, exitFatal)
} }
@@ -844,7 +675,7 @@ func TestRunSecondScanFails(t *testing.T) {
t.Errorf("stderr = %q, want %q", got, want) t.Errorf("stderr = %q, want %q", got, want)
} }
if got := stdout(); got != "" { if got := stdout.String(); got != "" {
t.Errorf("stdout = %q, want nothing (data only)", got) t.Errorf("stdout = %q, want nothing (data only)", got)
} }
+22 -11
View File
@@ -1,9 +1,14 @@
#!/bin/sh #!/bin/sh
# script/bootstrap: install all dependencies needed to build and develop # script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent; assumes nothing is present (not git, make, or # this repo. Idempotent: every install is guarded by a check so already
# go). Base tooling comes from nix, apt, brew, or apk (detected in that # installed tools are skipped. Base tooling comes from nix, apt, brew,
# order). golangci-lint and prettier are never installed: they run via # or apk (detected in that order); assumes nothing is present (not git,
# docker only (script/lint, script/fmt, script/fmt-check). # make, or go). Neither the linter nor the Markdown formatter is
# installed: golangci-lint (script/lint) and prettier (script/fmt,
# script/fmt-check) run via docker only, pinned by hash, so their only
# prerequisite is a working docker — which is warned about, not
# installed, because everything except linting and formatting works
# without it.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -58,19 +63,25 @@ missing() {
main() { main() {
cd "$ROOT" cd "$ROOT"
# Deliberately unpinned, so presence is the whole check: go.mod # System tooling, deliberately unpinned: these come from the host
# governs the Go version, and reproducible builds run in the # package manager and whatever version it ships is what the host
# digest-pinned Docker images. # gets, so a presence check is the right check. The repo pins no
# system toolchain versions — the Go language version is governed by
# go.mod, and builds that must be reproducible run in the Docker
# image, whose base images are pinned by digest.
if missing git; then pkg_install git git git git; fi if missing git; then pkg_install git git git git; fi
if missing make; then pkg_install gnumake make make make; fi if missing make; then pkg_install gnumake make make make; fi
if missing go; then pkg_install go golang go go; fi if missing go; then pkg_install go golang go go; fi
# Warn, do not fail: only the targets named below, and the # Linting and Markdown formatting run via docker only, so docker is
# pre-commit hook, need docker. # their prerequisite rather than something bootstrap installs. Warn,
# do not fail: everything except `make lint`, `make fmt` and
# `make fmt-check` — and, through them, `make check`, `make docker`
# and the pre-commit hook — works without it.
if missing docker; then if missing docker; then
echo "bootstrap: WARNING: docker not found; make lint, make fmt," >&2 echo "bootstrap: WARNING: docker not found; make lint, make fmt," >&2
echo "bootstrap: make fmt-check, make check, make docker and" >&2 echo "bootstrap: make fmt-check, make check and make docker" >&2
echo "bootstrap: make test-race require it." >&2 echo "bootstrap: require it." >&2
fi fi
go mod download go mod download
+22 -7
View File
@@ -1,19 +1,34 @@
#!/bin/sh #!/bin/sh
# script/cibuild: run the CI build. The Gitea workflow runs this on # script/cibuild: run the CI build. The Gitea workflow runs this on
# push. The Dockerfile runs every gate make check runs, as build steps, # push.
# so a successful build means the repo is green.
# #
# Without a fresh CHECK_EPOCH, a rebuild of an unchanged checkout serves # The Dockerfile runs the gates individually as build steps, not the
# the gate layers from cache and passes having run none of them. The # make check aggregate: the lint stage runs the gofmt check,
# process id goes in with the epoch so two runs started in the same # script/verify-lint-image-pin, golangci-lint config verify and
# second still differ. # golangci-lint run; the markdown stage runs the prettier check; the
# build stage, dropped to an unprivileged user, runs make test. None of
# make lint, make fmt-check or make check appears, because each runs
# docker, and docker cannot run inside a docker build. Nothing is
# skipped by that — the linter, gofmt and prettier are invoked directly
# in their stages, and the build stage's COPY --from lines make those
# stages prerequisites, so BuildKit must finish them first. Between the
# three stages everything make check would run has run, which is why a
# successful build here implies the repo is green.
#
# That implication holds only because of CHECK_EPOCH. A COPY layer is
# invalidated only by changed content, and a rebuild of an unchanged
# checkout sends the same content, so without a fresh value here Docker
# serves the gate layers from cache and the build reports a green it
# never earned. Passing the current epoch invalidates the gate
# layers on every run while leaving the pinned base images and
# go mod download cached; see the Dockerfile for the placement.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build --build-arg CHECK_EPOCH="$(date +%s)-$$" . docker build --build-arg CHECK_EPOCH="$(date +%s)" .
} }
main "$@" main "$@"
+10 -5
View File
@@ -1,9 +1,14 @@
#!/bin/sh #!/bin/sh
# script/docker: build the Docker image tagged with the project name. # script/docker: build the Docker image tagged with the project name.
# The tag comes from script/projectname. CHECK_EPOCH is passed for the # The tag comes from script/projectname.
# same reason script/cibuild passes it: without a fresh value an #
# unchanged tree is served from cache and this exits 0 having run no # CHECK_EPOCH is passed for the same reason script/cibuild passes it:
# gate. # without it Docker serves the Dockerfile's gate layers from cache on an
# unchanged tree and this exits 0 having run none of the lint stage's
# gates, the markdown stage's prettier gate or the builder stage's test
# gate. This is the set of gates a developer or reviewer runs by hand,
# so a cached pass here is the most misleading result the repo can
# produce. Dependency layers sit above the ARG and stay cached.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -12,7 +17,7 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
docker build \ docker build \
--build-arg CHECK_EPOCH="$(date +%s)-$$" \ --build-arg CHECK_EPOCH="$(date +%s)" \
-t "$("$SCRIPT_DIR/projectname")" \ -t "$("$SCRIPT_DIR/projectname")" \
. .
} }
+1 -6
View File
@@ -12,12 +12,7 @@ main() {
cd "$ROOT" cd "$ROOT"
status=0 status=0
# Under set -e a bare assignment would end the script when gofmt files="$(gofmt -s -l .)"
# fails (a Go file it cannot parse), and prettier would never run.
if ! files="$(gofmt -s -l .)"; then
echo "gofmt: failed; see its errors above" >&2
status=1
fi
if [ -n "$files" ]; then if [ -n "$files" ]; then
echo "gofmt: files not formatted:" >&2 echo "gofmt: files not formatted:" >&2
echo "$files" >&2 echo "$files" >&2
+13 -12
View File
@@ -1,17 +1,19 @@
#!/bin/sh #!/bin/sh
# script/lint: run the linter. golangci-lint is never installed on a # script/lint: run the linter. golangci-lint is never installed on a
# host: this builds Dockerfile.lint, which copies the repo into the # host: it runs via docker only, one way, everywhere — this builds
# digest-pinned golangci-lint image and lints as a build step, so a # Dockerfile.lint, which COPYs the repo into the digest-pinned
# successful build is a clean lint. A cold cache needs the network to # golangci-lint image and lints as a build step, so a successful build
# pull the image and for `go mod download`; once warm this runs offline # is a clean lint. The only prerequisite is a working docker. The gate
# until go.mod or go.sum changes. # steps make no network calls of their own, but Dockerfile.lint runs
# `go mod download` above them, so a cold cache does reach the network
# (as does pulling the pinned image); that layer stays cached, and once
# it is warm this runs offline until go.mod or go.sum changes.
# #
# Without a fresh CHECK_EPOCH docker serves the gate layers from cache # CHECK_EPOCH is what makes the result mean anything. Without it docker
# on an unchanged tree and this exits 0 having run no linter. The PID is # serves the gate layers from cache on an unchanged tree and this exits
# in the value because two lint runs land inside the same second easily. # 0 in well under a second having run no linter. The PID is in the value
# # as well as the epoch because two lint runs land inside the same second
# The image is never used, so --output=type=cacheonly writes none; # easily, and `date +%s` alone would cache the second one.
# without it every run leaves an untagged image behind.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
@@ -20,7 +22,6 @@ main() {
cd "$ROOT" cd "$ROOT"
docker build \ docker build \
--build-arg CHECK_EPOCH="$(date +%s)-$$" \ --build-arg CHECK_EPOCH="$(date +%s)-$$" \
--output=type=cacheonly \
-f Dockerfile.lint \ -f Dockerfile.lint \
. .
} }
-40
View File
@@ -1,40 +0,0 @@
#!/bin/sh
# script/test-race: run the test suite under the race detector. Not part
# of script/check.
#
# The race detector needs cgo and a C compiler, which the host build
# never uses, so the tests run in a golang image that has gcc. The
# checkout is mounted read-only, so the docker daemon must be local. The
# container starts with empty caches every time: each run downloads the
# dependencies and compiles them with the detector, which needs the
# network and takes minutes.
#
# The tests run as the calling user, never as root: several of them make
# a file unreadable and expect reading it to fail, and root reads it
# anyway. When the caller is root they run as nobody, and then the
# checkout must be readable by other users. Neither user has a home
# directory in the image, so HOME is /tmp, where Go puts its build cache.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# golang:1.25-trixie, 2026-10-04. Debian rather than the Alpine image the
# Dockerfile builds with, because this one includes gcc.
IMAGE="golang@sha256:2c4c60ef415fbfa5e90300722293bef36c5e63fae17570ce18f580af933dbd73"
main() {
user="$(id -u):$(id -g)"
if [ "$(id -u)" -eq 0 ]; then
user=65534:65534
fi
docker run --rm \
--user "$user" \
--env HOME=/tmp \
--env CGO_ENABLED=1 \
--volume "$ROOT:/src:ro" \
--workdir /src \
"$IMAGE" \
go test -race -timeout 60s ./...
}
main "$@"
+15 -8
View File
@@ -2,16 +2,23 @@
# script/verify-lint-image-pin: fail unless the golangci-lint image # script/verify-lint-image-pin: fail unless the golangci-lint image
# referenced by Dockerfile.lint and the one referenced by the main # referenced by Dockerfile.lint and the one referenced by the main
# Dockerfile's lint stage are the same image at the same digest. Our own # Dockerfile's lint stage are the same image at the same digest. Our own
# extension to scripts-to-rule-them-all, not one of its entrypoints; run # extension to scripts-to-rule-them-all, not one of its entrypoints.
# as a gate in both files. Nothing else keeps the two pins in sync, and
# a bump applied to one alone would lint the same tree against different
# rulesets, both green.
# #
# Do not hardcode the expected digest here: that is a third copy to keep # The linter version is pinned in two independent files. That is the
# in sync. # shape #42 turned into a build failure rather than tolerate: nothing
# else keeps the two in sync, and a bump applied to one file alone would
# leave `make lint` and the fail-fast lint stage of `make docker`
# linting the same tree against different rulesets, both green. This is
# the single guard that stops it, run as a gate in both files.
# #
# A reference that cannot be read is a hard failure, not a skip: two # It deliberately restates neither pin. A hardcoded expected digest here
# empty strings compare equal. # would be a third copy — one more thing to bump, and the same drift one
# file further out. It compares the two files to each other and knows
# nothing about which version is correct.
#
# A reference that cannot be read is a hard failure, not a skip: a
# comparison of two empty strings succeeds, which would turn this guard
# into exactly the unearned green it exists to prevent.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"