Author SHA1 Message Date
clawbot 282146a409 Reformat all Markdown with prettier
check / check (push) Successful in 2m2s
Mechanical result of `make fmt` now that prettier runs over Markdown:
README.md and TODO.md rewrapped to the house settings (4-space,
proseWrap always). REPO_POLICIES.md was already compliant. No prose was
changed by hand in this commit.

Model: opus-4-8
2026-09-21 07:32:12 +00:00
clawbot 2dd4b8c401 Restore Markdown formatting in fmt/fmt-check and pin prettier (closes #19)
script/fmt and script/fmt-check now run gofmt for Go and prettier for
Markdown; fmt-check reports each independently. prettier is pinned at
3.8.1 by package.json/yarn.lock (integrity hash); .prettierrc and
.prettierignore are the house settings copied from the prompts repo.
script/bootstrap installs node and yarn from the host package manager
and runs `yarn install --frozen-lockfile`.

The Markdown check runs in CI via the Dockerfile build stage, where
bootstrap provides prettier; it is removed from the lint stage because
the golangci-lint image has no node. JS manifests are copied before
bootstrap so the yarn layer caches, and node_modules is dockerignored.

node is an unpinned host runtime like git/make/go: nvm's glibc node does
not run on the musl/Alpine build image, so the canonical nvm route is
not usable here; prettier is the hash-pinned formatter. The wholesale
Markdown reformat follows in the next commit.

Model: opus-4-8
2026-09-21 07:31:53 +00:00
sneak 337b319542 Normalize the lint-image pin comments and FROM form (closes #25)
check / check (push) Successful in 1m9s
The `(Debian-based)` parenthetical broke the required
`# image:vX.Y.Z, YYYY-MM-DD` form and asserted a base change that never
happened (v2.12.1 was Debian too); the tag before the digest left three
FROM lines in one file using two conventions. Digest unchanged, in both
Dockerfile and Dockerfile.lint. The golang and alpine pin comments
already matched the required form.

script/verify-lint-image-pin parses these two FROM lines to keep them
identical and still matches the tagless form; its advice line drops the
now-meaningless "tag and digest". With no tag in either reference a
tag-only disagreement cannot arise; a tag reintroduced on one side is
caught as a plain mismatch.
2026-08-10 14:06:47 +00:00
clawbot d43c1d31ac Merge branch 'next': run all linting in Docker (closes #46)
check / check (push) Successful in 1m19s
2026-08-10 15:55:42 +02:00
sneak d4eaf5fed2 Run all linting in Docker via Dockerfile.lint (closes #46)
check / check (push) Successful in 1m38s
Per the owner ruling the linter runs in a container invoked through the
script/ entrypoint, never installed on a host. Dockerfile.lint COPYs
the repo into the digest-pinned golangci/golangci-lint:v2.12.2 image
and runs `golangci-lint config verify` and `golangci-lint run` as build
steps, so a successful build IS a clean lint. script/lint is reduced to
building it, and works with a remote docker daemon, where bind mounts
are impossible.

script/bootstrap loses the `go install`, the pin constants, the version
parser and verify_golangci_lint: with nothing linting on the host, the
$GOPATH/bin versus PATH shadowing they diagnosed has no subject. It
keeps the git/make/go presence checks and `go mod download`, and warns
rather than fails when docker is absent.

Traps for anyone changing this.

A lint build on an unchanged tree exits 0 in under a second having run
no linter -- #32 and #39 again. Caching is waived by ruling:
Dockerfile.lint carries ARG CHECK_EPOCH referenced inside every gate
RUN, because BuildKit hashes the expanded command and a declared but
unreferenced ARG invalidates nothing. script/lint passes
"$(date +%s)-$$"; the PID is there because two runs land in the same
second easily and a bare epoch would cache the second.

Nothing inside an image build may shell out to docker. The main
Dockerfile's lint stage therefore invokes golangci-lint directly rather
than `make lint`, and its build stage runs `make test` and
`make fmt-check` rather than the `make check` aggregate, which reaches
script/lint. Both stay `make` invocations rather than bare scripts
because the Makefile's `export CGO_ENABLED = 0` only reaches what it
invokes.

COPY --from=lint /usr/bin/golangci-lint becomes
COPY --from=lint /src/go.sum /dev/null. The copied binary was the only
edge forcing BuildKit to finish linting before the build stage starts;
dropping it without replacing the edge would have ended fail-fast
linting silently under a still-green build. That no-op copy is the
ordering edge canonical REPO_POLICIES.md prescribes. Nothing in the
build stage runs the linter now, so ENV PATH=/home/builder/go/bin:$PATH
goes with the `go install` that justified it.

script/verify-linter-pin is retired with its README entry: it compared
a linter binary against GOLANGCI_LINT_VERSION in script/bootstrap and
neither subject still exists. The drift moved rather than went away --
the linter is pinned twice, as the FROM line of Dockerfile.lint and the
FROM line of the Dockerfile lint stage, which is what #42 made a build
failure. script/verify-lint-image-pin compares those two references to
each other and restates neither pin; a hardcoded digest would be a
third copy and the same drift one file further out. It runs as a gate
in both files, and an unreadable reference is a hard failure rather
than a vacuous pass.

`golangci-lint config verify` is included per the ruling, and its
unpinned live HTTPS schema fetch was measured rather than assumed:
under --network none the pinned binary passes a valid config and
rejects an invalid one with the jsonschema error, so it validates
against a schema it embeds. That holds for the gate steps, none of
which makes a network call, but not for the build around them --
Dockerfile.lint runs `go mod download` above the gates, so a cold cache
needs the network and only a warm one lints offline, until go.mod or
go.sum changes.

Verified. `make lint` green with every PATH directory containing a
golangci-lint removed and `command -v golangci-lint` empty. Two
consecutive script/lint runs on an untouched tree both executed the
linter, 27.7s and 28.7s under distinct epochs with the COPY layer
CACHED above them. A planted unused variable failed script/lint with
that finding, and failed `make docker` at the lint stage with the build
stage stopped before its COPY --from=lint; reverted clean. The drift
guard fails on tag-only, digest-only and unreadable-reference cases,
naming both sides. `make check` green; `make docker` green in 5m35s
with all six gates executing and the test gate reporting real coverage
rather than a cached ok. In the builder image with the Go test cache
off, --user 0:0 still fails TestScanHardlinkRunFailsTogether where the
unprivileged user passes, so the non-root quirk is intact.
2026-08-10 13:47:31 +00:00
17 changed files with 1078 additions and 984 deletions
+1
View File
@@ -3,6 +3,7 @@
.DS_Store .DS_Store
sfdupes sfdupes
files.dat files.dat
node_modules
*.log *.log
*.out *.out
*.test *.test
+2
View File
@@ -0,0 +1,2 @@
node_modules/
yarn.lock
+4
View File
@@ -0,0 +1,4 @@
{
"tabWidth": 4,
"proseWrap": "always"
}
+60 -55
View File
@@ -1,6 +1,6 @@
# Lint stage — fast feedback on formatting and lint issues # Lint stage — fast feedback on formatting and lint issues
# golangci/golangci-lint:v2.12.2 (Debian-based), 2026-08-07 # golangci/golangci-lint:v2.12.2, 2026-08-07
FROM golangci/golangci-lint:v2.12.2@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240 AS lint
WORKDIR /src WORKDIR /src
COPY go.mod go.sum ./ COPY go.mod go.sum ./
RUN go mod download RUN go mod download
@@ -22,73 +22,70 @@ COPY . .
# (the pinned base image, go mod download) keeps its cache; only the # (the pinned base image, go mod download) keeps its cache; only the
# gates go cold. # gates go cold.
ARG CHECK_EPOCH ARG CHECK_EPOCH
RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
RUN echo "gate lint, epoch ${CHECK_EPOCH}" && make lint # The linter is invoked directly here, not through `make lint`. That
# target now runs `docker build -f Dockerfile.lint`, and a docker build
# cannot run a docker build: routing the gate through make would mean
# nesting docker inside this image. Same reason `make check` is gone
# from the build stage below.
#
# `make fmt-check` is not run in this stage: it now also runs prettier
# over Markdown, and this golangci-lint image has no node. The gate runs
# in the build stage below, where script/bootstrap installs node and
# prettier.
# The FROM above and the one in Dockerfile.lint pin the same linter
# twice, and nothing else keeps them in sync; this fails the build when
# they disagree. See the script for why it restates neither pin.
RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
script/verify-lint-image-pin
# Same config-schema check Dockerfile.lint runs, kept here so this build
# gates on exactly what script/lint gates on. It validates against a
# schema the pinned binary embeds, so it needs no network.
RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
golangci-lint config verify --config .golangci.yml
RUN echo "gate lint, epoch ${CHECK_EPOCH}" && \
golangci-lint run --config .golangci.yml ./...
# Build stage # Build stage
# golang:1.25-alpine, 2026-07-23 # golang:1.25-alpine, 2026-07-23
FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder FROM golang@sha256:56961d79ea8129efddcc0b8643fd8a5416b4e6228cfd477e3fd61deb2672c587 AS builder
# We never build or run as root. Create an unprivileged user and point # We never build or run as root. Create an unprivileged user and point
# HOME and the Go caches at its home so go build/test and golangci-lint # HOME and the Go caches at its home so go build and go test can write
# can write their caches when we drop to it below. $GOPATH/bin is on # their caches when we drop to it below. $GOPATH/bin is deliberately not
# PATH because that is where script/bootstrap's `go install` lands: a # on PATH: script/bootstrap no longer `go install`s anything (the linter
# tool bootstrap installs must be runnable afterwards, and bootstrap # runs from a pinned image, never from a host install), so nothing lands
# verifies its own installs against what PATH resolves, so leaving that # there and adding it would only widen what this image resolves.
# directory unsearched would make any install it performs both unusable
# and self-reported as shadowed. Nothing in this image is shadowed by
# it: the directory does not exist until bootstrap runs.
RUN adduser -D -u 1000 builder RUN adduser -D -u 1000 builder
ENV HOME=/home/builder ENV HOME=/home/builder
ENV GOPATH=/home/builder/go ENV GOPATH=/home/builder/go
ENV GOCACHE=/home/builder/.cache/go-build ENV GOCACHE=/home/builder/.cache/go-build
ENV PATH=/home/builder/go/bin:$PATH
WORKDIR /src WORKDIR /src
# Reuse the linter binary from the lint stage. This copy is load-bearing # No-op file copy whose only purpose is the build-graph edge: it is what
# twice over and must not be deleted as redundant now that bootstrap # makes this stage depend on the lint stage, and so what forces BuildKit
# below can install a linter of its own: # to finish fmt-check, the pin guard and lint before compilation and
# # tests start. Remove it and the fail-fast design dies silently — the
# - It is the only thing making this stage depend on the lint stage, # build stops gating on lint and still exits 0. It replaces a copy of
# so it is what forces BuildKit to finish fmt-check and lint before # the linter binary itself, which is no longer wanted here: nothing in
# compilation and tests start. Remove it and the fail-fast design # this stage runs the linter, because `make lint` is now a docker build
# dies silently: the build stops gating on lint and still exits 0. # and a docker build cannot run inside one.
# - Together with the check below it is what keeps the two stages on COPY --from=lint /src/go.sum /dev/null
# one toolchain: `make check` here runs the very binary the lint
# stage ran, not a second one that happens to agree. Bootstrap
# installing its own linter here instead would restore exactly the
# two-independent-toolchains problem the copy prevents (and cost a
# from-source build of the linter).
COPY --from=lint /usr/bin/golangci-lint /usr/local/bin/golangci-lint
# Fail the build, naming both versions, unless the binary that just
# arrived from the lint stage is the version script/bootstrap pins.
#
# Nothing else enforces that. The linter version is pinned in two
# independent places — the lint stage's image digest above and
# GOLANGCI_LINT_VERSION in script/bootstrap — and bumping one alone is
# an easy mistake. Without this check that mistake is invisible:
# bootstrap below would see a version that is not its pin, quietly
# rebuild the pinned one from source into a directory that is on PATH,
# verify that, and exit 0. The build would go green with the lint stage
# having linted at one version and `make check` at another, which is
# precisely the divergence the copy above exists to prevent.
#
# It runs here, before bootstrap, so that a reinstall cannot satisfy it,
# and it needs no CHECK_EPOCH: its only inputs are the copied binary and
# script/, so Docker invalidates this layer exactly when a cached result
# would stop being true.
COPY script/ script/
RUN script/verify-linter-pin /usr/local/bin/golangci-lint
# Install development prerequisites the same way a developer does, # Install development prerequisites the same way a developer does,
# rather than duplicating the installs inline. Only script/ (copied # rather than duplicating the installs inline. Only script/ and the
# above) and the dependency manifests are copied first, nothing else, so # dependency manifests are copied first, nothing else, so this layer
# this layer stays cached until the scripts or the dependencies change — # stays cached until the scripts or the dependencies change — bootstrap
# bootstrap ends in `go mod download`, which is why there is no separate # runs `go mod download` and `yarn install`, which is why there is no
# invocation of it here. # separate invocation of either here. The JS manifests (package.json,
COPY go.mod go.sum ./ # yarn.lock) are copied too so the yarn install layer caches alongside
# the Go one.
COPY script/ script/
COPY go.mod go.sum package.json yarn.lock ./
RUN script/bootstrap RUN script/bootstrap
COPY . . COPY . .
@@ -102,11 +99,19 @@ USER builder
# permission-denied test paths are exercised legitimately (root would # permission-denied test paths are exercised legitimately (root would
# bypass the chmod(0) the tests rely on). # bypass the chmod(0) the tests rely on).
# #
# The gates are the individual targets, not `make check`: that aggregate
# runs `script/lint`, which is now a docker build, and nothing inside an
# image build may shell out to docker. Lint is not skipped by this — it
# ran in the lint stage above, which this stage's COPY --from makes a
# prerequisite. `make`, not the scripts directly, because the Makefile's
# `export CGO_ENABLED = 0` applies only to what it invokes.
#
# Second per-stage declaration of the gate cache-buster; see the lint # Second per-stage declaration of the gate cache-buster; see the lint
# stage above for why one is not enough. It is placed after USER so the # stage above for why one is not enough. It is placed after USER so the
# drop to the unprivileged user still happens before the checks run. # drop to the unprivileged user still happens before the checks run.
ARG CHECK_EPOCH ARG CHECK_EPOCH
RUN echo "gate check, epoch ${CHECK_EPOCH}" && make check RUN echo "gate test, epoch ${CHECK_EPOCH}" && make test
RUN echo "gate fmt-check, epoch ${CHECK_EPOCH}" && make fmt-check
RUN make build RUN make build
+59
View File
@@ -0,0 +1,59 @@
# Lint-only image: this is how the linter runs, everywhere. The repo is
# COPYed into the pinned golangci-lint image and the linter runs as a
# build step, so a successful build IS a clean lint. golangci-lint is
# never installed on a host — one toolchain, pinned by digest, identical
# on a laptop and in CI — and this works even when the docker daemon is
# remote and bind mounts are impossible.
#
# script/lint builds this file. It is a separate image from the lint
# stage of the main Dockerfile because script/lint must not depend on
# the rest of that build; the two FROM lines are kept identical by
# script/verify-lint-image-pin, run as a gate below.
# golangci/golangci-lint:v2.12.2, 2026-08-07
FROM golangci/golangci-lint@sha256:5cceeef04e53efe1470638d4b4b4f5ceefd574955ab3941b2d9a68a8c9ad5240
WORKDIR /src
# Dependency layers first, so they stay cached across lint runs.
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Cache-buster for the gate layers, and only for them. Caching of the
# lint run is waived by ruling: COPY is invalidated only by changed
# content, so on an unchanged tree the gates below would be served from
# cache and this build would exit 0 in under a second having run no
# linter at all. That exact false green has bitten this repo twice
# already (#32, #39). script/lint passes a fresh value on every
# invocation.
#
# Each gate RUN must reference the value, because BuildKit hashes the
# expanded command and not the ARG declaration: a declared but
# unreferenced ARG invalidates nothing. The ARG sits below the
# dependency layers deliberately — everything above it keeps its cache,
# only the gates go cold.
ARG CHECK_EPOCH
# The linter version is pinned in two places, here and in the main
# Dockerfile's lint stage. Nothing else keeps them in sync, so a
# half-applied bump is a build failure; see the script.
RUN echo "gate lint-image-pin, epoch ${CHECK_EPOCH}" && \
script/verify-lint-image-pin
# Validates .golangci.yml against golangci-lint's JSON schema. The
# concern about this step was that it fetches that schema over a live,
# unpinned HTTPS call; measured on the pinned image, it does not. The
# binary carries the schema for its own version, so under
# `--network none` this both passes on a valid config and still rejects
# an invalid one with the jsonschema error. That holds for the gate
# steps generally — none of them makes a network call — but not for
# this build as a whole: `go mod download` above needs the network on a
# cold cache, and under `--network none` a first build fails there
# before reaching any gate. That layer stays cached, so only a warm
# cache lints offline, until go.mod or go.sum changes.
RUN echo "gate config verify, epoch ${CHECK_EPOCH}" && \
golangci-lint config verify --config .golangci.yml
RUN echo "gate lint, epoch ${CHECK_EPOCH}" && \
golangci-lint run --config .golangci.yml ./...
+385 -407
View File
@@ -2,19 +2,18 @@
## Description ## Description
`sfdupes` is an MIT-licensed Go CLI tool by `sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that
[@sneak](https://sneak.berlin) that quickly identifies *candidate* quickly identifies _candidate_ duplicate files — and, ultimately, entire
duplicate files — and, ultimately, entire duplicate directory trees — duplicate directory trees — across very large filesystems without reading full
across very large filesystems without reading full file contents. Files file contents. Files are considered duplicates when they have identical size,
are considered duplicates when they have identical size, identical identical SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
SHA-256 of their first 1024 bytes, and identical SHA-256 of their last 1024 bytes. This is a strong candidate signal, not proof of identical content
1024 bytes. This is a strong candidate signal, not proof of identical (the middle of the file is never read); the intended use is finding duplicate
content (the middle of the file is never read); the intended use is downloads and duplicated directory trees on multi-terabyte ZFS servers where
finding duplicate downloads and duplicated directory trees on reading every byte is prohibitively expensive. `scan` maintains a persistent
multi-terabyte ZFS servers where reading every byte is prohibitively SQLite database of file signatures that survives between runs, so it can be run
expensive. `scan` maintains a persistent SQLite database of file from cron and the reports can be generated at any time from the most recent
signatures that survives between runs, so it can be run from cron and scan.
the reports can be generated at any time from the most recent scan.
This README is the complete and authoritative specification. This README is the complete and authoritative specification.
@@ -28,91 +27,81 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes trees > dupetrees.tsv ./sfdupes trees > dupetrees.tsv
``` ```
`scan` walks one or more filesystem trees and maintains one database `scan` walks one or more filesystem trees and maintains one database record per
record per regular file (path, size, mtime, head hash, tail hash). The regular file (path, size, mtime, head hash, tail hash). The database persists
database persists between runs; a rescan only hashes files that are new between runs; a rescan only hashes files that are new or changed, and removes
or changed, and removes records for files that no longer exist. records for files that no longer exist. `report` reads the database and prints
`report` reads the database and prints the file-level duplicates the file-level duplicates report. `trees` reads the same database and prints the
report. `trees` reads the same database and prints the duplicate-tree duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
report. A missing/invalid subcommand — or a `scan` invocation with no with no `PATH` operand — prints a usage message and exits 2.
`PATH` operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
daily `sfdupes scan` cron job, with the reporting commands run cron job, with the reporting commands run interactively whenever needed; their
interactively whenever needed; their results are as fresh as the last results are as fresh as the last completed scan.
completed scan.
## Rationale ## Rationale
Duplicate finders that hash entire files do not scale to the target Duplicate finders that hash entire files do not scale to the target environment:
environment: ~10 million files and ~150 TB on possibly slow or busy ~10 million files and ~150 TB on possibly slow or busy disks (a ZFS pool under
disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and resilver). Reading at most 2 KiB per file — and only from files whose size at
only from files whose size at least one other file shares, since a least one other file shares, since a size-unique file cannot be a duplicate —
size-unique file cannot be a duplicate — makes a full-filesystem sweep makes a full-filesystem sweep tractable, and the signatures are kept in a
tractable, and the signatures are kept in a persistent database, so persistent database, so the expensive filesystem pass is incremental: a rescan
the expensive filesystem pass is incremental: a rescan re-hashes only re-hashes only files whose recorded mtime or size changed, and all analysis
files whose recorded mtime or size changed, and all analysis happens happens offline from the database alone. The end goal is not individual files
offline from the database alone. The end goal is but whole duplicated trees — duplicate extractions, duplicate downloads, copied
not individual files but whole duplicated trees — duplicate project trees — which an operator can consider removing as a unit.
extractions, duplicate downloads, copied project trees — which an
operator can consider removing as a unit.
## Design ## Design
Goals, in order: Goals, in order:
1. **Find whole duplicate trees, not just files.** The end goal is to 1. **Find whole duplicate trees, not just files.** The end goal is to identify
identify places where the exact same set of files and directories places where the exact same set of files and directories exists at two or
exists at two or more paths (duplicate extractions, duplicate more paths (duplicate extractions, duplicate downloads, copied project
downloads, copied project trees), so the operator can consider trees), so the operator can consider removing an entire subtree at once.
removing an entire subtree at once. File-level duplicate detection is File-level duplicate detection is the foundation; tree-level detection is
the foundation; tree-level detection is built on top of it. built on top of it.
2. **Never read full file contents.** At most 2 KiB is read per file 2. **Never read full file contents.** At most 2 KiB is read per file (first and
(first and last 1024 bytes), and only files whose size at least last 1024 bytes), and only files whose size at least one other file shares
one other file shares are read at all — a size-unique file cannot are read at all — a size-unique file cannot be a duplicate. Scale target:
be a duplicate. Scale target: tens of millions of files, ~150 TB tens of millions of files, ~150 TB filesystem, possibly slow or busy disks
filesystem, possibly slow or busy disks (ZFS pool under resilver). (ZFS pool under resilver). Holding one small record (path, size, mtime) per
Holding one small record (path, size, mtime) per file in memory file in memory during a scan is acceptable; holding every file's hashes is
during a scan is acceptable; holding every file's hashes is not not (they stay in the database).
(they stay in the database). 3. **Scan incrementally, analyze offline.** The expensive filesystem scan
3. **Scan incrementally, analyze offline.** The expensive filesystem maintains a persistent database; an unchanged file is never read again on a
scan maintains a persistent database; an unchanged file is never rescan. All analysis (`report`, `trees`) works from the database alone and
read again on a rescan. All analysis (`report`, `trees`) works from must never touch the scanned filesystem again. `scan` is designed to be
the database alone and must never touch the scanned filesystem cronned; the reports run at any time against the last completed scan.
again. `scan` is designed to be cronned; the reports run at any 4. **Clean stream separation.** Everything on stdout is machine-readable data.
time against the last completed scan. All progress, warnings, and summaries go to stderr. Never mix them.
4. **Clean stream separation.** Everything on stdout is machine-readable
data. All progress, warnings, and summaries go to stderr. Never mix
them.
### Constraints ### Constraints
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: - Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: `sfdupes`.
`sfdupes`. - Dependencies: standard library, `github.com/spf13/cobra` for the CLI, **one
- Dependencies: standard library, `github.com/spf13/cobra` for the progress-bar library** (`github.com/schollz/progressbar/v3`), and **one SQLite
CLI, **one progress-bar library** driver** (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
(`github.com/schollz/progressbar/v3`), and **one SQLite driver** `github.com/spf13/viper` is permitted if configuration-file support is ever
(`modernc.org/sqlite`, pure Go, so builds keep cgo disabled). needed, but is not currently used. No other third-party deps.
`github.com/spf13/viper` is permitted if configuration-file support
is ever needed, but is not currently used. No other third-party
deps.
- Cross-compilation is not a concern. Builds run with cgo disabled (the - Cross-compilation is not a concern. Builds run with cgo disabled (the
`Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go. `Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
- Analysis modes (`report`, `trees`) must be deterministic: identical - Analysis modes (`report`, `trees`) must be deterministic: identical database
database contents, identical output, regardless of the order in contents, identical output, regardless of the order in which records were
which records were inserted. inserted.
### Subcommands ### Subcommands
Three subcommands, all implemented: Three subcommands, all implemented:
1. `scan` — walk the filesystem and synchronize the database: one 1. `scan` — walk the filesystem and synchronize the database: one signature
signature record per regular file. record per regular file.
2. `report` — file-level duplicate report from the database. 2. `report` — file-level duplicate report from the database.
3. `trees` — tree-level duplicate report: reconstruct the directory 3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
hierarchy from the database records, compute a Merkle-style digest from the database records, compute a Merkle-style digest per directory, and
per directory, and report maximal groups of identical trees. report maximal groups of identical trees.
``` ```
sfdupes scan [--workers N] [-x] PATH... sfdupes scan [--workers N] [-x] PATH...
@@ -124,25 +113,22 @@ sfdupes trees > dupetrees.tsv
All three subcommands operate on a single SQLite database file: All three subcommands operate on a single SQLite database file:
- Location: the value of the `SFDUPES_DATABASE` environment variable - Location: the value of the `SFDUPES_DATABASE` environment variable when set
when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
There is no command-line flag. command-line flag.
- `scan` creates the database (and its parent directory) on first - `scan` creates the database (and its parent directory) on first use. `report`
use. `report` and `trees` require an existing database; a missing and `trees` require an existing database; a missing database file is a fatal
database file is a fatal error (exit 1) telling the user to run error (exit 1) telling the user to run `scan` first.
`scan` first. - The database uses WAL journal mode and a busy timeout, so running a report
- The database uses WAL journal mode and a busy timeout, so running a while a cron `scan` is in progress is safe. The filesystem is authoritative;
report while a cron `scan` is in progress is safe. The filesystem the database is an eventually-consistent reflection of it. Hashed records are
is authoritative; the database is an eventually-consistent committed in batched transactions while the scan is still running (keeping the
reflection of it. Hashed records are committed in batched WAL small and letting concurrent reports observe progress), so a report may
transactions while the scan is still running (keeping the WAL see a scan's changes partially applied, and a scan that dies partway leaves a
small and letting concurrent reports observe progress), so a valid database holding everything hashed so far; the next scan skips those
report may see a scan's changes partially applied, and a scan records and converges toward the filesystem.
that dies partway leaves a valid database holding everything - Schema (`PRAGMA user_version` is the schema version, currently 1; a database
hashed so far; the next scan skips those records and converges with any other version is a fatal error):
toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error):
```sql ```sql
CREATE TABLE files ( CREATE TABLE files (
@@ -154,167 +140,147 @@ All three subcommands operate on a single SQLite database file:
) WITHOUT ROWID; ) WITHOUT ROWID;
``` ```
Paths are stored as BLOBs because Unix paths are raw bytes, not Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
guaranteed UTF-8. `mtime` is used only for change detection; it is UTF-8. `mtime` is used only for change detection; it is not part of the
not part of the duplicate key. `head` and `tail` are empty strings duplicate key. `head` and `tail` are empty strings when the file has never
when the file has never been hashed because its size was unique as been hashed because its size was unique as of the last scan that covered it;
of the last scan that covered it; such records still define the such records still define the file for tree reconstruction but never
file for tree reconstruction but never participate in duplicate participate in duplicate groups.
groups.
### `scan` mode ### `scan` mode
`scan` requires one or more `PATH` operands naming the trees to scan. `scan` requires one or more `PATH` operands naming the trees to scan. There is
There is no default path; invoking `scan` with no operand is a usage no default path; invoking `scan` with no operand is a usage error (usage message
error (usage message on stderr, exit 2). An operand may be a directory on stderr, exit 2). An operand may be a directory or a regular file; an operand
or a regular file; an operand that does not exist is a fatal error that does not exist is a fatal error (exit 1). Because database records persist
(exit 1). Because database records persist between runs and are keyed between runs and are keyed by absolute path, each operand is resolved to an
by absolute path, each operand is resolved to an absolute, lexically absolute, lexically cleaned path (symlinks are not resolved) before walking, so
cleaned path (symlinks are not resolved) before walking, so results do results do not depend on the working directory. All operands belong to a single
not depend on the working directory. All operands belong to a single scan and are enumerated concurrently: every operand seeds the shared walk worker
scan and are enumerated concurrently: every operand seeds the shared pool. Overlapping operands are harmless — an operand that duplicates another or
walk worker pool. Overlapping operands are harmless — an operand that lies under another is dropped before walking, so every file is reached exactly
duplicates another or lies under another is dropped before walking, once and produces one database record.
so every file is reached exactly once and produces one database
record.
`scan` synchronizes the database with the filesystem state under the `scan` synchronizes the database with the filesystem state under the scanned
scanned operands: operands:
- Only a file whose size at least one other file shares is ever - Only a file whose size at least one other file shares is ever read: a
read: a size-unique file cannot be a duplicate, so it is recorded size-unique file cannot be a duplicate, so it is recorded without hashes
without hashes (`head` and `tail` empty). The size census covers (`head` and `tail` empty). The size census covers every file walked this scan
every file walked this scan plus every database record outside plus every database record outside the scanned operands, so a possible
the scanned operands, so a possible duplicate of a separately duplicate of a separately scanned tree is still recognized.
scanned tree is still recognized. - A file not yet in the database is inserted: hashed when its size is shared,
- A file not yet in the database is inserted: hashed when its size without hashes otherwise.
is shared, without hashes otherwise. - A file already in the database is **skipped without reading its contents**
- A file already in the database is **skipped without reading its when its lstat size equals the recorded size and its lstat mtime is not newer
contents** when its lstat size equals the recorded size and its than the recorded mtime. This is what makes a daily rescan cheap. Exception:
lstat mtime is not newer than the recorded mtime. This is what an unchanged file whose record lacks hashes is hashed — and its record updated
makes a daily rescan cheap. Exception: an unchanged file whose — once its size becomes shared, so hashing deferred by size-uniqueness happens
record lacks hashes is hashed — and its record updated — once its as soon as it could matter.
size becomes shared, so hashing deferred by size-uniqueness - A file whose mtime is newer than recorded, or whose size differs, is processed
happens as soon as it could matter. as if new: re-hashed, or recorded without hashes, per the shared-size rule.
- A file whose mtime is newer than recorded, or whose size differs, - A database record whose path lies under one of the scanned operands but was
is processed as if new: re-hashed, or recorded without hashes, not successfully processed this run is deleted. This removes records for
per the shared-size rule. deleted files. It also removes records for paths that failed to stat or hash
- A database record whose path lies under one of the scanned operands this run: the database only ever contains signatures verified by the most
but was not successfully processed this run is deleted. This recent scan that covered them (a subsequent successful scan re-adds such
removes records for deleted files. It also removes records for files).
paths that failed to stat or hash this run: the database only ever - Database records outside the scanned operands are untouched, so disjoint trees
contains signatures verified by the most recent scan that covered can be scanned on different schedules into the same database.
them (a subsequent successful scan re-adds such files).
- Database records outside the scanned operands are untouched, so
disjoint trees can be scanned on different schedules into the same
database.
`scan` runs **three sequential phases over the whole scan**. `scan` runs **three sequential phases over the whole scan**. Parallelism lives
Parallelism lives inside each phase; batched database writes begin inside each phase; batched database writes begin during the hash phase:
during the hash phase:
1. **walk + stat** — enumerate the trees under all `PATH` operands 1. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
concurrently with the walk worker pool: every operand seeds the with the walk worker pool: every operand seeds the shared queue, and each
shared queue, and each worker reads one directory at a time, worker reads one directory at a time, handing discovered subdirectories back
handing discovered subdirectories back to the queue and running to the queue and running `lstat` on each regular file as it is discovered
`lstat` on each regular file as it is discovered (while the (while the directory's metadata is still hot). Sequential directory
directory's metadata is still hot). Sequential directory enumeration is metadata-latency-bound and takes hours at tens of millions of
enumeration is metadata-latency-bound and takes hours at tens of files; per-directory parallelism is what makes the walk tractable on large
millions of files; per-directory parallelism is what makes the or busy pools. The walk builds the size census and resolves unchanged
walk tractable on large or busy pools. The walk builds the size already-hashed files on the fly; every other file is carried to the hash
census and resolves unchanged already-hashed files on the fly; phase as a (path, size, mtime) record.
every other file is carried to the hash phase as a (path, size, 2. **hash** — with the census complete, each carried file's size decides its
mtime) record. fate. Size-unique files are never read: new or changed ones are recorded
2. **hash** — with the census complete, each carried file's size without hashes in the update phase, unchanged unhashed ones simply keep
decides its fate. Size-unique files are never read: new or their records. Every file with a shared size is hashed by the worker pool:
changed ones are recorded without hashes in the update phase, read the first `min(1024, size)` bytes and the last `min(1024, size)` bytes
unchanged unhashed ones simply keep their records. Every file (one read when `size <= 1024`, since the two windows coincide) and compute
with a shared size is hashed by the worker pool: read the first the SHA-256 of each. Zero-length files have constant hashes and are never
`min(1024, size)` bytes and the last `min(1024, size)` bytes opened. Files are hashed in **inode order** (minimizing seeks on spinning
(one read when `size <= 1024`, since the two windows coincide) disks), and paths that are hard links to the same inode are **read once**,
and compute the SHA-256 of each. Zero-length files have constant all sharing the one result — a hard-link backup farm costs one read per
hashes and are never opened. Files are hashed in **inode order** inode, not per path. The phase total counts actual reads, so progress and
(minimizing seeks on spinning disks), and paths that are hard ETA are meaningful. Completed records are committed in batched transactions
links to the same inode are **read once**, all sharing the one **while hashing runs**, so a scan interrupted after hours keeps everything
result — a hard-link backup farm costs one read per inode, not hashed so far and the next scan resumes cheaply, skipping records already
per path. The phase total counts actual reads, so progress and written.
ETA are meaningful. Completed records are committed in batched 3. **update** — commit the final partial batch, the hash-less records for
transactions **while hashing runs**, so a scan interrupted after size-unique new and changed files, and the deletions for records the scan
hours keeps everything hashed so far and the next scan resumes did not verify (vanished files, plus paths that failed to stat or hash).
cheaply, skipping records already written.
3. **update** — commit the final partial batch, the hash-less
records for size-unique new and changed files, and the deletions
for records the scan did not verify (vanished files, plus paths
that failed to stat or hash).
Rules for the walk: Rules for the walk:
- Only regular files. Skip directories, symlinks (do not follow, - Only regular files. Skip directories, symlinks (do not follow, including
including symlink operands), sockets, FIFOs, and device nodes. symlink operands), sockets, FIFOs, and device nodes.
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; - Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
walking them would list every file once per snapshot). them would list every file once per snapshot).
- Filesystem boundaries are crossed by default. With `-x` - Filesystem boundaries are crossed by default. With `-x` (long form
(long form `--one-file-system`, following the GNU `du`/`rsync` `--one-file-system`, following the GNU `du`/`rsync` convention), never descend
convention), never descend into a directory on a different into a directory on a different filesystem than its `PATH` operand; each
filesystem than its `PATH` operand; each operand is bounded by its operand is bounded by its own filesystem.
own filesystem. - On any per-path error (permission denied, file vanished between passes,
- On any per-path error (permission denied, file vanished between unreadable): print a one-line warning to stderr, skip the path, and continue.
passes, unreadable): print a one-line warning to stderr, skip the Per-file errors never abort the run; the final summary reports how many were
path, and continue. Per-file errors never abort the run; the final skipped. As specified above, a skipped path that has a database record from an
summary reports how many were skipped. As specified above, a earlier scan loses that record; an unreadable directory subtree likewise loses
skipped path that has a database record from an earlier scan loses its records (accepted: the database mirrors what the latest scan could
that record; an unreadable directory subtree likewise loses its
records (accepted: the database mirrors what the latest scan could
actually verify). actually verify).
Concurrency: the walk phase (which also stats files) and the hash Concurrency: the walk phase (which also stats files) and the hash phase each use
phase each use a worker pool of `--workers` workers (default a worker pool of `--workers` workers (default `runtime.NumCPU()`); the walk
`runtime.NumCPU()`); the walk parallelizes across directories, parallelizes across directories, hashing across files. Both phases are
hashing across files. Both phases are seek-bound on spinning disks, seek-bound on spinning disks, so raising `--workers` well past the core count
so raising `--workers` well past the core count can help on pools can help on pools with many spindles. The main goroutine owns partitioning,
with many spindles. The main goroutine owns partitioning, database database writes, and progress rendering; progress display must never block the
writes, and progress rendering; progress display must never block workers.
the workers.
`scan` writes nothing to stdout. The summary line on stderr reports the `scan` writes nothing to stdout. The summary line on stderr reports the files
files seen this run broken down by disposition, plus skips: seen this run broken down by disposition, plus skips:
``` ```
scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
``` ```
(`removed` counts deleted database records, which are not part of the (`removed` counts deleted database records, which are not part of the files-seen
files-seen total.) total.)
### `report` mode ### `report` mode
`report` reads every record from the database and takes no positional `report` reads every record from the database and takes no positional arguments.
arguments.
**`report` must never touch the filesystem being analyzed.** It does not **`report` must never touch the filesystem being analyzed.** It does not stat,
stat, open, or otherwise access any path that appears in the records; its open, or otherwise access any path that appears in the records; its only I/O is
only I/O is reading the database and writing stdout/stderr. It must reading the database and writing stdout/stderr. It must produce identical output
produce identical output whether or not the scanned filesystem is still whether or not the scanned filesystem is still mounted.
mounted.
Processing: Processing:
- Records without hashes (size-unique when last scanned) are - Records without hashes (size-unique when last scanned) are excluded: their
excluded: their content is unknown, so they are never reported as content is unknown, so they are never reported as duplicates.
duplicates. - Group the remaining records by the key `(size, head_hash, tail_hash)`.
- Group the remaining records by the key
`(size, head_hash, tail_hash)`.
- Every group with two or more paths is a duplicate group. - Every group with two or more paths is a duplicate group.
- Within each group, sort paths lexicographically (byte order). The - Within each group, sort paths lexicographically (byte order). The first path
first path is the group's `first`; every other path is a `dupe`. is the group's `first`; every other path is a `dupe`.
- Order groups by size descending (biggest reclaimable space first), - Order groups by size descending (biggest reclaimable space first), tie-broken
tie-broken by `first` path ascending. Output must be fully by `first` path ascending. Output must be fully deterministic for a given
deterministic for a given database state. database state.
#### Report output format #### Report output format
TSV on stdout: a header line, then one row per duplicate file (N-1 rows TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
for a group of N): group of N):
``` ```
first dupe size first dupe size
@@ -322,94 +288,86 @@ first dupe size
/srv/a/big.iso /srv/c/big-copy2.iso 4294967296 /srv/a/big.iso /srv/c/big-copy2.iso 4294967296
``` ```
Summary to stderr: records read, number of duplicate groups, number of Summary to stderr: records read, number of duplicate groups, number of dupe
dupe files, and total reclaimable bytes (sum of `size` over all dupe files, and total reclaimable bytes (sum of `size` over all dupe rows) in human
rows) in human units. units.
### `trees` mode ### `trees` mode
`trees` reads the same database as `report` (no positional arguments) `trees` reads the same database as `report` (no positional arguments) and
and reports **entire duplicate directory trees**: directories under reports **entire duplicate directory trees**: directories under which the exact
which the exact same set of relative paths exists with the exact same same set of relative paths exists with the exact same file signatures.
file signatures.
**`trees` must never touch the filesystem being analyzed** — the same **`trees` must never touch the filesystem being analyzed** — the same rule as
rule as `report`. The directory hierarchy is reconstructed purely from `report`. The directory hierarchy is reconstructed purely from the paths in the
the paths in the records, split on `/`. records, split on `/`.
Definitions: Definitions:
- A file's **signature** is `(size, head_hash, tail_hash)` — mtime is - A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
informational and excluded. An unhashed record (empty hashes) has informational and excluded. An unhashed record (empty hashes) has unknown
unknown content: its signature is treated as unique to that file, content: its signature is treated as unique to that file, so a tree containing
so a tree containing an unhashed file never compares equal to any an unhashed file never compares equal to any other tree.
other tree. - A directory's **digest** is a SHA-256 Merkle digest computed bottom-up:
- A directory's **digest** is a SHA-256 Merkle digest computed serialize the directory's child entries — for a file child, its name and
bottom-up: serialize the directory's child entries — for a file signature; for a subdirectory child, its name and that subdirectory's digest —
child, its name and signature; for a subdirectory child, its name sort the serialized entries byte-lexicographically, and hash the
and that subdirectory's digest — sort the serialized entries concatenation. Names are part of the digest: two trees whose files differ only
byte-lexicographically, and hash the concatenation. Names are part in name are _not_ duplicates.
of the digest: two trees whose files differ only in name are *not* - Two directories are **duplicate trees** when their digests are equal. Equal
duplicates. digests imply equal recursive file count and equal total byte size.
- Two directories are **duplicate trees** when their digests are
equal. Equal digests imply equal recursive file count and equal
total byte size.
Known limitation (accepted): hard-linked paths are reported as Known limitation (accepted): hard-linked paths are reported as duplicates by
duplicates by `report` and count toward duplicate trees — their `report` and count toward duplicate trees — their content is genuinely identical
content is genuinely identical — even though they share storage, so — even though they share storage, so removing one reclaims no space. Inode
removing one reclaims no space. Inode identity is used during the identity is used during the scan to avoid redundant reads but is not persisted
scan to avoid redundant reads but is not persisted in the database. in the database.
Known limitation (accepted): only regular files that appear in the Known limitation (accepted): only regular files that appear in the database
database define a tree. Empty directories are invisible, and a file define a tree. Empty directories are invisible, and a file skipped during the
skipped during the scan (e.g. permission error) in one copy but not the scan (e.g. permission error) in one copy but not the other will make
other will make otherwise-identical trees compare as different. otherwise-identical trees compare as different.
Processing: Processing:
- Build the hierarchy, compute every directory's digest, and group - Build the hierarchy, compute every directory's digest, and group directories
directories by digest. Every group with two or more directories is a by digest. Every group with two or more directories is a duplicate-tree group.
duplicate-tree group. - **Report only maximal trees.** A group is suppressed when its members' parents
- **Report only maximal trees.** A group is suppressed when its are pairwise distinct directories that all share a single digest — such a
members' parents are pairwise distinct directories that all share a group is wholly implied by its parents' (or a further ancestor's) group.
single digest — such a group is wholly implied by its parents' (or a Groups containing sibling directories, or members whose parents differ, are
further ancestor's) group. Groups containing sibling directories, or always reported.
members whose parents differ, are always reported. - Within each group, sort paths lexicographically (byte order); the first path
- Within each group, sort paths lexicographically (byte order); the is `first`, every other path is a `dupe`.
first path is `first`, every other path is a `dupe`. - Order groups by total tree size descending, tie-broken by `first` path
- Order groups by total tree size descending, tie-broken by `first` ascending. Output must be fully deterministic for a given input.
path ascending. Output must be fully deterministic for a given
input.
#### Trees output format #### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
for a group of N). `files` is the recursive regular-file count of one group of N). `files` is the recursive regular-file count of one copy of the
copy of the tree; `size` is the recursive total byte size of one copy: tree; `size` is the recursive total byte size of one copy:
``` ```
first dupe files size first dupe files size
/srv/a/project /srv/backup/project 3417 104857600 /srv/a/project /srv/backup/project 3417 104857600
``` ```
Summary to stderr: records read, number of duplicate-tree groups, Summary to stderr: records read, number of duplicate-tree groups, number of dupe
number of dupe trees, and total reclaimable bytes (sum of `size` over trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
all dupe rows) in human units. units.
### Progress ### Progress
Use the progress-bar library for all scan progress; rendering in the Use the progress-bar library for all scan progress; rendering in the style of
style of `pv` is the model. All progress goes to stderr. `pv` is the model. All progress goes to stderr.
Each phase gets its own display, rendered the moment the phase Each phase gets its own display, rendered the moment the phase starts — a scan
starts — a scan must never look hung. Loading the existing-record must never look hung. Loading the existing-record index (`load`) and the walk
index (`load`) and the walk have no known totals while running: show have no known totals while running: show a live count, rate, and elapsed time
a live count, rate, and elapsed time (spinner-style, no percentage or (spinner-style, no percentage or ETA). The hash and update phases have exact
ETA). The hash and update phases totals — only files that actually need hashing appear in the hash total, so its
have exact totals — only files that actually need hashing appear in ETA is meaningful. Required elements for the bars with known totals:
the hash total, so its ETA is meaningful. Required elements for the
bars with known totals:
- elapsed time - elapsed time
- estimated time remaining - estimated time remaining
@@ -424,114 +382,136 @@ hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
Additional requirements: Additional requirements:
- When stderr is not a TTY, do not emit ANSI redraws: print a plain - When stderr is not a TTY, do not emit ANSI redraws: print a plain one-line
one-line progress update no more often than every 5 seconds instead. progress update no more often than every 5 seconds instead.
- Progress updates are driven from the main goroutine and must be - Progress updates are driven from the main goroutine and must be non-blocking
non-blocking with respect to the worker pool. with respect to the worker pool.
- `report` and `trees` modes need no progress display, only their - `report` and `trees` modes need no progress display, only their stderr
stderr summaries. summaries.
### Error handling and exit codes ### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings. - `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., a `PATH` operand does not exist, the - `1`: fatal error (e.g., a `PATH` operand does not exist, the database cannot
database cannot be created/opened/read/written, a missing database be created/opened/read/written, a missing database for `report`/`trees`,
for `report`/`trees`, stdout write failure). stdout write failure).
- `2`: usage error (including `scan` with no `PATH` operand and - `2`: usage error (including `scan` with no `PATH` operand and `report`/`trees`
`report`/`trees` with any positional argument). with any positional argument).
## Entrypoints ## Entrypoints
This repository adheres to the This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: the normalized executables in `script/` are the entrypoints standard: the normalized executables in `script/` are the entrypoints for the
for the development workflow, and the `Makefile` targets are thin development workflow, and the `Makefile` targets are thin shims that call them.
shims that call them. Every script is POSIX `sh`, resolves the Every script is POSIX `sh`, resolves the repository root itself so it can be run
repository root itself so it can be run from any working directory, from any working directory, and may be invoked directly. The provided
and may be invoked directly. The provided entrypoints are: entrypoints are:
- `script/bootstrap` — install everything needed to build and - `script/bootstrap` — install everything needed to build and develop this
develop this repository, idempotently, assuming nothing is repository, idempotently, assuming nothing is present. `git`, `make`, `go`,
present. `git`, `make`, and `go` come from the first of nix, apt, and `node` come from the first of nix, apt, brew, or apk found on the host,
brew, or apk found on the host, and are presence-checked only. and are presence-checked only; `node` is an unpinned host runtime like the
`golangci-lint` is treated differently: it is checked against the rest, because nvm's prebuilt node is glibc-linked and does not run on this
version pinned in the script (the version the `Dockerfile` lint repo's musl/Alpine build image. The Markdown formatter itself — `prettier` —
stage runs) and reinstalled with `go install` whenever the is pinned by `yarn.lock`'s integrity hash and installed with
installed version differs — older or newer, not merely absent — `yarn install --frozen-lockfile`. `golangci-lint` is deliberately **not**
because a host on any other version lints against different rules installed: it runs from a digest-pinned image via `script/lint` and never from
than CI. After installing, the script verifies the pin against the a host install, so there is no host copy to drift from the pin. A missing
`golangci-lint` that `PATH` actually resolves; if a different copy `docker` is warned about rather than installed or treated as fatal —
shadows the install, bootstrap fails, naming both the install everything except linting works without it. Ends with `go mod download` and
directory and the shadowing binary, rather than reporting a the `yarn` install.
success the gate would not honour. Ends with `go mod download`.
- `script/setup` — make a fresh clone ready for development: runs - `script/setup` — make a fresh clone ready for development: runs
`script/bootstrap`, then `script/install-precommit`. `script/bootstrap`, then `script/install-precommit`.
- `script/projectname` — print this project's name (`sfdupes`). - `script/projectname` — print this project's name (`sfdupes`). Scripts that
Scripts that need the name call it, so they stay identical across need the name call it, so they stay identical across repositories.
repositories. - `script/test` — run the test suite with a 30-second timeout and coverage
- `script/test` — run the test suite with a 30-second timeout and enabled, rerunning verbosely on failure so the logs show which test failed.
coverage enabled, rerunning verbosely on failure so the logs show - `script/lint` — run the linter. It builds `Dockerfile.lint`, which copies the
which test failed. repository into the digest-pinned `golangci/golangci-lint` image and runs
- `script/lint` — run `golangci-lint` over the module with the `golangci-lint config verify` and `golangci-lint run` as build steps, so a
repository's `.golangci.yml`. successful build is a clean lint. The linter is never run on the host, which
- `script/fmt` — format the Go sources in place (`gofmt -s -w`). makes a working `docker` the one prerequisite for linting — and therefore for
Markdown is not formatted. `make check` and the pre-commit hook. Offline machines: the gate steps
- `script/fmt-check` — the read-only counterpart of `script/fmt`: themselves make no network calls. `golangci-lint run` does not, and neither
prints any unformatted file and exits non-zero instead of writing. does `golangci-lint config verify` — it validates against a schema the pinned
- `script/check` — run `script/test`, `script/lint`, and binary embeds, measured under `--network none` to both pass a valid config and
`script/fmt-check`, in that order. Modifies nothing. reject an invalid one. The build around them does. `Dockerfile.lint` runs
- `script/docker` — build the Docker image, tagged with the name `go mod download` before the gates and this module has external dependencies,
from `script/projectname`. The `Dockerfile` runs the gates as so a first lint on a machine with a cold BuildKit cache reaches the network
build steps, so this is also the check a developer or reviewer there (as well as pulling the pinned image); under `--network none` it fails
runs by hand. at that step, before any gate. That layer sits above the gates and stays
- `script/cibuild` — build the Docker image untagged. This is what cached, so once it is warm `script/lint` — and with it `make check` — runs
the Gitea workflow runs on push; because the gates run as build entirely offline, until `go.mod` or `go.sum` changes and the download layer
steps, a successful build implies the repository is green. goes cold again. Because the daemon only ever sees a build context, this works
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` when the docker daemon is remote and bind mounts are impossible.
must be a no-op (a resulting change to `go.mod` or `go.sum` fails - `script/fmt` — format in place: `gofmt -s -w` for Go sources and `prettier`
the commit), then `script/check`. for Markdown (`--tab-width 4 --prose-wrap always`, the house settings, also
- `script/install-precommit` — install the git pre-commit hook that carried in `.prettierrc`). prettier is the pinned devDependency in
runs `script/precommit`. The hook is written to the common git `package.json`/`yarn.lock`, installed by `script/bootstrap`.
directory, so the main checkout and every worktree share it. - `script/fmt-check` — the read-only counterpart of `script/fmt`: runs both
- `script/verify-linter-pin` — fail unless a `golangci-lint` binary checks, reports each independently so it is clear which failed, and exits
(given as its argument, default whatever `PATH` resolves) is non-zero if either found unformatted files instead of writing.
exactly the version `script/bootstrap` pins, naming both versions - `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in
if not. The `Dockerfile` build stage runs it on the linter it that order. Modifies nothing. Needs `docker`, because `script/lint` does.
copies out of the lint stage: the version is pinned independently - `script/docker` — build the Docker image, tagged with the name from
in the lint stage's image digest and in `script/bootstrap`, and `script/projectname`. The `Dockerfile` runs the gates as build steps, so this
bumping one alone would otherwise be absorbed silently by is also the check a developer or reviewer runs by hand.
bootstrap rebuilding its pin from source, leaving the two stages - `script/cibuild` — build the Docker image untagged. This is what the Gitea
on different linters under a green build. The pin is read from workflow runs on push; because the gates run as build steps, a successful
`script/bootstrap`, which stays its single source of truth. build implies the repository is green.
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a
no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then
`script/check`.
- `script/install-precommit` — install the git pre-commit hook that runs
`script/precommit`. The hook is written to the common git directory, so the
main checkout and every worktree share it.
- `script/verify-lint-image-pin` — fail unless the `golangci/golangci-lint`
reference in `Dockerfile.lint` and the one in the `Dockerfile` lint stage are
the same image at the same digest, naming both if not. The linter is pinned in
those two files and nothing else keeps them in sync, so a bump applied to one
alone would leave `make lint` and the `Dockerfile`'s fail-fast lint stage
checking the same tree against different rulesets, both green. The guard
restates neither pin — a third copy would be the same drift one file further
out — and runs as a gate in both files, so `make lint`, `make check` and
`make docker` all catch it.
`script/docker` and `script/cibuild` both pass a freshly computed `script/verify-linter-pin` used to live here. It compared a linter binary
`CHECK_EPOCH` build argument, and the `Dockerfile`'s gate steps against a version pin in `script/bootstrap`, and both of its subjects are gone:
reference it. Without that, an unchanged tree lets Docker serve the no linter binary is copied between build stages any more, and bootstrap pins no
gate layers from cache and the build exits 0 having executed no tests version because it installs no linter. The drift it existed to catch has moved
and no lint — a green it never earned. `CHECK_EPOCH` invalidates the from binary-versus-pin to pin-versus-pin, which is what
gate layers on every run while leaving the pinned base images and the `script/verify-lint-image-pin` above checks.
dependency layers cached.
`script/lint`, `script/docker` and `script/cibuild` all pass a freshly computed
`CHECK_EPOCH` build argument, and the gate steps in `Dockerfile.lint` and
`Dockerfile` reference it. Without that, an unchanged tree lets Docker serve the
gate layers from cache and the build exits 0 having executed no tests and no
lint — a green it never earned, and one this repository has produced twice.
`CHECK_EPOCH` invalidates the gate layers on every run while leaving the pinned
base images and the dependency layers cached. `script/lint`'s value carries the
process id as well as the epoch, because two lint runs land inside the same
second easily and a bare epoch would cache the second one.
## Build ## Build
The `script/` entrypoints above are where the implementations live; The `script/` entrypoints above are where the implementations live; the
the `Makefile` targets are shims onto them, except `build`, which `Makefile` targets are shims onto them, except `build`, which carries the
carries the compile recipe: compile recipe:
- `make` / `make build` — build the `sfdupes` binary (cgo - `make` / `make build` — build the `sfdupes` binary (cgo disabled); building is
disabled); building is the default target. the default target.
- `make bootstrap` — install the build and development - `make bootstrap` — install the build and development dependencies.
dependencies. - `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook.
- `make setup` — prepare a fresh clone: `bootstrap` plus the - `make test` — run the test suite (30-second timeout; reruns with `-v` on
pre-commit hook. failure).
- `make test` — run the test suite (30-second timeout; reruns with - `make lint` — run `golangci-lint` with the repo config, in Docker (see
`-v` on failure). `script/lint`); requires `docker`.
- `make lint` — run `golangci-lint` with the repo config. - `make fmt` / `make fmt-check` — format Go and Markdown sources / verify both
- `make fmt` / `make fmt-check` — format Go sources / verify without writing.
formatting without writing. - `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. `docker`, via `lint`.
- `make docker` — build the Docker image, which runs `make check` as - `make docker` — build the Docker image, which runs the gates as build stages.
a build stage.
- `make hooks` — install the pre-commit hook. - `make hooks` — install the pre-commit hook.
- `make clean` — remove the binary. - `make clean` — remove the binary.
@@ -541,8 +521,8 @@ All of the following, run in this directory, must pass:
1. `make check` passes (tests, lint, `gofmt`). 1. `make check` passes (tests, lint, `gofmt`).
2. `make docker` succeeds. 2. `make docker` succeeds.
3. Smoke test — create a throwaway tree in a temp dir (never test 3. Smoke test — create a throwaway tree in a temp dir (never test against real
against real data): data):
```sh ```sh
d=$(mktemp -d) d=$(mktemp -d)
@@ -574,30 +554,29 @@ All of the following, run in this directory, must pass:
./sfdupes report ./sfdupes report
``` ```
(The scan database lives inside `$d` here purely for test hygiene; (The scan database lives inside `$d` here purely for test hygiene; scanning
scanning `$d` therefore also records the SQLite file itself, which `$d` therefore also records the SQLite file itself, which is harmless.)
is harmless.)
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
form one group (two dupe rows, `first` is the lexicographically group (two dupe rows, `first` is the lexicographically smallest path);
smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
`t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group; `t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
`tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and `empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere; groups
`tiny3` appear nowhere; groups ordered by size descending. ordered by size descending.
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
`$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3` (implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
appears nowhere (its file set differs by name). differs by name).
Expected from the second `report` (after the modify/delete rescan): Expected from the second `report` (after the modify/delete rescan):
`one.bin` has left its group (its content changed), so `one.bin` has left its group (its content changed), so
`copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is `copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is gone from the
gone from the database. database.
The test suite automates this scenario (see `scan_test.go`), plus a The test suite automates this scenario (see `scan_test.go`), plus a negative
negative check: `report` and `trees` operate on the database alone check: `report` and `trees` operate on the database alone and never touch
and never touch the scanned filesystem. the scanned filesystem.
## TODO ## TODO
@@ -605,11 +584,10 @@ Tracked in [TODO.md](TODO.md).
## Non-goals ## Non-goals
- No full-content verification, no byte-for-byte compare, no deletion - No full-content verification, no byte-for-byte compare, no deletion or linking
or linking of duplicates. The reports are advisory; acting on them is of duplicates. The reports are advisory; acting on them is the user's job.
the user's job. - No persistence beyond the SQLite database described above; no export/import
- No persistence beyond the SQLite database described above; no formats.
export/import formats.
- No daemon or filesystem watcher; scheduling rescans is cron's job. - No daemon or filesystem watcher; scheduling rescans is cron's job.
## License ## License
+344 -295
View File
@@ -1,337 +1,386 @@
# Workflow # Workflow
- take an issue from the `1.0.0` milestone on the tracker; work not - take an issue from the `1.0.0` milestone on the tracker; work not yet on the
yet on the tracker gets filed as an issue first tracker gets filed as an issue first
- branch (from `main`) - branch (from `main`)
- do the work, with tests, in small focused commits - do the work, with tests, in small focused commits
- record it at the top of Completed Steps (`TODO.md` changes in the - record it at the top of Completed Steps (`TODO.md` changes in the same commit
same commit as the work) as the work)
- push the branch and open a PR whose title ends with - push the branch and open a PR whose title ends with ` (closes #N)`
` (closes #N)` - an independent review gates the merge; every finding is addressed or
- an independent review gates the merge; every finding is addressed explicitly rebutted on the PR
or explicitly rebutted on the PR
- merge to `main` once the review passes - merge to `main` once the review passes
# Status # Status
- pre-1.0 - pre-1.0
- the Gitea tracker is authoritative for the pre-1.0 backlog: the - the Gitea tracker is authoritative for the pre-1.0 backlog: the open issues
open issues under the `1.0.0` milestone are what remains before under the `1.0.0` milestone are what remains before the tag, and this file
the tag, and this file records history and process, not the queue records history and process, not the queue
# Next Step # Next Step
- take the next issue from the `1.0.0` milestone on the tracker: - take the next issue from the `1.0.0` milestone on the tracker:
https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is https://git.eeqj.de/sneak/sfdupes/milestone/17 — the milestone is the source
the source of truth for what is left before 1.0.0. Individual of truth for what is left before 1.0.0. Individual issues are deliberately not
issues are deliberately not restated here; a copy in this file restated here; a copy in this file drifts out of date the moment the tracker
drifts out of date the moment the tracker moves moves
# Completed Steps # Completed Steps
- install the Docker build stage's prerequisites by running - restore Markdown formatting in `script/fmt`/`fmt-check` and reformat all
`script/bootstrap` instead of `apk add --no-cache make` inline Markdown to the house prettier settings (2026-09-21, closes
(2026-08-09, branch `dockerfile-bootstrap`, closes #42): canonical https://git.eeqj.de/sneak/sfdupes/issues/19)
`REPO_POLICIES.md:97` requires it, and the inline install left the - fix the lint-image pin comments and `FROM` form in `Dockerfile` and
build stage maintaining its own notion of the toolchain — exactly `Dockerfile.lint` (2026-08-10, branch `next`, closes
the divergence #24 exists to close, one layer down. The stage now https://git.eeqj.de/sneak/sfdupes/issues/25): dropped the false
copies `script/` plus `go.mod`/`go.sum` and runs `script/bootstrap`, `(Debian-based)` parenthetical (v2.12.1 was Debian too) and the redundant tag,
which ends in `go mod download`, so the separate invocation of that so both pins are the policy `# image:vX.Y.Z, YYYY-MM-DD` comment over a bare
is gone. `COPY --from=lint /usr/bin/golangci-lint` stays, and moves `FROM image@sha256:...`. Digest unchanged. `script/verify-lint-image-pin`
above the bootstrap layer. It is the only edge making this stage parses those `FROM` lines and still matches the tagless form; its advice line
depend on the lint stage, so deleting it as redundant would end lost the now meaningless "tag and digest". With no tag in either reference, a
fail-fast linting silently. Letting bootstrap install its own linter tag-only disagreement no longer exists — a one-sided tag is caught as a plain
here would have reintroduced the second toolchain and paid for a mismatch.
from-source build of it. What makes the two stages provably one
toolchain rather than two that happen to agree is a new - run all linting in Docker via `Dockerfile.lint` and `script/lint` (2026-08-10,
`script/verify-linter-pin`, run in the build stage on the binary branch `next`, closes https://git.eeqj.de/sneak/sfdupes/issues/46): per the
that arrives from the lint stage, before bootstrap: it fails the owner ruling, the linter runs inside a container invoked through the `script/`
build naming both versions unless that binary is the version entrypoint and is never installed on a host. New root `Dockerfile.lint` COPYs
`script/bootstrap` pins. Bootstrap's own check could not serve that the repo into the digest-pinned `golangci/golangci-lint:v2.12.2` image and
purpose — it reinstalls its pin from source and then verifies runs `golangci-lint config verify` and `golangci-lint run` as build steps, so
whatever `PATH` resolves, so drift self-heals silently and a lint a successful build IS a clean lint; `script/lint` is reduced to building it.
stage image bumped on its own would lint at the new version while `script/bootstrap` loses the `go install`, the pin constants, the version
`make check` ran at the old one, green. The linter version is pinned parser and `verify_golangci_lint` outright rather than hardening them — with
in two independent places (the lint stage image digest and nothing linting on the host, the `$GOPATH/bin` versus `PATH` problem that
motivated them has no subject — and now warns rather than fails when `docker`
is absent. Two traps handled. A lint build on an unchanged tree returns
success in well under a second having run no linter, which is
https://git.eeqj.de/sneak/sfdupes/issues/32 and
https://git.eeqj.de/sneak/sfdupes/issues/39 again, so `Dockerfile.lint`
carries `ARG CHECK_EPOCH` referenced inside every gate `RUN` (BuildKit hashes
the expanded command, not the declaration) and `script/lint` passes
`"$(date +%s)-$$"` — the PID matters because two lint runs land inside the
same second easily. And nothing inside an image build may shell out to docker,
so the main `Dockerfile`'s lint stage now invokes `golangci-lint` directly
instead of `make lint`, and its build stage runs `make test` and
`make fmt-check` instead of the `make check` aggregate (`make`, not the
scripts bare, because the Makefile's `export CGO_ENABLED = 0` only reaches
what it invokes). `COPY --from=lint` `/usr/bin/golangci-lint` is replaced by
`COPY --from=lint /src/go.sum /dev/null`: the copied binary was the only edge
forcing BuildKit to finish linting before the build stage starts, and dropping
it without replacing the edge would have ended fail-fast linting silently
under a still-green build. That is canonical `REPO_POLICIES.md:107`'s ordering
edge, restored. `ENV PATH=/home/builder/go/bin:$PATH` is gone with the
`go install` that justified it. `script/verify-linter-pin` is retired, deleted
along with its README entry, because both of its subjects ceased to exist in
the same change: it compared a linter binary against `GOLANGCI_LINT_VERSION`
in `script/bootstrap`, and there is now neither a binary crossing between
stages nor a version pin in bootstrap. The drift it guarded has not gone away,
it has moved — the linter is still pinned twice, now as the `FROM` line of
`Dockerfile.lint` and the `FROM` line of the `Dockerfile` lint stage, with
nothing syncing them, which is exactly what
https://git.eeqj.de/sneak/sfdupes/issues/42 made a build failure. Its
replacement is one new `script/verify-lint-image-pin`, run as a gate in both
files, which compares the two references to each other and deliberately
restates neither: a hardcoded expected digest would be a third copy and the
same drift one file further out. `golangci-lint config verify` is included per
the ruling, and the concern about its unpinned live HTTPS schema fetch was
measured rather than assumed — under `--network none` the pinned binary both
passes a valid config and rejects an invalid one with the jsonschema error, so
it validates from an embedded schema and makes no network call of its own. The
README scopes that to the gate steps rather than to linting as a whole:
`Dockerfile.lint` runs `go mod download` above them, so a cold cache still
needs the network and only a warm one lints offline. Verified: `make lint`
green with every `PATH` directory containing a `golangci-lint` removed
(`/home/user/go/bin`, `/home/user/.local/bin`, `/usr/local/bin`;
`command -v golangci-lint` empty); two consecutive `script/lint` runs on an
untouched tree both executed the linter, 27.7s and 28.7s in the lint step
under distinct epochs with the `COPY . .` layer `CACHED` above them, at 42.2s
and 41.8s wall clock — the no-cache rule was not weakened to shorten that.
Negative control: a planted `var unusedIssue46Sentinel = 1` failed
`script/lint` with
`report.go:173:5: var unusedIssue46Sentinel is unused (unused)`, and failed
`make docker` at `[lint 9/9]` with the build stage stopped at `[builder 3/12]`
— `COPY --from=lint`, `script/bootstrap`, the test gate and `make build` all
zero occurrences — then reverted clean. The drift guard fails on a tag-only
disagreement, on a digest-only disagreement, and on an unreadable reference,
naming both sides. `make docker` green in 5m35s with all six gates executing
under one epoch (lint 37.6s, test 25.2s reporting
`ok sneak.berlin/go/sfdupes 1.938s coverage: 88.5%`, not `(cached)`). The
non-root quirk still holds: in the builder image with the Go test cache off,
`--user 0:0` fails `TestScanHardlinkRunFailsTogether` (exit 1) where the
unprivileged user passes (exit 0). Noted for follow-up, not fixed here:
`golangci-lint` warns that the `gomodguard` linter is deprecated since v2.12.0
in favour of `gomodguard_v2`.
- install the Docker build stage's prerequisites by running `script/bootstrap`
instead of `apk add --no-cache make` inline (2026-08-09, branch
`dockerfile-bootstrap`, closes #42): canonical `REPO_POLICIES.md:97` requires
it, and the inline install left the build stage maintaining its own notion of
the toolchain — exactly the divergence #24 exists to close, one layer down.
The stage now copies `script/` plus `go.mod`/`go.sum` and runs
`script/bootstrap`, which ends in `go mod download`, so the separate
invocation of that is gone. `COPY --from=lint /usr/bin/golangci-lint` stays,
and moves above the bootstrap layer. It is the only edge making this stage
depend on the lint stage, so deleting it as redundant would end fail-fast
linting silently. Letting bootstrap install its own linter here would have
reintroduced the second toolchain and paid for a from-source build of it. What
makes the two stages provably one toolchain rather than two that happen to
agree is a new `script/verify-linter-pin`, run in the build stage on the
binary that arrives from the lint stage, before bootstrap: it fails the build
naming both versions unless that binary is the version `script/bootstrap`
pins. Bootstrap's own check could not serve that purpose — it reinstalls its
pin from source and then verifies whatever `PATH` resolves, so drift
self-heals silently and a lint stage image bumped on its own would lint at the
new version while `make check` ran at the old one, green. The linter version
is pinned in two independent places (the lint stage image digest and
`GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a `GOLANGCI_LINT_VERSION`) and nothing else keeps them in sync, so a
half-applied bump is now a build failure. The pin is read out of half-applied bump is now a build failure. The pin is read out of
`script/bootstrap`, which stays the single source of truth; a pin `script/bootstrap`, which stays the single source of truth; a pin that cannot
that cannot be read is a hard failure, not a skip. The check needs be read is a hard failure, not a skip. The check needs no `CHECK_EPOCH`: its
no `CHECK_EPOCH`: its only inputs are the copied binary and only inputs are the copied binary and `script/`, so Docker invalidates the
`script/`, so Docker invalidates the layer exactly when a cached layer exactly when a cached result would stop being true, and it is documented
result would stop being true, and it is documented with the other with the other entrypoints in the README. `$GOPATH/bin` joins `PATH` because
entrypoints in the README. `$GOPATH/bin` joins `PATH` because that is where bootstrap's `go install` lands and bootstrap verifies its
that is where bootstrap's `go install` lands and bootstrap verifies installs against what `PATH` resolves — nothing in the image is shadowed by
its installs against what `PATH` resolves — nothing in the image is it, the directory does not exist until bootstrap runs. Everything added sits
shadowed by it, the directory does not exist until bootstrap runs. above `ARG CHECK_EPOCH`, and the `chown` and `USER builder` still precede
Everything added sits above `ARG CHECK_EPOCH`, and the `chown` and `make check`. Verified: the guard fails the build with both versions named
`USER builder` still precede `make check`. Verified: the guard fails when the lint stage's linter is faked to a different version, and an
the build with both versions named when the lint stage's linter is unmodified build still passes it; bootstrap runs clean under Alpine's `sh` and
faked to a different version, and an unmodified build still passes its `apk` branch, installing `git` and `make` and finding the copied linter
it; bootstrap runs clean under Alpine's `sh` and its `apk` branch, already at the pin; a second build served the bootstrap and dependency layers
installing `git` and `make` and finding the copied `CACHED` while both gates ran with a fresh epoch; a planted `unused` finding
linter already at the pin; a second build served the bootstrap and failed the build at the lint gate in 48.9s with the build stage's `make check`
dependency layers `CACHED` while both gates ran with a fresh epoch; never starting; and the suite run in the image as `--user 0:0` fails
a planted `unused` finding failed the build at the lint gate in `TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged user is
48.9s with the build stage's `make check` never starting; and the still load-bearing. That last check needs the Go test cache disabled — the
suite run in the image as `--user 0:0` fails first attempt reported `ok ... (cached)` as root, reusing the result the
`TestScanHardlinkRunFailsTogether`, so the drop to the unprivileged build-time run had left in the shared cache, which would have read as a pass.
user is still load-bearing. That last check needs the Go test cache Build wall time, on a shared host running many concurrent builds and so noisy:
disabled — the first attempt reported `ok ... (cached)` as root, 2m13s on an unchanged tree, 2m17s and 4m29s for two builds after a source
reusing the result the build-time run had left in the shared cache, change, 5m14s cold. Only the cold one breaches the policy ceiling, and not
which would have read as a pass. Build wall time, on a shared host because of this change — `chown -R builder:builder /src /home/builder` walks
running many concurrent builds and so noisy: 2m13s on an unchanged the module cache and re-runs on every source change, and it alone varied
tree, 2m17s and 4m29s for two builds after a source change, 5m14s between 77s and 210s across those four builds, which is also the whole spread
cold. Only the cold one breaches the policy ceiling, and not because in the totals. The same cold measurement against `main` is 5m03s with a 209s
of this change — `chown -R builder:builder /src /home/builder` walks `chown`. Filed as #43
the module cache and re-runs on every source change, and it alone - bust the Docker layer cache for the gate steps, so `script/cibuild` and
varied between 77s and 210s across those four builds, which is also `script/docker` cannot report a green they did not earn (2026-08-09, branch
the whole spread in the totals. The same cold measurement against `cibuild-cache-bust`, closes #32): both scripts were bare `docker build`
`main` is 5m03s with a 209s `chown`. Filed as #43 invocations with no cache control, and the `Dockerfile` copies the tree before
- bust the Docker layer cache for the gate steps, so `script/cibuild` running its gates, so on an unchanged tree Docker served those layers from
and `script/docker` cannot report a green they did not earn cache and the build exited 0 having executed nothing. That is not hypothetical
(2026-08-09, branch `cibuild-cache-bust`, closes #32): both scripts here — every merge this repo has done is a non-fast-forward merge of an
were bare `docker build` invocations with no cache control, and the undiverged branch, so each merge commit's tree is byte-identical to the branch
`Dockerfile` copies the tree before running its gates, so on an head's and each merge CI run was almost certainly a full cache hit; and PR
unchanged tree Docker served those layers from cache and the build #31's reviewer found `make docker` returning success as a 17-layer cache hit,
exited 0 having executed nothing. That is not hypothetical here — catching it only by being suspicious. The fix is `ARG CHECK_EPOCH` with the
every merge this repo has done is a non-fast-forward merge of an scripts passing `--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or
undiverged branch, so each merge commit's tree is byte-identical to break it. `ARG` is scoped per stage and this `Dockerfile` has three gates
the branch head's and each merge CI run was almost certainly a full across two — `make fmt-check` and `make lint` in the lint stage, `make check`
cache hit; and PR #31's reviewer found `make docker` returning in the build stage — so a single declaration would have left one stage
success as a 17-layer cache hit, catching it only by being silently cacheable; it is declared in both. And BuildKit hashes the expanded
suspicious. The fix is `ARG CHECK_EPOCH` with the scripts passing command, not the declaration, so a declared-but-unreferenced `ARG` invalidates
`--build-arg CHECK_EPOCH="$(date +%s)"`. Two details make or break nothing: each gate `RUN` echoes the epoch, which also puts the value in the
it. `ARG` is scoped per stage and this `Dockerfile` has three gates build log as evidence the layer really ran. Placement is below the dependency
across two — `make fmt-check` and `make lint` in the lint stage, layers on purpose — a build that goes cold every time would be a different
`make check` in the build stage — so a single declaration would have bug, not a fix. Verified by running each script twice back to back on an
left one stage silently cacheable; it is declared in both. And unchanged tree under `BUILDKIT_PROGRESS=plain`: all three gates executed on
BuildKit hashes the expanded command, not the declaration, so a all four runs, each with a fresh epoch in the log (`script/cibuild` 78.8s then
declared-but-unreferenced `ARG` invalidates nothing: each gate `RUN` 61.1s; `script/docker` 61.1s then 53.4s), and twelve steps were still served
echoes the epoch, which also puts the value in the build log as `CACHED` in the steady state — both `go mod download`s, `apk add`, `adduser`,
evidence the layer really ran. Placement is below the dependency the `chown`, every `go.mod`/`go.sum` and source copy, the linter copy out of
layers on purpose — a build that goes cold every time would be a the lint stage, and the binary copy into the runtime stage. The lint stage
different bug, not a fix. Verified by running each script twice back still gates the build stage: with a deliberate `unused` finding planted in the
to back on an unchanged tree under `BUILDKIT_PROGRESS=plain`: all tree, the build failed at `make lint` in 36.1s and the build-stage
three gates executed on all four runs, each with a fresh epoch in `make check` never started. The build stage also still drops to the
the log (`script/cibuild` 78.8s then 61.1s; `script/docker` 61.1s unprivileged `builder` user before `make check`, which the suite depends on
then 53.4s), and twelve steps were still served `CACHED` in the rather than merely prefers: forcing the same image to run the tests as root
steady state — both `go mod download`s, `apk add`, `adduser`, the fails `TestScanHardlinkRunFailsTogether`, because root reads straight through
`chown`, every `go.mod`/`go.sum` and source copy, the linter copy the `chmod(0)` the test uses to prove hard links are read once. This is the
out of the lint stage, and the binary copy into the runtime stage. local fix only; propagating it to the canonical templates is `prompts` #26
The lint stage still gates the build stage: with a deliberate - check the installed golangci-lint version in `script/bootstrap` instead of
`unused` finding planted in the tree, the build failed at only its presence (2026-08-09, branch `bootstrap-version-check`, closes #24):
`make lint` in 36.1s and the build-stage `make check` never started. `missing golangci-lint` meant any linter already on `PATH` satisfied the
The build stage also still drops to the unprivileged `builder` user check, so the pin was never consulted and the v2.12.2 bump from #3 was inert
before `make check`, which the suite depends on rather than merely on every host that already had one — this host ran v2.10.1 against a v2.12.2
prefers: forcing the same image to run the tests as root fails pin, `make check` went green, and `make docker` then rejected the same commit
`TestScanHardlinkRunFailsTogether`, because root reads straight with findings the local gate never saw. The version now lives in one place,
through the `chmod(0)` the test uses to prove hard links are read `GOLANGCI_LINT_VERSION`, with the `go install` module ref derived from it so a
once. This is the local fix only; propagating it to the canonical bump cannot half-apply; a `golangci_lint_version` helper parses
templates is `prompts` #26 `golangci-lint --version` (taking the field after the word `version` and
- check the installed golangci-lint version in `script/bootstrap` tolerating an optional leading `v`, which the module ref carries and the
instead of only its presence (2026-08-09, branch binary's output does not), and any version that is not the pin — older, newer,
`bootstrap-version-check`, closes #24): `missing golangci-lint` meant absent or unparseable — is reinstalled. The install is then verified against
any linter already on `PATH` satisfied the check, so the pin was never the binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
consulted and the v2.12.2 bump from #3 was inert on every host that `GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes first on
already had one — this host ran v2.10.1 against a v2.12.2 pin, `PATH`, so a wrong-version one sitting ahead of it — nix, apt, brew, apk, or
`make check` went green, and `make docker` then rejected the same the `/usr/local/bin` copy the `Dockerfile` builder stage makes — would swallow
commit with findings the local gate never saw. The version now lives the install and leave the local gate disagreeing with CI under an affirmative
in one place, `GOLANGCI_LINT_VERSION`, with the `go install` module `bootstrap complete`. Bootstrap now re-reads the effective version after
ref derived from it so a bump cannot half-apply; a installing and, on a mismatch, prints both paths and both versions to stderr
`golangci_lint_version` helper parses `golangci-lint --version` and exits non-zero instead of claiming success; it does not reorder anyone's
(taking the field after the word `version` and tolerating an optional
leading `v`, which the module ref carries and the binary's output does
not), and any version that is not the pin — older, newer, absent or
unparseable — is reinstalled. The install is then verified against the
binary `PATH` actually resolves: `go install` writes into `GOBIN` (or
`GOPATH/bin`) while `make lint` runs whichever `golangci-lint` comes
first on `PATH`, so a wrong-version one sitting ahead of it — nix,
apt, brew, apk, or the `/usr/local/bin` copy the `Dockerfile` builder
stage makes — would swallow the install and leave the local gate
disagreeing with CI under an affirmative `bootstrap complete`.
Bootstrap now re-reads the effective version after installing and, on
a mismatch, prints both paths and both versions to stderr and exits
non-zero instead of claiming success; it does not reorder anyone's
`PATH` or delete their binary. The `--version` call keeps its stderr `PATH` or delete their binary. The `--version` call keeps its stderr
connected, so a present-but-broken binary says why rather than connected, so a present-but-broken binary says why rather than reinstalling
reinstalling forever in silence, and is bounded by `timeout(1)` where forever in silence, and is bounded by `timeout(1)` where that exists, so a
that exists, so a wedged binary cannot hang bootstrap. `git`, `make` wedged binary cannot hang bootstrap. `git`, `make` and `go` keep their
and `go` keep their presence-only checks and now say why in a presence-only checks and now say why in a comment: they are host
comment: they are host package-manager tools the repo deliberately package-manager tools the repo deliberately does not pin, with `go.mod`
does not pin, with `go.mod` governing the language version and the governing the language version and the digest-pinned images covering
digest-pinned images covering reproducible builds. Verified on this reproducible builds. Verified on this host by bootstrapping from v2.10.1 to
host by bootstrapping from v2.10.1 to v2.12.2 and running it again to v2.12.2 and running it again to a no-op, plus stub runs of the real script
a no-op, plus stub runs of the real script under `dash` covering a under `dash` covering a thirteen-input parse matrix (absent, older, newer,
thirteen-input parse matrix (absent, older, newer, host-style, host-style, image-style, leading-`v`, stderr-only, empty, non-zero exit,
image-style, leading-`v`, stderr-only, empty, non-zero exit, impostor impostor binary, `(devel)`, trailing `version`), a shadowed install that must
binary, `(devel)`, trailing `version`), a shadowed install that must exit non-zero, an install destination not on `PATH` at all, `GOBIN` set, and a
exit non-zero, an install destination not on `PATH` at all, `GOBIN` wedged binary that must hit the timeout; `make check` and `make lint` are
set, and a wedged binary that must hit the timeout; `make check` and clean at v2.12.2, so v2.10.1 was not hiding any findings on `main`
`make lint` are clean at v2.12.2, so v2.10.1 was not hiding any
findings on `main`
- unwind the hash worker pool on the error path (2026-08-09, branch - unwind the hash worker pool on the error path (2026-08-09, branch
`hash-pool-cleanup`, closes #6): `hashPhase` used to return the `hash-pool-cleanup`, closes #6): `hashPhase` used to return the moment
moment `recordRun` failed and abandon the pool — the feeder parked `recordRun` failed and abandon the pool — the feeder parked forever on a full
forever on a full `jobs` channel and every worker on a full `jobs` channel and every worker on a full `results` channel. That only stopped
`results` channel. That only stopped being invisible when #4 landed being invisible when #4 landed and `runScan` began unwinding instead of
and `runScan` began unwinding instead of calling `os.Exit`. The calling `os.Exit`. The pool is now an owned, context-aware `hashPool`: every
pool is now an owned, context-aware `hashPool`: every blocking send blocking send in the feeder and the workers selects on `ctx.Done()`, `jobs` is
in the feeder and the workers selects on `ctx.Done()`, `jobs` is closed on every path out, and `hashPhase` defers `pool.stop()`, which cancels
closed on every path out, and `hashPhase` defers `pool.stop()`, and then drains `results` until the last goroutine has exited — draining is
which cancels and then drains `results` until the last goroutine what frees a worker already parked on a send. `ctx` is threaded from
has exited — draining is what frees a worker already parked on a `cmd.Context()` through `runScan`, `syncScan`, both worker pools and the whole
send. `ctx` is threaded from `cmd.Context()` through `runScan`, database layer (it is the first parameter everywhere), so #5 can hand this
`syncScan`, both worker pools and the whole database layer (it is path a signal and needs to add nothing else. The walk pool never leaked,
the first parameter everywhere), so #5 can hand this path a signal because `walkPhase` always drains its events to close, but it has the same
and needs to add nothing else. The walk pool never leaked, because unbounded-send shape and #5 will give it an early return, so it gets the same
`walkPhase` always drains its events to close, but it has the same treatment plus a `ctx.Err()` guard after the walk: a cancelled walk yields a
unbounded-send shape and #5 will give it an early return, so it partial size census, and every file it never reached looks vanished to the
gets the same treatment plus a `ctx.Err()` guard after the walk: a update phase. That phase's own `BeginTx` fails on the same cancelled context
cancelled walk yields a partial size census, and every file it never before deleting anything, so the guard is defence in depth rather than the
reached looks vanished to the update phase. That phase's own only barrier — but it is the one that survives #5 deciding an interrupted scan
`BeginTx` fails on the same cancelled context before deleting may commit what it has. Tests drive `run(scan)` against a database whose
anything, so the guard is defence in depth rather than the only insert trigger aborts, and assert both that the scan fails instead of hanging
barrier — but it is the one that survives #5 deciding an interrupted and that `runtime.NumGoroutine()` polls back to its pre-scan baseline; a
scan may commit what it has. Tests drive `run(scan)` against a second set cancels a scan part-way through the walk — deterministically, by
database whose insert trigger aborts, and assert both that the scan counting the scan's own consultations of `ctx.Done()` rather than racing a
fails instead of hanging and that `runtime.NumGoroutine()` polls timer — and asserts that it stops at the guard holding a partial census and a
back to its pre-scan baseline; a second set cancels a scan part-way still-populated record index, with every record intact. The remaining
through the walk — deterministically, by counting the scan's own cancellation branches of both pools are covered by direct tests of
consultations of `ctx.Done()` rather than racing a timer — and `sendEvent`, the walk workers, `dispatchDirs`, `feedHashJobs`, `hashWorker`
asserts that it stops at the guard holding a partial census and a and `hashPhase`
still-populated record index, with every record intact. The - guarantee the database is closed on every fatal exit path (2026-08-09, branch
remaining cancellation branches of both pools are covered by direct `db-close-on-fatal`, closes #4): `fatalf` and its `os.Exit(1)` are gone, so
tests of `sendEvent`, the walk workers, `dispatchDirs`, the deferred `db.Close()` — and with it the SQLite WAL checkpoint — now
`feedHashJobs`, `hashWorker` and `hashPhase` actually runs when a subcommand fails; `runScan`, `runReport`, `runTrees`,
- guarantee the database is closed on every fatal exit path `loadRecords` and `resolveRoots` return errors instead. The single exit point
(2026-08-09, branch `db-close-on-fatal`, closes #4): `fatalf` and is `run` in `main.go`: it maps a `fatalError` (anything a subcommand returned)
its `os.Exit(1)` are gone, so the deferred `db.Close()` — and with to exit 1 and cobra's own argument and flag errors to exit 2, which keeps a
it the SQLite WAL checkpoint — now actually runs when a subcommand runtime failure from being reported as a usage error or printing the usage
fails; `runScan`, `runReport`, `runTrees`, `loadRecords` and text. New `main_test.go` drives the CLI in-process and asserts the exit codes
`resolveRoots` return errors instead. The single exit point is `run` from README §Error handling plus the stdout/stderr split, including that a
in `main.go`: it maps a `fatalError` (anything a subcommand fatal error raised after the database is open leaves no `-wal`/`-shm` sidecar
returned) to exit 1 and cobra's own argument and flag errors to exit behind for `scan`, `report` or `trees`
2, which keeps a runtime failure from being reported as a usage - update golangci-lint to v2.12.2 with the canonical config (2026-08-09, branch
error or printing the usage text. New `main_test.go` drives the CLI `golangci-v2.12.2`, merged as `38a01bd`, closes #3): bumped the pinned linter
in-process and asserts the exit codes from README §Error handling in the `Dockerfile` lint stage and `script/bootstrap` from v2.12.1 to v2.12.2,
plus the stdout/stderr split, including that a fatal error raised and replaced `.golangci.yml` with the canonical file — the linter settings
after the database is open leaves no `-wal`/`-shm` sidecar behind
for `scan`, `report` or `trees`
- update golangci-lint to v2.12.2 with the canonical config
(2026-08-09, branch `golangci-v2.12.2`, merged as `38a01bd`,
closes #3): bumped the pinned linter in the `Dockerfile` lint
stage and `script/bootstrap` from v2.12.1 to v2.12.2, and replaced
`.golangci.yml` with the canonical file — the linter settings
(`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under (`lll`, `funlen`, `cyclop`, `dupl` thresholds) now live under
`linters.settings` per the v2 schema, so they are actually `linters.settings` per the v2 schema, so they are actually applied; no new
applied; no new lint findings surfaced lint findings surfaced
- convert Makefile targets to scripts-to-rule-them-all `script/` - convert Makefile targets to scripts-to-rule-them-all `script/` entrypoints
entrypoints like the other managed repos (2026-07-26, commit like the other managed repos (2026-07-26, commit `3abeacf`, closes #1): all 12
`3abeacf`, closes #1): all 12 `script/` entrypoints exist `script/` entrypoints exist (`bootstrap`, `setup`, `projectname`, `test`,
(`bootstrap`, `setup`, `projectname`, `test`, `lint`, `fmt`, `lint`, `fmt`, `fmt-check`, `check`, `docker`, `cibuild`, `precommit`,
`fmt-check`, `check`, `docker`, `cibuild`, `precommit`, `install-precommit`) and every Makefile target is now a thin shim over them,
`install-precommit`) and every Makefile target is now a thin shim matching the other managed repos
over them, matching the other managed repos
- make the binary the default Make target (2026-07-24, branch - make the binary the default Make target (2026-07-24, branch
`make-default-target`): plain `make` now builds `sfdupes` `make-default-target`): plain `make` now builds `sfdupes` (previously it ran
(previously it ran `check` plus `build`); `make build` remains as `check` plus `build`); `make build` remains as an alias
an alias - scan-wide phases, concurrent operands, batched updates (2026-07-24, branch
- scan-wide phases, concurrent operands, batched updates (2026-07-24, `scan-wide-phases`): all operands seed the shared walk pool and every pass
branch `scan-wide-phases`): all operands seed the shared walk pool runs once over the whole scan, so totals and ETAs are scan-global; the
and every pass runs once over the whole scan, so totals and ETAs per-operand walk/hash/update cycles and their stderr announcements are gone;
are scan-global; the per-operand walk/hash/update cycles and their the update pass commits in batched transactions — the filesystem is
stderr announcements are gone; the update pass commits in batched authoritative and the database an eventually-consistent reflection, so
transactions — the filesystem is authoritative and the database an scan-level atomicity is not required
eventually-consistent reflection, so scan-level atomicity is not
required
- split the stat pass back out of the walk (2026-07-24, branch - split the stat pass back out of the walk (2026-07-24, branch
`parallel-phases`): phases are strictly sequential again — walk, `parallel-phases`): phases are strictly sequential again — walk, stat, hash,
stat, hash, update per operand — with parallelism only inside each update per operand — with parallelism only inside each phase; the walk
phase; the walk enumerates paths with per-directory workers and the enumerates paths with per-directory workers and the stat pass lstats them with
stat pass lstats them with per-file workers, restoring the exact per-file workers, restoring the exact total/ETA stat bar
total/ETA stat bar - announce each operand on stderr before its passes (2026-07-24, branch
- announce each operand on stderr before its passes (2026-07-24, `scan-operand-progress`): with per-operand walk/hash/update cycles, a
branch `scan-operand-progress`): with per-operand walk/hash/update multi-operand run (e.g. `scan /srv/*`) showed pass totals that looked like the
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals whole run's — an operator watching operand 3 of 14 hash 300k files concluded
that looked like the whole run's — an operator watching operand 3 of 20M files were being skipped
14 hash 300k files concluded 20M files were being skipped - parallel walk (2026-07-24, branch `parallel-walk`): the walk pass was a single
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass goroutine and took hours at ~20M files on a busy pool (observed: 22M files in
was a single goroutine and took hours at ~20M files on a busy pool 4h on a ZFS server); it is now a per-directory worker-pool traversal that
(observed: 22M files in 4h on a ZFS server); it is now a records size/mtime during the walk (folding away the separate stat pass,
per-directory worker-pool traversal that records size/mtime during halving metadata I/O), and each `PATH` operand commits in its own transaction
the walk (folding away the separate stat pass, halving metadata so an interrupted scan keeps completed operands
I/O), and each `PATH` operand commits in its own transaction so an
interrupted scan keeps completed operands
- persistent scan database (2026-07-24, branch `persistent-database`): - persistent scan database (2026-07-24, branch `persistent-database`): `scan`
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure now maintains a SQLite database (`modernc.org/sqlite`, pure Go, cgo stays
Go, cgo stays disabled) keyed by absolute path that survives between disabled) keyed by absolute path that survives between runs — a rescan hashes
runs — a rescan hashes only new or changed files (by mtime/size), only new or changed files (by mtime/size), deletes records for files vanished
deletes records for files vanished from under the scanned operands, from under the scanned operands, and leaves records outside them untouched, so
and leaves records outside them untouched, so `scan` can be cronned `scan` can be cronned daily; `report` and `trees` read the database (no
daily; `report` and `trees` read the database (no positional positional arguments) instead of a scan stream. Database at
arguments) instead of a scan stream. Database at `/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; WAL
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`; journaling plus a single-transaction update keep a report run during a scan
WAL journaling plus a single-transaction update keep a report run safe
during a scan safe - add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag `v0.0.1`,
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag and push `main` plus tags (2026-07-23)
`v0.0.1`, and push `main` plus tags (2026-07-23)
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required - `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
`PATH...` operands via cobra flags replacing the `/srv` `-root` `PATH...` operands via cobra flags replacing the `/srv` `-root` default; new
default; new `-x`/`--one-file-system` flag (GNU convention) to stop `-x`/`--one-file-system` flag (GNU convention) to stop at filesystem
at filesystem boundaries, which are crossed by default boundaries, which are crossed by default
- bring the repo into full policy compliance (2026-07-23, branch - bring the repo into full policy compliance (2026-07-23, branch
`repo-policy-compliance`; checklist below) `repo-policy-compliance`; checklist below)
- `git init` with README-only first commit; code baseline committed on - `git init` with README-only first commit; code baseline committed on `main`
`main` (2026-07-22) (2026-07-22)
- implement `scan`, `report`, and `trees` subcommands (pre-git history) - implement `scan`, `report`, and `trees` subcommands (pre-git history)
# Future Steps # Future Steps
- possible later features (explicitly out of scope per README): - possible later features (explicitly out of scope per README): full-content
full-content verification of candidates, removal-script helpers verification of candidates, removal-script helpers
# Repo Policy Compliance # Repo Policy Compliance
Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing Audited 2026-07-22 against `REPO_POLICIES.md` (2026-07-06), the existing repo
repo checklist, and the Go styleguide. Code is already gofmt-clean, so no checklist, and the Go styleguide. Code is already gofmt-clean, so no standalone
standalone formatting commit is needed. formatting commit is needed.
- [x] `.gitignore` missing — the compiled `sfdupes` binary and - [x] `.gitignore` missing — the compiled `sfdupes` binary and `files.dat` sit
`files.dat` sit untracked in the tree; needs OS/editor/Go untracked in the tree; needs OS/editor/Go artifacts plus secrets patterns
artifacts plus secrets patterns
- [x] `.editorconfig` missing - [x] `.editorconfig` missing
- [x] `LICENSE` missing and README has no License section (MIT assumed - [x] `LICENSE` missing and README has no License section (MIT assumed from
from house convention — user to confirm) house convention — user to confirm)
- [x] `REPO_POLICIES.md` missing from repo root - [x] `REPO_POLICIES.md` missing from repo root
- [x] `.golangci.yml` missing (install canonical copy); code must then - [x] `.golangci.yml` missing (install canonical copy); code must then pass
pass `make lint` (150 findings fixed; `make lint` is clean) `make lint` (150 findings fixed; `make lint` is clean)
- [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, - [x] `Makefile` lacks required targets `test`, `lint`, `fmt`, `fmt-check`,
`fmt-check`, `docker`, `hooks`; `check` currently depends on `docker`, `hooks`; `check` currently depends on `build`, which writes the
`build`, which writes the binary (`make check` must not modify binary (`make check` must not modify files)
files) - [x] no tests — `go test ./...` has nothing to run; policy requires real tests
- [x] no tests — `go test ./...` has nothing to run; policy requires with a 30-second timeout and the conditional `-v` rerun pattern (suite
real tests with a 30-second timeout and the conditional `-v` covers parsing, grouping, digests, suppression, hashing, and the scan
rerun pattern (suite covers parsing, grouping, digests, pipeline; 64% coverage)
suppression, hashing, and the scan pipeline; 64% coverage) - [x] `Dockerfile` missing — Go multistage with hash-pinned images: fail-fast
- [x] `Dockerfile` missing — Go multistage with hash-pinned images: lint stage, build stage running `make check`
fail-fast lint stage, build stage running `make check`
- [x] `.dockerignore` missing - [x] `.dockerignore` missing
- [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, - [x] `.gitea/workflows/check.yml` missing (`docker build .` on push, checkout
checkout action pinned by commit SHA) action pinned by commit SHA)
- [x] README lacks required sections: Description first line - [x] README lacks required sections: Description first line
(name/purpose/category/license/author), Getting Started, (name/purpose/category/license/author), Getting Started, Rationale, TODO,
Rationale, TODO, License, Author License, Author
- [x] README non-goal "no git repository setup and no CI" is stale now - [x] README non-goal "no git repository setup and no CI" is stale now that the
that the repo is under git with CI repo is under git with CI
- [x] pre-commit hook not installed (`make hooks` once the target - [x] pre-commit hook not installed (`make hooks` once the target exists)
exists)
Accepted divergences (no action): Accepted divergences (no action):
- flat single-package layout with `.go` files in the repo root — fine - flat single-package layout with `.go` files in the repo root — fine for a
for a small single-binary tool per the Go styleguide; the tracker small single-binary tool per the Go styleguide; the tracker audit agrees
audit agrees - `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` (pure-Go
- `go test` runs without `-race` — the repo mandates `CGO_ENABLED=0` builds) and the race detector requires cgo
(pure-Go builds) and the race detector requires cgo
+5
View File
@@ -0,0 +1,5 @@
{
"devDependencies": {
"prettier": "3.8.1"
}
}
+39 -96
View File
@@ -2,30 +2,20 @@
# script/bootstrap: install all dependencies needed to build and develop # script/bootstrap: install all dependencies needed to build and develop
# this repo. Idempotent: every install is guarded by a check so already # this repo. Idempotent: every install is guarded by a check so already
# installed tools are skipped. Base tooling comes from nix, apt, brew, # installed tools are skipped. Base tooling comes from nix, apt, brew,
# or apk (detected in that order); assumes nothing is present. # or apk (detected in that order); assumes nothing is present (not git,
# golangci-lint is installed via `go install` pinned to the same version # make, or go). The linter is NOT installed: golangci-lint runs via
# the Dockerfile lint stage uses (never "latest"), and is reinstalled # docker only (script/lint), pinned by image digest, so the only lint
# whenever the installed version differs from that pin. The install is # prerequisite is a working docker — which is warned about, not
# then verified against the binary PATH actually resolves: if the pin is # installed, because everything except linting works without it.
# still not what would run, bootstrap fails instead of reporting
# success.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Pinned versions, 2026-08-07 (same version as the Dockerfile lint stage). # yarn provides prettier, which formats Markdown. yarn is a tool, like
# This is the single source of truth for the linter version: the module # node/git/make/go below; the reference that governs formatting output is
# ref below and the version comparison in main() are both derived from # prettier, pinned by yarn.lock's integrity hash and installed by
# it, so a bump here cannot half-apply. Written without a leading "v", # `yarn install --frozen-lockfile`.
# the way `golangci-lint --version` reports it. YARN_VERSION="1.22.22"
GOLANGCI_LINT_VERSION="2.12.2"
GOLANGCI_LINT_MODULE="github.com/golangci/golangci-lint/v2/cmd/golangci-lint"
GOLANGCI_LINT_REF="$GOLANGCI_LINT_MODULE@v$GOLANGCI_LINT_VERSION"
# Seconds to allow `golangci-lint --version` to run. Bootstrap now
# executes the binary rather than merely locating it, so a wedged one
# must not hang the script.
GOLANGCI_LINT_VERSION_TIMEOUT="30"
PKGMGR="" PKGMGR=""
SUDO="" SUDO=""
@@ -74,70 +64,18 @@ missing() {
! command -v "$1" >/dev/null 2>&1 ! command -v "$1" >/dev/null 2>&1
} }
# Echo the installed golangci-lint version, or nothing when the tool is ensure_node() {
# absent. The binary reports e.g. if ! missing node; then return 0; fi
# golangci-lint has version X.Y.Z built with go1.26.5 from abc1234 ... pkg_install nodejs nodejs node nodejs
# so the version is the field after the literal word "version", and it
# carries no leading "v" (the module ref does). Some builds do print a
# leading "v", so strip one if present and compare bare versions.
#
# Only stdout is parsed; the binary's stderr is deliberately left
# connected to ours so that a present-but-broken linter (missing shared
# library, wrong architecture) says why instead of silently yielding the
# empty string. The call is bounded by timeout(1) where that exists —
# stock macOS has no timeout(1), and there the call runs unbounded, as
# it did before this check was version-aware.
golangci_lint_version() {
command -v golangci-lint >/dev/null 2>&1 || return 0
if command -v timeout >/dev/null 2>&1; then
timeout "$GOLANGCI_LINT_VERSION_TIMEOUT" golangci-lint --version
else
golangci-lint --version
fi | awk '
{
for (i = 1; i < NF; i++) {
if ($i == "version") {
v = $(i + 1)
sub(/^v/, "", v)
print v
exit
}
}
}
'
} }
# Confirm that the golangci-lint just installed is the one that will ensure_yarn() {
# actually run. `go install` writes into "$(go env GOBIN)" (or if ! missing yarn; then return 0; fi
# "$(go env GOPATH)/bin"), but `make lint` runs whatever PATH resolves if ! missing corepack; then
# first. When a wrong-version binary sits ahead of that directory — a corepack enable >/dev/null 2>&1 || true
# nix profile, apt, brew, apk, or a tarball in /usr/local/bin — the corepack prepare "yarn@$YARN_VERSION" --activate
# install lands behind the shadow and changes nothing the gate uses. else
# Exiting 0 there would leave the local gate linting against a different pkg_install yarn yarn yarn yarn
# ruleset than CI while claiming success, which is the failure this
# whole check exists to prevent. Diagnose and stop: naming both paths is
# what makes it fixable. Reordering PATH or deleting someone else's
# binary is not bootstrap's call.
verify_golangci_lint() {
# Forget any remembered command locations first: the install may have
# created a binary in a directory the shell already searched.
hash -r 2>/dev/null || true
goinstalldir="$(go env GOBIN)"
if [ -z "$goinstalldir" ]; then
goinstalldir="$(go env GOPATH)/bin"
fi
resolved="$(command -v golangci-lint 2>/dev/null || true)"
effective="$(golangci_lint_version)"
if [ "$effective" != "$GOLANGCI_LINT_VERSION" ]; then
echo "bootstrap: installed golangci-lint $GOLANGCI_LINT_VERSION into" \
"$goinstalldir, but the golangci-lint on PATH is" \
"${resolved:-not resolvable} and reports" \
"${effective:-no parseable version}" >&2
echo "bootstrap: the install is shadowed or unreachable; put" \
"$goinstalldir ahead of it on PATH (or remove the shadowing" \
"binary) and re-run" >&2
exit 1
fi fi
} }
@@ -154,20 +92,25 @@ main() {
if missing make; then pkg_install gnumake make make make; fi if missing make; then pkg_install gnumake make make make; fi
if missing go; then pkg_install go golang go go; fi if missing go; then pkg_install go golang go go; fi
# Lint tooling, pinned via go install (installs into # node runs prettier and is an unpinned host tool for the same reason
# "$(go env GOPATH)/bin"; ensure that is on your PATH). Unlike the # git/make/go are: it comes from the host package manager, whatever
# system tools above this is version-checked, not presence-checked: # version it ships. It is not installed via nvm the way the canonical
# the Dockerfile lint stage runs a digest-pinned linter, so a host # template does, because nvm's prebuilt node is glibc-linked and does
# running any other version lints against different rules and # not run on this repo's musl/Alpine build image. prettier — the tool
# `make check` can go green on a commit CI then rejects. Any version # whose version affects formatting output — is pinned by yarn.lock.
# that is not the pin — older or newer — is reinstalled, and the ensure_node
# install is then verified to be the binary PATH resolves. ensure_yarn
installed="$(golangci_lint_version)" yarn install --frozen-lockfile
if [ "$installed" != "$GOLANGCI_LINT_VERSION" ]; then
echo "bootstrap: golangci-lint ${installed:-absent or unparseable}," \ # Linting runs via docker only (script/lint), so docker is a lint
"want $GOLANGCI_LINT_VERSION; installing" # prerequisite rather than something bootstrap installs. Warn, do
go install "$GOLANGCI_LINT_REF" # not fail: everything except `make lint` — and, through it,
verify_golangci_lint # `make check`, `make docker` and the pre-commit hook — works
# without it.
if missing docker; then
echo "bootstrap: WARNING: docker not found; make lint, make check" >&2
echo "bootstrap: and make docker require it. Install docker to" >&2
echo "bootstrap: run the linter." >&2
fi fi
go mod download go mod download
+15 -4
View File
@@ -1,8 +1,19 @@
#!/bin/sh #!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs make fmt-check, # script/cibuild: run the CI build. The Gitea workflow runs this on
# make lint and make check (via the script/ entrypoints) as build steps, # push.
# so a successful build implies all checks pass. The Gitea workflow runs #
# this on push. # The Dockerfile runs the gates individually as build steps, not the
# make check aggregate: the lint stage runs make fmt-check,
# script/verify-lint-image-pin, golangci-lint config verify and
# golangci-lint run; the build stage, dropped to an unprivileged user,
# runs make test and make fmt-check. Neither make lint nor make check
# appears, because both reach script/lint, which is itself a docker
# build, and a docker build cannot run inside one. Lint is not skipped
# by that — the linter is invoked directly in the lint stage, and the
# build stage's COPY --from=lint makes that stage a prerequisite, so
# BuildKit must finish it first. Between the two stages everything
# make check would run has run, which is why a successful build here
# implies the repo is green.
# #
# That implication holds only because of CHECK_EPOCH. A COPY layer is # That implication holds only because of CHECK_EPOCH. A COPY layer is
# invalidated by changed content, and a merge commit's tree is # invalidated by changed content, and a merge commit's tree is
+5 -5
View File
@@ -4,11 +4,11 @@
# #
# CHECK_EPOCH is passed for the same reason script/cibuild passes it: # CHECK_EPOCH is passed for the same reason script/cibuild passes it:
# without it Docker serves the Dockerfile's gate layers from cache on an # without it Docker serves the Dockerfile's gate layers from cache on an
# unchanged tree and this exits 0 having run neither the lint stage nor # unchanged tree and this exits 0 having run neither the lint stage's
# the builder stage's make check. This is the gate a developer or # gates nor the builder stage's test and fmt-check gates. This is the
# reviewer runs by hand, so a cached pass here is the most misleading # set of gates a developer or reviewer runs by hand, so a cached pass
# result the repo can produce. Dependency layers sit above the ARG and # here is the most misleading result the repo can produce. Dependency
# stay cached. # layers sit above the ARG and stay cached.
set -eu set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
+12 -1
View File
@@ -1,12 +1,23 @@
#!/bin/sh #!/bin/sh
# script/fmt: format all files (writes). # script/fmt: format all files (writes). gofmt for Go, prettier for
# Markdown. prettier is the pinned devDependency in package.json/
# yarn.lock; script/bootstrap installs it (see run_prettier).
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
run_prettier() {
if ! command -v yarn >/dev/null 2>&1; then
echo "fmt: yarn not found; run script/bootstrap first" >&2
exit 1
fi
yarn run prettier "$@"
}
main() { main() {
cd "$ROOT" cd "$ROOT"
gofmt -s -w . gofmt -s -w .
run_prettier --write '**/*.md' --tab-width 4 --prose-wrap always
} }
main "$@" main "$@"
+21 -2
View File
@@ -1,18 +1,37 @@
#!/bin/sh #!/bin/sh
# script/fmt-check: check formatting (read-only). Same scope as # script/fmt-check: check formatting (read-only). Same scope as
# script/fmt, but fails instead of writing. # script/fmt: gofmt for Go, prettier for Markdown. Both run every time
# and each reports independently, so a failure names which formatter is
# unhappy; the script exits non-zero if either found unformatted files.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
run_prettier() {
if ! command -v yarn >/dev/null 2>&1; then
echo "fmt-check: yarn not found; run script/bootstrap first" >&2
exit 1
fi
yarn run prettier "$@"
}
main() { main() {
cd "$ROOT" cd "$ROOT"
rc=0
files="$(gofmt -s -l .)" files="$(gofmt -s -l .)"
if [ -n "$files" ]; then if [ -n "$files" ]; then
echo "gofmt: files not formatted:" >&2 echo "gofmt: files not formatted:" >&2
echo "$files" >&2 echo "$files" >&2
exit 1 rc=1
fi fi
if ! run_prettier --check '**/*.md' --tab-width 4 --prose-wrap always; then
echo "prettier: Markdown not formatted; run make fmt" >&2
rc=1
fi
exit "$rc"
} }
main "$@" main "$@"
+19 -2
View File
@@ -1,12 +1,29 @@
#!/bin/sh #!/bin/sh
# script/lint: run the linter. # script/lint: run the linter. golangci-lint is never installed on a
# host: it runs via docker only, one way, everywhere — this builds
# Dockerfile.lint, which COPYs the repo into the digest-pinned
# golangci-lint image and lints as a build step, so a successful build
# is a clean lint. The only prerequisite is a working docker. The gate
# steps make no network calls of their own, but Dockerfile.lint runs
# `go mod download` above them, so a cold cache does reach the network
# (as does pulling the pinned image); that layer stays cached, and once
# it is warm this runs offline until go.mod or go.sum changes.
#
# CHECK_EPOCH is what makes the result mean anything. Without it docker
# serves the gate layers from cache on an unchanged tree and this exits
# 0 in well under a second having run no linter. The PID is in the value
# as well as the epoch because two lint runs land inside the same second
# easily, and `date +%s` alone would cache the second one.
set -eu set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)" ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() { main() {
cd "$ROOT" cd "$ROOT"
golangci-lint run --config .golangci.yml ./... docker build \
--build-arg CHECK_EPOCH="$(date +%s)-$$" \
-f Dockerfile.lint \
.
} }
main "$@" main "$@"
+84
View File
@@ -0,0 +1,84 @@
#!/bin/sh
# script/verify-lint-image-pin: fail unless the golangci-lint image
# referenced by Dockerfile.lint and the one referenced by the main
# Dockerfile's lint stage are the same image at the same digest. Our own
# extension to scripts-to-rule-them-all, not one of its entrypoints.
#
# The linter version is pinned in two independent files. That is the
# shape #42 turned into a build failure rather than tolerate: nothing
# else keeps the two in sync, and a bump applied to one file alone would
# leave `make lint` and the fail-fast lint stage of `make docker`
# linting the same tree against different rulesets, both green. This is
# the single guard that stops it, run as a gate in both files.
#
# It deliberately restates neither pin. A hardcoded expected digest here
# would be a third copy — one more thing to bump, and the same drift one
# file further out. It compares the two files to each other and knows
# nothing about which version is correct.
#
# A reference that cannot be read is a hard failure, not a skip: a
# comparison of two empty strings succeeds, which would turn this guard
# into exactly the unearned green it exists to prevent.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
LINT_DOCKERFILE="Dockerfile.lint"
MAIN_DOCKERFILE="Dockerfile"
# Echo the single golangci-lint image reference in the named Dockerfile.
# Scans every argument of every FROM instruction rather than assuming a
# field position, so `FROM --platform=... img AS stage` reads correctly.
# Exits non-zero, with a diagnosis, unless there is exactly one.
lint_image_ref() {
file="$1"
if [ ! -f "$file" ]; then
echo "verify-lint-image-pin: $file: not found" >&2
return 1
fi
refs="$(
awk '
toupper($1) == "FROM" {
for (i = 2; i <= NF; i++) {
if ($i ~ /^golangci\/golangci-lint[:@]/) {
print $i
}
}
}
' "$file"
)"
count="$(printf '%s' "$refs" | grep -c . || true)"
if [ "$count" -ne 1 ]; then
echo "verify-lint-image-pin: $file: expected exactly one" \
"golangci/golangci-lint FROM reference, found $count" >&2
return 1
fi
printf '%s\n' "$refs"
}
main() {
cd "$ROOT"
lint_ref="$(lint_image_ref "$LINT_DOCKERFILE")"
main_ref="$(lint_image_ref "$MAIN_DOCKERFILE")"
if [ "$lint_ref" != "$main_ref" ]; then
echo "verify-lint-image-pin: the linter image is pinned twice and" \
"the two pins disagree:" >&2
echo "verify-lint-image-pin: $LINT_DOCKERFILE: $lint_ref" >&2
echo "verify-lint-image-pin: $MAIN_DOCKERFILE: $main_ref" >&2
echo "verify-lint-image-pin: bump both FROM lines together so" \
"script/lint and the Dockerfile lint stage keep running the" \
"same linter" >&2
exit 1
fi
echo "verify-lint-image-pin: $LINT_DOCKERFILE and $MAIN_DOCKERFILE" \
"agree on $lint_ref"
}
main "$@"
-102
View File
@@ -1,102 +0,0 @@
#!/bin/sh
# script/verify-linter-pin: fail unless a golangci-lint binary is exactly
# the version script/bootstrap pins. Takes the binary to check as its
# argument, defaulting to whatever PATH resolves. Our own extension to
# scripts-to-rule-them-all, not one of its entrypoints.
#
# The Dockerfile build stage runs this on the linter it copies out of the
# lint stage, before anything else runs there. Without it, drift between
# the two stages is silently absorbed: script/bootstrap reinstalls its
# pinned version from source, verifies that, and the build goes green
# with the lint stage having linted at one version and `make check`
# having run at another. Bumping the lint stage image alone is enough to
# produce that, and this is the check that turns it into a build failure
# naming both versions.
#
# The pin is read out of script/bootstrap rather than restated here.
# script/bootstrap is the single source of truth for the linter version,
# and a second hardcoded copy of it is exactly the drift this script
# exists to catch. A pin that cannot be read is therefore a hard failure
# and not a skip: silently comparing against an empty string would turn
# this check into the kind of unearned green it was written to stop.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Seconds to allow `golangci-lint --version` to run, so a wedged binary
# stops the build instead of hanging it. Bounded by timeout(1) where that
# exists; stock macOS has none, and there the call runs unbounded.
VERSION_TIMEOUT="30"
version_output() {
if command -v timeout >/dev/null 2>&1; then
timeout "$VERSION_TIMEOUT" "$1" --version
else
"$1" --version
fi
}
main() {
# Resolve the argument before changing directory, so a relative path
# means what the caller meant by it.
bin="${1:-golangci-lint}"
resolved="$(command -v "$bin" 2>/dev/null || true)"
cd "$ROOT"
pin="$(
sed -n 's/^GOLANGCI_LINT_VERSION="\([^"]*\)".*/\1/p' script/bootstrap
)"
if [ -z "$pin" ]; then
echo "verify-linter-pin: no GOLANGCI_LINT_VERSION assignment found" \
"in script/bootstrap; that file is the single source of truth" \
"for the linter version and this check cannot run without it" >&2
exit 1
fi
if [ -z "$resolved" ]; then
echo "verify-linter-pin: $bin: not found (pin is $pin)" >&2
exit 1
fi
# Same output shape script/bootstrap parses:
# golangci-lint has version X.Y.Z built with go1.26.5 from abc1234
# so the version is the field after the literal word "version", with
# any leading "v" stripped. stderr is left connected so a binary that
# cannot execute (wrong architecture, missing shared library) says why
# rather than being reported as merely unparseable.
if ! out="$(version_output "$resolved")"; then
echo "verify-linter-pin: $resolved --version failed; the binary" \
"cannot be executed or timed out (pin is $pin)" >&2
exit 1
fi
found="$(
echo "$out" | awk '
{
for (i = 1; i < NF; i++) {
if ($i == "version") {
v = $(i + 1)
sub(/^v/, "", v)
print v
exit
}
}
}
'
)"
if [ "$found" != "$pin" ]; then
echo "verify-linter-pin: $resolved reports" \
"${found:-no parseable version}, but script/bootstrap pins" \
"$pin" >&2
echo "verify-linter-pin: these must be the same version — bump the" \
"Dockerfile lint stage image and GOLANGCI_LINT_VERSION in" \
"script/bootstrap together" >&2
exit 1
fi
echo "verify-linter-pin: $resolved is $found, matching the" \
"script/bootstrap pin"
}
main "$@"
+8
View File
@@ -0,0 +1,8 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
prettier@3.8.1:
version "3.8.1"
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==