fatalf called os.Exit(1), which does not run deferred functions, so
every defer db.Close() was dead on the fatal path: the SQLite WAL was
left uncheckpointed and the -wal/-shm sidecars were left for the next
process to recover. It also made those paths impossible to exercise
in-process.
fatalf is gone. runScan, runReport, runTrees, loadRecords and
resolveRoots return their errors, so the deferred close always runs,
and the only exit point is run() in main.go.
Mapping errors to exit codes needs care: cobra prints the error and
the command's usage text for anything RunE returns, and main mapped
every Execute() error to exit 2. A runtime failure is not a usage
problem, so the runE adapter silences both for the subcommands and
marks their errors fatalError; run() reports a fatalError as
"sfdupes: ..." on stderr and exits 1, and leaves everything else --
cobra's own argument, flag and unknown-command errors, which cobra has
already reported with its usage text -- on exit 2. The bare
"sfdupes" invocation still prints usage and exits 2.
Exit codes and message text are unchanged: 0 on success even with
per-file warnings, 1 fatal, 2 usage, per README section "Error
handling and exit codes". Everything on stdout is still data only.
main_test.go drives the CLI in-process and covers all three: a fatal
error raised after the database is open (a database with no files
table) closes it and leaves no -wal or -shm behind for scan, report
and trees; a nonexistent PATH operand is fatal, not usage, and prints
no usage text; the usage errors still exit 2; and a scan that skipped
an unreadable file still exits 0.
The Next Step section still named the scripts-to-rule-them-all
conversion, which landed in 3abeacf on 2026-07-26 and closed#1
without recording itself in Completed Steps. TODO.md therefore
pointed the next contributor at finished work.
The backlog is no longer file-shaped: the Gitea tracker is
authoritative, with the open issues under the 1.0.0 milestone
defining what remains before the tag. Next Step now names the
milestone instead of restating a single issue that will drift, and
Status says so explicitly.
Workflow is reconciled with how work actually happens now: take an
issue, branch, implement with tests, record the result in the same
commit, open a PR titled "... (closes #N)", pass an independent
review, merge.
The scripts-to-rule-them-all conversion gains its missing Completed
Steps entry, and the golangci-lint entry now records its merge
commit 38a01bd and issue #3 rather than only the branch date.
Docs-only: TODO.md is the only file touched.
Bump the pinned golangci-lint from v2.12.1 to v2.12.2 in the
Dockerfile lint stage (tagged, digest-pinned Debian image) and in
script/bootstrap (go install ref). Replace .golangci.yml with the
canonical config: linter settings (lll, funlen, cyclop, dupl) move
under linters.settings per the v2 schema so they are actually
applied, and the redundant issues.exclude-use-default key is
dropped. No new lint findings surfaced; make check is green.
Plain make now builds sfdupes (previously the default was all =
check + build); make build remains as an alias, so the Dockerfile
and existing habits keep working. The sfdupes target is phony: go
build's own cache decides what to recompile.
All PATH operands belong to a single scan: every operand seeds the
shared walk worker pool, and each pass (walk, stat, hash, update)
runs exactly once over the whole scan, so pass totals, percentages,
and ETAs are scan-global. The per-operand walk/hash/update cycles and
their stderr operand announcements are gone; duplicate paths from
overlapping operands are deduplicated before stat.
The update pass now commits in batched transactions (10k changes per
batch) instead of one scan-wide transaction: the filesystem is
authoritative and the database is an eventually-consistent reflection
of it, so scan-level atomicity buys nothing, while batches keep the
WAL small and let concurrent reports observe progress.
Phases are strictly sequential again — walk, stat, hash, update per
operand — with parallelism only inside each phase. The walk
enumerates paths with per-directory workers (no lstat of file
entries); the stat pass lstats every collected path with per-file
workers, restoring its exact-total/ETA progress bar and per-file
parallelism inside wide flat directories.
With per-operand walk/hash/update cycles, a multi-operand invocation
(e.g. scan /srv/*) showed pass totals that looked like the whole
run's: an operator watching operand 3 of 14 hash 300k files concluded
the other 20M files were being skipped. Print the operand path and
its position before each cycle.
Replace the single-goroutine WalkDir traversal with a per-directory
worker pool: workers read directories concurrently and lstat entries
while each directory is fresh in cache, recording size and mtime
during the walk. This folds the separate stat pass away (halving
metadata I/O per run) and overlaps metadata latency, which dominated
on busy pools — a sequential walk of a ~22M-file tree was observed
taking over 4 hours.
Each PATH operand now loads its scope, walks, hashes, and commits in
its own transaction, so an interrupted scan keeps every operand
completed so far; a later overlapping operand sees the records
committed by earlier ones and reuses them unchanged.
The walk pass is a single goroutine; on a busy ZFS pool it manages
only a few thousand directory entries per second and takes hours at
~20M files. Respecify it as a worker-pool traversal that reads
directories concurrently and records size/mtime during the walk,
folding away the separate stat pass and halving metadata I/O. Each
PATH operand now commits in its own transaction so an interrupted
scan keeps the operands completed so far.
scan now synchronizes a database that survives between runs
(SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) instead of
emitting a stream: operands are resolved to absolute paths, unchanged
files (same size, mtime not newer than recorded) are never re-read, new
and changed files are hashed, and records under the scanned operands
that were not verified this run are deleted; records outside the
operands are untouched. All changes commit in a single transaction, and
WAL journaling with a busy timeout keeps a report run during a cron
scan safe.
report and trees read the database (no positional arguments); the
NUL-terminated stream format, its parser, and the malformed-record
handling are gone. The driver is modernc.org/sqlite (pure Go), so
builds keep cgo disabled.
scan will maintain a persistent database of file signatures
(default /var/lib/sfdupes/db.sqlite, overridable via
SFDUPES_DATABASE) that survives between runs; rescans hash only new
or changed files (mtime/size) and remove records for vanished files,
so scan can be cronned daily. report and trees will read the
database instead of a scan stream.
scan now takes one or more PATH operands (directories or regular
files) via cobra flags instead of the -root flag with its /srv
default; invoking scan with no operand is a usage error and a
nonexistent operand is fatal. Filesystem boundaries are crossed by
default; the new -x/--one-file-system flag (GNU du/rsync convention)
stops the walk at each operand's filesystem, implemented by comparing
lstat device IDs with build-tagged helpers for darwin's int32 Dev.
Verified against a real mounted disk image: default crosses, -x does
not, --one-file-system is identical to -x.
