TestSyncScanCancelledWalkKeepsRecords handed syncScan a context that
was already cancelled. loadIndex is the first thing syncScan does, and
its QueryContext fails on that context, so the scan returned before
startWalk was ever called: no walk ran, no pool started, no write path
was reachable, and all three of the test's assertions held for the
wrong reason. The post-walk ctx.Err() guard, which is the highest-stakes
line in the change, had no coverage at all — a panic in its body, or
deleting it outright, left the suite green.
Replace it with TestSyncScanCancelledMidWalkKeepsRecords, which cancels
during the walk and so reaches the guard holding a genuinely partial
census and a still-populated record index. The cancellation is driven
by the scan's own progress rather than by a timer: walkClock is a
context that cancels itself once its Done method has been consulted a
set number of times, and since every blocking channel operation in the
walk selects on Done — one consultation per event, a couple per
directory, against the index load's fixed three — a threshold set to a
quarter of the fixture's file count lands the cancellation deep inside
the walk on every run. The census settles at around 380 of 2000 files,
leaving some 1600 records that a complete-looking census would have
handed to the update phase as deletions.
The already-cancelled case is kept, renamed to what it actually tests
and with its goroutine assertion dropped, since nothing that could leak
is ever started.
Direct tests cover the remaining cancellation branches of both pools:
sendEvent abandoning a blocked send, walk workers dropping queued
directories, a walk worker abandoning its subdirectory hand-off,
dispatchDirs closing jobs on its way out, feedHashJobs doing the same,
hashWorker dropping queued runs, and hashPhase leaving its result loop.
Each is deterministic — the channels involved are unbuffered, unread or
pre-filled, so the cancellation case is the only one that can be ready.
Also correct two overstated claims. The hashLeakFiles comment described
a mechanism that does not occur: the surplus is absorbed exactly by the
two pool channels plus the workers in flight, so the feeder drains and
exits, and what an abandoned pool leaves parked is the workers and the
goroutine waiting on them. And the guard is defence in depth, not the
sole barrier against data loss: the update phase's BeginTx fails on the
same cancelled context before deleting anything today. The guard is
what keeps that true once an interrupted scan is allowed to commit what
it has.
hashPhase returned the moment recordRun failed and left the pool
running: the feeder parked forever on a full jobs channel and every
worker on a full results channel. Until #4 landed the process exited
before that mattered; now that runScan returns an error and unwinds,
the goroutines are a real leak.
The pool is now an owned hashPool. Its context is derived from the
scan's, every blocking send in the feeder and the workers selects on
ctx.Done(), the feeder closes jobs on every path out so the workers'
range always terminates, and hashPhase defers pool.stop(), which
cancels and then drains results until the last goroutine has exited.
Draining is the half that matters: a worker already parked on a send
cannot observe the cancellation until a receiver frees it.
ctx comes from cmd.Context() and is threaded through runScan,
syncScan, both worker pools and the database layer as the first
parameter throughout, so graceful interrupt handling has a path to
hook into rather than a pool to rewrite.
The walk pool never leaked, because walkPhase always drains its
events to close, but it has the same unbounded-send shape and gets
the same treatment, together with a ctx.Err() guard after the walk: a
cancelled walk leaves a partial size census, and the update phase
would read every file it never reached as vanished and delete its
record.
Tests drive the scan entry point against a database whose insert
trigger aborts, with a fixture large enough that the failure lands
partway through the hash phase with more runs queued than either pool
channel can hold, and assert that the scan fails instead of hanging
and that runtime.NumGoroutine polls back to its pre-scan baseline.
fatalf called os.Exit(1), which does not run deferred functions, so
every defer db.Close() was dead on the fatal path: the SQLite WAL was
left uncheckpointed and the -wal/-shm sidecars were left for the next
process to recover. It also made those paths impossible to exercise
in-process.
fatalf is gone. runScan, runReport, runTrees, loadRecords and
resolveRoots return their errors, so the deferred close always runs,
and the only exit point is run() in main.go.
Mapping errors to exit codes needs care: cobra prints the error and
the command's usage text for anything RunE returns, and main mapped
every Execute() error to exit 2. A runtime failure is not a usage
problem, so the runE adapter silences both for the subcommands and
marks their errors fatalError; run() reports a fatalError as
"sfdupes: ..." on stderr and exits 1, and leaves everything else --
cobra's own argument, flag and unknown-command errors, which cobra has
already reported with its usage text -- on exit 2. The bare
"sfdupes" invocation still prints usage and exits 2.
Exit codes and message text are unchanged: 0 on success even with
per-file warnings, 1 fatal, 2 usage, per README section "Error
handling and exit codes". Everything on stdout is still data only.
main_test.go drives the CLI in-process and covers all three: a fatal
error raised after the database is open (a database with no files
table) closes it and leaves no -wal or -shm behind for scan, report
and trees; a nonexistent PATH operand is fatal, not usage, and prints
no usage text; the usage errors still exit 2; and a scan that skipped
an unreadable file still exits 0.
Sort the hash queue by (device, inode) so reads proceed in inode
order, which minimizes seeking on spinning disks. Paths that are hard
links to the same inode form one run: the run is read once and every
path shares the result, so link farms (rsync --link-dest backups)
cost one read per inode instead of one per path. A run that fails to
read skips all of its paths.
Zero-length files have constant head/tail hashes; return them without
opening the file.
The hash progress total now counts actual reads (runs, not paths).
Hard-linked paths still appear in reports as duplicates — their
content is identical — though they share storage; noted in README.
On a database with tens of millions of records, indexing the existing
rows before the walk takes real single-core time with no output,
which is indistinguishable from a hang. Give the load its own
spinner, and render every phase display the moment the phase starts
instead of waiting for its first completed item.
Restructure scan into three phases: walk+stat, hash, update.
The stat pass is folded into the walk workers: each regular file is
lstatted as its directory is read, while the metadata is hot. The
walk builds a scan-wide size census (walked files plus records
outside the scan roots), and unchanged already-hashed files resolve
during the walk without further work.
Only files whose size at least one other file shares are ever read:
a size-unique file cannot be a duplicate, so it is recorded without
hashes (head and tail empty). When a later scan makes its size
shared, the file is hashed then, even if otherwise unchanged. report
excludes unhashed records; trees gives them a never-matching
signature so a tree containing one never compares equal to another.
Hashed records are committed in batched transactions while the hash
phase runs, so an interrupted scan keeps everything hashed so far
and the next run resumes cheaply. The hash phase total is exact,
giving a meaningful ETA.
Memory drops accordingly: the existing-record index holds only path,
size, mtime, and a hashed flag (no hash values); the walk carries one
small record per candidate file; overlapping operands are pruned up
front instead of deduplicating every walked path in a scan-wide set.
Files no bigger than one chunk are hashed with a single read.
All PATH operands belong to a single scan: every operand seeds the
shared walk worker pool, and each pass (walk, stat, hash, update)
runs exactly once over the whole scan, so pass totals, percentages,
and ETAs are scan-global. The per-operand walk/hash/update cycles and
their stderr operand announcements are gone; duplicate paths from
overlapping operands are deduplicated before stat.
The update pass now commits in batched transactions (10k changes per
batch) instead of one scan-wide transaction: the filesystem is
authoritative and the database is an eventually-consistent reflection
of it, so scan-level atomicity buys nothing, while batches keep the
WAL small and let concurrent reports observe progress.
Phases are strictly sequential again — walk, stat, hash, update per
operand — with parallelism only inside each phase. The walk
enumerates paths with per-directory workers (no lstat of file
entries); the stat pass lstats every collected path with per-file
workers, restoring its exact-total/ETA progress bar and per-file
parallelism inside wide flat directories.
With per-operand walk/hash/update cycles, a multi-operand invocation
(e.g. scan /srv/*) showed pass totals that looked like the whole
run's: an operator watching operand 3 of 14 hash 300k files concluded
the other 20M files were being skipped. Print the operand path and
its position before each cycle.
Replace the single-goroutine WalkDir traversal with a per-directory
worker pool: workers read directories concurrently and lstat entries
while each directory is fresh in cache, recording size and mtime
during the walk. This folds the separate stat pass away (halving
metadata I/O per run) and overlaps metadata latency, which dominated
on busy pools — a sequential walk of a ~22M-file tree was observed
taking over 4 hours.
Each PATH operand now loads its scope, walks, hashes, and commits in
its own transaction, so an interrupted scan keeps every operand
completed so far; a later overlapping operand sees the records
committed by earlier ones and reuses them unchanged.
scan now synchronizes a database that survives between runs
(SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) instead of
emitting a stream: operands are resolved to absolute paths, unchanged
files (same size, mtime not newer than recorded) are never re-read, new
and changed files are hashed, and records under the scanned operands
that were not verified this run are deleted; records outside the
operands are untouched. All changes commit in a single transaction, and
WAL journaling with a busy timeout keeps a report run during a cron
scan safe.
report and trees read the database (no positional arguments); the
NUL-terminated stream format, its parser, and the malformed-record
handling are gone. The driver is modernc.org/sqlite (pure Go), so
builds keep cgo disabled.
scan now takes one or more PATH operands (directories or regular
files) via cobra flags instead of the -root flag with its /srv
default; invoking scan with no operand is a usage error and a
nonexistent operand is fatal. Filesystem boundaries are crossed by
default; the new -x/--one-file-system flag (GNU du/rsync convention)
stops the walk at each operand's filesystem, implemented by comparing
lstat device IDs with build-tagged helpers for darwin's int32 Dev.
Verified against a real mounted disk image: default crosses, -x does
not, --one-file-system is identical to -x.
150 findings fixed: linter autofixes for whitespace style (wsl_v5,
nlreturn, noinlineerr), named returns removed, magic numbers replaced
with named constants, static errNotRegular error, explicit Close/Parse
error handling, unused cobra params renamed to _, and runReport/
runTrees/hashPass split into helpers to satisfy cyclop, gocognit, and
funlen. Behavior verified unchanged against the README smoke test.
