The rate was the phase's average, so a burst after a long wait read as
over 1,000 files a second. It now covers the items completed over the
last 5 seconds, or since the line before off a terminal; the ETA still
comes from the average. Once no item has completed for 10 seconds, the
rate gives way to "stalled" and where the phase waits, as "stalled
disk/db": disk, db, out (stdout) or cpu, or two of them. A line too
wide for the terminal loses its bar first, then its rate or those
words, so the phase, progress, elapsed time and ETA stay whole. The
content phase's last partial batch is now written under a display of
its own. The tests set the display's clock by hand.
Deviation: the stall words hold no time and name where the phase waits, not what it does.
Judgement call: trees' group phase, which works in memory, says "cpu".
Model: opus-5-5
info prints the database's path and source, the sizes of the database
file and its -wal file, when the database file last changed, the schema
version and the record counts, one per line as a name, a tab and the
value. It opens the database as report does and shows an open and a
count phase on stderr. The schema version and the counts come from one
read-only transaction, the counts from one query.
seedDatabase now closes test databases out of WAL mode, as scan does;
otherwise info's own connection creates an empty -wal file. The open
phase's name and text became constants for the linter, and the scan
command moved to newScanCommand to keep newRootCommand short enough.
Judgement call: the lines are written after the last display ends, so they skip the display's stdout writer.
Model: opus-5-5
report now runs in two phases, open and read, under the progress
display scan and trees use, until the summary line. open covers a wait
on a busy database. read counts the records, then reads them in the
same read-only transaction in signature order, the files_signature
index read backwards, so SQLite sorts nothing and the read ends at
exactly its count. The duplicate groups are formed in Go from runs of
equal signatures; only those of the size being read are held, and they
are written, ordered by first path, before the next size. The rows go
through the display's stdout writer. stdout and the summary line are
unchanged.
Judgement call: writing is part of read, not a phase of its own, since each size's rows are written while the read goes on.
Model: opus-5-5
scan recorded mtime in whole seconds, so a file rewritten in place at
the same size within the same second as its recorded mtime was classed
unchanged and kept its old hashes. The files table keeps mtime as whole
Unix seconds and gains mtime_nsec, the nanoseconds within that second.
scan holds the mtime as a time.Time and decides "newer" by comparing
Unix() and then Nanosecond(), so any time a filesystem can record
compares in the right order; After would misorder one too late for a
time.Time to hold without wrapping. The walk, a file given as an
operand, and the content phase's recheck all move over. PRAGMA
user_version stays 1, per the owner's ruling. README states what both
columns hold.
Model: opus-5-5
report now has SQLite group the records and put the rows in report
order, helped by a new files_signature index on (size, head, tail,
content), and writes each row as it reads it. trees reads the records
in path order, where all the paths under a directory come together, so
it computes each directory's digest as soon as the stream leaves it and
keeps only its path, parent, digest and totals. Output is unchanged.
The tests that called the removed in-memory grouping functions now group
records stored in a database. New tests check that both commands give
the same output whatever order the records were inserted in, and that a
stdout failure partway through a long report is reported as one.
Model: opus-5-5
report and trees already checked every stdout write and the final
flush. run now takes the stdout it hands to them, so tests pass a
closed file or a failing writer instead of swapping os.Stdout: a
closed stdout exits 1 with a one-line diagnostic, and the writer's
error reaches the caller.
README "Error handling" now states the two cases that never reach
sfdupes as a failed write: a pipe reader that exits early ends the
process with SIGPIPE, as with cat; and stdout closed with >&- is
replaced by /dev/null by the Go runtime before main runs, so the run
succeeds.
Model: opus-5-5
A path holding a tab or newline split a row of the report or trees
output. The path columns of both now write a backslash, tab, newline
and carriage return as \\, \t, \n and \r; every other byte is written
unchanged. Grouping and sorting still use the stored path. Warnings
on stderr are escaped the same way in warnf, so each stays one line.
In trees, the root directory's node now has the path "/" instead of
an empty string, and its children's paths start with a single slash.
README states the rule under "Report output format".
Model: opus-5-5
A file of 10 MiB or more now gets only its 64 KiB head and tail in the
hash phase, so its content is read only when it can be a duplicate. A
new content phase after the update phase finds every group of records,
anywhere in the database, that share size, head and tail and include
one without a content hash. It checks every member with lstat and, when
at least two pass, reads those without a content hash through the
existing worker pool; a stale file does not count as a match. report
and trees leave out records without a content hash. The README, help
text and TODO entry describe the gate; the schema stays at version 1.
Lint suppressed: gosec on the file open in hashContentOnly, as in
hashSignature, and on one chmod in a test.
Model: opus-5-5
scan now synchronizes a database that survives between runs
(SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) instead of
emitting a stream: operands are resolved to absolute paths, unchanged
files (same size, mtime not newer than recorded) are never re-read, new
and changed files are hashed, and records under the scanned operands
that were not verified this run are deleted; records outside the
operands are untouched. All changes commit in a single transaction, and
WAL journaling with a busy timeout keeps a report run during a cron
scan safe.
report and trees read the database (no positional arguments); the
NUL-terminated stream format, its parser, and the malformed-record
handling are gone. The driver is modernc.org/sqlite (pure Go), so
builds keep cgo disabled.
sneak
clawbot