Implement persistent SQLite scan database
All checks were successful
check / check (push) Successful in 3s

scan now synchronizes a database that survives between runs
(SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) instead of
emitting a stream: operands are resolved to absolute paths, unchanged
files (same size, mtime not newer than recorded) are never re-read, new
and changed files are hashed, and records under the scanned operands
that were not verified this run are deleted; records outside the
operands are untouched. All changes commit in a single transaction, and
WAL journaling with a busy timeout keeps a report run during a cron
scan safe.

report and trees read the database (no positional arguments); the
NUL-terminated stream format, its parser, and the malformed-record
handling are gone. The driver is modernc.org/sqlite (pure Go), so
builds keep cgo disabled.
This commit is contained in:
2026-07-24 03:08:49 +07:00
parent e0d578a707
commit d8fbcb32c2
12 changed files with 1229 additions and 379 deletions

View File

@@ -28,19 +28,14 @@ type treeNode struct {
totalSize int64
}
// runTrees implements the trees subcommand: it reads a scan stream from
// the named file (or stdin when absent or "-"), reconstructs the
// directory hierarchy from the record paths, computes a Merkle-style
// digest per directory, and prints maximal duplicate-tree groups as TSV
// on stdout. It never touches the scanned filesystem; its only I/O is
// the scan input, stdout, and stderr. args holds the positional
// arguments already validated by cobra (at most one).
func runTrees(args []string) {
in, name, closer := openScanInput(args)
defer closer()
recs, malformed := parseScanStream(in, name)
records := len(recs) + malformed
// runTrees implements the trees subcommand: it reads every record from
// the database, reconstructs the directory hierarchy from the record
// paths, computes a Merkle-style digest per directory, and prints
// maximal duplicate-tree groups as TSV on stdout. It never touches the
// scanned filesystem; its only I/O is the database, stdout, and
// stderr.
func runTrees() {
recs := loadRecords()
super, allDirs := buildHierarchy(recs)
super.compute()
@@ -78,9 +73,9 @@ func runTrees(args []string) {
}
fmt.Fprintf(os.Stderr,
"trees: %d records read%s, %d duplicate tree groups, %d dupe trees, %s reclaimable\n",
records, malformedNote(malformed), len(dupes), dupeTrees,
humanBytes(reclaimable))
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
"%s reclaimable\n",
len(recs), len(dupes), dupeTrees, humanBytes(reclaimable))
}
// buildHierarchy reconstructs the directory hierarchy from the record