Implement persistent SQLite scan database
All checks were successful
check / check (push) Successful in 3s
All checks were successful
check / check (push) Successful in 3s
scan now synchronizes a database that survives between runs (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) instead of emitting a stream: operands are resolved to absolute paths, unchanged files (same size, mtime not newer than recorded) are never re-read, new and changed files are hashed, and records under the scanned operands that were not verified this run are deleted; records outside the operands are untouched. All changes commit in a single transaction, and WAL journaling with a busy timeout keeps a report run during a cron scan safe. report and trees read the database (no positional arguments); the NUL-terminated stream format, its parser, and the malformed-record handling are gone. The driver is modernc.org/sqlite (pure Go), so builds keep cgo disabled.
This commit is contained in:
27
trees.go
27
trees.go
@@ -28,19 +28,14 @@ type treeNode struct {
|
||||
totalSize int64
|
||||
}
|
||||
|
||||
// runTrees implements the trees subcommand: it reads a scan stream from
|
||||
// the named file (or stdin when absent or "-"), reconstructs the
|
||||
// directory hierarchy from the record paths, computes a Merkle-style
|
||||
// digest per directory, and prints maximal duplicate-tree groups as TSV
|
||||
// on stdout. It never touches the scanned filesystem; its only I/O is
|
||||
// the scan input, stdout, and stderr. args holds the positional
|
||||
// arguments already validated by cobra (at most one).
|
||||
func runTrees(args []string) {
|
||||
in, name, closer := openScanInput(args)
|
||||
defer closer()
|
||||
|
||||
recs, malformed := parseScanStream(in, name)
|
||||
records := len(recs) + malformed
|
||||
// runTrees implements the trees subcommand: it reads every record from
|
||||
// the database, reconstructs the directory hierarchy from the record
|
||||
// paths, computes a Merkle-style digest per directory, and prints
|
||||
// maximal duplicate-tree groups as TSV on stdout. It never touches the
|
||||
// scanned filesystem; its only I/O is the database, stdout, and
|
||||
// stderr.
|
||||
func runTrees() {
|
||||
recs := loadRecords()
|
||||
|
||||
super, allDirs := buildHierarchy(recs)
|
||||
super.compute()
|
||||
@@ -78,9 +73,9 @@ func runTrees(args []string) {
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr,
|
||||
"trees: %d records read%s, %d duplicate tree groups, %d dupe trees, %s reclaimable\n",
|
||||
records, malformedNote(malformed), len(dupes), dupeTrees,
|
||||
humanBytes(reclaimable))
|
||||
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
|
||||
"%s reclaimable\n",
|
||||
len(recs), len(dupes), dupeTrees, humanBytes(reclaimable))
|
||||
}
|
||||
|
||||
// buildHierarchy reconstructs the directory hierarchy from the record
|
||||
|
||||
Reference in New Issue
Block a user