Keep the record count and total size in a totals table (closes #146) #148

Merged
clawbot merged 1 commits from issue-146-stored-totals into next 2026-10-27 18:59:37 +01:00
11 changed files with 322 additions and 83 deletions
+66 -29
View File
@@ -393,6 +393,20 @@ All four subcommands operate on a single SQLite database file:
content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
) WITHOUT ROWID;
CREATE INDEX files_signature ON files (size, head, tail, content);
CREATE TABLE totals (
records INTEGER NOT NULL, -- number of records in files
size INTEGER NOT NULL -- sizes of all records in files added up
);
INSERT INTO totals (records, size) VALUES (0, 0);
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
UPDATE totals SET records = records + 1, size = size + NEW.size;
END;
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
UPDATE totals SET records = records - 1, size = size - OLD.size;
END;
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
UPDATE totals SET size = size + NEW.size - OLD.size;
END;
```
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
@@ -406,6 +420,17 @@ All four subcommands operate on a single SQLite database file:
`files_signature` index lets `report` read the records in signature order
without sorting the table.
`totals` holds one row: the number of records in `files` and the sum of
their sizes. Three triggers on `files` keep that row current. Each runs in
the transaction of the change to `files` that fires it, so the row and the
records always agree. `info` reads both numbers from this row. `scan`,
`report` and `trees` read the number of records from it as the total of
their progress displays. Counting the records in `files` would read every
record, which takes minutes on a database of tens of millions. The triggers
make each write to `files` slower, so a first scan of many small files takes
about two fifths longer. On large files, where reading them takes most of
the time, the difference is small.
### Duplicate detection
Two files are duplicates when they have the same size and the same three hashes.
@@ -688,14 +713,18 @@ Processing:
database state.
`report` runs in two phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in signature order, largest size first. That is the order of the
`files_signature` index read backwards, so SQLite sorts nothing and returns the
first record at once. The records sharing a signature come together, and so do
the records of one size, so `report` holds only the duplicate groups of the size
it is reading, and writes them, ordered by `first` path, once it reaches a
smaller size. Its memory holds at most the duplicate paths of one size at a
time, never every record.
"Progress"):
- `open` opens the database.
- `read` gets the number of records from `totals`, then reads the records in
signature order, largest size first.
Signature order, largest size first, is the `files_signature` index read
backwards, so SQLite sorts nothing and returns the first record at once. The
records sharing a signature come together, and so do the records of one size, so
`report` holds only the duplicate groups of the size it is reading, and writes
them, ordered by `first` path, once it reaches a smaller size. Its memory holds
at most the duplicate paths of one size at a time, never every record.
#### Report output format
@@ -769,10 +798,13 @@ Processing:
ascending. Output must be fully deterministic for a given input.
`trees` runs in four phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in path order, building the hierarchy and computing the digests as it goes;
`group` groups the directories by digest and picks the maximal groups; `write`
writes the groups to stdout.
"Progress"):
- `open` opens the database.
- `read` gets the number of records from `totals`, then reads the records in
path order, building the hierarchy and computing the digests as it goes.
- `group` groups the directories by digest and picks the maximal groups.
- `write` writes the groups to stdout.
#### Trees output format
@@ -819,15 +851,19 @@ size of recorded files 150000000000000 (136.4 TiB)
database is in WAL mode, the newest changes may be in the `-wal` file alone.
- `schema version` is the database's `PRAGMA user_version`. A database with
another version than this build's is refused, as by `report`.
- `records` is the number of records. `size of recorded files` adds up the sizes
of all records, in bytes, then in human units.
- `records` is the number of records. `size of recorded files` is the sum of the
sizes of all records, in bytes, then in human units.
`info` runs in two phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `count` reads the schema version and
counts the records with one query, a single pass over the table, both in one
read-only transaction, so the numbers agree with each other while a scan is
committing. `info` writes its lines once the `count` display has ended, so on a
terminal they come after the displays.
"Progress"):
- `open` opens the database.
- `count` reads the schema version and the row of `totals` (see "Database").
Both reads run in one read-only transaction, so the numbers agree with each
other while a scan is committing.
`info` writes its lines once the `count` display has ended, so on a terminal
they come after the displays.
### Progress
@@ -843,8 +879,9 @@ progress, its elapsed time and its time remaining.
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
`[4/4] update`):
- `load` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `load` gets the number of records from `totals` first, then reads the records
against that number. Until it has the number, the display says it is counting
the records.
- `walk` has no exact total. Its estimate is the number of records the last scan
left under the operands, known once `load` is done and shown with a `~`. The
time remaining is unknown once the walk passes the estimate, and when there
@@ -857,8 +894,9 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `read` gets the number of records from `totals` first, then reads the records
against that number. Until it has the number, the display says it is counting
the records.
- `group` has the directories `read` found as its total, and `write` the
duplicate-tree groups.
@@ -866,17 +904,16 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count, writing
the rows to stdout as it goes. While the count runs, the display says it is
counting the records.
- `read` gets the number of records from `totals` first, then reads the records
against that number, writing the rows to stdout as it goes. Until it has the
number, the display says it is counting the records.
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `count` has no total either, because one query counts everything: the display
says it is counting the records until the query returns, and its time
remaining is unknown.
- `count` has no total either, because it reads one row: the display says it is
counting the records until it has the row, and its time remaining is unknown.
Each display is one line holding:
+8
View File
@@ -29,6 +29,14 @@
# Completed Steps
- the database keeps the number of records and the sum of their sizes in the one
row of a `totals` table, which three triggers on `files` keep current. `info`
reads that row, and so do `scan`'s `load` phase and the `read` phases of
`report` and `trees` for their totals; no count reads every record any more.
The triggers make each write to `files` slower. `scan` keeps SQLite's
temporary data in memory, which wins back part of that time (2026-10-27,
https://git.eeqj.de/sneak/sfdupes/issues/146)
- an operand at or inside a directory named `.zfs` is never scanned, whether
given alone or under another operand. `scan` warns about it, counts it as
skipped and does not walk it (2026-10-26,
+63 -17
View File
@@ -69,6 +69,49 @@ const createIndexSQL = `
CREATE INDEX files_signature ON files (size, head, tail, content)
`
// createTotalsSQL creates the totals table, which holds one row: the
// number of records and the sizes of all records added up. Counting
// them over files would read every record. The three triggers below
// keep the row current, in the transaction of each change to files.
const createTotalsSQL = `
CREATE TABLE totals (
records INTEGER NOT NULL,
size INTEGER NOT NULL
)
`
// insertTotalsSQL adds the one row of totals, for an empty files table.
const insertTotalsSQL = `
INSERT INTO totals (records, size) VALUES (0, 0)
`
// totalsAfterInsertSQL counts each inserted record into totals.
const totalsAfterInsertSQL = `
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
UPDATE totals SET records = records + 1, size = size + NEW.size;
END
`
// totalsAfterDeleteSQL takes each deleted record out of totals.
const totalsAfterDeleteSQL = `
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
UPDATE totals SET records = records - 1, size = size - OLD.size;
END
`
// totalsAfterUpdateSQL replaces a record's old size in totals with its
// new one. scan's upsert of an existing record is such an update.
const totalsAfterUpdateSQL = `
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
UPDATE totals SET size = size + NEW.size - OLD.size;
END
`
// recordCountSQL reads the number of records from totals.
const recordCountSQL = `
SELECT records FROM totals
`
// upsertSQL inserts one file record, replacing any existing record for
// the same path.
const upsertSQL = `
@@ -150,9 +193,14 @@ func createDatabaseDirectory(path string) error {
// scanParams are the connection parameters for scan: read-write, with
// WAL journaling and a busy timeout, so a report can run while a cron
// scan is in progress. closeScanDatabase leaves WAL mode again.
//
// Each write to files fires a trigger, so SQLite keeps a statement
// journal for it. temp_store(MEMORY) keeps that journal in memory,
// which costs far less than a temporary file.
const scanParams = "_pragma=busy_timeout(10000)" +
"&_pragma=journal_mode(WAL)" +
"&_pragma=synchronous(NORMAL)"
"&_pragma=synchronous(NORMAL)" +
"&_pragma=temp_store(MEMORY)"
// reportParams are the connection parameters for report, trees and
// info: read-only, with the same busy timeout. They set no journal
@@ -426,14 +474,14 @@ func createSchema(ctx context.Context, db *sql.DB) error {
defer func() { _ = tx.Rollback() }()
_, err = tx.ExecContext(ctx, createTableSQL)
if err != nil {
return fmt.Errorf("create schema: %w", err)
}
_, err = tx.ExecContext(ctx, createIndexSQL)
if err != nil {
return fmt.Errorf("create schema: %w", err)
for _, stmt := range []string{
createTableSQL, createIndexSQL, createTotalsSQL, insertTotalsSQL,
totalsAfterInsertSQL, totalsAfterDeleteSQL, totalsAfterUpdateSQL,
} {
_, err = tx.ExecContext(ctx, stmt)
if err != nil {
return fmt.Errorf("create schema: %w", err)
}
}
_, err = tx.ExecContext(ctx,
@@ -480,7 +528,7 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
var n int64
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
if err != nil {
return fmt.Errorf("count records: %w", err)
}
@@ -556,7 +604,7 @@ func loadSignatureRows(ctx context.Context, db *sql.DB,
var n int64
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
if err != nil {
return fmt.Errorf("count records: %w", err)
}
@@ -622,7 +670,7 @@ func (r *scanRec) setHashes(head, tail, content []byte) error {
func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
var n int64
err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
err := db.QueryRowContext(ctx, recordCountSQL).Scan(&n)
if err != nil {
return 0, fmt.Errorf("count records: %w", err)
}
@@ -630,12 +678,10 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
return n, nil
}
// databaseCountsSQL counts the records and adds up their sizes, in one
// pass over the table. A SUM over no rows is NULL, so on an empty table
// the SUM falls back to 0.
// databaseCountsSQL reads the number of records and the sizes of all
// records added up from totals.
const databaseCountsSQL = `
SELECT COUNT(*), COALESCE(SUM(size), 0)
FROM files
SELECT records, size FROM totals
`
// databaseCounts is what info reads from the database: its schema
+157 -10
View File
@@ -535,17 +535,12 @@ func TestLoadRowsRefuseWrongSizeHash(t *testing.T) {
}
}
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
// from the files_signature index alone, with nothing to sort: the
// first record comes at once, so report's read phase counts each
// record as it comes rather than waiting on SQLite for all of them.
func TestSignatureRowsNeedNoSort(t *testing.T) {
t.Parallel()
// queryPlan returns the detail of each line of SQLite's EXPLAIN QUERY
// PLAN for query.
func queryPlan(t *testing.T, db *sql.DB, query string) []string {
t.Helper()
db := openTestDB(t)
rows, err := db.QueryContext(t.Context(),
"EXPLAIN QUERY PLAN "+signatureRowsSQL)
rows, err := db.QueryContext(t.Context(), "EXPLAIN QUERY PLAN "+query)
if err != nil {
t.Fatal(err)
}
@@ -573,12 +568,164 @@ func TestSignatureRowsNeedNoSort(t *testing.T) {
t.Fatal(err)
}
return plan
}
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
// from the files_signature index alone, with nothing to sort: the
// first record comes at once, so report's read phase counts each
// record as it comes rather than waiting on SQLite for all of them.
func TestSignatureRowsNeedNoSort(t *testing.T) {
t.Parallel()
db := openTestDB(t)
plan := queryPlan(t, db, signatureRowsSQL)
want := []string{"SCAN files USING COVERING INDEX files_signature"}
if !slices.Equal(plan, want) {
t.Errorf("query plan = %q, want %q", plan, want)
}
}
// TestCountsReadOnlyTotals checks that SQLite answers the queries for
// the number of records and their total size from the one row of
// totals. A plan that reads files or its index reads every record.
func TestCountsReadOnlyTotals(t *testing.T) {
t.Parallel()
db := openTestDB(t)
for _, query := range []string{recordCountSQL, databaseCountsSQL} {
plan := queryPlan(t, db, query)
want := []string{"SCAN totals"}
if !slices.Equal(plan, want) {
t.Errorf("query plan of %q = %q, want %q", query, plan, want)
}
}
}
// TestCountsComeFromTotals sets the row of totals to numbers that files
// does not hold, then checks that scan's load phase, the read phases of
// trees and report, and info each report the numbers from totals.
func TestCountsComeFromTotals(t *testing.T) {
t.Parallel()
// files is empty, so a count over files gives 0 records of 0 bytes.
const records, size = 7, 9000
db := openTestDB(t)
_, err := db.ExecContext(t.Context(),
"UPDATE totals SET records = ?, size = ?", records, size)
if err != nil {
t.Fatal(err)
}
scanTotal, err := countRecords(t.Context(), db)
if err != nil {
t.Fatal(err)
}
var treesTotal, reportTotal int64
err = loadFileRows(t.Context(), db,
func(n int64) { treesTotal = n }, func(scanRec) {})
if err != nil {
t.Fatal(err)
}
err = loadSignatureRows(t.Context(), db,
func(n int64) { reportTotal = n },
func(scanRec) error { return nil })
if err != nil {
t.Fatal(err)
}
info, err := readDatabaseCounts(t.Context(), db)
if err != nil {
t.Fatal(err)
}
for name, got := range map[string]int64{
"scan's load": scanTotal,
"trees' read": treesTotal,
"report's read": reportTotal,
"info": info.records,
} {
if got != records {
t.Errorf("%s: %d records, want %d from totals",
name, got, records)
}
}
if info.size != size {
t.Errorf("info: size %d, want %d from totals", info.size, size)
}
}
// assertTotalsMatchFiles fails unless the row of totals holds the
// number of records and the sizes of all records added up, as counted
// over files.
func assertTotalsMatchFiles(t *testing.T, db *sql.DB) {
t.Helper()
got, err := readDatabaseCounts(t.Context(), db)
if err != nil {
t.Fatal(err)
}
var records, size int64
err = db.QueryRowContext(t.Context(),
"SELECT COUNT(*), COALESCE(SUM(size), 0) FROM files").
Scan(&records, &size)
if err != nil {
t.Fatal(err)
}
if got.records != records || got.size != size {
t.Errorf("totals = %d records of %d bytes, files holds %d of %d",
got.records, got.size, records, size)
}
}
// TestTotalsMatchFiles scans the smoke tree, rescans it after two files
// change size, and rescans it after two files are removed. After each
// scan the row of totals must agree with files.
func TestTotalsMatchFiles(t *testing.T) {
t.Parallel()
dir := buildSmokeTree(t)
db := openTestDB(t)
syncTree(t, db, dir)
assertTotalsMatchFiles(t, db)
writeFile(t, dir, "a/one.bin", pattern(10, 2500))
writeFile(t, dir, "t1/f1", pattern(30, 1000))
if st := syncTree(t, db, dir); st.updated != 2 {
t.Fatalf("rescan stats = %+v, want 2 updated", st)
}
assertTotalsMatchFiles(t, db)
for _, name := range []string{"a/unique.bin", "t2/sub/f2"} {
err := os.Remove(filepath.Join(dir, name))
if err != nil {
t.Fatal(err)
}
}
if st := syncTree(t, db, dir); st.removed != 2 {
t.Fatalf("rescan stats = %+v, want 2 removed", st)
}
assertTotalsMatchFiles(t, db)
}
func TestApplyChangesRoundTrip(t *testing.T) {
t.Parallel()
+6 -6
View File
@@ -26,11 +26,11 @@ func infoPhase(name string) string {
// it. Any database problem, including a missing database, is fatal.
//
// The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and count (count the records,
// with one query). Each display ends before the next phase starts, and
// before an error is returned to be printed. The details are written
// once the last display has ended, so no display is drawn while they
// are written.
// own on stderr: open (open the database) and count (read the number of
// records and their total size from the totals table). Each display
// ends before the next phase starts, and before an error is returned
// to be printed. The details are written once the last display has
// ended, so no display is drawn while they are written.
func runInfo(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
@@ -56,7 +56,7 @@ func runInfo(ctx context.Context, stdout io.Writer) error {
defer func() { _ = db.Close() }()
// One query counts everything, so the phase never has a total.
// One row holds both counts, so the phase never has a total.
prog = newProgress(progressParams{
label: infoPhase("count"), total: -1,
waitingFor: countingRecords,
+6 -5
View File
@@ -180,9 +180,9 @@ func capture(t *testing.T, std **os.File) func() string {
}
// brokenDatabase writes a database that opens cleanly and passes the
// schema-version check but has no files table, so the first query
// fails with the database already open: a fatal error on a path that
// owns an open database. It closes the database the way scan does.
// schema-version check but has no tables, so the first query fails
// with the database already open: a fatal error on a path that owns an
// open database. It closes the database the way scan does.
func brokenDatabase(t *testing.T) string {
t.Helper()
@@ -265,8 +265,9 @@ func TestRunFatalAfterOpenClosesDatabase(t *testing.T) {
assertFatalOutput(t, stderr.String(), stdout())
// Proof that the failure happened after the open: only a
// query against the opened database can report this.
if !strings.Contains(stderr.String(), "no such table: files") {
// query against the opened database can report this. Each
// subcommand's first query reads totals.
if !strings.Contains(stderr.String(), "no such table: totals") {
t.Errorf("stderr = %q, want the failure to come from a "+
"query on the open database", stderr.String())
}
+1 -1
View File
@@ -60,7 +60,7 @@ const fallbackWidth = 80
const spinnerMargin = 5
// countingRecords is what the display of a phase that reads the records
// says while it counts them, before it has its total.
// says until it has the number of records, its total.
const countingRecords = "counting the records"
// openPhase is the name of the phase in which report, trees and info
+5 -5
View File
@@ -45,11 +45,11 @@ func reportPhase(name string) string {
// problem, including a missing database, is fatal.
//
// The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and read (count the records,
// then read them in signature order, writing each size's duplicate
// groups to stdout once that size is read; see readReport). Each
// display ends before the next phase starts, and before an error is
// returned to be printed.
// own on stderr: open (open the database) and read (read the number of
// records from the totals table, then read the records in signature
// order, writing each size's duplicate groups to stdout once that size
// is read; see readReport). Each display ends before the next phase
// starts, and before an error is returned to be printed.
func runReport(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
+2 -2
View File
@@ -455,8 +455,8 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
// included, so its display counts against the number of records.
func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// Indexing tens of millions of records takes real time; without a
// display the scan looks hung before the walk begins. Counting them
// first gives the display its total.
// display the scan looks hung before the walk begins. The number of
// records, read first, gives the display its total.
prog := newProgress(progressParams{
label: scanPhase("load"), unit: scanUnit, total: -1,
waitingFor: countingRecords, stalledOn: "db",
+3 -3
View File
@@ -1228,9 +1228,9 @@ func TestScanPipeline(t *testing.T) {
// TestScanProgressNamesEachPhase checks the plain progress lines of a
// first scan, and of a rescan of one operand holding two of the
// database's records. Each phase's first line names it with its place
// among the four: load counts the records before reading them, and the
// walk estimates its total from the records under its operand, or says
// its time remaining is unknown when there are none.
// among the four: load gets the number of records before reading them,
// and the walk estimates its total from the records under its operand,
// or says its time remaining is unknown when there are none.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanProgressNamesEachPhase(t *testing.T) {
+5 -5
View File
@@ -40,11 +40,11 @@ func treesPhase(name string) string {
// database problem, including a missing database, is fatal.
//
// The work runs in four phases, each under a progress display of its
// own on stderr: open (open the database), read (count the records,
// then read them into the hierarchy), group (group the directories by
// digest) and write (write the groups to stdout). Each display ends
// before the next phase starts, and before an error is returned to be
// printed.
// own on stderr: open (open the database), read (read the number of
// records from the totals table, then read the records into the
// hierarchy), group (group the directories by digest) and write (write
// the groups to stdout). Each display ends before the next phase
// starts, and before an error is returned to be printed.
func runTrees(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {