diff --git a/README.md b/README.md index 6a4b5c3..7dc8fb3 100644 --- a/README.md +++ b/README.md @@ -393,6 +393,20 @@ All four subcommands operate on a single SQLite database file: content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples ) WITHOUT ROWID; CREATE INDEX files_signature ON files (size, head, tail, content); + CREATE TABLE totals ( + records INTEGER NOT NULL, -- number of records in files + size INTEGER NOT NULL -- sizes of all records in files added up + ); + INSERT INTO totals (records, size) VALUES (0, 0); + CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN + UPDATE totals SET records = records + 1, size = size + NEW.size; + END; + CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN + UPDATE totals SET records = records - 1, size = size - OLD.size; + END; + CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN + UPDATE totals SET size = size + NEW.size - OLD.size; + END; ``` Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed @@ -406,6 +420,17 @@ All four subcommands operate on a single SQLite database file: `files_signature` index lets `report` read the records in signature order without sorting the table. + `totals` holds one row: the number of records in `files` and the sum of + their sizes. Three triggers on `files` keep that row current. Each runs in + the transaction of the change to `files` that fires it, so the row and the + records always agree. `info` reads both numbers from this row. `scan`, + `report` and `trees` read the number of records from it as the total of + their progress displays. Counting the records in `files` would read every + record, which takes minutes on a database of tens of millions. The triggers + make each write to `files` slower, so a first scan of many small files takes + about two fifths longer. On large files, where reading them takes most of + the time, the difference is small. + ### Duplicate detection Two files are duplicates when they have the same size and the same three hashes. @@ -688,14 +713,18 @@ Processing: database state. `report` runs in two phases, each with its own progress display on stderr (see -"Progress"): `open` opens the database; `read` counts the records, then reads -them in signature order, largest size first. That is the order of the -`files_signature` index read backwards, so SQLite sorts nothing and returns the -first record at once. The records sharing a signature come together, and so do -the records of one size, so `report` holds only the duplicate groups of the size -it is reading, and writes them, ordered by `first` path, once it reaches a -smaller size. Its memory holds at most the duplicate paths of one size at a -time, never every record. +"Progress"): + +- `open` opens the database. +- `read` gets the number of records from `totals`, then reads the records in + signature order, largest size first. + +Signature order, largest size first, is the `files_signature` index read +backwards, so SQLite sorts nothing and returns the first record at once. The +records sharing a signature come together, and so do the records of one size, so +`report` holds only the duplicate groups of the size it is reading, and writes +them, ordered by `first` path, once it reaches a smaller size. Its memory holds +at most the duplicate paths of one size at a time, never every record. #### Report output format @@ -769,10 +798,13 @@ Processing: ascending. Output must be fully deterministic for a given input. `trees` runs in four phases, each with its own progress display on stderr (see -"Progress"): `open` opens the database; `read` counts the records, then reads -them in path order, building the hierarchy and computing the digests as it goes; -`group` groups the directories by digest and picks the maximal groups; `write` -writes the groups to stdout. +"Progress"): + +- `open` opens the database. +- `read` gets the number of records from `totals`, then reads the records in + path order, building the hierarchy and computing the digests as it goes. +- `group` groups the directories by digest and picks the maximal groups. +- `write` writes the groups to stdout. #### Trees output format @@ -819,15 +851,19 @@ size of recorded files 150000000000000 (136.4 TiB) database is in WAL mode, the newest changes may be in the `-wal` file alone. - `schema version` is the database's `PRAGMA user_version`. A database with another version than this build's is refused, as by `report`. -- `records` is the number of records. `size of recorded files` adds up the sizes - of all records, in bytes, then in human units. +- `records` is the number of records. `size of recorded files` is the sum of the + sizes of all records, in bytes, then in human units. `info` runs in two phases, each with its own progress display on stderr (see -"Progress"): `open` opens the database; `count` reads the schema version and -counts the records with one query, a single pass over the table, both in one -read-only transaction, so the numbers agree with each other while a scan is -committing. `info` writes its lines once the `count` display has ended, so on a -terminal they come after the displays. +"Progress"): + +- `open` opens the database. +- `count` reads the schema version and the row of `totals` (see "Database"). + Both reads run in one read-only transaction, so the numbers agree with each + other while a scan is committing. + +`info` writes its lines once the `count` display has ended, so on a terminal +they come after the displays. ### Progress @@ -843,8 +879,9 @@ progress, its elapsed time and its time remaining. A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`, `[4/4] update`): -- `load` counts the records first, then reads them against that count. While the - count runs, the display says it is counting the records. +- `load` gets the number of records from `totals` first, then reads the records + against that number. Until it has the number, the display says it is counting + the records. - `walk` has no exact total. Its estimate is the number of records the last scan left under the operands, known once `load` is done and shown with a `~`. The time remaining is unknown once the walk passes the estimate, and when there @@ -857,8 +894,9 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`, - `open` has no total. While it opens the database, waiting on a busy one included, the display says it is opening the database. -- `read` counts the records first, then reads them against that count. While the - count runs, the display says it is counting the records. +- `read` gets the number of records from `totals` first, then reads the records + against that number. Until it has the number, the display says it is counting + the records. - `group` has the directories `read` found as its total, and `write` the duplicate-tree groups. @@ -866,17 +904,16 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`, - `open` has no total. While it opens the database, waiting on a busy one included, the display says it is opening the database. -- `read` counts the records first, then reads them against that count, writing - the rows to stdout as it goes. While the count runs, the display says it is - counting the records. +- `read` gets the number of records from `totals` first, then reads the records + against that number, writing the rows to stdout as it goes. Until it has the + number, the display says it is counting the records. `info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"): - `open` has no total. While it opens the database, waiting on a busy one included, the display says it is opening the database. -- `count` has no total either, because one query counts everything: the display - says it is counting the records until the query returns, and its time - remaining is unknown. +- `count` has no total either, because it reads one row: the display says it is + counting the records until it has the row, and its time remaining is unknown. Each display is one line holding: diff --git a/TODO.md b/TODO.md index def2438..8cff644 100644 --- a/TODO.md +++ b/TODO.md @@ -29,6 +29,14 @@ # Completed Steps +- the database keeps the number of records and the sum of their sizes in the one + row of a `totals` table, which three triggers on `files` keep current. `info` + reads that row, and so do `scan`'s `load` phase and the `read` phases of + `report` and `trees` for their totals; no count reads every record any more. + The triggers make each write to `files` slower. `scan` keeps SQLite's + temporary data in memory, which wins back part of that time (2026-10-27, + https://git.eeqj.de/sneak/sfdupes/issues/146) + - an operand at or inside a directory named `.zfs` is never scanned, whether given alone or under another operand. `scan` warns about it, counts it as skipped and does not walk it (2026-10-26, diff --git a/db.go b/db.go index c7835b4..c878f60 100644 --- a/db.go +++ b/db.go @@ -69,6 +69,49 @@ const createIndexSQL = ` CREATE INDEX files_signature ON files (size, head, tail, content) ` +// createTotalsSQL creates the totals table, which holds one row: the +// number of records and the sizes of all records added up. Counting +// them over files would read every record. The three triggers below +// keep the row current, in the transaction of each change to files. +const createTotalsSQL = ` +CREATE TABLE totals ( + records INTEGER NOT NULL, + size INTEGER NOT NULL +) +` + +// insertTotalsSQL adds the one row of totals, for an empty files table. +const insertTotalsSQL = ` +INSERT INTO totals (records, size) VALUES (0, 0) +` + +// totalsAfterInsertSQL counts each inserted record into totals. +const totalsAfterInsertSQL = ` +CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN + UPDATE totals SET records = records + 1, size = size + NEW.size; +END +` + +// totalsAfterDeleteSQL takes each deleted record out of totals. +const totalsAfterDeleteSQL = ` +CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN + UPDATE totals SET records = records - 1, size = size - OLD.size; +END +` + +// totalsAfterUpdateSQL replaces a record's old size in totals with its +// new one. scan's upsert of an existing record is such an update. +const totalsAfterUpdateSQL = ` +CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN + UPDATE totals SET size = size + NEW.size - OLD.size; +END +` + +// recordCountSQL reads the number of records from totals. +const recordCountSQL = ` +SELECT records FROM totals +` + // upsertSQL inserts one file record, replacing any existing record for // the same path. const upsertSQL = ` @@ -150,9 +193,14 @@ func createDatabaseDirectory(path string) error { // scanParams are the connection parameters for scan: read-write, with // WAL journaling and a busy timeout, so a report can run while a cron // scan is in progress. closeScanDatabase leaves WAL mode again. +// +// Each write to files fires a trigger, so SQLite keeps a statement +// journal for it. temp_store(MEMORY) keeps that journal in memory, +// which costs far less than a temporary file. const scanParams = "_pragma=busy_timeout(10000)" + "&_pragma=journal_mode(WAL)" + - "&_pragma=synchronous(NORMAL)" + "&_pragma=synchronous(NORMAL)" + + "&_pragma=temp_store(MEMORY)" // reportParams are the connection parameters for report, trees and // info: read-only, with the same busy timeout. They set no journal @@ -426,14 +474,14 @@ func createSchema(ctx context.Context, db *sql.DB) error { defer func() { _ = tx.Rollback() }() - _, err = tx.ExecContext(ctx, createTableSQL) - if err != nil { - return fmt.Errorf("create schema: %w", err) - } - - _, err = tx.ExecContext(ctx, createIndexSQL) - if err != nil { - return fmt.Errorf("create schema: %w", err) + for _, stmt := range []string{ + createTableSQL, createIndexSQL, createTotalsSQL, insertTotalsSQL, + totalsAfterInsertSQL, totalsAfterDeleteSQL, totalsAfterUpdateSQL, + } { + _, err = tx.ExecContext(ctx, stmt) + if err != nil { + return fmt.Errorf("create schema: %w", err) + } } _, err = tx.ExecContext(ctx, @@ -480,7 +528,7 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64), var n int64 - err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n) + err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n) if err != nil { return fmt.Errorf("count records: %w", err) } @@ -556,7 +604,7 @@ func loadSignatureRows(ctx context.Context, db *sql.DB, var n int64 - err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n) + err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n) if err != nil { return fmt.Errorf("count records: %w", err) } @@ -622,7 +670,7 @@ func (r *scanRec) setHashes(head, tail, content []byte) error { func countRecords(ctx context.Context, db *sql.DB) (int64, error) { var n int64 - err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n) + err := db.QueryRowContext(ctx, recordCountSQL).Scan(&n) if err != nil { return 0, fmt.Errorf("count records: %w", err) } @@ -630,12 +678,10 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) { return n, nil } -// databaseCountsSQL counts the records and adds up their sizes, in one -// pass over the table. A SUM over no rows is NULL, so on an empty table -// the SUM falls back to 0. +// databaseCountsSQL reads the number of records and the sizes of all +// records added up from totals. const databaseCountsSQL = ` -SELECT COUNT(*), COALESCE(SUM(size), 0) -FROM files +SELECT records, size FROM totals ` // databaseCounts is what info reads from the database: its schema diff --git a/db_test.go b/db_test.go index 9268048..6d9da76 100644 --- a/db_test.go +++ b/db_test.go @@ -535,17 +535,12 @@ func TestLoadRowsRefuseWrongSizeHash(t *testing.T) { } } -// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL -// from the files_signature index alone, with nothing to sort: the -// first record comes at once, so report's read phase counts each -// record as it comes rather than waiting on SQLite for all of them. -func TestSignatureRowsNeedNoSort(t *testing.T) { - t.Parallel() +// queryPlan returns the detail of each line of SQLite's EXPLAIN QUERY +// PLAN for query. +func queryPlan(t *testing.T, db *sql.DB, query string) []string { + t.Helper() - db := openTestDB(t) - - rows, err := db.QueryContext(t.Context(), - "EXPLAIN QUERY PLAN "+signatureRowsSQL) + rows, err := db.QueryContext(t.Context(), "EXPLAIN QUERY PLAN "+query) if err != nil { t.Fatal(err) } @@ -573,12 +568,164 @@ func TestSignatureRowsNeedNoSort(t *testing.T) { t.Fatal(err) } + return plan +} + +// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL +// from the files_signature index alone, with nothing to sort: the +// first record comes at once, so report's read phase counts each +// record as it comes rather than waiting on SQLite for all of them. +func TestSignatureRowsNeedNoSort(t *testing.T) { + t.Parallel() + + db := openTestDB(t) + + plan := queryPlan(t, db, signatureRowsSQL) + want := []string{"SCAN files USING COVERING INDEX files_signature"} if !slices.Equal(plan, want) { t.Errorf("query plan = %q, want %q", plan, want) } } +// TestCountsReadOnlyTotals checks that SQLite answers the queries for +// the number of records and their total size from the one row of +// totals. A plan that reads files or its index reads every record. +func TestCountsReadOnlyTotals(t *testing.T) { + t.Parallel() + + db := openTestDB(t) + + for _, query := range []string{recordCountSQL, databaseCountsSQL} { + plan := queryPlan(t, db, query) + + want := []string{"SCAN totals"} + if !slices.Equal(plan, want) { + t.Errorf("query plan of %q = %q, want %q", query, plan, want) + } + } +} + +// TestCountsComeFromTotals sets the row of totals to numbers that files +// does not hold, then checks that scan's load phase, the read phases of +// trees and report, and info each report the numbers from totals. +func TestCountsComeFromTotals(t *testing.T) { + t.Parallel() + + // files is empty, so a count over files gives 0 records of 0 bytes. + const records, size = 7, 9000 + + db := openTestDB(t) + + _, err := db.ExecContext(t.Context(), + "UPDATE totals SET records = ?, size = ?", records, size) + if err != nil { + t.Fatal(err) + } + + scanTotal, err := countRecords(t.Context(), db) + if err != nil { + t.Fatal(err) + } + + var treesTotal, reportTotal int64 + + err = loadFileRows(t.Context(), db, + func(n int64) { treesTotal = n }, func(scanRec) {}) + if err != nil { + t.Fatal(err) + } + + err = loadSignatureRows(t.Context(), db, + func(n int64) { reportTotal = n }, + func(scanRec) error { return nil }) + if err != nil { + t.Fatal(err) + } + + info, err := readDatabaseCounts(t.Context(), db) + if err != nil { + t.Fatal(err) + } + + for name, got := range map[string]int64{ + "scan's load": scanTotal, + "trees' read": treesTotal, + "report's read": reportTotal, + "info": info.records, + } { + if got != records { + t.Errorf("%s: %d records, want %d from totals", + name, got, records) + } + } + + if info.size != size { + t.Errorf("info: size %d, want %d from totals", info.size, size) + } +} + +// assertTotalsMatchFiles fails unless the row of totals holds the +// number of records and the sizes of all records added up, as counted +// over files. +func assertTotalsMatchFiles(t *testing.T, db *sql.DB) { + t.Helper() + + got, err := readDatabaseCounts(t.Context(), db) + if err != nil { + t.Fatal(err) + } + + var records, size int64 + + err = db.QueryRowContext(t.Context(), + "SELECT COUNT(*), COALESCE(SUM(size), 0) FROM files"). + Scan(&records, &size) + if err != nil { + t.Fatal(err) + } + + if got.records != records || got.size != size { + t.Errorf("totals = %d records of %d bytes, files holds %d of %d", + got.records, got.size, records, size) + } +} + +// TestTotalsMatchFiles scans the smoke tree, rescans it after two files +// change size, and rescans it after two files are removed. After each +// scan the row of totals must agree with files. +func TestTotalsMatchFiles(t *testing.T) { + t.Parallel() + + dir := buildSmokeTree(t) + db := openTestDB(t) + + syncTree(t, db, dir) + assertTotalsMatchFiles(t, db) + + writeFile(t, dir, "a/one.bin", pattern(10, 2500)) + writeFile(t, dir, "t1/f1", pattern(30, 1000)) + + if st := syncTree(t, db, dir); st.updated != 2 { + t.Fatalf("rescan stats = %+v, want 2 updated", st) + } + + assertTotalsMatchFiles(t, db) + + for _, name := range []string{"a/unique.bin", "t2/sub/f2"} { + err := os.Remove(filepath.Join(dir, name)) + if err != nil { + t.Fatal(err) + } + } + + if st := syncTree(t, db, dir); st.removed != 2 { + t.Fatalf("rescan stats = %+v, want 2 removed", st) + } + + assertTotalsMatchFiles(t, db) +} + func TestApplyChangesRoundTrip(t *testing.T) { t.Parallel() diff --git a/info.go b/info.go index 3ec91ff..c30cac8 100644 --- a/info.go +++ b/info.go @@ -26,11 +26,11 @@ func infoPhase(name string) string { // it. Any database problem, including a missing database, is fatal. // // The work runs in two phases, each under a progress display of its -// own on stderr: open (open the database) and count (count the records, -// with one query). Each display ends before the next phase starts, and -// before an error is returned to be printed. The details are written -// once the last display has ended, so no display is drawn while they -// are written. +// own on stderr: open (open the database) and count (read the number of +// records and their total size from the totals table). Each display +// ends before the next phase starts, and before an error is returned +// to be printed. The details are written once the last display has +// ended, so no display is drawn while they are written. func runInfo(ctx context.Context, stdout io.Writer) error { dbPath, source, err := databasePath() if err != nil { @@ -56,7 +56,7 @@ func runInfo(ctx context.Context, stdout io.Writer) error { defer func() { _ = db.Close() }() - // One query counts everything, so the phase never has a total. + // One row holds both counts, so the phase never has a total. prog = newProgress(progressParams{ label: infoPhase("count"), total: -1, waitingFor: countingRecords, diff --git a/main_test.go b/main_test.go index cca359a..cdb385c 100644 --- a/main_test.go +++ b/main_test.go @@ -180,9 +180,9 @@ func capture(t *testing.T, std **os.File) func() string { } // brokenDatabase writes a database that opens cleanly and passes the -// schema-version check but has no files table, so the first query -// fails with the database already open: a fatal error on a path that -// owns an open database. It closes the database the way scan does. +// schema-version check but has no tables, so the first query fails +// with the database already open: a fatal error on a path that owns an +// open database. It closes the database the way scan does. func brokenDatabase(t *testing.T) string { t.Helper() @@ -265,8 +265,9 @@ func TestRunFatalAfterOpenClosesDatabase(t *testing.T) { assertFatalOutput(t, stderr.String(), stdout()) // Proof that the failure happened after the open: only a - // query against the opened database can report this. - if !strings.Contains(stderr.String(), "no such table: files") { + // query against the opened database can report this. Each + // subcommand's first query reads totals. + if !strings.Contains(stderr.String(), "no such table: totals") { t.Errorf("stderr = %q, want the failure to come from a "+ "query on the open database", stderr.String()) } diff --git a/progress.go b/progress.go index 85e79d4..3b112c3 100644 --- a/progress.go +++ b/progress.go @@ -60,7 +60,7 @@ const fallbackWidth = 80 const spinnerMargin = 5 // countingRecords is what the display of a phase that reads the records -// says while it counts them, before it has its total. +// says until it has the number of records, its total. const countingRecords = "counting the records" // openPhase is the name of the phase in which report, trees and info diff --git a/report.go b/report.go index 5db1217..f780335 100644 --- a/report.go +++ b/report.go @@ -45,11 +45,11 @@ func reportPhase(name string) string { // problem, including a missing database, is fatal. // // The work runs in two phases, each under a progress display of its -// own on stderr: open (open the database) and read (count the records, -// then read them in signature order, writing each size's duplicate -// groups to stdout once that size is read; see readReport). Each -// display ends before the next phase starts, and before an error is -// returned to be printed. +// own on stderr: open (open the database) and read (read the number of +// records from the totals table, then read the records in signature +// order, writing each size's duplicate groups to stdout once that size +// is read; see readReport). Each display ends before the next phase +// starts, and before an error is returned to be printed. func runReport(ctx context.Context, stdout io.Writer) error { dbPath, source, err := databasePath() if err != nil { diff --git a/scan.go b/scan.go index a16dd84..ac49c7e 100644 --- a/scan.go +++ b/scan.go @@ -455,8 +455,8 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { // included, so its display counts against the number of records. func (s *scanState) loadIndex(ctx context.Context, roots []string) error { // Indexing tens of millions of records takes real time; without a - // display the scan looks hung before the walk begins. Counting them - // first gives the display its total. + // display the scan looks hung before the walk begins. The number of + // records, read first, gives the display its total. prog := newProgress(progressParams{ label: scanPhase("load"), unit: scanUnit, total: -1, waitingFor: countingRecords, stalledOn: "db", diff --git a/scan_test.go b/scan_test.go index 58ab1db..4ed8ae1 100644 --- a/scan_test.go +++ b/scan_test.go @@ -1228,9 +1228,9 @@ func TestScanPipeline(t *testing.T) { // TestScanProgressNamesEachPhase checks the plain progress lines of a // first scan, and of a rescan of one operand holding two of the // database's records. Each phase's first line names it with its place -// among the four: load counts the records before reading them, and the -// walk estimates its total from the records under its operand, or says -// its time remaining is unknown when there are none. +// among the four: load gets the number of records before reading them, +// and the walk estimates its total from the records under its operand, +// or says its time remaining is unknown when there are none. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestScanProgressNamesEachPhase(t *testing.T) { diff --git a/trees.go b/trees.go index 1d2e689..3006931 100644 --- a/trees.go +++ b/trees.go @@ -40,11 +40,11 @@ func treesPhase(name string) string { // database problem, including a missing database, is fatal. // // The work runs in four phases, each under a progress display of its -// own on stderr: open (open the database), read (count the records, -// then read them into the hierarchy), group (group the directories by -// digest) and write (write the groups to stdout). Each display ends -// before the next phase starts, and before an error is returned to be -// printed. +// own on stderr: open (open the database), read (read the number of +// records from the totals table, then read the records into the +// hierarchy), group (group the directories by digest) and write (write +// the groups to stdout). Each display ends before the next phase +// starts, and before an error is returned to be printed. func runTrees(ctx context.Context, stdout io.Writer) error { dbPath, source, err := databasePath() if err != nil {