Keep the record count and total size in a totals table (closes #146)

info counted the records and added up their sizes with a query that
reads every record, which takes minutes on a large database. The
schema now has a totals table of one row, kept current by three
triggers on files, in the transaction of each insert, delete and size
update. info, scan's load phase and the read phases of report and
trees read that row instead of counting over files. scan writes files
as before.

Model: opus-5-5

README.md

@@ -421,15 +421,11 @@ All four subcommands operate on a single SQLite database file:
     without sorting the table.
 
     `totals` holds one row: the number of records in `files` and the sum of
-    their sizes. Three triggers on `files` keep that row current. Each runs in
+    their sizes. Its three triggers keep that row current. Each runs in the
-    the transaction of the change to `files` that fires it, so the row and the
+    transaction of the change to `files` that fires it, so the row and the
-    records always agree. `info` reads both numbers from this row. `scan`,
+    records always agree. `info` and the progress displays read the number of
-    `report` and `trees` read the number of records from it as the total of
+    records from this row. Counting them in `files` would read every record,
-    their progress displays. Counting the records in `files` would read every
+    which takes minutes on a database of tens of millions.
-    record, which takes minutes on a database of tens of millions. The triggers
-    make each write to `files` slower, so a first scan of many small files takes
-    about two fifths longer. On large files, where reading them takes most of
-    the time, the difference is small.
 
 ### Duplicate detection
 
@@ -851,8 +847,8 @@ size of recorded files	150000000000000 (136.4 TiB)
   database is in WAL mode, the newest changes may be in the `-wal` file alone.
 - `schema version` is the database's `PRAGMA user_version`. A database with
   another version than this build's is refused, as by `report`.
-- `records` is the number of records. `size of recorded files` is the sum of the
+- `records` is the number of records. `size of recorded files` adds up the sizes
-  sizes of all records, in bytes, then in human units.
+  of all records, in bytes, then in human units.
 
 `info` runs in two phases, each with its own progress display on stderr (see
 "Progress"):

TODO.md

@@ -32,10 +32,8 @@
 - the database keeps the number of records and the sum of their sizes in the one
   row of a `totals` table, which three triggers on `files` keep current. `info`
   reads that row, and so do `scan`'s `load` phase and the `read` phases of
-  `report` and `trees` for their totals; no count reads every record any more.
+  `report` and `trees` for their totals; no count reads every record any more
-  The triggers make each write to `files` slower. `scan` keeps SQLite's
+  (2026-10-27, https://git.eeqj.de/sneak/sfdupes/issues/146)
-  temporary data in memory, which wins back part of that time (2026-10-27,
-  https://git.eeqj.de/sneak/sfdupes/issues/146)
 
 - an operand at or inside a directory named `.zfs` is never scanned, whether
   given alone or under another operand. `scan` warns about it, counts it as

db.go

@@ -193,14 +193,9 @@ func createDatabaseDirectory(path string) error {
 // scanParams are the connection parameters for scan: read-write, with
 // WAL journaling and a busy timeout, so a report can run while a cron
 // scan is in progress. closeScanDatabase leaves WAL mode again.
-//
-// Each write to files fires a trigger, so SQLite keeps a statement
-// journal for it. temp_store(MEMORY) keeps that journal in memory,
-// which costs far less than a temporary file.
 const scanParams = "_pragma=busy_timeout(10000)" +
 	"&_pragma=journal_mode(WAL)" +
-	"&_pragma=synchronous(NORMAL)" +
+	"&_pragma=synchronous(NORMAL)"
-	"&_pragma=temp_store(MEMORY)"
 
 // reportParams are the connection parameters for report, trees and
 // info: read-only, with the same busy timeout. They set no journal

db_test.go

@@ -588,9 +588,9 @@ func TestSignatureRowsNeedNoSort(t *testing.T) {
 	}
 }
 
-// TestCountsReadOnlyTotals checks that SQLite answers the queries for
+// TestCountsReadOnlyTotals checks that SQLite answers each query that
-// the number of records and their total size from the one row of
+// counts the records from the one row of totals. A plan that reads
-// totals. A plan that reads files or its index reads every record.
+// files or its index reads every record.
 func TestCountsReadOnlyTotals(t *testing.T) {
 	t.Parallel()
 

progress.go

@@ -60,7 +60,7 @@ const fallbackWidth = 80
 const spinnerMargin = 5
 
 // countingRecords is what the display of a phase that reads the records
-// says until it has the number of records, its total.
+// says while it counts them, before it has its total.
 const countingRecords = "counting the records"
 
 // openPhase is the name of the phase in which report, trees and info

report.go

@@ -45,11 +45,11 @@ func reportPhase(name string) string {
 // problem, including a missing database, is fatal.
 //
 // The work runs in two phases, each under a progress display of its
-// own on stderr: open (open the database) and read (read the number of
+// own on stderr: open (open the database) and read (count the records,
-// records from the totals table, then read the records in signature
+// then read them in signature order, writing each size's duplicate
-// order, writing each size's duplicate groups to stdout once that size
+// groups to stdout once that size is read; see readReport). Each
-// is read; see readReport). Each display ends before the next phase
+// display ends before the next phase starts, and before an error is
-// starts, and before an error is returned to be printed.
+// returned to be printed.
 func runReport(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {

scan_test.go

@@ -1228,9 +1228,9 @@ func TestScanPipeline(t *testing.T) {
 // TestScanProgressNamesEachPhase checks the plain progress lines of a
 // first scan, and of a rescan of one operand holding two of the
 // database's records. Each phase's first line names it with its place
-// among the four: load gets the number of records before reading them,
+// among the four: load counts the records before reading them, and the
-// and the walk estimates its total from the records under its operand,
+// walk estimates its total from the records under its operand, or says
-// or says its time remaining is unknown when there are none.
+// its time remaining is unknown when there are none.
 //
 //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
 func TestScanProgressNamesEachPhase(t *testing.T) {

trees.go

@@ -40,11 +40,11 @@ func treesPhase(name string) string {
 // database problem, including a missing database, is fatal.
 //
 // The work runs in four phases, each under a progress display of its
-// own on stderr: open (open the database), read (read the number of
+// own on stderr: open (open the database), read (count the records,
-// records from the totals table, then read the records into the
+// then read them into the hierarchy), group (group the directories by
-// hierarchy), group (group the directories by digest) and write (write
+// digest) and write (write the groups to stdout). Each display ends
-// the groups to stdout). Each display ends before the next phase
+// before the next phase starts, and before an error is returned to be
-// starts, and before an error is returned to be printed.
+// printed.
 func runTrees(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {