Keep the record count and total size in a totals table (closes #146)

info counted the records and added up their sizes with a query that
reads every record, which takes minutes on a large database. The
schema now has a totals table of one row, kept current by three
triggers on files, in the transaction of each insert, delete and size
update. info, scan's load phase and the read phases of report and
trees read that row instead of counting over files. scan writes files
with the same statements as before. Each of those writes now fires a
trigger and is slower, so scan keeps SQLite's temporary data in
memory, which wins back part of that time.

Model: opus-5-5

README.md

@@ -421,11 +421,15 @@ All four subcommands operate on a single SQLite database file:
     without sorting the table.
 
     `totals` holds one row: the number of records in `files` and the sum of
-    their sizes. Its three triggers keep that row current. Each runs in the
+    their sizes. Three triggers on `files` keep that row current. Each runs in
-    transaction of the change to `files` that fires it, so the row and the
+    the transaction of the change to `files` that fires it, so the row and the
-    records always agree. `info` and the progress displays read the number of
+    records always agree. `info` reads both numbers from this row. `scan`,
-    records from this row. Counting them in `files` would read every record,
+    `report` and `trees` read the number of records from it as the total of
-    which takes minutes on a database of tens of millions.
+    their progress displays. Counting the records in `files` would read every
+    record, which takes minutes on a database of tens of millions. The triggers
+    make each write to `files` slower, so a first scan of many small files takes
+    about two fifths longer. On large files, where reading them takes most of
+    the time, the difference is small.
 
 ### Duplicate detection
 
@@ -847,8 +851,8 @@ size of recorded files	150000000000000 (136.4 TiB)
   database is in WAL mode, the newest changes may be in the `-wal` file alone.
 - `schema version` is the database's `PRAGMA user_version`. A database with
   another version than this build's is refused, as by `report`.
-- `records` is the number of records. `size of recorded files` adds up the sizes
+- `records` is the number of records. `size of recorded files` is the sum of the
-  of all records, in bytes, then in human units.
+  sizes of all records, in bytes, then in human units.
 
 `info` runs in two phases, each with its own progress display on stderr (see
 "Progress"):

TODO.md

@@ -32,8 +32,10 @@
 - the database keeps the number of records and the sum of their sizes in the one
   row of a `totals` table, which three triggers on `files` keep current. `info`
   reads that row, and so do `scan`'s `load` phase and the `read` phases of
-  `report` and `trees` for their totals; no count reads every record any more
+  `report` and `trees` for their totals; no count reads every record any more.
-  (2026-10-27, https://git.eeqj.de/sneak/sfdupes/issues/146)
+  The triggers make each write to `files` slower. `scan` keeps SQLite's
+  temporary data in memory, which wins back part of that time (2026-10-27,
+  https://git.eeqj.de/sneak/sfdupes/issues/146)
 
 - an operand at or inside a directory named `.zfs` is never scanned, whether
   given alone or under another operand. `scan` warns about it, counts it as

db.go

@@ -193,9 +193,14 @@ func createDatabaseDirectory(path string) error {
 // scanParams are the connection parameters for scan: read-write, with
 // WAL journaling and a busy timeout, so a report can run while a cron
 // scan is in progress. closeScanDatabase leaves WAL mode again.
+//
+// Each write to files fires a trigger, so SQLite keeps a statement
+// journal for it. temp_store(MEMORY) keeps that journal in memory,
+// which costs far less than a temporary file.
 const scanParams = "_pragma=busy_timeout(10000)" +
 	"&_pragma=journal_mode(WAL)" +
-	"&_pragma=synchronous(NORMAL)"
+	"&_pragma=synchronous(NORMAL)" +
+	"&_pragma=temp_store(MEMORY)"
 
 // reportParams are the connection parameters for report, trees and
 // info: read-only, with the same busy timeout. They set no journal

db_test.go

@@ -588,9 +588,9 @@ func TestSignatureRowsNeedNoSort(t *testing.T) {
 	}
 }
 
-// TestCountsReadOnlyTotals checks that SQLite answers each query that
+// TestCountsReadOnlyTotals checks that SQLite answers the queries for
-// counts the records from the one row of totals. A plan that reads
+// the number of records and their total size from the one row of
-// files or its index reads every record.
+// totals. A plan that reads files or its index reads every record.
 func TestCountsReadOnlyTotals(t *testing.T) {
 	t.Parallel()
 

progress.go

@@ -60,7 +60,7 @@ const fallbackWidth = 80
 const spinnerMargin = 5
 
 // countingRecords is what the display of a phase that reads the records
-// says while it counts them, before it has its total.
+// says until it has the number of records, its total.
 const countingRecords = "counting the records"
 
 // openPhase is the name of the phase in which report, trees and info

report.go

@@ -45,11 +45,11 @@ func reportPhase(name string) string {
 // problem, including a missing database, is fatal.
 //
 // The work runs in two phases, each under a progress display of its
-// own on stderr: open (open the database) and read (count the records,
+// own on stderr: open (open the database) and read (read the number of
-// then read them in signature order, writing each size's duplicate
+// records from the totals table, then read the records in signature
-// groups to stdout once that size is read; see readReport). Each
+// order, writing each size's duplicate groups to stdout once that size
-// display ends before the next phase starts, and before an error is
+// is read; see readReport). Each display ends before the next phase
-// returned to be printed.
+// starts, and before an error is returned to be printed.
 func runReport(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {

scan_test.go

@@ -1228,9 +1228,9 @@ func TestScanPipeline(t *testing.T) {
 // TestScanProgressNamesEachPhase checks the plain progress lines of a
 // first scan, and of a rescan of one operand holding two of the
 // database's records. Each phase's first line names it with its place
-// among the four: load counts the records before reading them, and the
+// among the four: load gets the number of records before reading them,
-// walk estimates its total from the records under its operand, or says
+// and the walk estimates its total from the records under its operand,
-// its time remaining is unknown when there are none.
+// or says its time remaining is unknown when there are none.
 //
 //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
 func TestScanProgressNamesEachPhase(t *testing.T) {

trees.go

@@ -40,11 +40,11 @@ func treesPhase(name string) string {
 // database problem, including a missing database, is fatal.
 //
 // The work runs in four phases, each under a progress display of its
-// own on stderr: open (open the database), read (count the records,
+// own on stderr: open (open the database), read (read the number of
-// then read them into the hierarchy), group (group the directories by
+// records from the totals table, then read the records into the
-// digest) and write (write the groups to stdout). Each display ends
+// hierarchy), group (group the directories by digest) and write (write
-// before the next phase starts, and before an error is returned to be
+// the groups to stdout). Each display ends before the next phase
-// printed.
+// starts, and before an error is returned to be printed.
 func runTrees(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {