Keep the record count and total size in a totals table (closes #146)

info counted the records and added up their sizes with a query that
reads every record, which takes minutes on a large database. The
schema now has a totals table of one row, kept current by three
triggers on files, in the transaction of each insert, delete and size
update. info, scan's load phase and the read phases of report and
trees read that row instead of counting over files. scan writes files
as before.

Model: opus-5-5

README.md

@@ -709,18 +709,14 @@ Processing:
   database state.
 
 `report` runs in two phases, each with its own progress display on stderr (see
-"Progress"):
+"Progress"): `open` opens the database; `read` gets the number of records from
-
+`totals`, then reads the records in signature order, largest size first. That is
-- `open` opens the database.
+the order of the `files_signature` index read backwards, so SQLite sorts nothing
-- `read` gets the number of records from `totals`, then reads the records in
+and returns the first record at once. The records sharing a signature come
-  signature order, largest size first.
+together, and so do the records of one size, so `report` holds only the
-
+duplicate groups of the size it is reading, and writes them, ordered by `first`
-Signature order, largest size first, is the `files_signature` index read
+path, once it reaches a smaller size. Its memory holds at most the duplicate
-backwards, so SQLite sorts nothing and returns the first record at once. The
+paths of one size at a time, never every record.
-records sharing a signature come together, and so do the records of one size, so
-`report` holds only the duplicate groups of the size it is reading, and writes
-them, ordered by `first` path, once it reaches a smaller size. Its memory holds
-at most the duplicate paths of one size at a time, never every record.
 
 #### Report output format
 
@@ -794,13 +790,10 @@ Processing:
   ascending. Output must be fully deterministic for a given input.
 
 `trees` runs in four phases, each with its own progress display on stderr (see
-"Progress"):
+"Progress"): `open` opens the database; `read` gets the number of records from
-
+`totals`, then reads the records in path order, building the hierarchy and
-- `open` opens the database.
+computing the digests as it goes; `group` groups the directories by digest and
-- `read` gets the number of records from `totals`, then reads the records in
+picks the maximal groups; `write` writes the groups to stdout.
-  path order, building the hierarchy and computing the digests as it goes.
-- `group` groups the directories by digest and picks the maximal groups.
-- `write` writes the groups to stdout.
 
 #### Trees output format
 
@@ -851,15 +844,10 @@ size of recorded files	150000000000000 (136.4 TiB)
   of all records, in bytes, then in human units.
 
 `info` runs in two phases, each with its own progress display on stderr (see
-"Progress"):
+"Progress"): `open` opens the database; `count` reads the schema version and the
-
+row of `totals` (see "Database") in one read-only transaction, so the numbers
-- `open` opens the database.
+agree with each other while a scan is committing. `info` writes its lines once
-- `count` reads the schema version and the row of `totals` (see "Database").
+the `count` display has ended, so on a terminal they come after the displays.
-  Both reads run in one read-only transaction, so the numbers agree with each
-  other while a scan is committing.
-
-`info` writes its lines once the `count` display has ended, so on a terminal
-they come after the displays.
 
 ### Progress
 

db_test.go

@@ -606,65 +606,6 @@ func TestCountsReadOnlyTotals(t *testing.T) {
 	}
 }
 
-// TestCountsComeFromTotals sets the row of totals to numbers that files
-// does not hold, then checks that scan's load phase, the read phases of
-// trees and report, and info each report the numbers from totals.
-func TestCountsComeFromTotals(t *testing.T) {
-	t.Parallel()
-
-	// files is empty, so a count over files gives 0 records of 0 bytes.
-	const records, size = 7, 9000
-
-	db := openTestDB(t)
-
-	_, err := db.ExecContext(t.Context(),
-		"UPDATE totals SET records = ?, size = ?", records, size)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	scanTotal, err := countRecords(t.Context(), db)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	var treesTotal, reportTotal int64
-
-	err = loadFileRows(t.Context(), db,
-		func(n int64) { treesTotal = n }, func(scanRec) {})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	err = loadSignatureRows(t.Context(), db,
-		func(n int64) { reportTotal = n },
-		func(scanRec) error { return nil })
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	info, err := readDatabaseCounts(t.Context(), db)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	for name, got := range map[string]int64{
-		"scan's load":   scanTotal,
-		"trees' read":   treesTotal,
-		"report's read": reportTotal,
-		"info":          info.records,
-	} {
-		if got != records {
-			t.Errorf("%s: %d records, want %d from totals",
-				name, got, records)
-		}
-	}
-
-	if info.size != size {
-		t.Errorf("info: size %d, want %d from totals", info.size, size)
-	}
-}
-
 // assertTotalsMatchFiles fails unless the row of totals holds the
 // number of records and the sizes of all records added up, as counted
 // over files.