Keep the record count and total size in a totals table (closes #146)

info counted the records and added up their sizes with a query that
reads every record, which takes minutes on a large database. The
schema now has a totals table of one row, kept current by three
triggers on files, in the transaction of each insert, delete and size
update. info, scan's load phase and the read phases of report and
trees read that row instead of counting over files. scan writes files
as before.

Model: opus-5-5

README.md

@@ -709,14 +709,18 @@ Processing:
   database state.
 
 `report` runs in two phases, each with its own progress display on stderr (see
-"Progress"): `open` opens the database; `read` gets the number of records from
+"Progress"):
-`totals`, then reads the records in signature order, largest size first. That is
+
-the order of the `files_signature` index read backwards, so SQLite sorts nothing
+- `open` opens the database.
-and returns the first record at once. The records sharing a signature come
+- `read` gets the number of records from `totals`, then reads the records in
-together, and so do the records of one size, so `report` holds only the
+  signature order, largest size first.
-duplicate groups of the size it is reading, and writes them, ordered by `first`
+
-path, once it reaches a smaller size. Its memory holds at most the duplicate
+Signature order, largest size first, is the `files_signature` index read
-paths of one size at a time, never every record.
+backwards, so SQLite sorts nothing and returns the first record at once. The
+records sharing a signature come together, and so do the records of one size, so
+`report` holds only the duplicate groups of the size it is reading, and writes
+them, ordered by `first` path, once it reaches a smaller size. Its memory holds
+at most the duplicate paths of one size at a time, never every record.
 
 #### Report output format
 
@@ -790,10 +794,13 @@ Processing:
   ascending. Output must be fully deterministic for a given input.
 
 `trees` runs in four phases, each with its own progress display on stderr (see
-"Progress"): `open` opens the database; `read` gets the number of records from
+"Progress"):
-`totals`, then reads the records in path order, building the hierarchy and
+
-computing the digests as it goes; `group` groups the directories by digest and
+- `open` opens the database.
-picks the maximal groups; `write` writes the groups to stdout.
+- `read` gets the number of records from `totals`, then reads the records in
+  path order, building the hierarchy and computing the digests as it goes.
+- `group` groups the directories by digest and picks the maximal groups.
+- `write` writes the groups to stdout.
 
 #### Trees output format
 
@@ -844,10 +851,15 @@ size of recorded files	150000000000000 (136.4 TiB)
   of all records, in bytes, then in human units.
 
 `info` runs in two phases, each with its own progress display on stderr (see
-"Progress"): `open` opens the database; `count` reads the schema version and the
+"Progress"):
-row of `totals` (see "Database") in one read-only transaction, so the numbers
+
-agree with each other while a scan is committing. `info` writes its lines once
+- `open` opens the database.
-the `count` display has ended, so on a terminal they come after the displays.
+- `count` reads the schema version and the row of `totals` (see "Database").
+  Both reads run in one read-only transaction, so the numbers agree with each
+  other while a scan is committing.
+
+`info` writes its lines once the `count` display has ended, so on a terminal
+they come after the displays.
 
 ### Progress
 

db_test.go

@@ -606,6 +606,65 @@ func TestCountsReadOnlyTotals(t *testing.T) {
 	}
 }
 
+// TestCountsComeFromTotals sets the row of totals to numbers that files
+// does not hold, then checks that scan's load phase, the read phases of
+// trees and report, and info each report the numbers from totals.
+func TestCountsComeFromTotals(t *testing.T) {
+	t.Parallel()
+
+	// files is empty, so a count over files gives 0 records of 0 bytes.
+	const records, size = 7, 9000
+
+	db := openTestDB(t)
+
+	_, err := db.ExecContext(t.Context(),
+		"UPDATE totals SET records = ?, size = ?", records, size)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	scanTotal, err := countRecords(t.Context(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	var treesTotal, reportTotal int64
+
+	err = loadFileRows(t.Context(), db,
+		func(n int64) { treesTotal = n }, func(scanRec) {})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	err = loadSignatureRows(t.Context(), db,
+		func(n int64) { reportTotal = n },
+		func(scanRec) error { return nil })
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	info, err := readDatabaseCounts(t.Context(), db)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for name, got := range map[string]int64{
+		"scan's load":   scanTotal,
+		"trees' read":   treesTotal,
+		"report's read": reportTotal,
+		"info":          info.records,
+	} {
+		if got != records {
+			t.Errorf("%s: %d records, want %d from totals",
+				name, got, records)
+		}
+	}
+
+	if info.size != size {
+		t.Errorf("info: size %d, want %d from totals", info.size, size)
+	}
+}
+
 // assertTotalsMatchFiles fails unless the row of totals holds the
 // number of records and the sizes of all records added up, as counted
 // over files.