Compare commits
1
Commits
next
...
61ee49a905
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
61ee49a905 |
@@ -393,6 +393,20 @@ All four subcommands operate on a single SQLite database file:
|
|||||||
content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
|
content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
|
||||||
) WITHOUT ROWID;
|
) WITHOUT ROWID;
|
||||||
CREATE INDEX files_signature ON files (size, head, tail, content);
|
CREATE INDEX files_signature ON files (size, head, tail, content);
|
||||||
|
CREATE TABLE totals (
|
||||||
|
records INTEGER NOT NULL, -- number of records in files
|
||||||
|
size INTEGER NOT NULL -- sizes of all records in files added up
|
||||||
|
);
|
||||||
|
INSERT INTO totals (records, size) VALUES (0, 0);
|
||||||
|
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
|
||||||
|
UPDATE totals SET records = records + 1, size = size + NEW.size;
|
||||||
|
END;
|
||||||
|
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
|
||||||
|
UPDATE totals SET records = records - 1, size = size - OLD.size;
|
||||||
|
END;
|
||||||
|
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
|
||||||
|
UPDATE totals SET size = size + NEW.size - OLD.size;
|
||||||
|
END;
|
||||||
```
|
```
|
||||||
|
|
||||||
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
|
Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
|
||||||
@@ -406,6 +420,17 @@ All four subcommands operate on a single SQLite database file:
|
|||||||
`files_signature` index lets `report` read the records in signature order
|
`files_signature` index lets `report` read the records in signature order
|
||||||
without sorting the table.
|
without sorting the table.
|
||||||
|
|
||||||
|
`totals` holds one row: the number of records in `files` and the sum of
|
||||||
|
their sizes. Three triggers on `files` keep that row current. Each runs in
|
||||||
|
the transaction of the change to `files` that fires it, so the row and the
|
||||||
|
records always agree. `info` reads both numbers from this row. `scan`,
|
||||||
|
`report` and `trees` read the number of records from it as the total of
|
||||||
|
their progress displays. Counting the records in `files` would read every
|
||||||
|
record, which takes minutes on a database of tens of millions. The triggers
|
||||||
|
make each write to `files` slower, so a first scan of many small files takes
|
||||||
|
about two fifths longer. On large files, where reading them takes most of
|
||||||
|
the time, the difference is small.
|
||||||
|
|
||||||
### Duplicate detection
|
### Duplicate detection
|
||||||
|
|
||||||
Two files are duplicates when they have the same size and the same three hashes.
|
Two files are duplicates when they have the same size and the same three hashes.
|
||||||
@@ -688,14 +713,18 @@ Processing:
|
|||||||
database state.
|
database state.
|
||||||
|
|
||||||
`report` runs in two phases, each with its own progress display on stderr (see
|
`report` runs in two phases, each with its own progress display on stderr (see
|
||||||
"Progress"): `open` opens the database; `read` counts the records, then reads
|
"Progress"):
|
||||||
them in signature order, largest size first. That is the order of the
|
|
||||||
`files_signature` index read backwards, so SQLite sorts nothing and returns the
|
- `open` opens the database.
|
||||||
first record at once. The records sharing a signature come together, and so do
|
- `read` gets the number of records from `totals`, then reads the records in
|
||||||
the records of one size, so `report` holds only the duplicate groups of the size
|
signature order, largest size first.
|
||||||
it is reading, and writes them, ordered by `first` path, once it reaches a
|
|
||||||
smaller size. Its memory holds at most the duplicate paths of one size at a
|
Signature order, largest size first, is the `files_signature` index read
|
||||||
time, never every record.
|
backwards, so SQLite sorts nothing and returns the first record at once. The
|
||||||
|
records sharing a signature come together, and so do the records of one size, so
|
||||||
|
`report` holds only the duplicate groups of the size it is reading, and writes
|
||||||
|
them, ordered by `first` path, once it reaches a smaller size. Its memory holds
|
||||||
|
at most the duplicate paths of one size at a time, never every record.
|
||||||
|
|
||||||
#### Report output format
|
#### Report output format
|
||||||
|
|
||||||
@@ -769,10 +798,13 @@ Processing:
|
|||||||
ascending. Output must be fully deterministic for a given input.
|
ascending. Output must be fully deterministic for a given input.
|
||||||
|
|
||||||
`trees` runs in four phases, each with its own progress display on stderr (see
|
`trees` runs in four phases, each with its own progress display on stderr (see
|
||||||
"Progress"): `open` opens the database; `read` counts the records, then reads
|
"Progress"):
|
||||||
them in path order, building the hierarchy and computing the digests as it goes;
|
|
||||||
`group` groups the directories by digest and picks the maximal groups; `write`
|
- `open` opens the database.
|
||||||
writes the groups to stdout.
|
- `read` gets the number of records from `totals`, then reads the records in
|
||||||
|
path order, building the hierarchy and computing the digests as it goes.
|
||||||
|
- `group` groups the directories by digest and picks the maximal groups.
|
||||||
|
- `write` writes the groups to stdout.
|
||||||
|
|
||||||
#### Trees output format
|
#### Trees output format
|
||||||
|
|
||||||
@@ -819,15 +851,19 @@ size of recorded files 150000000000000 (136.4 TiB)
|
|||||||
database is in WAL mode, the newest changes may be in the `-wal` file alone.
|
database is in WAL mode, the newest changes may be in the `-wal` file alone.
|
||||||
- `schema version` is the database's `PRAGMA user_version`. A database with
|
- `schema version` is the database's `PRAGMA user_version`. A database with
|
||||||
another version than this build's is refused, as by `report`.
|
another version than this build's is refused, as by `report`.
|
||||||
- `records` is the number of records. `size of recorded files` adds up the sizes
|
- `records` is the number of records. `size of recorded files` is the sum of the
|
||||||
of all records, in bytes, then in human units.
|
sizes of all records, in bytes, then in human units.
|
||||||
|
|
||||||
`info` runs in two phases, each with its own progress display on stderr (see
|
`info` runs in two phases, each with its own progress display on stderr (see
|
||||||
"Progress"): `open` opens the database; `count` reads the schema version and
|
"Progress"):
|
||||||
counts the records with one query, a single pass over the table, both in one
|
|
||||||
read-only transaction, so the numbers agree with each other while a scan is
|
- `open` opens the database.
|
||||||
committing. `info` writes its lines once the `count` display has ended, so on a
|
- `count` reads the schema version and the row of `totals` (see "Database").
|
||||||
terminal they come after the displays.
|
Both reads run in one read-only transaction, so the numbers agree with each
|
||||||
|
other while a scan is committing.
|
||||||
|
|
||||||
|
`info` writes its lines once the `count` display has ended, so on a terminal
|
||||||
|
they come after the displays.
|
||||||
|
|
||||||
### Progress
|
### Progress
|
||||||
|
|
||||||
@@ -843,8 +879,9 @@ progress, its elapsed time and its time remaining.
|
|||||||
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
|
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
|
||||||
`[4/4] update`):
|
`[4/4] update`):
|
||||||
|
|
||||||
- `load` counts the records first, then reads them against that count. While the
|
- `load` gets the number of records from `totals` first, then reads the records
|
||||||
count runs, the display says it is counting the records.
|
against that number. Until it has the number, the display says it is counting
|
||||||
|
the records.
|
||||||
- `walk` has no exact total. Its estimate is the number of records the last scan
|
- `walk` has no exact total. Its estimate is the number of records the last scan
|
||||||
left under the operands, known once `load` is done and shown with a `~`. The
|
left under the operands, known once `load` is done and shown with a `~`. The
|
||||||
time remaining is unknown once the walk passes the estimate, and when there
|
time remaining is unknown once the walk passes the estimate, and when there
|
||||||
@@ -857,8 +894,9 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
|
|||||||
|
|
||||||
- `open` has no total. While it opens the database, waiting on a busy one
|
- `open` has no total. While it opens the database, waiting on a busy one
|
||||||
included, the display says it is opening the database.
|
included, the display says it is opening the database.
|
||||||
- `read` counts the records first, then reads them against that count. While the
|
- `read` gets the number of records from `totals` first, then reads the records
|
||||||
count runs, the display says it is counting the records.
|
against that number. Until it has the number, the display says it is counting
|
||||||
|
the records.
|
||||||
- `group` has the directories `read` found as its total, and `write` the
|
- `group` has the directories `read` found as its total, and `write` the
|
||||||
duplicate-tree groups.
|
duplicate-tree groups.
|
||||||
|
|
||||||
@@ -866,17 +904,16 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
|
|||||||
|
|
||||||
- `open` has no total. While it opens the database, waiting on a busy one
|
- `open` has no total. While it opens the database, waiting on a busy one
|
||||||
included, the display says it is opening the database.
|
included, the display says it is opening the database.
|
||||||
- `read` counts the records first, then reads them against that count, writing
|
- `read` gets the number of records from `totals` first, then reads the records
|
||||||
the rows to stdout as it goes. While the count runs, the display says it is
|
against that number, writing the rows to stdout as it goes. Until it has the
|
||||||
counting the records.
|
number, the display says it is counting the records.
|
||||||
|
|
||||||
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
|
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
|
||||||
|
|
||||||
- `open` has no total. While it opens the database, waiting on a busy one
|
- `open` has no total. While it opens the database, waiting on a busy one
|
||||||
included, the display says it is opening the database.
|
included, the display says it is opening the database.
|
||||||
- `count` has no total either, because one query counts everything: the display
|
- `count` has no total either, because it reads one row: the display says it is
|
||||||
says it is counting the records until the query returns, and its time
|
counting the records until it has the row, and its time remaining is unknown.
|
||||||
remaining is unknown.
|
|
||||||
|
|
||||||
Each display is one line holding:
|
Each display is one line holding:
|
||||||
|
|
||||||
|
|||||||
@@ -29,6 +29,14 @@
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- the database keeps the number of records and the sum of their sizes in the one
|
||||||
|
row of a `totals` table, which three triggers on `files` keep current. `info`
|
||||||
|
reads that row, and so do `scan`'s `load` phase and the `read` phases of
|
||||||
|
`report` and `trees` for their totals; no count reads every record any more.
|
||||||
|
The triggers make each write to `files` slower. `scan` keeps SQLite's
|
||||||
|
temporary data in memory, which wins back part of that time (2026-10-27,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/146)
|
||||||
|
|
||||||
- an operand at or inside a directory named `.zfs` is never scanned, whether
|
- an operand at or inside a directory named `.zfs` is never scanned, whether
|
||||||
given alone or under another operand. `scan` warns about it, counts it as
|
given alone or under another operand. `scan` warns about it, counts it as
|
||||||
skipped and does not walk it (2026-10-26,
|
skipped and does not walk it (2026-10-26,
|
||||||
|
|||||||
@@ -69,6 +69,49 @@ const createIndexSQL = `
|
|||||||
CREATE INDEX files_signature ON files (size, head, tail, content)
|
CREATE INDEX files_signature ON files (size, head, tail, content)
|
||||||
`
|
`
|
||||||
|
|
||||||
|
// createTotalsSQL creates the totals table, which holds one row: the
|
||||||
|
// number of records and the sizes of all records added up. Counting
|
||||||
|
// them over files would read every record. The three triggers below
|
||||||
|
// keep the row current, in the transaction of each change to files.
|
||||||
|
const createTotalsSQL = `
|
||||||
|
CREATE TABLE totals (
|
||||||
|
records INTEGER NOT NULL,
|
||||||
|
size INTEGER NOT NULL
|
||||||
|
)
|
||||||
|
`
|
||||||
|
|
||||||
|
// insertTotalsSQL adds the one row of totals, for an empty files table.
|
||||||
|
const insertTotalsSQL = `
|
||||||
|
INSERT INTO totals (records, size) VALUES (0, 0)
|
||||||
|
`
|
||||||
|
|
||||||
|
// totalsAfterInsertSQL counts each inserted record into totals.
|
||||||
|
const totalsAfterInsertSQL = `
|
||||||
|
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
|
||||||
|
UPDATE totals SET records = records + 1, size = size + NEW.size;
|
||||||
|
END
|
||||||
|
`
|
||||||
|
|
||||||
|
// totalsAfterDeleteSQL takes each deleted record out of totals.
|
||||||
|
const totalsAfterDeleteSQL = `
|
||||||
|
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
|
||||||
|
UPDATE totals SET records = records - 1, size = size - OLD.size;
|
||||||
|
END
|
||||||
|
`
|
||||||
|
|
||||||
|
// totalsAfterUpdateSQL replaces a record's old size in totals with its
|
||||||
|
// new one. scan's upsert of an existing record is such an update.
|
||||||
|
const totalsAfterUpdateSQL = `
|
||||||
|
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
|
||||||
|
UPDATE totals SET size = size + NEW.size - OLD.size;
|
||||||
|
END
|
||||||
|
`
|
||||||
|
|
||||||
|
// recordCountSQL reads the number of records from totals.
|
||||||
|
const recordCountSQL = `
|
||||||
|
SELECT records FROM totals
|
||||||
|
`
|
||||||
|
|
||||||
// upsertSQL inserts one file record, replacing any existing record for
|
// upsertSQL inserts one file record, replacing any existing record for
|
||||||
// the same path.
|
// the same path.
|
||||||
const upsertSQL = `
|
const upsertSQL = `
|
||||||
@@ -150,9 +193,14 @@ func createDatabaseDirectory(path string) error {
|
|||||||
// scanParams are the connection parameters for scan: read-write, with
|
// scanParams are the connection parameters for scan: read-write, with
|
||||||
// WAL journaling and a busy timeout, so a report can run while a cron
|
// WAL journaling and a busy timeout, so a report can run while a cron
|
||||||
// scan is in progress. closeScanDatabase leaves WAL mode again.
|
// scan is in progress. closeScanDatabase leaves WAL mode again.
|
||||||
|
//
|
||||||
|
// Each write to files fires a trigger, so SQLite keeps a statement
|
||||||
|
// journal for it. temp_store(MEMORY) keeps that journal in memory,
|
||||||
|
// which costs far less than a temporary file.
|
||||||
const scanParams = "_pragma=busy_timeout(10000)" +
|
const scanParams = "_pragma=busy_timeout(10000)" +
|
||||||
"&_pragma=journal_mode(WAL)" +
|
"&_pragma=journal_mode(WAL)" +
|
||||||
"&_pragma=synchronous(NORMAL)"
|
"&_pragma=synchronous(NORMAL)" +
|
||||||
|
"&_pragma=temp_store(MEMORY)"
|
||||||
|
|
||||||
// reportParams are the connection parameters for report, trees and
|
// reportParams are the connection parameters for report, trees and
|
||||||
// info: read-only, with the same busy timeout. They set no journal
|
// info: read-only, with the same busy timeout. They set no journal
|
||||||
@@ -426,14 +474,14 @@ func createSchema(ctx context.Context, db *sql.DB) error {
|
|||||||
|
|
||||||
defer func() { _ = tx.Rollback() }()
|
defer func() { _ = tx.Rollback() }()
|
||||||
|
|
||||||
_, err = tx.ExecContext(ctx, createTableSQL)
|
for _, stmt := range []string{
|
||||||
if err != nil {
|
createTableSQL, createIndexSQL, createTotalsSQL, insertTotalsSQL,
|
||||||
return fmt.Errorf("create schema: %w", err)
|
totalsAfterInsertSQL, totalsAfterDeleteSQL, totalsAfterUpdateSQL,
|
||||||
}
|
} {
|
||||||
|
_, err = tx.ExecContext(ctx, stmt)
|
||||||
_, err = tx.ExecContext(ctx, createIndexSQL)
|
if err != nil {
|
||||||
if err != nil {
|
return fmt.Errorf("create schema: %w", err)
|
||||||
return fmt.Errorf("create schema: %w", err)
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = tx.ExecContext(ctx,
|
_, err = tx.ExecContext(ctx,
|
||||||
@@ -480,7 +528,7 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
|||||||
|
|
||||||
var n int64
|
var n int64
|
||||||
|
|
||||||
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("count records: %w", err)
|
return fmt.Errorf("count records: %w", err)
|
||||||
}
|
}
|
||||||
@@ -556,7 +604,7 @@ func loadSignatureRows(ctx context.Context, db *sql.DB,
|
|||||||
|
|
||||||
var n int64
|
var n int64
|
||||||
|
|
||||||
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("count records: %w", err)
|
return fmt.Errorf("count records: %w", err)
|
||||||
}
|
}
|
||||||
@@ -622,7 +670,7 @@ func (r *scanRec) setHashes(head, tail, content []byte) error {
|
|||||||
func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
|
func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
|
||||||
var n int64
|
var n int64
|
||||||
|
|
||||||
err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
err := db.QueryRowContext(ctx, recordCountSQL).Scan(&n)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, fmt.Errorf("count records: %w", err)
|
return 0, fmt.Errorf("count records: %w", err)
|
||||||
}
|
}
|
||||||
@@ -630,12 +678,10 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
|
|||||||
return n, nil
|
return n, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// databaseCountsSQL counts the records and adds up their sizes, in one
|
// databaseCountsSQL reads the number of records and the sizes of all
|
||||||
// pass over the table. A SUM over no rows is NULL, so on an empty table
|
// records added up from totals.
|
||||||
// the SUM falls back to 0.
|
|
||||||
const databaseCountsSQL = `
|
const databaseCountsSQL = `
|
||||||
SELECT COUNT(*), COALESCE(SUM(size), 0)
|
SELECT records, size FROM totals
|
||||||
FROM files
|
|
||||||
`
|
`
|
||||||
|
|
||||||
// databaseCounts is what info reads from the database: its schema
|
// databaseCounts is what info reads from the database: its schema
|
||||||
|
|||||||
+157
-10
@@ -535,17 +535,12 @@ func TestLoadRowsRefuseWrongSizeHash(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
|
// queryPlan returns the detail of each line of SQLite's EXPLAIN QUERY
|
||||||
// from the files_signature index alone, with nothing to sort: the
|
// PLAN for query.
|
||||||
// first record comes at once, so report's read phase counts each
|
func queryPlan(t *testing.T, db *sql.DB, query string) []string {
|
||||||
// record as it comes rather than waiting on SQLite for all of them.
|
t.Helper()
|
||||||
func TestSignatureRowsNeedNoSort(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := openTestDB(t)
|
rows, err := db.QueryContext(t.Context(), "EXPLAIN QUERY PLAN "+query)
|
||||||
|
|
||||||
rows, err := db.QueryContext(t.Context(),
|
|
||||||
"EXPLAIN QUERY PLAN "+signatureRowsSQL)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -573,12 +568,164 @@ func TestSignatureRowsNeedNoSort(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return plan
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
|
||||||
|
// from the files_signature index alone, with nothing to sort: the
|
||||||
|
// first record comes at once, so report's read phase counts each
|
||||||
|
// record as it comes rather than waiting on SQLite for all of them.
|
||||||
|
func TestSignatureRowsNeedNoSort(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db := openTestDB(t)
|
||||||
|
|
||||||
|
plan := queryPlan(t, db, signatureRowsSQL)
|
||||||
|
|
||||||
want := []string{"SCAN files USING COVERING INDEX files_signature"}
|
want := []string{"SCAN files USING COVERING INDEX files_signature"}
|
||||||
if !slices.Equal(plan, want) {
|
if !slices.Equal(plan, want) {
|
||||||
t.Errorf("query plan = %q, want %q", plan, want)
|
t.Errorf("query plan = %q, want %q", plan, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestCountsReadOnlyTotals checks that SQLite answers the queries for
|
||||||
|
// the number of records and their total size from the one row of
|
||||||
|
// totals. A plan that reads files or its index reads every record.
|
||||||
|
func TestCountsReadOnlyTotals(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
db := openTestDB(t)
|
||||||
|
|
||||||
|
for _, query := range []string{recordCountSQL, databaseCountsSQL} {
|
||||||
|
plan := queryPlan(t, db, query)
|
||||||
|
|
||||||
|
want := []string{"SCAN totals"}
|
||||||
|
if !slices.Equal(plan, want) {
|
||||||
|
t.Errorf("query plan of %q = %q, want %q", query, plan, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCountsComeFromTotals sets the row of totals to numbers that files
|
||||||
|
// does not hold, then checks that scan's load phase, the read phases of
|
||||||
|
// trees and report, and info each report the numbers from totals.
|
||||||
|
func TestCountsComeFromTotals(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// files is empty, so a count over files gives 0 records of 0 bytes.
|
||||||
|
const records, size = 7, 9000
|
||||||
|
|
||||||
|
db := openTestDB(t)
|
||||||
|
|
||||||
|
_, err := db.ExecContext(t.Context(),
|
||||||
|
"UPDATE totals SET records = ?, size = ?", records, size)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
scanTotal, err := countRecords(t.Context(), db)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var treesTotal, reportTotal int64
|
||||||
|
|
||||||
|
err = loadFileRows(t.Context(), db,
|
||||||
|
func(n int64) { treesTotal = n }, func(scanRec) {})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = loadSignatureRows(t.Context(), db,
|
||||||
|
func(n int64) { reportTotal = n },
|
||||||
|
func(scanRec) error { return nil })
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
info, err := readDatabaseCounts(t.Context(), db)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, got := range map[string]int64{
|
||||||
|
"scan's load": scanTotal,
|
||||||
|
"trees' read": treesTotal,
|
||||||
|
"report's read": reportTotal,
|
||||||
|
"info": info.records,
|
||||||
|
} {
|
||||||
|
if got != records {
|
||||||
|
t.Errorf("%s: %d records, want %d from totals",
|
||||||
|
name, got, records)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if info.size != size {
|
||||||
|
t.Errorf("info: size %d, want %d from totals", info.size, size)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertTotalsMatchFiles fails unless the row of totals holds the
|
||||||
|
// number of records and the sizes of all records added up, as counted
|
||||||
|
// over files.
|
||||||
|
func assertTotalsMatchFiles(t *testing.T, db *sql.DB) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
got, err := readDatabaseCounts(t.Context(), db)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var records, size int64
|
||||||
|
|
||||||
|
err = db.QueryRowContext(t.Context(),
|
||||||
|
"SELECT COUNT(*), COALESCE(SUM(size), 0) FROM files").
|
||||||
|
Scan(&records, &size)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got.records != records || got.size != size {
|
||||||
|
t.Errorf("totals = %d records of %d bytes, files holds %d of %d",
|
||||||
|
got.records, got.size, records, size)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTotalsMatchFiles scans the smoke tree, rescans it after two files
|
||||||
|
// change size, and rescans it after two files are removed. After each
|
||||||
|
// scan the row of totals must agree with files.
|
||||||
|
func TestTotalsMatchFiles(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := buildSmokeTree(t)
|
||||||
|
db := openTestDB(t)
|
||||||
|
|
||||||
|
syncTree(t, db, dir)
|
||||||
|
assertTotalsMatchFiles(t, db)
|
||||||
|
|
||||||
|
writeFile(t, dir, "a/one.bin", pattern(10, 2500))
|
||||||
|
writeFile(t, dir, "t1/f1", pattern(30, 1000))
|
||||||
|
|
||||||
|
if st := syncTree(t, db, dir); st.updated != 2 {
|
||||||
|
t.Fatalf("rescan stats = %+v, want 2 updated", st)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertTotalsMatchFiles(t, db)
|
||||||
|
|
||||||
|
for _, name := range []string{"a/unique.bin", "t2/sub/f2"} {
|
||||||
|
err := os.Remove(filepath.Join(dir, name))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if st := syncTree(t, db, dir); st.removed != 2 {
|
||||||
|
t.Fatalf("rescan stats = %+v, want 2 removed", st)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertTotalsMatchFiles(t, db)
|
||||||
|
}
|
||||||
|
|
||||||
func TestApplyChangesRoundTrip(t *testing.T) {
|
func TestApplyChangesRoundTrip(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -26,11 +26,11 @@ func infoPhase(name string) string {
|
|||||||
// it. Any database problem, including a missing database, is fatal.
|
// it. Any database problem, including a missing database, is fatal.
|
||||||
//
|
//
|
||||||
// The work runs in two phases, each under a progress display of its
|
// The work runs in two phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database) and count (count the records,
|
// own on stderr: open (open the database) and count (read the number of
|
||||||
// with one query). Each display ends before the next phase starts, and
|
// records and their total size from the totals table). Each display
|
||||||
// before an error is returned to be printed. The details are written
|
// ends before the next phase starts, and before an error is returned
|
||||||
// once the last display has ended, so no display is drawn while they
|
// to be printed. The details are written once the last display has
|
||||||
// are written.
|
// ended, so no display is drawn while they are written.
|
||||||
func runInfo(ctx context.Context, stdout io.Writer) error {
|
func runInfo(ctx context.Context, stdout io.Writer) error {
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -56,7 +56,7 @@ func runInfo(ctx context.Context, stdout io.Writer) error {
|
|||||||
|
|
||||||
defer func() { _ = db.Close() }()
|
defer func() { _ = db.Close() }()
|
||||||
|
|
||||||
// One query counts everything, so the phase never has a total.
|
// One row holds both counts, so the phase never has a total.
|
||||||
prog = newProgress(progressParams{
|
prog = newProgress(progressParams{
|
||||||
label: infoPhase("count"), total: -1,
|
label: infoPhase("count"), total: -1,
|
||||||
waitingFor: countingRecords,
|
waitingFor: countingRecords,
|
||||||
|
|||||||
+6
-5
@@ -180,9 +180,9 @@ func capture(t *testing.T, std **os.File) func() string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// brokenDatabase writes a database that opens cleanly and passes the
|
// brokenDatabase writes a database that opens cleanly and passes the
|
||||||
// schema-version check but has no files table, so the first query
|
// schema-version check but has no tables, so the first query fails
|
||||||
// fails with the database already open: a fatal error on a path that
|
// with the database already open: a fatal error on a path that owns an
|
||||||
// owns an open database. It closes the database the way scan does.
|
// open database. It closes the database the way scan does.
|
||||||
func brokenDatabase(t *testing.T) string {
|
func brokenDatabase(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
@@ -265,8 +265,9 @@ func TestRunFatalAfterOpenClosesDatabase(t *testing.T) {
|
|||||||
assertFatalOutput(t, stderr.String(), stdout())
|
assertFatalOutput(t, stderr.String(), stdout())
|
||||||
|
|
||||||
// Proof that the failure happened after the open: only a
|
// Proof that the failure happened after the open: only a
|
||||||
// query against the opened database can report this.
|
// query against the opened database can report this. Each
|
||||||
if !strings.Contains(stderr.String(), "no such table: files") {
|
// subcommand's first query reads totals.
|
||||||
|
if !strings.Contains(stderr.String(), "no such table: totals") {
|
||||||
t.Errorf("stderr = %q, want the failure to come from a "+
|
t.Errorf("stderr = %q, want the failure to come from a "+
|
||||||
"query on the open database", stderr.String())
|
"query on the open database", stderr.String())
|
||||||
}
|
}
|
||||||
|
|||||||
+1
-1
@@ -60,7 +60,7 @@ const fallbackWidth = 80
|
|||||||
const spinnerMargin = 5
|
const spinnerMargin = 5
|
||||||
|
|
||||||
// countingRecords is what the display of a phase that reads the records
|
// countingRecords is what the display of a phase that reads the records
|
||||||
// says while it counts them, before it has its total.
|
// says until it has the number of records, its total.
|
||||||
const countingRecords = "counting the records"
|
const countingRecords = "counting the records"
|
||||||
|
|
||||||
// openPhase is the name of the phase in which report, trees and info
|
// openPhase is the name of the phase in which report, trees and info
|
||||||
|
|||||||
@@ -45,11 +45,11 @@ func reportPhase(name string) string {
|
|||||||
// problem, including a missing database, is fatal.
|
// problem, including a missing database, is fatal.
|
||||||
//
|
//
|
||||||
// The work runs in two phases, each under a progress display of its
|
// The work runs in two phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database) and read (count the records,
|
// own on stderr: open (open the database) and read (read the number of
|
||||||
// then read them in signature order, writing each size's duplicate
|
// records from the totals table, then read the records in signature
|
||||||
// groups to stdout once that size is read; see readReport). Each
|
// order, writing each size's duplicate groups to stdout once that size
|
||||||
// display ends before the next phase starts, and before an error is
|
// is read; see readReport). Each display ends before the next phase
|
||||||
// returned to be printed.
|
// starts, and before an error is returned to be printed.
|
||||||
func runReport(ctx context.Context, stdout io.Writer) error {
|
func runReport(ctx context.Context, stdout io.Writer) error {
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -455,8 +455,8 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
|||||||
// included, so its display counts against the number of records.
|
// included, so its display counts against the number of records.
|
||||||
func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||||
// Indexing tens of millions of records takes real time; without a
|
// Indexing tens of millions of records takes real time; without a
|
||||||
// display the scan looks hung before the walk begins. Counting them
|
// display the scan looks hung before the walk begins. The number of
|
||||||
// first gives the display its total.
|
// records, read first, gives the display its total.
|
||||||
prog := newProgress(progressParams{
|
prog := newProgress(progressParams{
|
||||||
label: scanPhase("load"), unit: scanUnit, total: -1,
|
label: scanPhase("load"), unit: scanUnit, total: -1,
|
||||||
waitingFor: countingRecords, stalledOn: "db",
|
waitingFor: countingRecords, stalledOn: "db",
|
||||||
|
|||||||
+3
-3
@@ -1228,9 +1228,9 @@ func TestScanPipeline(t *testing.T) {
|
|||||||
// TestScanProgressNamesEachPhase checks the plain progress lines of a
|
// TestScanProgressNamesEachPhase checks the plain progress lines of a
|
||||||
// first scan, and of a rescan of one operand holding two of the
|
// first scan, and of a rescan of one operand holding two of the
|
||||||
// database's records. Each phase's first line names it with its place
|
// database's records. Each phase's first line names it with its place
|
||||||
// among the four: load counts the records before reading them, and the
|
// among the four: load gets the number of records before reading them,
|
||||||
// walk estimates its total from the records under its operand, or says
|
// and the walk estimates its total from the records under its operand,
|
||||||
// its time remaining is unknown when there are none.
|
// or says its time remaining is unknown when there are none.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
func TestScanProgressNamesEachPhase(t *testing.T) {
|
func TestScanProgressNamesEachPhase(t *testing.T) {
|
||||||
|
|||||||
@@ -40,11 +40,11 @@ func treesPhase(name string) string {
|
|||||||
// database problem, including a missing database, is fatal.
|
// database problem, including a missing database, is fatal.
|
||||||
//
|
//
|
||||||
// The work runs in four phases, each under a progress display of its
|
// The work runs in four phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database), read (count the records,
|
// own on stderr: open (open the database), read (read the number of
|
||||||
// then read them into the hierarchy), group (group the directories by
|
// records from the totals table, then read the records into the
|
||||||
// digest) and write (write the groups to stdout). Each display ends
|
// hierarchy), group (group the directories by digest) and write (write
|
||||||
// before the next phase starts, and before an error is returned to be
|
// the groups to stdout). Each display ends before the next phase
|
||||||
// printed.
|
// starts, and before an error is returned to be printed.
|
||||||
func runTrees(ctx context.Context, stdout io.Writer) error {
|
func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
Reference in New Issue
Block a user