4 Commits
Author SHA1 Message Date
clawbot e9e09effc6 Add sfdupes info: show the database's details (closes #117)
check / check (push) Waiting to run
info prints the database's path and source, the sizes of the database
file and its -wal file, when the database file last changed, the schema
version and the record counts, one per line as a name, a tab and the
value. It opens the database as report does and shows an open and a
count phase on stderr. The schema version and the counts come from one
read-only transaction, the counts from one query.

seedDatabase now closes test databases out of WAL mode, as scan does;
otherwise info's own connection creates an empty -wal file. The open
phase's name and text became constants for the linter, and the scan
command moved to newScanCommand to keep newRootCommand short enough.

Judgement call: the lines are written after the last display ends, so they skip the display's stdout writer.

Model: opus-5-5
2026-10-16 22:41:16 +02:00
clawbot ebe4d4dbeb report: show its phases as N of M, with progress, elapsed time and ETA (closes #114)
check / check (push) Canceled after 0s
report now runs in two phases, open and read, under the progress
display scan and trees use, until the summary line. open covers a wait
on a busy database. read counts the records, then reads them in the
same read-only transaction in signature order, the files_signature
index read backwards, so SQLite sorts nothing and the read ends at
exactly its count. The duplicate groups are formed in Go from runs of
equal signatures; only those of the size being read are held, and they
are written, ordered by first path, before the next size. The rows go
through the display's stdout writer. stdout and the summary line are
unchanged.

Judgement call: writing is part of read, not a phase of its own, since each size's rows are written while the read goes on.

Model: opus-5-5
2026-10-16 21:44:55 +02:00
clawbot 8e0facdcd3 trees: show its phases as N of M, with progress, elapsed time and ETA (closes #113)
check / check (push) Canceled after 0s
trees now runs in four phases, open, read, group and write, each under
the progress display scan uses, until the summary line. open covers a
wait on a busy database. read counts the records and reads them in one
read-only transaction, so the two agree during a scan; group counts the
directories, and write the groups. The rows reach stdout through a
writer. When stdout and stderr are the same file, it clears the display
and holds its lock while writing, so no row shares a line with the
display on a shared terminal; otherwise it writes without the lock, so
a slow reader of stdout cannot stop the display. Each rate names what
its phase counts.

Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches.

Model: opus-5-5
2026-10-16 21:00:14 +02:00
clawbot 8dc73aa3f5 scan: show each phase as N of M, with progress, elapsed time and ETA, also while it waits (closes #112)
check / check (push) Canceled after 0s
Every scan display names its phase and its place among the five, as
"[2/5] walk", with its progress, elapsed time and ETA. load counts the
records first; the walk estimates its total from the records under the
operands; the content phase adds up the sizes of the matching groups
and says it is searching until the first record returns. Each display
redraws every 100 ms on a terminal and every 5 seconds in plain mode.

On a terminal no line reaches the last column: a line too wide drops
its rate, then its bar. After the terminal is made narrower than the
widest line drawn, the display makes a new library object, whose erase
fits.

Judgement call: load and the content checks draw the bar too, once their totals are known.
Judgement call: the bar is 10 columns wide, not the width the line leaves.

Model: opus-5-5
2026-10-16 19:11:10 +02:00
16 changed files with 1776 additions and 301 deletions
+148 -50
View File
@@ -38,8 +38,9 @@ database persists between runs; a rescan only hashes files that are new or
changed, or that may have gained a duplicate since the last scan, and removes
records for files that no longer exist. `report` reads the database and prints
the file-level duplicates report. `trees` reads the same database and prints the
duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
with no `PATH` operand — prints a usage message and exits 2.
duplicate-tree report. `info` prints the database's details: its path, the sizes
of its files and the number of its records. A missing/invalid subcommand — or a
`scan` invocation with no `PATH` operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
@@ -109,9 +110,9 @@ As a file `/etc/cron.d/sfdupes`:
- A scan still running when the next one starts carries on. The new one fails at
once, and the lines cron mails include
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
- `report` and `trees` need only read access to the database (see "Database").
Under the usual umask of `022` the first scan creates it readable by every
user, so an unprivileged user can run them against root's database.
- `report`, `trees` and `info` need only read access to the database (see
"Database"). Under the usual umask of `022` the first scan creates it readable
by every user, so an unprivileged user can run them against root's database.
### Reading the reports
@@ -199,12 +200,13 @@ Goals, in order:
under resilver). Holding one small record (path, size, mtime) per file in
memory during a scan is acceptable; holding every file's hashes is not (they
stay in the database). The reporting commands do not hold every file's
hashes either: `report` lets SQLite group and order the records and writes
each row as it reads it, so its memory does not grow with the database, and
`trees` reads the records in path order and keeps each directory's path,
digest and totals, plus the hashes of only the files in the directories
holding the record being read, so its memory grows with the number of
directories and with the size of the largest directory.
hashes either: `report` reads the records in signature order and holds the
duplicate paths of one size at a time, so its memory grows with the
duplicates of the size that has the most of them, and `trees` reads the
records in path order and keeps each directory's path, digest and totals,
plus the hashes of only the files in the directories holding the record
being read, so its memory grows with the number of directories and with the
size of the largest directory.
3. **Scan incrementally, analyze offline.** The expensive filesystem scan
maintains a persistent database; an unchanged file is never read again on a
rescan, except to compute its content hash once a file of 10 MiB or more
@@ -234,7 +236,7 @@ Goals, in order:
### Subcommands
Three subcommands, all implemented:
Four subcommands, all implemented:
1. `scan` — walk the filesystem and synchronize the database: one signature
record per regular file.
@@ -242,11 +244,14 @@ Three subcommands, all implemented:
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
from the database records, compute a Merkle-style digest per directory, and
report maximal groups of identical trees.
4. `info` — the database's details: its path, the sizes of its files, its
schema version and the counts of its records.
```
sfdupes scan [--workers N] [-x] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
sfdupes info
sfdupes --version
sfdupes [command] --help
```
@@ -261,7 +266,7 @@ stdout.
### Database
All three subcommands operate on a single SQLite database file:
All four subcommands operate on a single SQLite database file:
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
@@ -284,13 +289,13 @@ All three subcommands operate on a single SQLite database file:
symlinks resolved. After a `cd` through a symlink to a directory, a path
starting `../` therefore names a file beside the directory the symlink leads
to, as it does for any other program. `scan` says `new database` when no
file is at that path as it starts. `report` and `trees` print the line
before they open the database, so the error for a missing database comes
after it.
file is at that path as it starts. `report`, `trees` and `info` print the
line before they open the database, so the error for a missing database
comes after it.
- `scan` creates the database (and its parent directory) on first use. `report`
and `trees` require an existing database; a missing database file is a fatal
error (exit 1) telling the user to run `scan` first.
- `scan` creates the database (and its parent directory) on first use. `report`,
`trees` and `info` require an existing database; a missing database file is a
fatal error (exit 1) telling the user to run `scan` first.
- Only one `scan` runs against a database at a time. For its whole run, `scan`
holds an exclusive `flock(2)` lock on a lock file beside the database, named
by appending `.lock` to the database's path with symlinks resolved
@@ -306,8 +311,8 @@ All three subcommands operate on a single SQLite database file:
on. The lock file is created on first use, open to its owner only, and left in
place: a leftover file blocks nothing, because the lock ends with the process
holding it however it ends, a fatal error or an interrupt included, and
deleting the file while a scan runs would let a second scan start. `report`
and `trees` never take the lock, so they run during a scan.
deleting the file while a scan runs would let a second scan start. `report`,
`trees` and `info` never take the lock, so they run during a scan.
- While `scan` runs, the database is in WAL journal mode with a busy timeout, so
running a report while a cron `scan` is in progress is safe. The filesystem is
authoritative; the database is an eventually-consistent reflection of it.
@@ -323,9 +328,9 @@ All three subcommands operate on a single SQLite database file:
needs the database to itself: a `scan` that starts while a report is still
reading waits for it up to the 10-second busy timeout, then fails; a `scan`
that ends while a report has the database open warns and leaves the database
in WAL mode until the next scan. `report` writes each row as it reads it, so
it is still reading while its output is paused (a pager, a stalled pipe), and
a `scan` started then fails after the busy timeout.
in WAL mode until the next scan. `report` writes its rows while it reads the
records, so it is still reading while its output is paused (a pager, a stalled
pipe), and a `scan` started then fails after the busy timeout.
- A `scan` writes to the database and the files beside it at these points (its
phases are described under "`scan` mode"):
- As it starts, before the walk: the directory, an empty database file and
@@ -340,10 +345,10 @@ All three subcommands operate on a single SQLite database file:
- In the content phase: content hashes, again committed 10,000 at a time.
- At the end, as it closes the database: the `-wal` and `-shm` files go
away, unless a report still has the database open (see above).
- `report` and `trees` open the database read-only and need only read access to
the database file, and no write access to its directory. While the database is
in WAL mode they also read the `-wal` and `-shm` files beside it, which SQLite
creates with the database file's permissions.
- `report`, `trees` and `info` open the database read-only and need only read
access to the database file, and no write access to its directory. While the
database is in WAL mode they also read the `-wal` and `-shm` files beside it,
which SQLite creates with the database file's permissions.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
with any other version is a fatal error. `scan` creates the schema and sets
the version in one transaction, so a first scan stopped while doing so leaves
@@ -379,8 +384,8 @@ All three subcommands operate on a single SQLite database file:
a file of 10 MiB or more, `content` stays empty until the content phase of a
scan (see "`scan` mode" below) has read the file. A record with an empty
`content` is never part of a duplicate group, though it still defines the
file for tree reconstruction. The `files_signature` index lets SQLite group
the records by signature for `report` without sorting the whole table.
file for tree reconstruction. The `files_signature` index lets `report` read
the records in signature order without sorting the table.
### Duplicate detection
@@ -604,11 +609,8 @@ total.)
**`report` must never touch the filesystem being analyzed.** It does not stat,
open, or otherwise access any path that appears in the records; its only I/O is
reading the database, writing stdout/stderr, and the temporary file SQLite sorts
in when the duplicate rows do not fit in memory. SQLite puts that file in
`$SQLITE_TMPDIR` or `$TMPDIR` when set, otherwise in `/var/tmp` (or `/tmp`), and
deletes it as soon as it has opened it. `report` must produce identical output
whether or not the scanned filesystem is still mounted.
reading the database and writing stdout/stderr. `report` must produce identical
output whether or not the scanned filesystem is still mounted.
Processing:
@@ -622,6 +624,16 @@ Processing:
by `first` path ascending. Output must be fully deterministic for a given
database state.
`report` runs in two phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in signature order, largest size first. That is the order of the
`files_signature` index read backwards, so SQLite sorts nothing and returns the
first record at once. The records sharing a signature come together, and so do
the records of one size, so `report` holds only the duplicate groups of the size
it is reading, and writes them, ordered by `first` path, once it reaches a
smaller size. Its memory holds at most the duplicate paths of one size at a
time, never every record.
#### Report output format
TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
@@ -695,6 +707,12 @@ Processing:
- Order groups by total tree size descending, tie-broken by `first` path
ascending. Output must be fully deterministic for a given input.
`trees` runs in four phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in path order, building the hierarchy and computing the digests as it goes;
`group` groups the directories by digest and picks the maximal groups; `write`
writes the groups to stdout.
#### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
@@ -713,15 +731,60 @@ Summary to stderr: records read, number of duplicate-tree groups, number of dupe
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
units.
### `info` mode
`info` takes no positional arguments and prints the database's details on
stdout, one per line: a name, a tab, and the value. It opens the database the
way `report` does, read-only and without the scan lock, so it runs during a scan
and needs only read access (see "Database"). Like `report`, it never touches the
scanned filesystem.
```
database /var/lib/sfdupes/db.sqlite (default)
database file size 2147483648 (2.0 GiB)
-wal file size 4124152 (3.9 MiB)
modified 2026-10-16T12:02:50Z
schema version 1
records 12345678
records with head and tail hashes 1234567
records with content hash 123456
size of recorded files 150000000000000 (136.4 TiB)
```
- `database` is the path and where it came from, as on the line every subcommand
prints first on stderr (see "Database").
- `database file size` and `-wal file size` are in bytes, then in human units.
The `-wal` line is there only when a `-wal` file is beside the database, as
while a scan runs.
- `modified` is the time the database file was last changed, in UTC. While the
database is in WAL mode, the newest changes may be in the `-wal` file alone.
- `schema version` is the database's `PRAGMA user_version`. A database with
another version than this build's is refused, as by `report`.
- `records` is the number of records. `records with head and tail hashes` and
`records with content hash` count the records that have those hashes (see
"Database"), and `size of recorded files` adds up the sizes of all records, in
bytes, then in human units.
`info` runs in two phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `count` reads the schema version and
counts the records with one query, a single pass over the table, both in one
read-only transaction, so the numbers agree with each other while a scan is
committing. `info` writes its lines once the `count` display has ended, so on a
terminal they come after the displays.
### Progress
Use the progress-bar library for all scan progress; rendering in the style of
`pv` is the model. All progress goes to stderr.
Use the progress-bar library for all progress of `scan`, `report`, `trees` and
`info`; rendering in the style of `pv` is the model. All progress goes to
stderr.
A scan must always show that it is working and not hung. Each phase gets its own
display from the moment it starts, naming the phase and its place among the five
(`[1/5] load`, `[2/5] walk`, `[3/5] hash`, `[4/5] update`, `[5/5] content`) and
showing its progress, its elapsed time and its estimated time remaining (ETA):
`scan`, `report`, `trees` and `info` must always show that they are working and
not hung. Each phase gets its own display from the moment it starts, naming the
phase and its place among the command's phases and showing its progress, its
elapsed time and its estimated time remaining (ETA).
A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
`[4/5] update`, `[5/5] content`):
- `load` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
@@ -738,12 +801,39 @@ showing its progress, its elapsed time and its estimated time remaining (ETA):
and ETA, start there. The files to read then get a display of their own, with
their count as its total.
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
`[4/4] write`, see "`trees` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `group` has the directories `read` found as its total, and `write` the
duplicate-tree groups.
`report` has two phases (`[1/2] open`, `[2/2] read`, see "`report` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count, writing
the rows to stdout as it goes. While the count runs, the display says it is
counting the records.
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `count` has no total either, because one query counts everything: the display
says it is counting the records until the query returns, and its ETA is
unknown.
Each display is one line holding:
- the phase and its place among the phases
- a `[m/n] x%` display (items processed / total items, percent); before a phase
has a total, the items processed, or what the phase is waiting for
- current rate (items/s)
- current rate (items/s), naming the items the phase counts: files, records,
directories or groups
- elapsed time
- estimated time remaining, or `unknown`
@@ -756,7 +846,8 @@ Example shape (exact layout is flexible, content is not):
Additional requirements:
- On a terminal, a phase with an exact total (every phase but `walk`, once it
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
the `open` phase of `report` and `trees`, and both phases of `info`, once it
has counted what it will do) is drawn as its line followed by the progress-bar
library's bar, 10 columns wide, which shows the percent in place of the line's
own. Any other phase, and a phase whose bar has no room, is drawn as the
@@ -770,7 +861,9 @@ Additional requirements:
gives way to the spinner. Only when those four alone are too wide, as on a
terminal narrower than about 70 columns hours into a scan of millions of
files, is the line cut short at the terminal's edge, so the end of the ETA is
lost first.
lost first. A terminal made narrower while a phase runs gets the same: the
next draw erases the line within the new width and fits the line to it, and
the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once
@@ -779,21 +872,26 @@ Additional requirements:
goroutine counts the items, and each display draws them from a goroutine of
its own.
- A warning printed during a phase always lands on a line of its own, never
inside the progress display.
inside the progress display. So does every line written to stdout when stdout
and stderr are the same file (the same device and inode): one terminal, as in
a plain `sfdupes report` or `sfdupes trees`, or one pipe or file that both are
sent to. When they are not, stdout is written without waiting on the display,
and the display keeps being drawn while a slow reader of stdout holds up a
write.
- A phase that stops short of its total, as an interrupted one does, is left
showing the count it stopped at rather than filled up.
- `report` and `trees` modes need no progress display, only their stderr
summaries.
### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is
already running against the same database, the database cannot be
created/opened/read/written, a missing database for `report`/`trees`, stdout
write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see below).
created/opened/read/written, a missing database for `report`/`trees`/`info`,
stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see
below).
- `2`: usage error (including `scan` with no `PATH` operand, `scan` with
`--workers` below 1, and `report`/`trees` with any positional argument).
`--workers` below 1, and `report`/`trees`/`info` with any positional
argument).
A stdout write failure, such as a full disk, is reported in one line on stderr
and exits 1. Two cases never reach sfdupes as a failed write:
+40 -6
View File
@@ -29,6 +29,40 @@
# Completed Steps
- new subcommand `info` prints the database's details on stdout, one per line as
a name, a tab and the value: its path and where it came from, the sizes of the
database file and of its `-wal` file when there is one, when the database file
was last changed, the schema version, the number of records, how many have
head and tail hashes and how many a content hash, and the sizes of all records
added up. It opens the database as `report` does, and shows two phases on
stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come
from one read-only transaction, the counts from one query (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/117)
- `report` shows its work on stderr in two phases, `[1/2] open` and
`[2/2] read`, with the progress display `scan` and `trees` use, from the
moment it starts until the summary line: `open` says it is opening the
database, also while it waits on a busy one; `read` counts the records, then
reads them against that count in signature order, which SQLite takes from the
`files_signature` index without sorting, so the first record comes at once.
`report` forms the duplicate groups itself, holding only those of the size it
is reading, and writes them ordered by first path before it reads the next
size. stdout and the summary line are unchanged, and SQLite no longer needs a
temporary file to sort in (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/114)
- `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`,
`[3/4] group` and `[4/4] write`, each with the progress display `scan` uses,
from the moment it starts until the summary line: `open` says it is opening
the database, also while it waits on a busy one; `read` counts the records,
then reads them against that count; `group` counts the directories it groups,
and `write` the groups it writes. A display's rate names what its phase
counts: files, records, directories or groups. stdout and the summary line are
unchanged. When stdout and stderr are the same file, as one terminal is, none
of stdout's lines shares a line with the display; when they are not, the
display keeps being drawn while stdout waits on a slow reader (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/113)
- each `scan` phase's progress display names the phase and its place among the
five, as `[2/5] walk`, from the moment it starts, with its progress, elapsed
time and ETA: `load` counts the records first, the walk's total is estimated
@@ -37,12 +71,12 @@
until the first one is returned. On a terminal a phase with an exact total
draws its line followed by the library's bar, and any other a spinner followed
by the line; every display is drawn again several times a second. No line
reaches the terminal's last column, however narrow: a line too wide leaves out
its rate, then its bar, keeping the phase, progress, elapsed time and ETA
whole, and is cut short only when those alone do not fit, as below about 70
columns late in a large scan. In plain mode the line is printed every 5
seconds whether or not an item completed (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/112)
reaches the terminal's last column, however narrow, also after the terminal is
made narrower during a phase: a line too wide leaves out its rate, then its
bar, keeping the phase, progress, elapsed time and ETA whole, and is cut short
only when those alone do not fit, as below about 70 columns late in a large
scan. In plain mode the line is printed every 5 seconds whether or not an item
completed (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/112)
- `scan`, `report` and `trees` first print a line on stderr naming the database
they use, as an absolute path, and whether that path is the default or comes
+124 -50
View File
@@ -61,8 +61,8 @@ CREATE TABLE files (
) WITHOUT ROWID
`
// createIndexSQL indexes the records by signature, so report can have
// SQLite group them without sorting the whole table.
// createIndexSQL indexes the records by signature, so report can read
// them in signature order without sorting the table.
const createIndexSQL = `
CREATE INDEX files_signature ON files (size, head, tail, content)
`
@@ -79,7 +79,8 @@ ON CONFLICT (path) DO UPDATE SET
content = excluded.content
`
// errNoDatabase reports a missing database file for report/trees.
// errNoDatabase reports a missing database file for report, trees and
// info.
var errNoDatabase = errors.New(
"no database (run \"sfdupes scan\" first, or set " + databaseEnv + ")")
@@ -131,9 +132,9 @@ const scanParams = "_pragma=busy_timeout(10000)" +
"&_pragma=journal_mode(WAL)" +
"&_pragma=synchronous(NORMAL)"
// reportParams are the connection parameters for report and trees:
// read-only, with the same busy timeout. They set no journal mode,
// because setting one is a write.
// reportParams are the connection parameters for report, trees and
// info: read-only, with the same busy timeout. They set no journal
// mode, because setting one is a write.
const reportParams = "mode=ro" +
"&_pragma=busy_timeout(10000)" +
"&_pragma=query_only(1)"
@@ -262,9 +263,9 @@ func closeScanDatabase(ctx context.Context, db *sql.DB, path string) {
_ = db.Close()
}
// openReportDatabase opens an existing database for the report and
// trees subcommands. A missing database file is an error directing the
// user to run scan first; the schema version must match exactly.
// openReportDatabase opens an existing database for the report, trees
// and info subcommands. A missing database file is an error directing
// the user to run scan first; the schema version must match exactly.
func openReportDatabase(ctx context.Context,
path string,
) (*sql.DB, error) {
@@ -403,10 +404,32 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
return v, nil
}
// loadFileRows streams every record to fn in path order: byte order,
// which is the order of the primary key, so SQLite does not sort.
func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
rows, err := db.QueryContext(ctx,
// loadFileRows gives total the number of records, then streams every
// record to fn in path order: byte order, which is the order of the
// primary key, so SQLite does not sort. The count and the records are
// read in one transaction, so they agree while a scan is committing.
func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
fn func(r scanRec),
) error {
// Everything goes through tx: db has one connection, so a query on
// db would wait for tx forever.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return fmt.Errorf("read records: %w", err)
}
defer func() { _ = tx.Rollback() }()
var n int64
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
if err != nil {
return fmt.Errorf("count records: %w", err)
}
total(n)
rows, err := tx.QueryContext(ctx,
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
"FROM files ORDER BY path")
if err != nil {
@@ -441,77 +464,74 @@ func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
return nil
}
// dupeRowsSQL selects every record in a duplicate group, with the
// group's first path. A group is the records with a content hash that
// share a size, head, tail, and content, when there are two or more of
// them. The rows come in report order: groups by size descending, then
// by first path, and each group's paths ascending.
const dupeRowsSQL = `
SELECT g.first, f.path, f.size
FROM files AS f
JOIN (
SELECT size, head, tail, content, MIN(path) AS first
FROM files
WHERE content <> ''
GROUP BY size, head, tail, content
HAVING COUNT(*) > 1
) AS g USING (size, head, tail, content)
ORDER BY f.size DESC, g.first, f.path
// signatureRowsSQL selects every record's signature and path in
// signature order, largest size first. That is the order of the
// files_signature index read backwards, so SQLite reads the index alone
// and returns the first row at once, with nothing to sort.
const signatureRowsSQL = `
SELECT size, head, tail, content, path
FROM files
ORDER BY size DESC, head DESC, tail DESC, content DESC
`
// loadDupeRows streams the rows of dupeRowsSQL to fn and returns the
// number of records in the database. The count and the rows are read
// in one transaction, so they agree while a scan is committing. An
// error from fn stops the reading and is returned as it is.
func loadDupeRows(ctx context.Context, db *sql.DB,
fn func(first, path string, size int64) error,
) (int, error) {
// loadSignatureRows gives total the number of records, then streams
// every record's signature and path to fn, without its mtime, in the
// order of signatureRowsSQL. The count and the records are read in one
// transaction, so they agree while a scan is committing. An error from
// fn stops the reading and is returned as it is.
func loadSignatureRows(ctx context.Context, db *sql.DB,
total func(n int64), fn func(r scanRec) error,
) error {
// Everything goes through tx: the report connection is the only
// one, so a query on db would wait for tx forever.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return 0, fmt.Errorf("read records: %w", err)
return fmt.Errorf("read records: %w", err)
}
defer func() { _ = tx.Rollback() }()
var records int
var n int64
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&records)
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
if err != nil {
return 0, fmt.Errorf("read records: %w", err)
return fmt.Errorf("count records: %w", err)
}
rows, err := tx.QueryContext(ctx, dupeRowsSQL)
total(n)
rows, err := tx.QueryContext(ctx, signatureRowsSQL)
if err != nil {
return 0, fmt.Errorf("read records: %w", err)
return fmt.Errorf("read records: %w", err)
}
defer func() { _ = rows.Close() }()
for rows.Next() {
var (
first, path []byte
size int64
path []byte
r scanRec
)
err = rows.Scan(&first, &path, &size)
err = rows.Scan(&r.size, &r.head, &r.tail, &r.content, &path)
if err != nil {
return 0, fmt.Errorf("read record: %w", err)
return fmt.Errorf("read record: %w", err)
}
err = fn(string(first), string(path), size)
r.path = string(path)
err = fn(r)
if err != nil {
return 0, err
return err
}
}
err = rows.Err()
if err != nil {
return 0, fmt.Errorf("read records: %w", err)
return fmt.Errorf("read records: %w", err)
}
return records, nil
return nil
}
// countRecords returns the number of records in the database.
@@ -526,6 +546,60 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
return n, nil
}
// databaseCountsSQL counts the records, those with head and tail
// hashes and those with a content hash, and adds up their sizes, in one
// pass over the table. A SUM over no rows is NULL, so on an empty table
// each SUM falls back to 0.
const databaseCountsSQL = `
SELECT COUNT(*),
COALESCE(SUM(head <> '' AND tail <> ''), 0),
COALESCE(SUM(content <> ''), 0),
COALESCE(SUM(size), 0)
FROM files
`
// databaseCounts is what info reads from the database: its schema
// version and the counts of databaseCountsSQL.
type databaseCounts struct {
schemaVersion int
records int64 // every record
headTail int64 // records with head and tail hashes
content int64 // records with a content hash
size int64 // the sizes of all records added up
}
// readDatabaseCounts reads the schema version, then the counts of
// databaseCountsSQL, in one read-only transaction, so both describe the
// database at one moment while a scan is committing.
func readDatabaseCounts(ctx context.Context,
db *sql.DB,
) (databaseCounts, error) {
var c databaseCounts
// Everything goes through tx: the report connection is the only
// one, so a query on db would wait for tx forever.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return c, fmt.Errorf("count records: %w", err)
}
defer func() { _ = tx.Rollback() }()
err = tx.QueryRowContext(ctx, "PRAGMA user_version").
Scan(&c.schemaVersion)
if err != nil {
return c, fmt.Errorf("read schema version: %w", err)
}
err = tx.QueryRowContext(ctx, databaseCountsSQL).
Scan(&c.records, &c.headTail, &c.content, &c.size)
if err != nil {
return c, fmt.Errorf("count records: %w", err)
}
return c, nil
}
// loadFileMeta streams every record's path, size, mtime, and whether
// it carries hashes to fn. Scan change detection needs no hash
// values, and skipping the hash columns keeps the scan's in-memory
+117 -1
View File
@@ -284,12 +284,128 @@ func TestCloseScanDatabaseWhileReportOpen(t *testing.T) {
defer func() { _ = reportDB.Close() }()
err = loadFileRows(t.Context(), reportDB, func(scanRec) {})
err = loadFileRows(t.Context(), reportDB, func(int64) {},
func(scanRec) {})
if err != nil {
t.Fatalf("loadFileRows: %v", err)
}
}
// TestLoadRowsCountAgreeDuringScan has a scan commit records after
// loadFileRows, as trees calls it, or loadSignatureRows, as report
// calls it, has counted the records and before it reads them, as when
// trees or report runs during a scan: each must read exactly as many
// records as it counted.
func TestLoadRowsCountAgreeDuringScan(t *testing.T) {
t.Parallel()
loads := map[string]func(context.Context, *sql.DB, func(int64),
func(scanRec)) error{
"loadFileRows": loadFileRows,
"loadSignatureRows": func(ctx context.Context, db *sql.DB,
total func(int64), fn func(scanRec),
) error {
return loadSignatureRows(ctx, db, total, func(r scanRec) error {
fn(r)
return nil
})
},
}
for name, load := range loads {
t.Run(name, func(t *testing.T) {
t.Parallel()
path := testDBPath(t)
scanDB, err := openScanDatabase(t.Context(), path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = scanDB.Close() }()
recs := smokeTreeRecs()
err = applyChanges(t.Context(), scanDB, recs[:3], nil, nil)
if err != nil {
t.Fatal(err)
}
reportDB, err := openReportDatabase(t.Context(), path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = reportDB.Close() }()
var counted, read int64
err = load(t.Context(), reportDB, func(n int64) {
counted = n
err := applyChanges(t.Context(), scanDB, recs[3:], nil, nil)
if err != nil {
t.Error(err)
}
}, func(scanRec) { read++ })
if err != nil {
t.Fatal(err)
}
if counted != 3 || read != counted {
t.Errorf("counted %d records and read %d, want 3 and 3",
counted, read)
}
})
}
}
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
// from the files_signature index alone, with nothing to sort: the
// first record comes at once, so report's read phase counts each
// record as it comes rather than waiting on SQLite for all of them.
func TestSignatureRowsNeedNoSort(t *testing.T) {
t.Parallel()
db := openTestDB(t)
rows, err := db.QueryContext(t.Context(),
"EXPLAIN QUERY PLAN "+signatureRowsSQL)
if err != nil {
t.Fatal(err)
}
defer func() { _ = rows.Close() }()
var plan []string
for rows.Next() {
var (
id, parent, unused int
detail string
)
err = rows.Scan(&id, &parent, &unused, &detail)
if err != nil {
t.Fatal(err)
}
plan = append(plan, detail)
}
err = rows.Err()
if err != nil {
t.Fatal(err)
}
want := []string{"SCAN files USING COVERING INDEX files_signature"}
if !slices.Equal(plan, want) {
t.Errorf("query plan = %q, want %q", plan, want)
}
}
func TestApplyChangesRoundTrip(t *testing.T) {
t.Parallel()
+128
View File
@@ -0,0 +1,128 @@
package main
import (
"context"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"strings"
"time"
)
// infoPhase names one phase of info with its place among all of them,
// as "[2/2] count", for the phase's progress display. The list here is
// the one place the phases are counted.
func infoPhase(name string) string {
return phaseLabel([]string{openPhase, "count"}, name)
}
// runInfo implements the info subcommand: it prints the database's
// details on stdout (see writeInfo). It opens the database as report
// does, read-only and without the scan lock, and never touches the
// scanned filesystem. It names the database on stderr before it opens
// it. Any database problem, including a missing database, is fatal.
//
// The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and count (count the records,
// with one query). Each display ends before the next phase starts, and
// before an error is returned to be printed. The details are written
// once the last display has ended, so no display is drawn while they
// are written.
func runInfo(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr, "info: database %s (%s)\n",
escapePath(dbPath), source)
// Opening can wait on a busy database for as long as the busy
// timeout.
prog := newProgress(progressParams{
label: infoPhase(openPhase), total: -1,
waitingFor: openingDatabase,
})
db, err := openReportDatabase(ctx, dbPath)
prog.finish()
if err != nil {
return err
}
defer func() { _ = db.Close() }()
// One query counts everything, so the phase never has a total.
prog = newProgress(progressParams{
label: infoPhase("count"), total: -1,
waitingFor: countingRecords,
})
counts, err := readDatabaseCounts(ctx, db)
prog.finish()
if err != nil {
return fmt.Errorf("database %s: %w", dbPath, err)
}
return writeInfo(stdout, dbPath, source, counts)
}
// writeInfo writes the details of the database at dbPath to stdout,
// one per line, as a name, a tab and the value (README "`info` mode"):
// the path and source as databasePath returns them, the sizes of the
// database file and of its -wal file when there is one, when the
// database file was last changed, and the schema version and record
// counts that readDatabaseCounts read.
func writeInfo(stdout io.Writer, dbPath, source string,
counts databaseCounts,
) error {
// SQLite keeps the -wal file beside the database file, named from
// its path with symlinks resolved.
file, err := filepath.EvalSymlinks(dbPath)
if err != nil {
return fmt.Errorf("database: %w", err)
}
dbFile, err := os.Stat(file)
if err != nil {
return fmt.Errorf("database: %w", err)
}
walFile, err := os.Stat(file + "-wal")
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return fmt.Errorf("database: %w", err)
}
var b strings.Builder
fmt.Fprintf(&b, "database\t%s (%s)\n", escapePath(dbPath), source)
fmt.Fprintf(&b, "database file size\t%d (%s)\n",
dbFile.Size(), humanBytes(dbFile.Size()))
if walFile != nil {
fmt.Fprintf(&b, "-wal file size\t%d (%s)\n",
walFile.Size(), humanBytes(walFile.Size()))
}
fmt.Fprintf(&b, "modified\t%s\n",
dbFile.ModTime().UTC().Format(time.RFC3339))
fmt.Fprintf(&b, "schema version\t%d\n", counts.schemaVersion)
fmt.Fprintf(&b, "records\t%d\n", counts.records)
fmt.Fprintf(&b, "records with head and tail hashes\t%d\n",
counts.headTail)
fmt.Fprintf(&b, "records with content hash\t%d\n", counts.content)
fmt.Fprintf(&b, "size of recorded files\t%d (%s)\n",
counts.size, humanBytes(counts.size))
_, err = io.WriteString(stdout, b.String())
if err != nil {
return fmt.Errorf("write stdout: %w", err)
}
return nil
}
+168
View File
@@ -0,0 +1,168 @@
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// infoOpenLine is the plain line of info's open phase when it opens the
// database at once; it prints it as the phase starts and as it ends.
const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown"
// infoCountLine is the plain line of info's count phase when the count
// takes under a second; it prints it as the phase starts and as it
// ends.
const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " +
"eta unknown"
// infoRecs are four records: a duplicate pair with every hash, a file
// of 20 MiB with head and tail hashes but no content hash yet, and a
// file never hashed.
func infoRecs() []scanRec {
return []scanRec{
{size: 5, head: "h", tail: "t", content: "c", path: "/d/a"},
{size: 5, head: "h", tail: "t", content: "c", path: "/d/b"},
{size: 20 << 20, head: "bh", tail: "bt", path: "/d/big"},
{size: 7, path: "/d/unique"},
}
}
// TestRunInfo runs info on an empty and on a filled database, with
// stdout and stderr the same file that is not a terminal, and checks
// everything written there: the line naming the database, each phase's
// line as it starts and as it ends, and then the details, each on a
// line of its own. No -wal file is beside a database scan has closed.
func TestRunInfo(t *testing.T) {
cases := map[string]struct {
recs []scanRec
want string // the details after the database file's time
}{
"empty": {
want: "schema version\t1\n" +
"records\t0\n" +
"records with head and tail hashes\t0\n" +
"records with content hash\t0\n" +
"size of recorded files\t0 (0 B)\n",
},
"filled": {
recs: infoRecs(),
want: "schema version\t1\n" +
"records\t4\n" +
"records with head and tail hashes\t3\n" +
"records with content hash\t2\n" +
"size of recorded files\t20971537 (20.0 MiB)\n",
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
path := seedDatabase(t, tc.recs)
t.Setenv(databaseEnv, path)
file, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
stderr := captureStderr(t)
code := run([]string{cmdInfo}, os.Stderr, os.Stderr)
if code != exitOK {
t.Fatalf("run(info) = %d, want %d; stderr: %s",
code, exitOK, stderr())
}
named := path + " (from " + databaseEnv + ")\n"
want := "info: database " + named +
infoOpenLine + "\n" + infoOpenLine + "\n" +
infoCountLine + "\n" + infoCountLine + "\n" +
"database\t" + named +
fmt.Sprintf("database file size\t%d (%s)\n",
file.Size(), humanBytes(file.Size())) +
"modified\t" + file.ModTime().UTC().Format(time.RFC3339) +
"\n" + tc.want
if got := stderr(); got != want {
t.Errorf("stdout and stderr = %q, want %q", got, want)
}
})
}
}
// TestRunInfoDuringScan runs info while a scan holds the lock and has
// the database open in WAL mode, with records committed to the -wal
// file only. info must count those records and show the -wal file's
// size, also when the database path is a symlink: SQLite keeps the
// -wal file beside the symlink's target.
func TestRunInfoDuringScan(t *testing.T) {
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
db, err := openScanDatabase(t.Context(), path)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
err = applyChanges(t.Context(), db, infoRecs(), nil, nil)
if err != nil {
t.Fatal(err)
}
holdScanLock(t, path)
wal, err := os.Stat(path + "-wal")
if err != nil {
t.Fatal(err)
}
for _, named := range []string{path, link} {
t.Run(filepath.Base(named), func(t *testing.T) {
t.Setenv(databaseEnv, named)
stdout := runStdout(t, cmdInfo)
for _, want := range []string{
"database\t" + named + " (from " + databaseEnv + ")\n",
fmt.Sprintf("\n-wal file size\t%d (%s)\n",
wal.Size(), humanBytes(wal.Size())),
"\nrecords\t4\n",
} {
if !strings.Contains(stdout, want) {
t.Errorf("stdout = %q, want it to hold %q", stdout, want)
}
}
})
}
}
func TestRunInfoNeedsOnlyReadAccess(t *testing.T) {
// README "`info` mode": like report, info needs only read access to
// the database file and none to its directory.
path := testDBPath(t)
t.Setenv(databaseEnv, path)
scanFixture(t)
makeReadOnly(t, path)
stdout := runStdout(t, cmdInfo)
if want := "\nrecords\t2\n"; !strings.Contains(stdout, want) {
t.Errorf("stdout = %q, want it to hold %q", stdout, want)
}
if strings.Contains(stdout, "-wal") {
t.Errorf("stdout = %q, want no -wal file between scans", stdout)
}
}
+41 -24
View File
@@ -8,13 +8,14 @@
// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
// or larger. scan maintains a persistent SQLite database of file
// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
// that the reporting subcommands read.
// that the other subcommands read.
//
// Usage:
//
// sfdupes scan [--workers N] [-x] PATH...
// sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv
// sfdupes info
// sfdupes --version
//
// See README.md for the complete specification.
@@ -44,6 +45,7 @@ const (
cmdScan = "scan"
cmdReport = "report"
cmdTrees = "trees"
cmdInfo = "info"
)
// errNoSubcommand is returned by the root command when it is invoked
@@ -73,8 +75,8 @@ func main() {
// exit code. It is the program's single exit point: the subcommands
// return their errors instead of exiting, so every deferred cleanup —
// above all closing the database, which checkpoints the SQLite WAL —
// runs before the process ends. The report and trees subcommands write
// their data to stdout.
// runs before the process ends. The report, trees and info subcommands
// write their data to stdout.
func run(args []string, stdout, stderr io.Writer) int {
// A nil slice makes cobra fall back to os.Args, which would let a
// test binary's own flags reach the command tree.
@@ -148,6 +150,41 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
root.Flags().BoolVarP(&showVersion, "version", "v", false,
"print the version to stdout")
reportCmd := &cobra.Command{
Use: cmdReport,
Short: "Read the scan database and print the file-level duplicates report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runReport(ctx, stdout)
}),
}
treesCmd := &cobra.Command{
Use: cmdTrees,
Short: "Read the scan database and print the duplicate-tree report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runTrees(ctx, stdout)
}),
}
infoCmd := &cobra.Command{
Use: cmdInfo,
Short: "Read the scan database and print its details",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runInfo(ctx, stdout)
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
return root
}
// newScanCommand builds the scan subcommand, with its --workers and -x
// flags.
func newScanCommand() *cobra.Command {
var (
scanWorkers int
scanOneFS bool
@@ -172,27 +209,7 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
reportCmd := &cobra.Command{
Use: cmdReport,
Short: "Read the scan database and print the file-level duplicates report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runReport(ctx, stdout)
}),
}
treesCmd := &cobra.Command{
Use: cmdTrees,
Short: "Read the scan database and print the duplicate-tree report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runTrees(ctx, stdout)
}),
}
root.AddCommand(scanCmd, reportCmd, treesCmd)
return root
return scanCmd
}
// checkScanWorkers rejects a scan --workers value below 1. That is a
+49 -19
View File
@@ -284,10 +284,16 @@ func assertFatalOutput(t *testing.T, stderr, stdout string) {
}
func TestRunMissingDatabaseIsFatal(t *testing.T) {
// README §Database: report and trees need an existing database; a
// missing one exits 1 with a message telling the user to run scan,
// after the line naming the database.
for _, name := range []string{cmdReport, cmdTrees} {
// README §Database: report, trees and info need an existing
// database; a missing one exits 1 with a message telling the user to
// run scan, after the line naming the database.
openLines := map[string]string{
cmdReport: reportOpenLine,
cmdTrees: treesOpenLine,
cmdInfo: infoOpenLine,
}
for name, openLine := range openLines {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
@@ -303,8 +309,10 @@ func TestRunMissingDatabaseIsFatal(t *testing.T) {
t.Errorf("run(%s) = %d, want %d", name, code, exitFatal)
}
// Each shows its open phase, as it starts and as it ends,
// before the error.
want := name + ": database " + path + " (from " + databaseEnv +
")\n" +
")\n" + openLine + "\n" + openLine + "\n" +
"sfdupes: " + path + ": no database (run \"sfdupes " +
"scan\" first, or set " + databaseEnv + ")\n"
if got := stderr(); got != want {
@@ -505,8 +513,9 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
// exist at the default path on the machine running the test, so
// nothing here may create or read one: scan gets an operand that
// does not exist, which stops it before it takes the lock, and
// report and trees a cancelled context, which stops them before
// they read anything.
// report, trees and info a cancelled context, which stops them
// before they read anything. They show their open phase after the
// line.
t.Setenv(databaseEnv, "")
// Whether scan calls the database new depends on that machine too.
@@ -534,13 +543,22 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
run: func() error {
return runReport(cancelledContext(t), io.Discard)
},
want: "report: database " + defaultDatabasePath + " (default)\n",
want: "report: database " + defaultDatabasePath + " (default)\n" +
reportOpenLine + "\n" + reportOpenLine + "\n",
},
cmdTrees: {
run: func() error {
return runTrees(cancelledContext(t), io.Discard)
},
want: "trees: database " + defaultDatabasePath + " (default)\n",
want: "trees: database " + defaultDatabasePath + " (default)\n" +
treesOpenLine + "\n" + treesOpenLine + "\n",
},
cmdInfo: {
run: func() error {
return runInfo(cancelledContext(t), io.Discard)
},
want: "info: database " + defaultDatabasePath + " (default)\n" +
infoOpenLine + "\n" + infoOpenLine + "\n",
},
}
@@ -568,6 +586,7 @@ func TestRunUsageErrors(t *testing.T) {
"scan without paths": {[]string{cmdScan}, usageMarker},
"report with args": {[]string{cmdReport, "x"}, usageMarker},
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
"unknown subcommand": {[]string{"nope"}, "unknown command"},
}
@@ -908,11 +927,16 @@ func TestRunReportSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want)
}
want = "report: database " + path + " (from " + databaseEnv + ")\n" +
"report: 2 records read, 1 duplicate groups, 1 dupe files, " +
// Its phases come between the two lines; see
// TestRunReportProgressNamesEachPhase.
first := "report: database " + path + " (from " + databaseEnv + ")\n"
last := "\nreport: 2 records read, 1 duplicate groups, 1 dupe files, " +
"300 B reclaimable\n"
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
if got := stderr(); !strings.HasPrefix(got, first) ||
!strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to start %q and end %q",
got, first, last)
}
assertNoSidecars(t, path)
@@ -942,11 +966,16 @@ func TestRunTreesSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want)
}
want = "trees: database " + path + " (from " + databaseEnv + ")\n" +
"trees: 2 records read, 1 duplicate tree groups, 1 dupe trees, " +
"300 B reclaimable\n"
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
// Its phases come between the two lines; see
// TestRunTreesProgressNamesEachPhase.
first := "trees: database " + path + " (from " + databaseEnv + ")\n"
last := "\ntrees: 2 records read, 1 duplicate tree groups, " +
"1 dupe trees, 300 B reclaimable\n"
if got := stderr(); !strings.HasPrefix(got, first) ||
!strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to start %q and end %q",
got, first, last)
}
assertNoSidecars(t, path)
@@ -1238,7 +1267,7 @@ func TestRunReportsDuringScan(t *testing.T) {
func TestRunStdoutClosedIsFatal(t *testing.T) {
// README §Error handling: a stdout write failure exits 1, reported
// in one line on stderr.
for _, name := range []string{cmdReport, cmdTrees} {
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
t.Run(name, func(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
@@ -1288,6 +1317,7 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
cases := map[string]func(context.Context, io.Writer) error{
cmdReport: runReport,
cmdTrees: runTrees,
cmdInfo: runInfo,
}
for name, fn := range cases {
+118 -20
View File
@@ -2,9 +2,11 @@ package main
import (
"fmt"
"io"
"math"
"os"
"slices"
"strings"
"sync"
"time"
@@ -45,6 +47,18 @@ const spinnerMargin = 5
// around them, and the terminal's last column.
const barMargin = barWidth + 11
// countingRecords is what the display of a phase that reads the records
// says while it counts them, before it has its total.
const countingRecords = "counting the records"
// openPhase is the name of the phase in which report, trees and info
// open the database, and openingDatabase what its display says while
// it does.
const (
openPhase = "open"
openingDatabase = "opening the database"
)
// stderrIsTTY reports whether stderr is attached to a terminal.
func stderrIsTTY() bool {
return term.IsTerminal(int(os.Stderr.Fd()))
@@ -71,6 +85,7 @@ func phaseLabel(phases []string, phase string) string {
// progressParams describe one phase's progress display as it starts.
type progressParams struct {
label string // the phase and its place, from phaseLabel
unit string // what the phase counts, as "files"
total int64 // the items the phase will complete; -1 when not known
estimate bool // total is an estimate, which the count may pass
waitingFor string // what the phase waits for until setTotal is called
@@ -89,7 +104,8 @@ type progressParams struct {
// below it. Every method must be called from the main goroutine. A nil
// *progress is a valid no-display receiver: every method is a no-op, so
// batched database flushes during the hash phase can reuse the update
// phase's helpers without rendering anything.
// phase's helpers without rendering anything, and its writer writes
// with no display to clear.
type progress struct {
width func() int // reads the terminal's width; nil when not a TTY
stop chan struct{} // closed by finish to end the draws
@@ -98,7 +114,9 @@ type progress struct {
mu sync.Mutex
bar *progressbar.ProgressBar // what the library draws; nil until then
withBar bool // bar is the library's bar, not its spinner
widest int // the widest line bar has drawn, in columns
label string
unit string
total int64
estimate bool
waitingFor string
@@ -126,8 +144,9 @@ func startProgress(params progressParams, width func() int,
now := time.Now()
p := &progress{
width: width, stop: make(chan struct{}), done: make(chan struct{}),
label: params.label, total: params.total, estimate: params.estimate,
waitingFor: params.waitingFor, start: now, rateStart: now,
label: params.label, unit: params.unit, total: params.total,
estimate: params.estimate, waitingFor: params.waitingFor,
start: now, rateStart: now,
}
// Show the zero state at once: the first item may take minutes,
@@ -176,15 +195,20 @@ func (p *progress) draw() {
text, withBar = p.line(width-barMargin, false)
}
margin := barMargin
if !withBar {
text, _ = p.line(width-spinnerMargin, true)
margin = spinnerMargin
text, _ = p.line(width-margin, true)
}
// The library makes a bar or a spinner. A display that turns from
// one to the other, as its line grows or the terminal is resized,
// gets a new one.
if p.bar == nil || p.withBar != withBar {
p.newBar(withBar)
// The library makes a bar or a spinner, and before each draw erases
// its line with as many spaces as the widest line it has drawn. A
// display gets a new one when it turns from one to the other, as its
// line grows or the terminal is resized, and when the terminal has
// been made too narrow for that erase, which would wrap and leave a
// line behind at every draw.
if p.bar == nil || p.withBar != withBar || p.widest >= width {
p.newBar(withBar, width)
}
if withBar {
@@ -192,16 +216,23 @@ func (p *progress) draw() {
}
p.bar.Describe(text)
// The library drew the text and its margin, less the terminal's last
// column, which the margin keeps clear.
p.widest = max(p.widest, len(text)+margin-1)
}
// newBar has the library make what draw draws: a bar for the phase's
// total when withBar is set, and a spinner when not. Either shows the
// text draw gives it, and nothing of its own besides the spinner, or
// the bar and its percent. p.mu must be held.
func (p *progress) newBar(withBar bool) {
func (p *progress) newBar(withBar bool, width int) {
if p.bar != nil {
// Erase what was drawn before.
_ = p.bar.Clear()
// Erase what was drawn before, short of the last column of a
// terminal that may have been made narrower since: the library's
// own erase is as wide as the widest line it drew.
blank := strings.Repeat(" ", min(p.widest, width-1))
fmt.Fprint(os.Stderr, "\r"+blank+"\r")
}
total := int64(-1) // the library's spinner
@@ -220,7 +251,7 @@ func (p *progress) newBar(withBar bool) {
progressbar.OptionSetElapsedTime(false),
progressbar.OptionSetPredictTime(false),
)
p.withBar = withBar
p.withBar, p.widest = withBar, 0
}
// setTotal gives a phase that started without a total its total, once
@@ -273,6 +304,73 @@ func (p *progress) warnf(format string, args ...any) {
fmt.Fprintln(os.Stderr, msg)
}
// progressWriter writes a phase's output to out, which may be the file
// the phase's display is drawn on, as stdout is in a plain "sfdupes
// trees" on a terminal. It holds what is written to it and writes that
// to out in pieces of about ioBufSize bytes. When out is the file
// stderr is, each piece is written under the display's lock with the
// display cleared first, as warnf does for a warning: no draw lands in
// the middle of a piece, and the next draw shows the display again, on
// the line below. Every write to it must end with a newline, as a row
// written with one fmt.Fprintf does, so that each piece ends a line.
// Flush writes what it holds once the last row is written.
type progressWriter struct {
// p is the display to clear, nil unless out is the file stderr is.
p *progress
out io.Writer
buf []byte
}
// writer returns a progressWriter writing to out around p's display.
// It clears the display and takes its lock only when out is the file
// stderr is: the same device and inode, as os.SameFile compares them,
// which is one terminal, or one pipe or file that both are sent to.
// Otherwise it writes to out without the lock, so the display keeps
// being drawn while a write to out waits for a slow reader.
func (p *progress) writer(out io.Writer) *progressWriter {
w := &progressWriter{out: out}
if f, ok := out.(*os.File); ok {
outInfo, outErr := f.Stat()
errInfo, errErr := os.Stderr.Stat()
if outErr == nil && errErr == nil && os.SameFile(outInfo, errInfo) {
w.p = p
}
}
return w
}
// Write holds b, and writes what it holds to out once that comes to
// ioBufSize bytes.
func (w *progressWriter) Write(b []byte) (int, error) {
w.buf = append(w.buf, b...)
if len(w.buf) < ioBufSize {
return len(b), nil
}
return len(b), w.Flush()
}
// Flush writes what w holds to out. When out is the file stderr is, the
// display is cleared first and not drawn again until it is written.
func (w *progressWriter) Flush() error {
if w.p != nil {
w.p.mu.Lock()
defer w.p.mu.Unlock()
if w.p.bar != nil {
_ = w.p.bar.Clear()
}
}
_, err := w.out.Write(w.buf)
w.buf = w.buf[:0]
return err
}
// finish ends the phase's display, drawn once more as the phase ends.
// A phase stopped short of its total, as an interrupted one is, shows
// the count it stopped at rather than a full bar.
@@ -297,9 +395,9 @@ func (p *progress) finish() {
//
// [3/5] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s
//
// without the percent when percent is false, as before the library's
// bar, which shows its own. A phase with no total shows the items done
// instead, or what it waits for.
// where "files" is the phase's unit, without the percent when percent
// is false, as before the library's bar, which shows its own. A phase
// with no total shows the items done instead, or what it waits for.
//
// The phase and its place, the progress, the elapsed time and the ETA
// are kept whole. A line wider than room leaves out the rate, or what
@@ -323,7 +421,7 @@ func (p *progress) line(room int, percent bool) (string, bool) {
}
head = p.label + ": " + p.itemsDone(percent) + ", "
extra = fmt.Sprintf("%.0f files/s", rate)
extra = fmt.Sprintf("%.0f %s/s", rate, p.unit)
tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(rate))
}
@@ -341,11 +439,11 @@ func (p *progress) line(room int, percent bool) (string, bool) {
// itemsDone formats the items done, as "[1234/5000] 25%", with "~"
// before a total that is an estimate and without the percent when
// percent is false, or as "1234 files" when the phase has no total.
// p.mu must be held.
// percent is false, or as "1234 files", in the phase's unit, when the
// phase has no total. p.mu must be held.
func (p *progress) itemsDone(percent bool) string {
if p.total < 0 {
return fmt.Sprintf("%d files", p.count)
return fmt.Sprintf("%d %s", p.count, p.unit)
}
approx := ""
+239 -19
View File
@@ -1,6 +1,8 @@
package main
import (
"fmt"
"io"
"math"
"os"
"path/filepath"
@@ -82,34 +84,48 @@ func TestProgressLine(t *testing.T) {
}{{
p: &progress{
label: scanPhase("load"), total: -1,
waitingFor: "counting the records",
waitingFor: countingRecords,
},
want: "[1/5] load: counting the records, elapsed 10s, eta unknown",
}, {
p: &progress{label: scanPhase("hash"), total: 100, count: 20},
p: &progress{
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
},
want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
}, {
p: &progress{label: scanPhase("hash"), total: 100},
p: &progress{label: scanPhase("hash"), unit: scanUnit, total: 100},
want: "[3/5] hash: [0/100] 0%, 0 files/s, elapsed 10s, eta unknown",
}, {
p: &progress{label: scanPhase("update"), total: 0},
p: &progress{label: scanPhase("update"), unit: scanUnit, total: 0},
want: "[4/5] update: [0/0] 100%, 0 files/s, elapsed 10s, eta 0s",
}, {
p: &progress{
label: scanPhase("walk"), total: 100, estimate: true, count: 20,
label: scanPhase("walk"), unit: scanUnit, total: 100,
estimate: true, count: 20,
},
want: "[2/5] walk: [20/~100] 20%, 2 files/s, elapsed 10s, eta 40s",
}, {
// Past the estimate the walk cannot tell how long it has left.
p: &progress{
label: scanPhase("walk"), total: 100, estimate: true, count: 120,
label: scanPhase("walk"), unit: scanUnit, total: 100,
estimate: true, count: 120,
},
want: "[2/5] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " +
"eta unknown",
}, {
// A first scan's walk has no estimate.
p: &progress{label: scanPhase("walk"), total: -1, count: 20},
p: &progress{
label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20,
},
want: "[2/5] walk: 20 files, 2 files/s, elapsed 10s, eta unknown",
}, {
// The rate is in what the phase counts.
p: &progress{
label: treesPhase("group"), unit: "directories", total: 100,
count: 20,
},
want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " +
"eta 40s",
}}
for _, tc := range cases {
@@ -129,7 +145,7 @@ func TestLineFitsRoom(t *testing.T) {
ago := time.Now().Add(-10 * time.Second)
hash := &progress{
label: scanPhase("hash"), total: 100, count: 20,
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
start: ago, rateStart: ago,
}
content := &progress{
@@ -183,7 +199,8 @@ func TestSetTotalStartsRate(t *testing.T) {
waited := time.Now().Add(-20 * time.Second)
p := &progress{
label: scanPhase("load"), total: -1, start: waited, rateStart: waited,
label: scanPhase("load"), unit: scanUnit, total: -1, start: waited,
rateStart: waited,
}
p.setTotal(100)
@@ -226,7 +243,9 @@ func awaitOutput(t *testing.T, stderr func() string, want string,
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestPlainProgressPrintsAtStartAndEnd(t *testing.T) {
stderr := captureStderr(t)
p := newProgress(progressParams{label: scanPhase("hash"), total: 1000})
p := newProgress(progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 1000,
})
for range 1000 {
p.increment()
@@ -252,7 +271,7 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) {
stderr := captureStderr(t)
p := startProgress(progressParams{
label: scanPhase("load"), total: -1,
waitingFor: "counting the records",
waitingFor: countingRecords,
}, nil, testRedraw)
awaitOutput(t, stderr,
@@ -289,8 +308,14 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) {
params progressParams
want string // how the line shows a count of 50
}{
{progressParams{label: scanPhase("walk"), total: -1}, "50 files,"},
{progressParams{label: scanPhase("hash"), total: 100}, "[50/100],"},
{
progressParams{label: scanPhase("walk"), unit: scanUnit, total: -1},
"50 files,",
},
{
progressParams{label: scanPhase("hash"), unit: scanUnit, total: 100},
"[50/100],",
},
}
for _, tc := range cases {
@@ -341,8 +366,9 @@ func drawLate(stderr func() string, params progressParams, width int,
}
// TestTerminalLinesKeepETA checks the line each display shows on an
// 80-column terminal at a large scan's counts and times: the rate is
// left out, so that the elapsed time and the ETA stay whole.
// 80-column terminal at a large scan's counts and times, and the line
// of trees' read phase at the same counts: the rate is left out, so
// that the elapsed time and the ETA stay whole.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalLinesKeepETA(t *testing.T) {
@@ -354,18 +380,29 @@ func TestTerminalLinesKeepETA(t *testing.T) {
}{{
// Past its estimate, the walk cannot tell how long it has left.
params: progressParams{
label: scanPhase("walk"), total: 1200000, estimate: true,
label: scanPhase("walk"), unit: scanUnit, total: 1200000,
estimate: true,
},
want: "[2/5] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " +
"eta unknown",
}, {
params: progressParams{label: scanPhase("hash"), total: 9876543},
params: progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
},
want: "[3/5] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s",
}, {
params: progressParams{label: scanPhase("content"), total: 9876543},
params: progressParams{
label: scanPhase("content"), unit: scanUnit, total: 9876543,
},
want: "[5/5] content: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s",
}, {
params: progressParams{
label: treesPhase("read"), unit: "records", total: 9876543,
},
want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s",
}}
for _, tc := range cases {
@@ -389,7 +426,9 @@ func TestTerminalLinesFitWidth(t *testing.T) {
for _, width := range []int{120, 100, 80, 72, 60, 50, 40} {
for _, total := range []int64{9876543, 20000} {
params := progressParams{label: scanPhase("content"), total: total}
params := progressParams{
label: scanPhase("content"), unit: scanUnit, total: total,
}
out := drawLate(stderr, params, width, total/8)
if width == 120 && !strings.Contains(out, "% |") {
@@ -410,6 +449,68 @@ func TestTerminalLinesFitWidth(t *testing.T) {
}
}
// TestTerminalNarrowedRedrawsInPlace draws a large scan's display, two
// and a half hours into the phase, on a terminal 120 columns wide, then
// makes the terminal narrower than that line and draws it again a few
// times. Nothing written after that, the erase before each redraw
// included, may reach the narrower terminal's last column: it would
// wrap, and every redraw would leave a line behind. At these widths the
// spinner stays a spinner and the bar stays a bar.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
stderr := captureStderr(t)
cases := []struct {
params progressParams
narrow int
}{
{
progressParams{label: scanPhase("walk"), unit: scanUnit, total: -1},
60,
},
{
progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
},
90,
},
}
for _, tc := range cases {
width := 120
// No timed draws: the test draws.
p := startProgress(tc.params, func() int { return width }, time.Hour)
p.mu.Lock()
p.count = 1234567
p.start = p.start.Add(-150 * time.Minute)
p.rateStart = p.start
p.mu.Unlock()
p.draw()
width = tc.narrow
before := len(stderr())
p.draw()
p.draw()
p.finish()
lines := strings.FieldsFuncSeq(stderr()[before:], func(r rune) bool {
return r == '\r' || r == '\n'
})
for line := range lines {
if n := utf8.RuneCountInString(line); n >= width {
t.Errorf("%d columns written on a terminal made %d wide: %q",
n, width, line)
}
}
}
}
// TestProgressWarningsOnOwnLines drives the terminal displays, spinner
// and bar, through a run of warnings with no items between them, as when
// the walk meets many unreadable paths, for many redraws: every warning
@@ -459,6 +560,125 @@ func TestProgressWarningsOnOwnLines(t *testing.T) {
}
}
// TestProgressWriterRowsOnOwnLines drives the terminal displays,
// spinner and bar, through a run of rows written through the display's
// writer to the same terminal, as stdout is in a plain "sfdupes trees",
// each written out at once, for many redraws: the terminal must show
// every row whole, in order, on a line of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressWriterRowsOnOwnLines(t *testing.T) {
stderr := captureStderr(t)
cases := []progressParams{
{
label: treesPhase("open"), total: -1,
waitingFor: "opening the database",
},
{label: treesPhase("write"), unit: "groups", total: 10},
}
var want []string
for _, params := range cases {
p := startProgress(params, terminalWidth, testRedraw)
out := p.writer(os.Stderr)
// No pause between rows: one is garbled only if a redraw lands
// on its line.
for start := time.Now(); time.Since(start) < 50*testRedraw; {
row := fmt.Sprintf("row %d", len(want))
want = append(want, row)
_, err := fmt.Fprintln(out, row)
if err == nil {
err = out.Flush()
}
if err != nil {
t.Fatal(err)
}
}
p.finish()
}
// A terminal shows each line as the text after its last carriage
// return.
var shown []string
for line := range strings.SplitSeq(stderr(), "\n") {
if strings.Contains(line, "row ") {
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
}
}
if len(shown) != len(want) {
t.Fatalf("terminal shows %d rows, want %d", len(shown), len(want))
}
for i := range want {
if shown[i] != want[i] {
t.Fatalf("terminal shows %q, want %q", shown[i], want[i])
}
}
}
// TestProgressWriterStalledStdoutKeepsLines writes a row larger than a
// pipe holds through the display's writer to a pipe nobody reads yet,
// as stdout is when its reader has stopped while stderr goes to a file:
// while the write waits, the display must go on printing its line.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressWriterStalledStdoutKeepsLines(t *testing.T) {
stderr := captureStderr(t)
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_ = w.Close()
_ = r.Close()
})
p := startProgress(progressParams{label: treesPhase("write"), total: 1},
nil, testRedraw)
out := p.writer(w)
written := make(chan error, 1)
go func() {
_, writeErr := fmt.Fprintln(out, strings.Repeat("x", ioBufSize))
written <- writeErr
}()
// Pass or fail, the pipe is then read so that the write ends, and
// the display ends before stderr is put back.
t.Cleanup(func() {
go func() {
_, _ = io.Copy(io.Discard, r)
}()
writeErr := <-written
if writeErr != nil {
t.Error(writeErr)
}
p.finish()
})
// The first byte read shows that the write has started; the rest
// of the row waits until the pipe is read.
_, err = r.Read(make([]byte, 1))
if err != nil {
t.Fatal(err)
}
line := "[4/4] write: "
awaitOutput(t, stderr, line, strings.Count(stderr(), line)+3)
}
// TestTerminalStoppedShortKeepsCount checks that the terminal display
// of a phase that stops before its total, as an interrupted one does,
// is left showing the count it stopped at rather than a full bar.
+188 -46
View File
@@ -1,11 +1,12 @@
package main
import (
"bufio"
"context"
"database/sql"
"fmt"
"io"
"os"
"slices"
"strings"
"time"
)
@@ -29,13 +30,25 @@ type scanRec struct {
path string
}
// reportPhase names one phase of report with its place among all of
// them, as "[2/2] read", for the phase's progress display. The list
// here is the one place the phases are counted.
func reportPhase(name string) string {
return phaseLabel([]string{openPhase, "read"}, name)
}
// runReport implements the report subcommand: it prints the file-level
// duplicates report as TSV on stdout. SQLite groups and orders the
// records, and each row is written as it is read, so no group is held
// in memory. It never touches the scanned filesystem; its only I/O is
// the database (with SQLite's temporary sort file), stdout, and stderr.
// It names the database on stderr before it opens it. Any database
// duplicates report as TSV on stdout. It never touches the scanned
// filesystem; its only I/O is the database, stdout, and stderr. It
// names the database on stderr before it opens it. Any database
// problem, including a missing database, is fatal.
//
// The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and read (count the records,
// then read them in signature order, writing each size's duplicate
// groups to stdout once that size is read; see readReport). Each
// display ends before the next phase starts, and before an error is
// returned to be printed.
func runReport(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
@@ -45,65 +58,194 @@ func runReport(ctx context.Context, stdout io.Writer) error {
fmt.Fprintf(os.Stderr, "report: database %s (%s)\n",
escapePath(dbPath), source)
// Opening can wait on a busy database for as long as the busy
// timeout.
prog := newProgress(progressParams{
label: reportPhase(openPhase), total: -1,
waitingFor: openingDatabase,
})
db, err := openReportDatabase(ctx, dbPath)
prog.finish()
if err != nil {
return err
}
defer func() { _ = db.Close() }()
out := bufio.NewWriterSize(stdout, ioBufSize)
prog = newProgress(progressParams{
label: reportPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords,
})
report, err := readReport(ctx, db, dbPath, stdout, prog)
_, err = fmt.Fprintln(out, "first\tdupe\tsize")
if err != nil {
return fmt.Errorf("write stdout: %w", err)
}
var (
groups, dupeFiles int
reclaimable int64
writeErr error
)
records, err := loadDupeRows(ctx, db,
func(first, path string, size int64) error {
// A group's first path is its first row; every other
// path is a dupe.
if path == first {
groups++
return nil
}
_, writeErr = fmt.Fprintf(out, "%s\t%s\t%d\n",
escapePath(first), escapePath(path), size)
dupeFiles++
reclaimable += size
return writeErr
})
if writeErr != nil {
return fmt.Errorf("write stdout: %w", writeErr)
}
prog.finish()
if err != nil {
return fmt.Errorf("database %s: %w", dbPath, err)
}
err = out.Flush()
if err != nil {
return fmt.Errorf("write stdout: %w", err)
return err
}
fmt.Fprintf(os.Stderr,
"report: %d records read, %d duplicate groups, %d dupe files, "+
"%s reclaimable\n",
records, groups, dupeFiles, humanBytes(reclaimable))
report.records, report.dupeGroups, report.dupeFiles,
humanBytes(report.reclaimable))
return nil
}
// readReport gives prog the number of records in db, the database at
// dbPath, as its total, then reads every record in signature order,
// counting each on prog, and writes the report to stdout: the header
// line, then the rows of each size's duplicate groups once that size is
// read. stdout may be the terminal prog is drawn on, so the lines go
// through prog's writer. It returns what it wrote, with the totals for
// the summary line.
func readReport(ctx context.Context, db *sql.DB, dbPath string,
stdout io.Writer, prog *progress,
) (*reportWriter, error) {
out := prog.writer(stdout)
report := &reportWriter{out: out}
_, err := fmt.Fprintln(out, "first\tdupe\tsize")
if err != nil {
return nil, fmt.Errorf("write stdout: %w", err)
}
var writeErr error
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
prog.increment()
writeErr = report.add(r)
return writeErr
})
if writeErr != nil {
return nil, fmt.Errorf("write stdout: %w", writeErr)
}
if err != nil {
return nil, fmt.Errorf("database %s: %w", dbPath, err)
}
err = report.finish()
if err == nil {
err = out.Flush()
}
if err != nil {
return nil, fmt.Errorf("write stdout: %w", err)
}
return report, nil
}
// reportWriter writes the report's rows to out from the records, added
// in signature order, largest size first, as loadSignatureRows reads
// them. The records sharing a signature come together, and so do the
// records of one size, so it holds only the duplicate groups of the
// size being read, and writes them, ordered by first path, once a
// record of a smaller size comes. It counts what the summary line
// reports.
type reportWriter struct {
out io.Writer
sig scanRec // the signature being read; its path is not used
paths []string // the paths read so far with sig
groups [][]string // the duplicate groups of sig.size, each sorted
records, dupeGroups, dupeFiles int
reclaimable int64
}
// add takes the next record. One with the signature being read joins
// its paths; any other ends them, and one of a smaller size first
// writes the groups of the size before.
func (w *reportWriter) add(r scanRec) error {
w.records++
// A record without a content hash has unknown content (README
// "Database"), so it is never a duplicate.
if r.content == "" {
return nil
}
if r.size == w.sig.size && r.head == w.sig.head &&
r.tail == w.sig.tail && r.content == w.sig.content {
w.paths = append(w.paths, r.path)
return nil
}
w.endGroup()
if r.size != w.sig.size {
err := w.writeGroups()
if err != nil {
return err
}
}
w.sig = r
w.paths = append(w.paths, r.path)
return nil
}
// endGroup ends the paths read with w.sig: two or more are a duplicate
// group, held with its paths sorted until writeGroups writes it.
func (w *reportWriter) endGroup() {
if len(w.paths) < minGroupSize {
// A path with no duplicate is dropped, and its slice reused.
w.paths = w.paths[:0]
return
}
slices.Sort(w.paths)
w.groups = append(w.groups, w.paths)
w.paths = nil
}
// writeGroups writes the duplicate groups held, all of size w.sig.size,
// ordered by first path: for each group, a row for every path but the
// first. No two groups share a first path, since no two records share
// a path.
func (w *reportWriter) writeGroups() error {
slices.SortFunc(w.groups, func(a, b []string) int {
return strings.Compare(a[0], b[0])
})
for _, g := range w.groups {
for _, path := range g[1:] {
_, err := fmt.Fprintf(w.out, "%s\t%s\t%d\n",
escapePath(g[0]), escapePath(path), w.sig.size)
if err != nil {
return err
}
w.dupeFiles++
w.reclaimable += w.sig.size
}
w.dupeGroups++
}
w.groups = nil
return nil
}
// finish writes the groups of the last size read.
func (w *reportWriter) finish() error {
w.endGroup()
return w.writeGroups()
}
// escapePath returns a path as it is written in a report column (README
// "Report output format"): a backslash, tab, newline or carriage return
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
+190 -25
View File
@@ -9,6 +9,7 @@ import (
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"testing"
"time"
@@ -28,7 +29,8 @@ func awkwardPairRecs() []scanRec {
}
}
// seedDatabase writes recs into a fresh database and returns its path.
// seedDatabase writes recs into a fresh database, closes it as scan
// does, out of WAL mode, and returns its path.
func seedDatabase(t *testing.T, recs []scanRec) string {
t.Helper()
@@ -44,10 +46,7 @@ func seedDatabase(t *testing.T, recs []scanRec) string {
t.Fatal(err)
}
err = db.Close()
if err != nil {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, path)
return path
}
@@ -60,27 +59,41 @@ type dupeGroup struct {
}
// dupeGroups returns the duplicate groups report reads from db, in
// report order.
// report order, from the rows it writes: a group's rows come together,
// each naming the group's first path.
func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
t.Helper()
var groups []dupeGroup
var out bytes.Buffer
_, err := loadDupeRows(t.Context(), db,
func(first, path string, size int64) error {
if path == first {
groups = append(groups, dupeGroup{size: size})
}
g := &groups[len(groups)-1]
g.paths = append(g.paths, path)
return nil
})
_, err := readReport(t.Context(), db, "", &out, nil)
if err != nil {
t.Fatal(err)
}
var groups []dupeGroup
rows := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n")
for _, row := range rows[1:] { // after the header
fields := strings.Split(row, "\t")
if len(fields) != 3 {
t.Fatalf("row %q, want first, dupe and size", row)
}
size, err := strconv.ParseInt(fields[2], 10, 64)
if err != nil {
t.Fatal(err)
}
if len(groups) == 0 || groups[len(groups)-1].paths[0] != fields[0] {
groups = append(groups,
dupeGroup{size: size, paths: []string{fields[0]}})
}
g := &groups[len(groups)-1]
g.paths = append(g.paths, fields[1])
}
return groups
}
@@ -118,26 +131,147 @@ func TestRunReportEscapesPaths(t *testing.T) {
}
func TestReportStdoutFailsWhileReading(t *testing.T) {
// Each row holds two paths longer than dir, so the report is more
// than twice the stdout buffer and stdout fails while rows are
// still being read, not at the final flush.
t.Parallel()
// Every two records are a duplicate pair of a size of their own, and
// a pair's row is written once a smaller size is read. Each row
// holds two paths longer than dir, so the rows come to more than
// twice the stdout buffer, and stdout fails while records are still
// being read, not once every record is.
dir := "/" + strings.Repeat("d", 4096)
recs := make([]scanRec, ioBufSize/len(dir))
recs := make([]scanRec, 2*ioBufSize/len(dir))
for i := range recs {
recs[i] = scanRec{
size: 1, head: "h", tail: "t", content: "c",
size: int64(i/2) + 1, head: "h", tail: "t", content: "c",
path: fmt.Sprintf("%s/%d", dir, i),
}
}
t.Setenv(databaseEnv, seedDatabase(t, recs))
db := openTestDB(t)
err := runReport(t.Context(), failingWriter{})
err := applyChanges(t.Context(), db, recs, nil, nil)
if err != nil {
t.Fatal(err)
}
// Counts the records read, and draws nothing.
prog := &progress{}
_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
if !errors.Is(err, errWriteFailed) ||
!strings.HasPrefix(err.Error(), "write stdout: ") {
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
}
if prog.count >= int64(len(recs)) {
t.Errorf("stdout failed after all %d records were read, want it "+
"to stop the read", prog.count)
}
}
// reportOpenLine is the plain line of report's open phase when it opens
// the database at once; it prints it as the phase starts and as it ends.
const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown"
// smokeReport is report's stdout for the smoke-test records.
const smokeReport = "first\tdupe\tsize\n" +
"/d/t1/f1\t/d/t2/f1\t3000\n" +
"/d/t1/f1\t/d/t3/f1\t3000\n" +
"/d/t1/sub/f2\t/d/t2/sub/f2\t100\n" +
"/d/t1/sub/f2\t/d/t3/sub/f2renamed\t100\n"
// TestRunReportProgressNamesEachPhase runs report on the smoke-test
// records with stderr not a terminal, and checks every line it prints
// there. After the line naming the database, each phase prints a line
// naming it with its place among the two as it starts and another as
// it ends; read counts the 6 records before it reads them. The summary
// comes last, and stdout holds the report alone.
func TestRunReportProgressNamesEachPhase(t *testing.T) {
path := seedDatabase(t, smokeTreeRecs())
t.Setenv(databaseEnv, path)
stderr := captureStderr(t)
var stdout bytes.Buffer
code := run([]string{cmdReport}, &stdout, os.Stderr)
if code != exitOK {
t.Fatalf("run(report) = %d, want %d; stderr: %s",
code, exitOK, stderr())
}
if got := stdout.String(); got != smokeReport {
t.Errorf("stdout = %q, want %q", got, smokeReport)
}
// How each line starts; the rate at the end of a phase varies.
want := []string{
"report: database " + path + " (from " + databaseEnv + ")",
reportOpenLine,
reportOpenLine,
"[2/2] read: counting the records, elapsed 0s, eta unknown",
"[2/2] read: [6/6] 100%, ",
"report: 6 records read, 2 duplicate groups, 4 dupe files, " +
"6.1 KiB reclaimable",
}
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
if len(lines) != len(want) {
t.Fatalf("stderr lines = %q, want %d lines starting %q",
lines, len(want), want)
}
for i := range want {
if !strings.HasPrefix(lines[i], want[i]) {
t.Errorf("stderr line %d = %q, want it to start %q",
i+1, lines[i], want[i])
}
}
}
// TestReadReportOnTerminal writes the smoke-test report to the terminal
// its read phase's display is drawn on, as a plain "sfdupes report"
// does: the terminal must show the header and every row whole, each on
// a line of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestReadReportOnTerminal(t *testing.T) {
stderr := captureStderr(t)
db := openTestDB(t)
err := applyChanges(t.Context(), db, smokeTreeRecs(), nil, nil)
if err != nil {
t.Fatal(err)
}
// No timed draws: the display is drawn as it starts and as it ends.
p := startProgress(progressParams{
label: reportPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords,
}, terminalWidth, time.Hour)
_, err = readReport(t.Context(), db, "", os.Stderr, p)
p.finish()
if err != nil {
t.Fatal(err)
}
// A terminal shows each line as the text after its last carriage
// return.
var shown []string
for line := range strings.SplitSeq(stderr(), "\n") {
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
}
want := strings.Split(strings.TrimSuffix(smokeReport, "\n"), "\n")
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
t.Errorf("terminal shows lines %q, want %q first", shown, want)
}
}
func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
@@ -354,6 +488,37 @@ func TestDupeGroupsTieBreak(t *testing.T) {
}
}
func TestDupeGroupsOrderedByFirstPath(t *testing.T) {
t.Parallel()
// report reads the groups of size 50 in signature order, here /a,
// /z, /k, which is neither the order of their first paths nor its
// reverse, and must write them ordered by first path. The group of
// size 7 comes after them all, though its first path sorts first.
recs := []scanRec{
{size: 50, head: "a", tail: "a", content: "a", path: "/k/2"},
{size: 50, head: "a", tail: "a", content: "a", path: "/k/1"},
{size: 50, head: "b", tail: "b", content: "b", path: "/z/1"},
{size: 50, head: "b", tail: "b", content: "b", path: "/z/2"},
{size: 50, head: "c", tail: "c", content: "c", path: "/a/2"},
{size: 50, head: "c", tail: "c", content: "c", path: "/a/1"},
{size: 7, head: "d", tail: "d", content: "d", path: "/0/2"},
{size: 7, head: "d", tail: "d", content: "d", path: "/0/1"},
}
groups := dupeGroupsOf(t, recs)
got := make([]string, 0, len(groups))
for _, g := range groups {
got = append(got, g.paths[0])
}
want := []string{"/a/1", "/k/1", "/z/1", "/0/1"}
if !slices.Equal(got, want) {
t.Errorf("groups' first paths = %q, want %q", got, want)
}
}
func TestDupeGroupsDeterministic(t *testing.T) {
t.Parallel()
+11 -6
View File
@@ -84,6 +84,10 @@ type fileMeta struct {
hashed bool
}
// scanUnit is what every scan phase counts, as its progress display
// names it.
const scanUnit = "files"
// scanPhase names one phase of a scan with its place among all of them,
// as "[2/5] walk", for the phase's progress display. The list here is
// the one place the phases are counted.
@@ -441,8 +445,8 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// display the scan looks hung before the walk begins. Counting them
// first gives the display its total.
prog := newProgress(progressParams{
label: scanPhase("load"), total: -1,
waitingFor: "counting the records",
label: scanPhase("load"), unit: scanUnit, total: -1,
waitingFor: countingRecords,
})
defer prog.finish()
@@ -492,7 +496,8 @@ func (s *scanState) walkPhase(
}
prog := newProgress(progressParams{
label: scanPhase("walk"), total: estimate, estimate: true,
label: scanPhase("walk"), unit: scanUnit, total: estimate,
estimate: true,
})
var (
@@ -683,7 +688,7 @@ func (s *scanState) readRuns(ctx context.Context, workers int,
defer pool.stop()
prog := newProgress(progressParams{
label: scanPhase(label), total: int64(len(runs)),
label: scanPhase(label), unit: scanUnit, total: int64(len(runs)),
})
defer prog.finish()
@@ -772,7 +777,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
total := len(s.batch) + len(s.sentinels) + len(deletes)
prog := newProgress(progressParams{
label: scanPhase("update"), total: int64(total),
label: scanPhase("update"), unit: scanUnit, total: int64(total),
})
defer prog.finish()
@@ -860,7 +865,7 @@ func (s *scanState) contentCandidates(
// record. The checks start there, against a count of the records
// the search returns, and so do their rate and ETA.
prog := newProgress(progressParams{
label: scanPhase("content"), total: -1,
label: scanPhase("content"), unit: scanUnit, total: -1,
waitingFor: "searching the database",
})
defer prog.finish()
+3 -3
View File
@@ -1478,7 +1478,7 @@ func dbRecords(t *testing.T, db *sql.DB) []scanRec {
var recs []scanRec
err := loadFileRows(t.Context(), db, func(r scanRec) {
err := loadFileRows(t.Context(), db, func(int64) {}, func(r scanRec) {
recs = append(recs, r)
})
if err != nil {
@@ -1550,7 +1550,7 @@ func assertSmokeTreeGroups(t *testing.T, dir string, db *sql.DB) {
super, dirs := dbTree(t, db)
tg := collectTreeGroups(dirs, super)
tg := collectTreeGroups(dirs, super, nil)
if len(tg) != 1 {
t.Fatalf("len(tree groups) = %d, want 1", len(tg))
}
@@ -2092,7 +2092,7 @@ func TestTreesUnhashedNeverEqual(t *testing.T) {
for name, unknown := range cases {
super, dirs := treeOf(t, append(slices.Clone(shared), unknown...))
if tg := collectTreeGroups(dirs, super); len(tg) != 0 {
if tg := collectTreeGroups(dirs, super, nil); len(tg) != 0 {
t.Errorf("%s: tree groups = %d, want 0 (the files may differ)",
name, len(tg))
}
+105 -25
View File
@@ -1,9 +1,9 @@
package main
import (
"bufio"
"context"
"crypto/sha256"
"database/sql"
"fmt"
"io"
"os"
@@ -24,6 +24,13 @@ type treeNode struct {
totalSize int64
}
// treesPhase names one phase of trees with its place among all of them,
// as "[2/4] read", for the phase's progress display. The list here is
// the one place the phases are counted.
func treesPhase(name string) string {
return phaseLabel([]string{openPhase, "read", "group", "write"}, name)
}
// runTrees implements the trees subcommand: it reads every record from
// the database in path order, reconstructs the directory hierarchy from
// the record paths, computes a Merkle-style digest per directory, and
@@ -31,6 +38,13 @@ type treeNode struct {
// touches the scanned filesystem; its only I/O is the database, stdout,
// and stderr. It names the database on stderr before it opens it. Any
// database problem, including a missing database, is fatal.
//
// The work runs in four phases, each under a progress display of its
// own on stderr: open (open the database), read (count the records,
// then read them into the hierarchy), group (group the directories by
// digest) and write (write the groups to stdout). Each display ends
// before the next phase starts, and before an error is returned to be
// printed.
func runTrees(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
@@ -40,34 +54,99 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
fmt.Fprintf(os.Stderr, "trees: database %s (%s)\n",
escapePath(dbPath), source)
// Opening can wait on a busy database for as long as the busy
// timeout.
prog := newProgress(progressParams{
label: treesPhase(openPhase), total: -1,
waitingFor: openingDatabase,
})
db, err := openReportDatabase(ctx, dbPath)
prog.finish()
if err != nil {
return err
}
defer func() { _ = db.Close() }()
records := 0
tree := newTreeBuilder()
err = loadFileRows(ctx, db, func(r scanRec) {
records++
tree.add(r)
prog = newProgress(progressParams{
label: treesPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords,
})
records, super, allDirs, err := readTree(ctx, db, prog)
prog.finish()
if err != nil {
return fmt.Errorf("database %s: %w", dbPath, err)
}
prog = newProgress(progressParams{
label: treesPhase("group"), unit: "directories",
total: int64(len(allDirs)),
})
dupes := collectTreeGroups(allDirs, super, prog)
prog.finish()
prog = newProgress(progressParams{
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
})
dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
prog.finish()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr,
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
"%s reclaimable\n",
records, len(dupes), dupeTrees, humanBytes(reclaimable))
return nil
}
// readTree gives prog the number of records in db as its total, then
// reads every record in path order into the directory hierarchy,
// counting each on prog. It returns the number of records read, the
// super-root and every directory.
func readTree(ctx context.Context, db *sql.DB,
prog *progress,
) (int, *treeNode, []*treeNode, error) {
records := 0
tree := newTreeBuilder()
err := loadFileRows(ctx, db, prog.setTotal, func(r scanRec) {
records++
prog.increment()
tree.add(r)
})
if err != nil {
return 0, nil, nil, err
}
super, allDirs := tree.finish()
dupes := collectTreeGroups(allDirs, super)
return records, super, allDirs, nil
}
out := bufio.NewWriterSize(stdout, ioBufSize)
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
// a row for every member of each group but its first, counting each
// group on prog once its rows are written. stdout may be the terminal
// prog is drawn on, so the lines go through prog's writer. It returns
// the number of rows (dupe trees) and the bytes they could reclaim.
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
prog *progress,
) (int, int64, error) {
out := prog.writer(stdout)
_, err = fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
if err != nil {
return fmt.Errorf("write stdout: %w", err)
return 0, 0, fmt.Errorf("write stdout: %w", err)
}
dupeTrees := 0
@@ -81,25 +160,22 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
escapePath(first.path), escapePath(n.path),
first.fileCount, first.totalSize)
if err != nil {
return fmt.Errorf("write stdout: %w", err)
return 0, 0, fmt.Errorf("write stdout: %w", err)
}
dupeTrees++
reclaimable += first.totalSize
}
prog.increment()
}
err = out.Flush()
if err != nil {
return fmt.Errorf("write stdout: %w", err)
return 0, 0, fmt.Errorf("write stdout: %w", err)
}
fmt.Fprintf(os.Stderr,
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
"%s reclaimable\n",
records, len(dupes), dupeTrees, humanBytes(reclaimable))
return nil
return dupeTrees, reclaimable, nil
}
// treeBuilder reconstructs the directory hierarchy from records added
@@ -221,14 +297,18 @@ func fileEntry(name string, r scanRec) string {
"\x00" + r.head + "\x00" + r.tail + "\x00" + content
}
// collectTreeGroups groups directories by digest and returns every
// maximal group with two or more members, each group's members sorted
// by path, groups ordered by tree size descending then by first path
// ascending.
func collectTreeGroups(allDirs []*treeNode, super *treeNode) [][]*treeNode {
// collectTreeGroups groups directories by digest, counting each one on
// prog as it is grouped, and returns every maximal group with two or
// more members, each group's members sorted by path, groups ordered by
// tree size descending then by first path ascending.
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
prog *progress,
) [][]*treeNode {
groups := make(map[[sha256.Size]byte][]*treeNode)
for _, d := range allDirs {
groups[d.digest] = append(groups[d.digest], d)
prog.increment()
}
var dupes [][]*treeNode
+107 -7
View File
@@ -3,8 +3,11 @@ package main
import (
"bytes"
"database/sql"
"os"
"slices"
"strings"
"testing"
"time"
)
// Signature hashes shared by the smoke-test records.
@@ -17,6 +20,11 @@ const (
f2Content = "f2c"
)
// treesOpenLine is the plain line of trees' open phase when it opens
// the database at once; it prints it as the phase starts and as it ends.
const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
"eta unknown"
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
// /d/t2 are identical, /d/t3 differs from them only by one filename.
func smokeTreeRecs() []scanRec {
@@ -38,7 +46,7 @@ func dbTree(t *testing.T, db *sql.DB) (*treeNode, []*treeNode) {
tree := newTreeBuilder()
err := loadFileRows(t.Context(), db, tree.add)
err := loadFileRows(t.Context(), db, func(int64) {}, tree.add)
if err != nil {
t.Fatal(err)
}
@@ -170,7 +178,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
t.Fatalf("directory paths = %q, want %q", got, want)
}
groups := collectTreeGroups(dirs, super)
groups := collectTreeGroups(dirs, super, nil)
gotGroups := groupPaths(groups)
wantGroups := [][]string{{"/a", "/c"}}
@@ -203,6 +211,98 @@ func TestRunTreesEscapesPaths(t *testing.T) {
}
}
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
// records with stderr not a terminal, and checks every line it prints
// there. After the line naming the database, each phase prints a line
// naming it with its place among the four as it starts and another as
// it ends. read counts the 6 records before it reads them, group counts
// the 8 directories, and write the 1 group. The summary comes last, and
// stdout holds the report alone.
func TestRunTreesProgressNamesEachPhase(t *testing.T) {
path := seedDatabase(t, smokeTreeRecs())
t.Setenv(databaseEnv, path)
stderr := captureStderr(t)
var stdout bytes.Buffer
code := run([]string{cmdTrees}, &stdout, os.Stderr)
if code != exitOK {
t.Fatalf("run(trees) = %d, want %d; stderr: %s",
code, exitOK, stderr())
}
report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
if got := stdout.String(); got != report {
t.Errorf("stdout = %q, want %q", got, report)
}
// How each line starts; the rate at the end of a phase varies.
want := []string{
"trees: database " + path + " (from " + databaseEnv + ")",
treesOpenLine,
treesOpenLine,
"[2/4] read: counting the records, elapsed 0s, eta unknown",
"[2/4] read: [6/6] 100%, ",
"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown",
"[3/4] group: [8/8] 100%, ",
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown",
"[4/4] write: [1/1] 100%, ",
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
}
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
if len(lines) != len(want) {
t.Fatalf("stderr lines = %q, want %d lines starting %q",
lines, len(want), want)
}
for i := range want {
if !strings.HasPrefix(lines[i], want[i]) {
t.Errorf("stderr line %d = %q, want it to start %q",
i+1, lines[i], want[i])
}
}
}
// TestWriteTreeGroupsOnTerminal writes the smoke-test report to the
// terminal its write phase's display is drawn on, as a plain "sfdupes
// trees" does: the terminal must show the header and the row whole,
// each on a line of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestWriteTreeGroupsOnTerminal(t *testing.T) {
stderr := captureStderr(t)
super, dirs := treeOf(t, smokeTreeRecs())
dupes := collectTreeGroups(dirs, super, nil)
// No timed draws: the display is drawn as it starts and as it ends.
p := startProgress(progressParams{
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
}, terminalWidth, time.Hour)
_, _, err := writeTreeGroups(os.Stderr, dupes, p)
p.finish()
if err != nil {
t.Fatal(err)
}
// A terminal shows each line as the text after its last carriage
// return.
var shown []string
for line := range strings.SplitSeq(stderr(), "\n") {
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
}
want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
t.Errorf("terminal shows lines %q, want %q first", shown, want)
}
}
func TestTreeDigests(t *testing.T) {
t.Parallel()
@@ -254,7 +354,7 @@ func TestCollectTreeGroupsMaximal(t *testing.T) {
super, dirs := treeOf(t, smokeTreeRecs())
groups := collectTreeGroups(dirs, super)
groups := collectTreeGroups(dirs, super, nil)
want := [][]string{{"/d/t1", "/d/t2"}}
if got := groupPaths(groups); !slices.EqualFunc(got, want,
@@ -278,14 +378,14 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
super, dirs := treeOf(t, recs)
forward := groupPaths(collectTreeGroups(dirs, super))
forward := groupPaths(collectTreeGroups(dirs, super, nil))
reversed := slices.Clone(recs)
slices.Reverse(reversed)
superR, dirsR := treeOf(t, reversed)
backward := groupPaths(collectTreeGroups(dirsR, superR))
backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
if !slices.EqualFunc(forward, backward, slices.Equal) {
t.Fatalf("output depends on record order: %v vs %v",
forward, backward)
@@ -304,7 +404,7 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super))
got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/x1", "/p/x2"}}
if !slices.EqualFunc(got, want, slices.Equal) {
@@ -326,7 +426,7 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super))
got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/a/x", "/q/b/x"}}
if !slices.EqualFunc(got, want, slices.Equal) {