Compare commits
4
Commits
a9b69e911b
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e9e09effc6 | ||
|
|
ebe4d4dbeb | ||
|
|
8e0facdcd3 | ||
|
|
8dc73aa3f5 |
@@ -38,8 +38,9 @@ database persists between runs; a rescan only hashes files that are new or
|
||||
changed, or that may have gained a duplicate since the last scan, and removes
|
||||
records for files that no longer exist. `report` reads the database and prints
|
||||
the file-level duplicates report. `trees` reads the same database and prints the
|
||||
duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
|
||||
with no `PATH` operand — prints a usage message and exits 2.
|
||||
duplicate-tree report. `info` prints the database's details: its path, the sizes
|
||||
of its files and the number of its records. A missing/invalid subcommand — or a
|
||||
`scan` invocation with no `PATH` operand — prints a usage message and exits 2.
|
||||
|
||||
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
|
||||
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
|
||||
@@ -109,9 +110,9 @@ As a file `/etc/cron.d/sfdupes`:
|
||||
- A scan still running when the next one starts carries on. The new one fails at
|
||||
once, and the lines cron mails include
|
||||
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
|
||||
- `report` and `trees` need only read access to the database (see "Database").
|
||||
Under the usual umask of `022` the first scan creates it readable by every
|
||||
user, so an unprivileged user can run them against root's database.
|
||||
- `report`, `trees` and `info` need only read access to the database (see
|
||||
"Database"). Under the usual umask of `022` the first scan creates it readable
|
||||
by every user, so an unprivileged user can run them against root's database.
|
||||
|
||||
### Reading the reports
|
||||
|
||||
@@ -199,12 +200,13 @@ Goals, in order:
|
||||
under resilver). Holding one small record (path, size, mtime) per file in
|
||||
memory during a scan is acceptable; holding every file's hashes is not (they
|
||||
stay in the database). The reporting commands do not hold every file's
|
||||
hashes either: `report` lets SQLite group and order the records and writes
|
||||
each row as it reads it, so its memory does not grow with the database, and
|
||||
`trees` reads the records in path order and keeps each directory's path,
|
||||
digest and totals, plus the hashes of only the files in the directories
|
||||
holding the record being read, so its memory grows with the number of
|
||||
directories and with the size of the largest directory.
|
||||
hashes either: `report` reads the records in signature order and holds the
|
||||
duplicate paths of one size at a time, so its memory grows with the
|
||||
duplicates of the size that has the most of them, and `trees` reads the
|
||||
records in path order and keeps each directory's path, digest and totals,
|
||||
plus the hashes of only the files in the directories holding the record
|
||||
being read, so its memory grows with the number of directories and with the
|
||||
size of the largest directory.
|
||||
3. **Scan incrementally, analyze offline.** The expensive filesystem scan
|
||||
maintains a persistent database; an unchanged file is never read again on a
|
||||
rescan, except to compute its content hash once a file of 10 MiB or more
|
||||
@@ -234,7 +236,7 @@ Goals, in order:
|
||||
|
||||
### Subcommands
|
||||
|
||||
Three subcommands, all implemented:
|
||||
Four subcommands, all implemented:
|
||||
|
||||
1. `scan` — walk the filesystem and synchronize the database: one signature
|
||||
record per regular file.
|
||||
@@ -242,11 +244,14 @@ Three subcommands, all implemented:
|
||||
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
|
||||
from the database records, compute a Merkle-style digest per directory, and
|
||||
report maximal groups of identical trees.
|
||||
4. `info` — the database's details: its path, the sizes of its files, its
|
||||
schema version and the counts of its records.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] PATH...
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
sfdupes --version
|
||||
sfdupes [command] --help
|
||||
```
|
||||
@@ -261,7 +266,7 @@ stdout.
|
||||
|
||||
### Database
|
||||
|
||||
All three subcommands operate on a single SQLite database file:
|
||||
All four subcommands operate on a single SQLite database file:
|
||||
|
||||
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
|
||||
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
|
||||
@@ -284,13 +289,13 @@ All three subcommands operate on a single SQLite database file:
|
||||
symlinks resolved. After a `cd` through a symlink to a directory, a path
|
||||
starting `../` therefore names a file beside the directory the symlink leads
|
||||
to, as it does for any other program. `scan` says `new database` when no
|
||||
file is at that path as it starts. `report` and `trees` print the line
|
||||
before they open the database, so the error for a missing database comes
|
||||
after it.
|
||||
file is at that path as it starts. `report`, `trees` and `info` print the
|
||||
line before they open the database, so the error for a missing database
|
||||
comes after it.
|
||||
|
||||
- `scan` creates the database (and its parent directory) on first use. `report`
|
||||
and `trees` require an existing database; a missing database file is a fatal
|
||||
error (exit 1) telling the user to run `scan` first.
|
||||
- `scan` creates the database (and its parent directory) on first use. `report`,
|
||||
`trees` and `info` require an existing database; a missing database file is a
|
||||
fatal error (exit 1) telling the user to run `scan` first.
|
||||
- Only one `scan` runs against a database at a time. For its whole run, `scan`
|
||||
holds an exclusive `flock(2)` lock on a lock file beside the database, named
|
||||
by appending `.lock` to the database's path with symlinks resolved
|
||||
@@ -306,8 +311,8 @@ All three subcommands operate on a single SQLite database file:
|
||||
on. The lock file is created on first use, open to its owner only, and left in
|
||||
place: a leftover file blocks nothing, because the lock ends with the process
|
||||
holding it however it ends, a fatal error or an interrupt included, and
|
||||
deleting the file while a scan runs would let a second scan start. `report`
|
||||
and `trees` never take the lock, so they run during a scan.
|
||||
deleting the file while a scan runs would let a second scan start. `report`,
|
||||
`trees` and `info` never take the lock, so they run during a scan.
|
||||
- While `scan` runs, the database is in WAL journal mode with a busy timeout, so
|
||||
running a report while a cron `scan` is in progress is safe. The filesystem is
|
||||
authoritative; the database is an eventually-consistent reflection of it.
|
||||
@@ -323,9 +328,9 @@ All three subcommands operate on a single SQLite database file:
|
||||
needs the database to itself: a `scan` that starts while a report is still
|
||||
reading waits for it up to the 10-second busy timeout, then fails; a `scan`
|
||||
that ends while a report has the database open warns and leaves the database
|
||||
in WAL mode until the next scan. `report` writes each row as it reads it, so
|
||||
it is still reading while its output is paused (a pager, a stalled pipe), and
|
||||
a `scan` started then fails after the busy timeout.
|
||||
in WAL mode until the next scan. `report` writes its rows while it reads the
|
||||
records, so it is still reading while its output is paused (a pager, a stalled
|
||||
pipe), and a `scan` started then fails after the busy timeout.
|
||||
- A `scan` writes to the database and the files beside it at these points (its
|
||||
phases are described under "`scan` mode"):
|
||||
- As it starts, before the walk: the directory, an empty database file and
|
||||
@@ -340,10 +345,10 @@ All three subcommands operate on a single SQLite database file:
|
||||
- In the content phase: content hashes, again committed 10,000 at a time.
|
||||
- At the end, as it closes the database: the `-wal` and `-shm` files go
|
||||
away, unless a report still has the database open (see above).
|
||||
- `report` and `trees` open the database read-only and need only read access to
|
||||
the database file, and no write access to its directory. While the database is
|
||||
in WAL mode they also read the `-wal` and `-shm` files beside it, which SQLite
|
||||
creates with the database file's permissions.
|
||||
- `report`, `trees` and `info` open the database read-only and need only read
|
||||
access to the database file, and no write access to its directory. While the
|
||||
database is in WAL mode they also read the `-wal` and `-shm` files beside it,
|
||||
which SQLite creates with the database file's permissions.
|
||||
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
|
||||
with any other version is a fatal error. `scan` creates the schema and sets
|
||||
the version in one transaction, so a first scan stopped while doing so leaves
|
||||
@@ -379,8 +384,8 @@ All three subcommands operate on a single SQLite database file:
|
||||
a file of 10 MiB or more, `content` stays empty until the content phase of a
|
||||
scan (see "`scan` mode" below) has read the file. A record with an empty
|
||||
`content` is never part of a duplicate group, though it still defines the
|
||||
file for tree reconstruction. The `files_signature` index lets SQLite group
|
||||
the records by signature for `report` without sorting the whole table.
|
||||
file for tree reconstruction. The `files_signature` index lets `report` read
|
||||
the records in signature order without sorting the table.
|
||||
|
||||
### Duplicate detection
|
||||
|
||||
@@ -604,11 +609,8 @@ total.)
|
||||
|
||||
**`report` must never touch the filesystem being analyzed.** It does not stat,
|
||||
open, or otherwise access any path that appears in the records; its only I/O is
|
||||
reading the database, writing stdout/stderr, and the temporary file SQLite sorts
|
||||
in when the duplicate rows do not fit in memory. SQLite puts that file in
|
||||
`$SQLITE_TMPDIR` or `$TMPDIR` when set, otherwise in `/var/tmp` (or `/tmp`), and
|
||||
deletes it as soon as it has opened it. `report` must produce identical output
|
||||
whether or not the scanned filesystem is still mounted.
|
||||
reading the database and writing stdout/stderr. `report` must produce identical
|
||||
output whether or not the scanned filesystem is still mounted.
|
||||
|
||||
Processing:
|
||||
|
||||
@@ -622,6 +624,16 @@ Processing:
|
||||
by `first` path ascending. Output must be fully deterministic for a given
|
||||
database state.
|
||||
|
||||
`report` runs in two phases, each with its own progress display on stderr (see
|
||||
"Progress"): `open` opens the database; `read` counts the records, then reads
|
||||
them in signature order, largest size first. That is the order of the
|
||||
`files_signature` index read backwards, so SQLite sorts nothing and returns the
|
||||
first record at once. The records sharing a signature come together, and so do
|
||||
the records of one size, so `report` holds only the duplicate groups of the size
|
||||
it is reading, and writes them, ordered by `first` path, once it reaches a
|
||||
smaller size. Its memory holds at most the duplicate paths of one size at a
|
||||
time, never every record.
|
||||
|
||||
#### Report output format
|
||||
|
||||
TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
|
||||
@@ -695,6 +707,12 @@ Processing:
|
||||
- Order groups by total tree size descending, tie-broken by `first` path
|
||||
ascending. Output must be fully deterministic for a given input.
|
||||
|
||||
`trees` runs in four phases, each with its own progress display on stderr (see
|
||||
"Progress"): `open` opens the database; `read` counts the records, then reads
|
||||
them in path order, building the hierarchy and computing the digests as it goes;
|
||||
`group` groups the directories by digest and picks the maximal groups; `write`
|
||||
writes the groups to stdout.
|
||||
|
||||
#### Trees output format
|
||||
|
||||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
||||
@@ -713,15 +731,60 @@ Summary to stderr: records read, number of duplicate-tree groups, number of dupe
|
||||
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
|
||||
units.
|
||||
|
||||
### `info` mode
|
||||
|
||||
`info` takes no positional arguments and prints the database's details on
|
||||
stdout, one per line: a name, a tab, and the value. It opens the database the
|
||||
way `report` does, read-only and without the scan lock, so it runs during a scan
|
||||
and needs only read access (see "Database"). Like `report`, it never touches the
|
||||
scanned filesystem.
|
||||
|
||||
```
|
||||
database /var/lib/sfdupes/db.sqlite (default)
|
||||
database file size 2147483648 (2.0 GiB)
|
||||
-wal file size 4124152 (3.9 MiB)
|
||||
modified 2026-10-16T12:02:50Z
|
||||
schema version 1
|
||||
records 12345678
|
||||
records with head and tail hashes 1234567
|
||||
records with content hash 123456
|
||||
size of recorded files 150000000000000 (136.4 TiB)
|
||||
```
|
||||
|
||||
- `database` is the path and where it came from, as on the line every subcommand
|
||||
prints first on stderr (see "Database").
|
||||
- `database file size` and `-wal file size` are in bytes, then in human units.
|
||||
The `-wal` line is there only when a `-wal` file is beside the database, as
|
||||
while a scan runs.
|
||||
- `modified` is the time the database file was last changed, in UTC. While the
|
||||
database is in WAL mode, the newest changes may be in the `-wal` file alone.
|
||||
- `schema version` is the database's `PRAGMA user_version`. A database with
|
||||
another version than this build's is refused, as by `report`.
|
||||
- `records` is the number of records. `records with head and tail hashes` and
|
||||
`records with content hash` count the records that have those hashes (see
|
||||
"Database"), and `size of recorded files` adds up the sizes of all records, in
|
||||
bytes, then in human units.
|
||||
|
||||
`info` runs in two phases, each with its own progress display on stderr (see
|
||||
"Progress"): `open` opens the database; `count` reads the schema version and
|
||||
counts the records with one query, a single pass over the table, both in one
|
||||
read-only transaction, so the numbers agree with each other while a scan is
|
||||
committing. `info` writes its lines once the `count` display has ended, so on a
|
||||
terminal they come after the displays.
|
||||
|
||||
### Progress
|
||||
|
||||
Use the progress-bar library for all scan progress; rendering in the style of
|
||||
`pv` is the model. All progress goes to stderr.
|
||||
Use the progress-bar library for all progress of `scan`, `report`, `trees` and
|
||||
`info`; rendering in the style of `pv` is the model. All progress goes to
|
||||
stderr.
|
||||
|
||||
A scan must always show that it is working and not hung. Each phase gets its own
|
||||
display from the moment it starts, naming the phase and its place among the five
|
||||
(`[1/5] load`, `[2/5] walk`, `[3/5] hash`, `[4/5] update`, `[5/5] content`) and
|
||||
showing its progress, its elapsed time and its estimated time remaining (ETA):
|
||||
`scan`, `report`, `trees` and `info` must always show that they are working and
|
||||
not hung. Each phase gets its own display from the moment it starts, naming the
|
||||
phase and its place among the command's phases and showing its progress, its
|
||||
elapsed time and its estimated time remaining (ETA).
|
||||
|
||||
A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
|
||||
`[4/5] update`, `[5/5] content`):
|
||||
|
||||
- `load` counts the records first, then reads them against that count. While the
|
||||
count runs, the display says it is counting the records.
|
||||
@@ -738,12 +801,39 @@ showing its progress, its elapsed time and its estimated time remaining (ETA):
|
||||
and ETA, start there. The files to read then get a display of their own, with
|
||||
their count as its total.
|
||||
|
||||
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
|
||||
`[4/4] write`, see "`trees` mode"):
|
||||
|
||||
- `open` has no total. While it opens the database, waiting on a busy one
|
||||
included, the display says it is opening the database.
|
||||
- `read` counts the records first, then reads them against that count. While the
|
||||
count runs, the display says it is counting the records.
|
||||
- `group` has the directories `read` found as its total, and `write` the
|
||||
duplicate-tree groups.
|
||||
|
||||
`report` has two phases (`[1/2] open`, `[2/2] read`, see "`report` mode"):
|
||||
|
||||
- `open` has no total. While it opens the database, waiting on a busy one
|
||||
included, the display says it is opening the database.
|
||||
- `read` counts the records first, then reads them against that count, writing
|
||||
the rows to stdout as it goes. While the count runs, the display says it is
|
||||
counting the records.
|
||||
|
||||
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
|
||||
|
||||
- `open` has no total. While it opens the database, waiting on a busy one
|
||||
included, the display says it is opening the database.
|
||||
- `count` has no total either, because one query counts everything: the display
|
||||
says it is counting the records until the query returns, and its ETA is
|
||||
unknown.
|
||||
|
||||
Each display is one line holding:
|
||||
|
||||
- the phase and its place among the phases
|
||||
- a `[m/n] x%` display (items processed / total items, percent); before a phase
|
||||
has a total, the items processed, or what the phase is waiting for
|
||||
- current rate (items/s)
|
||||
- current rate (items/s), naming the items the phase counts: files, records,
|
||||
directories or groups
|
||||
- elapsed time
|
||||
- estimated time remaining, or `unknown`
|
||||
|
||||
@@ -756,7 +846,8 @@ Example shape (exact layout is flexible, content is not):
|
||||
|
||||
Additional requirements:
|
||||
|
||||
- On a terminal, a phase with an exact total (every phase but `walk`, once it
|
||||
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
|
||||
the `open` phase of `report` and `trees`, and both phases of `info`, once it
|
||||
has counted what it will do) is drawn as its line followed by the progress-bar
|
||||
library's bar, 10 columns wide, which shows the percent in place of the line's
|
||||
own. Any other phase, and a phase whose bar has no room, is drawn as the
|
||||
@@ -770,7 +861,9 @@ Additional requirements:
|
||||
gives way to the spinner. Only when those four alone are too wide, as on a
|
||||
terminal narrower than about 70 columns hours into a scan of millions of
|
||||
files, is the line cut short at the terminal's edge, so the end of the ETA is
|
||||
lost first.
|
||||
lost first. A terminal made narrower while a phase runs gets the same: the
|
||||
next draw erases the line within the new width and fits the line to it, and
|
||||
the display goes on redrawing in place.
|
||||
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
|
||||
redraws: print the line the moment each phase starts, every 5 seconds while it
|
||||
runs (whether or not an item has completed, and never more often), and once
|
||||
@@ -779,21 +872,26 @@ Additional requirements:
|
||||
goroutine counts the items, and each display draws them from a goroutine of
|
||||
its own.
|
||||
- A warning printed during a phase always lands on a line of its own, never
|
||||
inside the progress display.
|
||||
inside the progress display. So does every line written to stdout when stdout
|
||||
and stderr are the same file (the same device and inode): one terminal, as in
|
||||
a plain `sfdupes report` or `sfdupes trees`, or one pipe or file that both are
|
||||
sent to. When they are not, stdout is written without waiting on the display,
|
||||
and the display keeps being drawn while a slow reader of stdout holds up a
|
||||
write.
|
||||
- A phase that stops short of its total, as an interrupted one does, is left
|
||||
showing the count it stopped at rather than filled up.
|
||||
- `report` and `trees` modes need no progress display, only their stderr
|
||||
summaries.
|
||||
|
||||
### Error handling and exit codes
|
||||
|
||||
- `0`: success, even if individual files were skipped with warnings.
|
||||
- `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is
|
||||
already running against the same database, the database cannot be
|
||||
created/opened/read/written, a missing database for `report`/`trees`, stdout
|
||||
write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see below).
|
||||
created/opened/read/written, a missing database for `report`/`trees`/`info`,
|
||||
stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see
|
||||
below).
|
||||
- `2`: usage error (including `scan` with no `PATH` operand, `scan` with
|
||||
`--workers` below 1, and `report`/`trees` with any positional argument).
|
||||
`--workers` below 1, and `report`/`trees`/`info` with any positional
|
||||
argument).
|
||||
|
||||
A stdout write failure, such as a full disk, is reported in one line on stderr
|
||||
and exits 1. Two cases never reach sfdupes as a failed write:
|
||||
|
||||
@@ -29,6 +29,40 @@
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- new subcommand `info` prints the database's details on stdout, one per line as
|
||||
a name, a tab and the value: its path and where it came from, the sizes of the
|
||||
database file and of its `-wal` file when there is one, when the database file
|
||||
was last changed, the schema version, the number of records, how many have
|
||||
head and tail hashes and how many a content hash, and the sizes of all records
|
||||
added up. It opens the database as `report` does, and shows two phases on
|
||||
stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come
|
||||
from one read-only transaction, the counts from one query (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/117)
|
||||
|
||||
- `report` shows its work on stderr in two phases, `[1/2] open` and
|
||||
`[2/2] read`, with the progress display `scan` and `trees` use, from the
|
||||
moment it starts until the summary line: `open` says it is opening the
|
||||
database, also while it waits on a busy one; `read` counts the records, then
|
||||
reads them against that count in signature order, which SQLite takes from the
|
||||
`files_signature` index without sorting, so the first record comes at once.
|
||||
`report` forms the duplicate groups itself, holding only those of the size it
|
||||
is reading, and writes them ordered by first path before it reads the next
|
||||
size. stdout and the summary line are unchanged, and SQLite no longer needs a
|
||||
temporary file to sort in (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/114)
|
||||
|
||||
- `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`,
|
||||
`[3/4] group` and `[4/4] write`, each with the progress display `scan` uses,
|
||||
from the moment it starts until the summary line: `open` says it is opening
|
||||
the database, also while it waits on a busy one; `read` counts the records,
|
||||
then reads them against that count; `group` counts the directories it groups,
|
||||
and `write` the groups it writes. A display's rate names what its phase
|
||||
counts: files, records, directories or groups. stdout and the summary line are
|
||||
unchanged. When stdout and stderr are the same file, as one terminal is, none
|
||||
of stdout's lines shares a line with the display; when they are not, the
|
||||
display keeps being drawn while stdout waits on a slow reader (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/113)
|
||||
|
||||
- each `scan` phase's progress display names the phase and its place among the
|
||||
five, as `[2/5] walk`, from the moment it starts, with its progress, elapsed
|
||||
time and ETA: `load` counts the records first, the walk's total is estimated
|
||||
@@ -37,12 +71,12 @@
|
||||
until the first one is returned. On a terminal a phase with an exact total
|
||||
draws its line followed by the library's bar, and any other a spinner followed
|
||||
by the line; every display is drawn again several times a second. No line
|
||||
reaches the terminal's last column, however narrow: a line too wide leaves out
|
||||
its rate, then its bar, keeping the phase, progress, elapsed time and ETA
|
||||
whole, and is cut short only when those alone do not fit, as below about 70
|
||||
columns late in a large scan. In plain mode the line is printed every 5
|
||||
seconds whether or not an item completed (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/112)
|
||||
reaches the terminal's last column, however narrow, also after the terminal is
|
||||
made narrower during a phase: a line too wide leaves out its rate, then its
|
||||
bar, keeping the phase, progress, elapsed time and ETA whole, and is cut short
|
||||
only when those alone do not fit, as below about 70 columns late in a large
|
||||
scan. In plain mode the line is printed every 5 seconds whether or not an item
|
||||
completed (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/112)
|
||||
|
||||
- `scan`, `report` and `trees` first print a line on stderr naming the database
|
||||
they use, as an absolute path, and whether that path is the default or comes
|
||||
|
||||
@@ -61,8 +61,8 @@ CREATE TABLE files (
|
||||
) WITHOUT ROWID
|
||||
`
|
||||
|
||||
// createIndexSQL indexes the records by signature, so report can have
|
||||
// SQLite group them without sorting the whole table.
|
||||
// createIndexSQL indexes the records by signature, so report can read
|
||||
// them in signature order without sorting the table.
|
||||
const createIndexSQL = `
|
||||
CREATE INDEX files_signature ON files (size, head, tail, content)
|
||||
`
|
||||
@@ -79,7 +79,8 @@ ON CONFLICT (path) DO UPDATE SET
|
||||
content = excluded.content
|
||||
`
|
||||
|
||||
// errNoDatabase reports a missing database file for report/trees.
|
||||
// errNoDatabase reports a missing database file for report, trees and
|
||||
// info.
|
||||
var errNoDatabase = errors.New(
|
||||
"no database (run \"sfdupes scan\" first, or set " + databaseEnv + ")")
|
||||
|
||||
@@ -131,9 +132,9 @@ const scanParams = "_pragma=busy_timeout(10000)" +
|
||||
"&_pragma=journal_mode(WAL)" +
|
||||
"&_pragma=synchronous(NORMAL)"
|
||||
|
||||
// reportParams are the connection parameters for report and trees:
|
||||
// read-only, with the same busy timeout. They set no journal mode,
|
||||
// because setting one is a write.
|
||||
// reportParams are the connection parameters for report, trees and
|
||||
// info: read-only, with the same busy timeout. They set no journal
|
||||
// mode, because setting one is a write.
|
||||
const reportParams = "mode=ro" +
|
||||
"&_pragma=busy_timeout(10000)" +
|
||||
"&_pragma=query_only(1)"
|
||||
@@ -262,9 +263,9 @@ func closeScanDatabase(ctx context.Context, db *sql.DB, path string) {
|
||||
_ = db.Close()
|
||||
}
|
||||
|
||||
// openReportDatabase opens an existing database for the report and
|
||||
// trees subcommands. A missing database file is an error directing the
|
||||
// user to run scan first; the schema version must match exactly.
|
||||
// openReportDatabase opens an existing database for the report, trees
|
||||
// and info subcommands. A missing database file is an error directing
|
||||
// the user to run scan first; the schema version must match exactly.
|
||||
func openReportDatabase(ctx context.Context,
|
||||
path string,
|
||||
) (*sql.DB, error) {
|
||||
@@ -403,10 +404,32 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
|
||||
return v, nil
|
||||
}
|
||||
|
||||
// loadFileRows streams every record to fn in path order: byte order,
|
||||
// which is the order of the primary key, so SQLite does not sort.
|
||||
func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
|
||||
rows, err := db.QueryContext(ctx,
|
||||
// loadFileRows gives total the number of records, then streams every
|
||||
// record to fn in path order: byte order, which is the order of the
|
||||
// primary key, so SQLite does not sort. The count and the records are
|
||||
// read in one transaction, so they agree while a scan is committing.
|
||||
func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
||||
fn func(r scanRec),
|
||||
) error {
|
||||
// Everything goes through tx: db has one connection, so a query on
|
||||
// db would wait for tx forever.
|
||||
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
|
||||
if err != nil {
|
||||
return fmt.Errorf("read records: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
var n int64
|
||||
|
||||
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
||||
if err != nil {
|
||||
return fmt.Errorf("count records: %w", err)
|
||||
}
|
||||
|
||||
total(n)
|
||||
|
||||
rows, err := tx.QueryContext(ctx,
|
||||
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
|
||||
"FROM files ORDER BY path")
|
||||
if err != nil {
|
||||
@@ -441,77 +464,74 @@ func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// dupeRowsSQL selects every record in a duplicate group, with the
|
||||
// group's first path. A group is the records with a content hash that
|
||||
// share a size, head, tail, and content, when there are two or more of
|
||||
// them. The rows come in report order: groups by size descending, then
|
||||
// by first path, and each group's paths ascending.
|
||||
const dupeRowsSQL = `
|
||||
SELECT g.first, f.path, f.size
|
||||
FROM files AS f
|
||||
JOIN (
|
||||
SELECT size, head, tail, content, MIN(path) AS first
|
||||
FROM files
|
||||
WHERE content <> ''
|
||||
GROUP BY size, head, tail, content
|
||||
HAVING COUNT(*) > 1
|
||||
) AS g USING (size, head, tail, content)
|
||||
ORDER BY f.size DESC, g.first, f.path
|
||||
// signatureRowsSQL selects every record's signature and path in
|
||||
// signature order, largest size first. That is the order of the
|
||||
// files_signature index read backwards, so SQLite reads the index alone
|
||||
// and returns the first row at once, with nothing to sort.
|
||||
const signatureRowsSQL = `
|
||||
SELECT size, head, tail, content, path
|
||||
FROM files
|
||||
ORDER BY size DESC, head DESC, tail DESC, content DESC
|
||||
`
|
||||
|
||||
// loadDupeRows streams the rows of dupeRowsSQL to fn and returns the
|
||||
// number of records in the database. The count and the rows are read
|
||||
// in one transaction, so they agree while a scan is committing. An
|
||||
// error from fn stops the reading and is returned as it is.
|
||||
func loadDupeRows(ctx context.Context, db *sql.DB,
|
||||
fn func(first, path string, size int64) error,
|
||||
) (int, error) {
|
||||
// loadSignatureRows gives total the number of records, then streams
|
||||
// every record's signature and path to fn, without its mtime, in the
|
||||
// order of signatureRowsSQL. The count and the records are read in one
|
||||
// transaction, so they agree while a scan is committing. An error from
|
||||
// fn stops the reading and is returned as it is.
|
||||
func loadSignatureRows(ctx context.Context, db *sql.DB,
|
||||
total func(n int64), fn func(r scanRec) error,
|
||||
) error {
|
||||
// Everything goes through tx: the report connection is the only
|
||||
// one, so a query on db would wait for tx forever.
|
||||
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("read records: %w", err)
|
||||
return fmt.Errorf("read records: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
var records int
|
||||
var n int64
|
||||
|
||||
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&records)
|
||||
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("read records: %w", err)
|
||||
return fmt.Errorf("count records: %w", err)
|
||||
}
|
||||
|
||||
rows, err := tx.QueryContext(ctx, dupeRowsSQL)
|
||||
total(n)
|
||||
|
||||
rows, err := tx.QueryContext(ctx, signatureRowsSQL)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("read records: %w", err)
|
||||
return fmt.Errorf("read records: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
first, path []byte
|
||||
size int64
|
||||
path []byte
|
||||
r scanRec
|
||||
)
|
||||
|
||||
err = rows.Scan(&first, &path, &size)
|
||||
err = rows.Scan(&r.size, &r.head, &r.tail, &r.content, &path)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("read record: %w", err)
|
||||
return fmt.Errorf("read record: %w", err)
|
||||
}
|
||||
|
||||
err = fn(string(first), string(path), size)
|
||||
r.path = string(path)
|
||||
|
||||
err = fn(r)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("read records: %w", err)
|
||||
return fmt.Errorf("read records: %w", err)
|
||||
}
|
||||
|
||||
return records, nil
|
||||
return nil
|
||||
}
|
||||
|
||||
// countRecords returns the number of records in the database.
|
||||
@@ -526,6 +546,60 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// databaseCountsSQL counts the records, those with head and tail
|
||||
// hashes and those with a content hash, and adds up their sizes, in one
|
||||
// pass over the table. A SUM over no rows is NULL, so on an empty table
|
||||
// each SUM falls back to 0.
|
||||
const databaseCountsSQL = `
|
||||
SELECT COUNT(*),
|
||||
COALESCE(SUM(head <> '' AND tail <> ''), 0),
|
||||
COALESCE(SUM(content <> ''), 0),
|
||||
COALESCE(SUM(size), 0)
|
||||
FROM files
|
||||
`
|
||||
|
||||
// databaseCounts is what info reads from the database: its schema
|
||||
// version and the counts of databaseCountsSQL.
|
||||
type databaseCounts struct {
|
||||
schemaVersion int
|
||||
records int64 // every record
|
||||
headTail int64 // records with head and tail hashes
|
||||
content int64 // records with a content hash
|
||||
size int64 // the sizes of all records added up
|
||||
}
|
||||
|
||||
// readDatabaseCounts reads the schema version, then the counts of
|
||||
// databaseCountsSQL, in one read-only transaction, so both describe the
|
||||
// database at one moment while a scan is committing.
|
||||
func readDatabaseCounts(ctx context.Context,
|
||||
db *sql.DB,
|
||||
) (databaseCounts, error) {
|
||||
var c databaseCounts
|
||||
|
||||
// Everything goes through tx: the report connection is the only
|
||||
// one, so a query on db would wait for tx forever.
|
||||
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
|
||||
if err != nil {
|
||||
return c, fmt.Errorf("count records: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
err = tx.QueryRowContext(ctx, "PRAGMA user_version").
|
||||
Scan(&c.schemaVersion)
|
||||
if err != nil {
|
||||
return c, fmt.Errorf("read schema version: %w", err)
|
||||
}
|
||||
|
||||
err = tx.QueryRowContext(ctx, databaseCountsSQL).
|
||||
Scan(&c.records, &c.headTail, &c.content, &c.size)
|
||||
if err != nil {
|
||||
return c, fmt.Errorf("count records: %w", err)
|
||||
}
|
||||
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// loadFileMeta streams every record's path, size, mtime, and whether
|
||||
// it carries hashes to fn. Scan change detection needs no hash
|
||||
// values, and skipping the hash columns keeps the scan's in-memory
|
||||
|
||||
+117
-1
@@ -284,12 +284,128 @@ func TestCloseScanDatabaseWhileReportOpen(t *testing.T) {
|
||||
|
||||
defer func() { _ = reportDB.Close() }()
|
||||
|
||||
err = loadFileRows(t.Context(), reportDB, func(scanRec) {})
|
||||
err = loadFileRows(t.Context(), reportDB, func(int64) {},
|
||||
func(scanRec) {})
|
||||
if err != nil {
|
||||
t.Fatalf("loadFileRows: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestLoadRowsCountAgreeDuringScan has a scan commit records after
|
||||
// loadFileRows, as trees calls it, or loadSignatureRows, as report
|
||||
// calls it, has counted the records and before it reads them, as when
|
||||
// trees or report runs during a scan: each must read exactly as many
|
||||
// records as it counted.
|
||||
func TestLoadRowsCountAgreeDuringScan(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
loads := map[string]func(context.Context, *sql.DB, func(int64),
|
||||
func(scanRec)) error{
|
||||
"loadFileRows": loadFileRows,
|
||||
"loadSignatureRows": func(ctx context.Context, db *sql.DB,
|
||||
total func(int64), fn func(scanRec),
|
||||
) error {
|
||||
return loadSignatureRows(ctx, db, total, func(r scanRec) error {
|
||||
fn(r)
|
||||
|
||||
return nil
|
||||
})
|
||||
},
|
||||
}
|
||||
|
||||
for name, load := range loads {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
path := testDBPath(t)
|
||||
|
||||
scanDB, err := openScanDatabase(t.Context(), path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
defer func() { _ = scanDB.Close() }()
|
||||
|
||||
recs := smokeTreeRecs()
|
||||
|
||||
err = applyChanges(t.Context(), scanDB, recs[:3], nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
reportDB, err := openReportDatabase(t.Context(), path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
defer func() { _ = reportDB.Close() }()
|
||||
|
||||
var counted, read int64
|
||||
|
||||
err = load(t.Context(), reportDB, func(n int64) {
|
||||
counted = n
|
||||
|
||||
err := applyChanges(t.Context(), scanDB, recs[3:], nil, nil)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}, func(scanRec) { read++ })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if counted != 3 || read != counted {
|
||||
t.Errorf("counted %d records and read %d, want 3 and 3",
|
||||
counted, read)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
|
||||
// from the files_signature index alone, with nothing to sort: the
|
||||
// first record comes at once, so report's read phase counts each
|
||||
// record as it comes rather than waiting on SQLite for all of them.
|
||||
func TestSignatureRowsNeedNoSort(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := openTestDB(t)
|
||||
|
||||
rows, err := db.QueryContext(t.Context(),
|
||||
"EXPLAIN QUERY PLAN "+signatureRowsSQL)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
defer func() { _ = rows.Close() }()
|
||||
|
||||
var plan []string
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
id, parent, unused int
|
||||
detail string
|
||||
)
|
||||
|
||||
err = rows.Scan(&id, &parent, &unused, &detail)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
plan = append(plan, detail)
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
want := []string{"SCAN files USING COVERING INDEX files_signature"}
|
||||
if !slices.Equal(plan, want) {
|
||||
t.Errorf("query plan = %q, want %q", plan, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestApplyChangesRoundTrip(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -0,0 +1,128 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// infoPhase names one phase of info with its place among all of them,
|
||||
// as "[2/2] count", for the phase's progress display. The list here is
|
||||
// the one place the phases are counted.
|
||||
func infoPhase(name string) string {
|
||||
return phaseLabel([]string{openPhase, "count"}, name)
|
||||
}
|
||||
|
||||
// runInfo implements the info subcommand: it prints the database's
|
||||
// details on stdout (see writeInfo). It opens the database as report
|
||||
// does, read-only and without the scan lock, and never touches the
|
||||
// scanned filesystem. It names the database on stderr before it opens
|
||||
// it. Any database problem, including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in two phases, each under a progress display of its
|
||||
// own on stderr: open (open the database) and count (count the records,
|
||||
// with one query). Each display ends before the next phase starts, and
|
||||
// before an error is returned to be printed. The details are written
|
||||
// once the last display has ended, so no display is drawn while they
|
||||
// are written.
|
||||
func runInfo(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr, "info: database %s (%s)\n",
|
||||
escapePath(dbPath), source)
|
||||
|
||||
// Opening can wait on a busy database for as long as the busy
|
||||
// timeout.
|
||||
prog := newProgress(progressParams{
|
||||
label: infoPhase(openPhase), total: -1,
|
||||
waitingFor: openingDatabase,
|
||||
})
|
||||
db, err := openReportDatabase(ctx, dbPath)
|
||||
|
||||
prog.finish()
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
// One query counts everything, so the phase never has a total.
|
||||
prog = newProgress(progressParams{
|
||||
label: infoPhase("count"), total: -1,
|
||||
waitingFor: countingRecords,
|
||||
})
|
||||
counts, err := readDatabaseCounts(ctx, db)
|
||||
|
||||
prog.finish()
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("database %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
return writeInfo(stdout, dbPath, source, counts)
|
||||
}
|
||||
|
||||
// writeInfo writes the details of the database at dbPath to stdout,
|
||||
// one per line, as a name, a tab and the value (README "`info` mode"):
|
||||
// the path and source as databasePath returns them, the sizes of the
|
||||
// database file and of its -wal file when there is one, when the
|
||||
// database file was last changed, and the schema version and record
|
||||
// counts that readDatabaseCounts read.
|
||||
func writeInfo(stdout io.Writer, dbPath, source string,
|
||||
counts databaseCounts,
|
||||
) error {
|
||||
// SQLite keeps the -wal file beside the database file, named from
|
||||
// its path with symlinks resolved.
|
||||
file, err := filepath.EvalSymlinks(dbPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("database: %w", err)
|
||||
}
|
||||
|
||||
dbFile, err := os.Stat(file)
|
||||
if err != nil {
|
||||
return fmt.Errorf("database: %w", err)
|
||||
}
|
||||
|
||||
walFile, err := os.Stat(file + "-wal")
|
||||
if err != nil && !errors.Is(err, fs.ErrNotExist) {
|
||||
return fmt.Errorf("database: %w", err)
|
||||
}
|
||||
|
||||
var b strings.Builder
|
||||
|
||||
fmt.Fprintf(&b, "database\t%s (%s)\n", escapePath(dbPath), source)
|
||||
fmt.Fprintf(&b, "database file size\t%d (%s)\n",
|
||||
dbFile.Size(), humanBytes(dbFile.Size()))
|
||||
|
||||
if walFile != nil {
|
||||
fmt.Fprintf(&b, "-wal file size\t%d (%s)\n",
|
||||
walFile.Size(), humanBytes(walFile.Size()))
|
||||
}
|
||||
|
||||
fmt.Fprintf(&b, "modified\t%s\n",
|
||||
dbFile.ModTime().UTC().Format(time.RFC3339))
|
||||
fmt.Fprintf(&b, "schema version\t%d\n", counts.schemaVersion)
|
||||
fmt.Fprintf(&b, "records\t%d\n", counts.records)
|
||||
fmt.Fprintf(&b, "records with head and tail hashes\t%d\n",
|
||||
counts.headTail)
|
||||
fmt.Fprintf(&b, "records with content hash\t%d\n", counts.content)
|
||||
fmt.Fprintf(&b, "size of recorded files\t%d (%s)\n",
|
||||
counts.size, humanBytes(counts.size))
|
||||
|
||||
_, err = io.WriteString(stdout, b.String())
|
||||
if err != nil {
|
||||
return fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
+168
@@ -0,0 +1,168 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// infoOpenLine is the plain line of info's open phase when it opens the
|
||||
// database at once; it prints it as the phase starts and as it ends.
|
||||
const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
|
||||
"eta unknown"
|
||||
|
||||
// infoCountLine is the plain line of info's count phase when the count
|
||||
// takes under a second; it prints it as the phase starts and as it
|
||||
// ends.
|
||||
const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " +
|
||||
"eta unknown"
|
||||
|
||||
// infoRecs are four records: a duplicate pair with every hash, a file
|
||||
// of 20 MiB with head and tail hashes but no content hash yet, and a
|
||||
// file never hashed.
|
||||
func infoRecs() []scanRec {
|
||||
return []scanRec{
|
||||
{size: 5, head: "h", tail: "t", content: "c", path: "/d/a"},
|
||||
{size: 5, head: "h", tail: "t", content: "c", path: "/d/b"},
|
||||
{size: 20 << 20, head: "bh", tail: "bt", path: "/d/big"},
|
||||
{size: 7, path: "/d/unique"},
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunInfo runs info on an empty and on a filled database, with
|
||||
// stdout and stderr the same file that is not a terminal, and checks
|
||||
// everything written there: the line naming the database, each phase's
|
||||
// line as it starts and as it ends, and then the details, each on a
|
||||
// line of its own. No -wal file is beside a database scan has closed.
|
||||
func TestRunInfo(t *testing.T) {
|
||||
cases := map[string]struct {
|
||||
recs []scanRec
|
||||
want string // the details after the database file's time
|
||||
}{
|
||||
"empty": {
|
||||
want: "schema version\t1\n" +
|
||||
"records\t0\n" +
|
||||
"records with head and tail hashes\t0\n" +
|
||||
"records with content hash\t0\n" +
|
||||
"size of recorded files\t0 (0 B)\n",
|
||||
},
|
||||
"filled": {
|
||||
recs: infoRecs(),
|
||||
want: "schema version\t1\n" +
|
||||
"records\t4\n" +
|
||||
"records with head and tail hashes\t3\n" +
|
||||
"records with content hash\t2\n" +
|
||||
"size of recorded files\t20971537 (20.0 MiB)\n",
|
||||
},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := seedDatabase(t, tc.recs)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
file, err := os.Stat(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run([]string{cmdInfo}, os.Stderr, os.Stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(info) = %d, want %d; stderr: %s",
|
||||
code, exitOK, stderr())
|
||||
}
|
||||
|
||||
named := path + " (from " + databaseEnv + ")\n"
|
||||
want := "info: database " + named +
|
||||
infoOpenLine + "\n" + infoOpenLine + "\n" +
|
||||
infoCountLine + "\n" + infoCountLine + "\n" +
|
||||
"database\t" + named +
|
||||
fmt.Sprintf("database file size\t%d (%s)\n",
|
||||
file.Size(), humanBytes(file.Size())) +
|
||||
"modified\t" + file.ModTime().UTC().Format(time.RFC3339) +
|
||||
"\n" + tc.want
|
||||
|
||||
if got := stderr(); got != want {
|
||||
t.Errorf("stdout and stderr = %q, want %q", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunInfoDuringScan runs info while a scan holds the lock and has
|
||||
// the database open in WAL mode, with records committed to the -wal
|
||||
// file only. info must count those records and show the -wal file's
|
||||
// size, also when the database path is a symlink: SQLite keeps the
|
||||
// -wal file beside the symlink's target.
|
||||
func TestRunInfoDuringScan(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
link := filepath.Join(t.TempDir(), "link.sqlite")
|
||||
|
||||
err := os.Symlink(path, link)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
db, err := openScanDatabase(t.Context(), path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
err = applyChanges(t.Context(), db, infoRecs(), nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
holdScanLock(t, path)
|
||||
|
||||
wal, err := os.Stat(path + "-wal")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
for _, named := range []string{path, link} {
|
||||
t.Run(filepath.Base(named), func(t *testing.T) {
|
||||
t.Setenv(databaseEnv, named)
|
||||
|
||||
stdout := runStdout(t, cmdInfo)
|
||||
|
||||
for _, want := range []string{
|
||||
"database\t" + named + " (from " + databaseEnv + ")\n",
|
||||
fmt.Sprintf("\n-wal file size\t%d (%s)\n",
|
||||
wal.Size(), humanBytes(wal.Size())),
|
||||
"\nrecords\t4\n",
|
||||
} {
|
||||
if !strings.Contains(stdout, want) {
|
||||
t.Errorf("stdout = %q, want it to hold %q", stdout, want)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunInfoNeedsOnlyReadAccess(t *testing.T) {
|
||||
// README "`info` mode": like report, info needs only read access to
|
||||
// the database file and none to its directory.
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
scanFixture(t)
|
||||
makeReadOnly(t, path)
|
||||
|
||||
stdout := runStdout(t, cmdInfo)
|
||||
|
||||
if want := "\nrecords\t2\n"; !strings.Contains(stdout, want) {
|
||||
t.Errorf("stdout = %q, want it to hold %q", stdout, want)
|
||||
}
|
||||
|
||||
if strings.Contains(stdout, "-wal") {
|
||||
t.Errorf("stdout = %q, want no -wal file between scans", stdout)
|
||||
}
|
||||
}
|
||||
@@ -8,13 +8,14 @@
|
||||
// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
|
||||
// or larger. scan maintains a persistent SQLite database of file
|
||||
// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
|
||||
// that the reporting subcommands read.
|
||||
// that the other subcommands read.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// sfdupes scan [--workers N] [-x] PATH...
|
||||
// sfdupes report > dupes.tsv
|
||||
// sfdupes trees > dupetrees.tsv
|
||||
// sfdupes info
|
||||
// sfdupes --version
|
||||
//
|
||||
// See README.md for the complete specification.
|
||||
@@ -44,6 +45,7 @@ const (
|
||||
cmdScan = "scan"
|
||||
cmdReport = "report"
|
||||
cmdTrees = "trees"
|
||||
cmdInfo = "info"
|
||||
)
|
||||
|
||||
// errNoSubcommand is returned by the root command when it is invoked
|
||||
@@ -73,8 +75,8 @@ func main() {
|
||||
// exit code. It is the program's single exit point: the subcommands
|
||||
// return their errors instead of exiting, so every deferred cleanup —
|
||||
// above all closing the database, which checkpoints the SQLite WAL —
|
||||
// runs before the process ends. The report and trees subcommands write
|
||||
// their data to stdout.
|
||||
// runs before the process ends. The report, trees and info subcommands
|
||||
// write their data to stdout.
|
||||
func run(args []string, stdout, stderr io.Writer) int {
|
||||
// A nil slice makes cobra fall back to os.Args, which would let a
|
||||
// test binary's own flags reach the command tree.
|
||||
@@ -148,6 +150,41 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
root.Flags().BoolVarP(&showVersion, "version", "v", false,
|
||||
"print the version to stdout")
|
||||
|
||||
reportCmd := &cobra.Command{
|
||||
Use: cmdReport,
|
||||
Short: "Read the scan database and print the file-level duplicates report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runReport(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
treesCmd := &cobra.Command{
|
||||
Use: cmdTrees,
|
||||
Short: "Read the scan database and print the duplicate-tree report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runTrees(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
infoCmd := &cobra.Command{
|
||||
Use: cmdInfo,
|
||||
Short: "Read the scan database and print its details",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runInfo(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
|
||||
|
||||
return root
|
||||
}
|
||||
|
||||
// newScanCommand builds the scan subcommand, with its --workers and -x
|
||||
// flags.
|
||||
func newScanCommand() *cobra.Command {
|
||||
var (
|
||||
scanWorkers int
|
||||
scanOneFS bool
|
||||
@@ -172,27 +209,7 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
|
||||
"do not cross filesystem boundaries")
|
||||
|
||||
reportCmd := &cobra.Command{
|
||||
Use: cmdReport,
|
||||
Short: "Read the scan database and print the file-level duplicates report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runReport(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
treesCmd := &cobra.Command{
|
||||
Use: cmdTrees,
|
||||
Short: "Read the scan database and print the duplicate-tree report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runTrees(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
root.AddCommand(scanCmd, reportCmd, treesCmd)
|
||||
|
||||
return root
|
||||
return scanCmd
|
||||
}
|
||||
|
||||
// checkScanWorkers rejects a scan --workers value below 1. That is a
|
||||
|
||||
+49
-19
@@ -284,10 +284,16 @@ func assertFatalOutput(t *testing.T, stderr, stdout string) {
|
||||
}
|
||||
|
||||
func TestRunMissingDatabaseIsFatal(t *testing.T) {
|
||||
// README §Database: report and trees need an existing database; a
|
||||
// missing one exits 1 with a message telling the user to run scan,
|
||||
// after the line naming the database.
|
||||
for _, name := range []string{cmdReport, cmdTrees} {
|
||||
// README §Database: report, trees and info need an existing
|
||||
// database; a missing one exits 1 with a message telling the user to
|
||||
// run scan, after the line naming the database.
|
||||
openLines := map[string]string{
|
||||
cmdReport: reportOpenLine,
|
||||
cmdTrees: treesOpenLine,
|
||||
cmdInfo: infoOpenLine,
|
||||
}
|
||||
|
||||
for name, openLine := range openLines {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
@@ -303,8 +309,10 @@ func TestRunMissingDatabaseIsFatal(t *testing.T) {
|
||||
t.Errorf("run(%s) = %d, want %d", name, code, exitFatal)
|
||||
}
|
||||
|
||||
// Each shows its open phase, as it starts and as it ends,
|
||||
// before the error.
|
||||
want := name + ": database " + path + " (from " + databaseEnv +
|
||||
")\n" +
|
||||
")\n" + openLine + "\n" + openLine + "\n" +
|
||||
"sfdupes: " + path + ": no database (run \"sfdupes " +
|
||||
"scan\" first, or set " + databaseEnv + ")\n"
|
||||
if got := stderr(); got != want {
|
||||
@@ -505,8 +513,9 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
// exist at the default path on the machine running the test, so
|
||||
// nothing here may create or read one: scan gets an operand that
|
||||
// does not exist, which stops it before it takes the lock, and
|
||||
// report and trees a cancelled context, which stops them before
|
||||
// they read anything.
|
||||
// report, trees and info a cancelled context, which stops them
|
||||
// before they read anything. They show their open phase after the
|
||||
// line.
|
||||
t.Setenv(databaseEnv, "")
|
||||
|
||||
// Whether scan calls the database new depends on that machine too.
|
||||
@@ -534,13 +543,22 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
run: func() error {
|
||||
return runReport(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "report: database " + defaultDatabasePath + " (default)\n",
|
||||
want: "report: database " + defaultDatabasePath + " (default)\n" +
|
||||
reportOpenLine + "\n" + reportOpenLine + "\n",
|
||||
},
|
||||
cmdTrees: {
|
||||
run: func() error {
|
||||
return runTrees(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "trees: database " + defaultDatabasePath + " (default)\n",
|
||||
want: "trees: database " + defaultDatabasePath + " (default)\n" +
|
||||
treesOpenLine + "\n" + treesOpenLine + "\n",
|
||||
},
|
||||
cmdInfo: {
|
||||
run: func() error {
|
||||
return runInfo(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "info: database " + defaultDatabasePath + " (default)\n" +
|
||||
infoOpenLine + "\n" + infoOpenLine + "\n",
|
||||
},
|
||||
}
|
||||
|
||||
@@ -568,6 +586,7 @@ func TestRunUsageErrors(t *testing.T) {
|
||||
"scan without paths": {[]string{cmdScan}, usageMarker},
|
||||
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
||||
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
||||
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
||||
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
||||
"unknown subcommand": {[]string{"nope"}, "unknown command"},
|
||||
}
|
||||
@@ -908,11 +927,16 @@ func TestRunReportSucceeds(t *testing.T) {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
want = "report: database " + path + " (from " + databaseEnv + ")\n" +
|
||||
"report: 2 records read, 1 duplicate groups, 1 dupe files, " +
|
||||
// Its phases come between the two lines; see
|
||||
// TestRunReportProgressNamesEachPhase.
|
||||
first := "report: database " + path + " (from " + databaseEnv + ")\n"
|
||||
last := "\nreport: 2 records read, 1 duplicate groups, 1 dupe files, " +
|
||||
"300 B reclaimable\n"
|
||||
if got := stderr(); got != want {
|
||||
t.Errorf("stderr = %q, want %q", got, want)
|
||||
|
||||
if got := stderr(); !strings.HasPrefix(got, first) ||
|
||||
!strings.HasSuffix(got, last) {
|
||||
t.Errorf("stderr = %q, want it to start %q and end %q",
|
||||
got, first, last)
|
||||
}
|
||||
|
||||
assertNoSidecars(t, path)
|
||||
@@ -942,11 +966,16 @@ func TestRunTreesSucceeds(t *testing.T) {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
want = "trees: database " + path + " (from " + databaseEnv + ")\n" +
|
||||
"trees: 2 records read, 1 duplicate tree groups, 1 dupe trees, " +
|
||||
"300 B reclaimable\n"
|
||||
if got := stderr(); got != want {
|
||||
t.Errorf("stderr = %q, want %q", got, want)
|
||||
// Its phases come between the two lines; see
|
||||
// TestRunTreesProgressNamesEachPhase.
|
||||
first := "trees: database " + path + " (from " + databaseEnv + ")\n"
|
||||
last := "\ntrees: 2 records read, 1 duplicate tree groups, " +
|
||||
"1 dupe trees, 300 B reclaimable\n"
|
||||
|
||||
if got := stderr(); !strings.HasPrefix(got, first) ||
|
||||
!strings.HasSuffix(got, last) {
|
||||
t.Errorf("stderr = %q, want it to start %q and end %q",
|
||||
got, first, last)
|
||||
}
|
||||
|
||||
assertNoSidecars(t, path)
|
||||
@@ -1238,7 +1267,7 @@ func TestRunReportsDuringScan(t *testing.T) {
|
||||
func TestRunStdoutClosedIsFatal(t *testing.T) {
|
||||
// README §Error handling: a stdout write failure exits 1, reported
|
||||
// in one line on stderr.
|
||||
for _, name := range []string{cmdReport, cmdTrees} {
|
||||
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
@@ -1288,6 +1317,7 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
|
||||
cases := map[string]func(context.Context, io.Writer) error{
|
||||
cmdReport: runReport,
|
||||
cmdTrees: runTrees,
|
||||
cmdInfo: runInfo,
|
||||
}
|
||||
|
||||
for name, fn := range cases {
|
||||
|
||||
+118
-20
@@ -2,9 +2,11 @@ package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
@@ -45,6 +47,18 @@ const spinnerMargin = 5
|
||||
// around them, and the terminal's last column.
|
||||
const barMargin = barWidth + 11
|
||||
|
||||
// countingRecords is what the display of a phase that reads the records
|
||||
// says while it counts them, before it has its total.
|
||||
const countingRecords = "counting the records"
|
||||
|
||||
// openPhase is the name of the phase in which report, trees and info
|
||||
// open the database, and openingDatabase what its display says while
|
||||
// it does.
|
||||
const (
|
||||
openPhase = "open"
|
||||
openingDatabase = "opening the database"
|
||||
)
|
||||
|
||||
// stderrIsTTY reports whether stderr is attached to a terminal.
|
||||
func stderrIsTTY() bool {
|
||||
return term.IsTerminal(int(os.Stderr.Fd()))
|
||||
@@ -71,6 +85,7 @@ func phaseLabel(phases []string, phase string) string {
|
||||
// progressParams describe one phase's progress display as it starts.
|
||||
type progressParams struct {
|
||||
label string // the phase and its place, from phaseLabel
|
||||
unit string // what the phase counts, as "files"
|
||||
total int64 // the items the phase will complete; -1 when not known
|
||||
estimate bool // total is an estimate, which the count may pass
|
||||
waitingFor string // what the phase waits for until setTotal is called
|
||||
@@ -89,7 +104,8 @@ type progressParams struct {
|
||||
// below it. Every method must be called from the main goroutine. A nil
|
||||
// *progress is a valid no-display receiver: every method is a no-op, so
|
||||
// batched database flushes during the hash phase can reuse the update
|
||||
// phase's helpers without rendering anything.
|
||||
// phase's helpers without rendering anything, and its writer writes
|
||||
// with no display to clear.
|
||||
type progress struct {
|
||||
width func() int // reads the terminal's width; nil when not a TTY
|
||||
stop chan struct{} // closed by finish to end the draws
|
||||
@@ -98,7 +114,9 @@ type progress struct {
|
||||
mu sync.Mutex
|
||||
bar *progressbar.ProgressBar // what the library draws; nil until then
|
||||
withBar bool // bar is the library's bar, not its spinner
|
||||
widest int // the widest line bar has drawn, in columns
|
||||
label string
|
||||
unit string
|
||||
total int64
|
||||
estimate bool
|
||||
waitingFor string
|
||||
@@ -126,8 +144,9 @@ func startProgress(params progressParams, width func() int,
|
||||
now := time.Now()
|
||||
p := &progress{
|
||||
width: width, stop: make(chan struct{}), done: make(chan struct{}),
|
||||
label: params.label, total: params.total, estimate: params.estimate,
|
||||
waitingFor: params.waitingFor, start: now, rateStart: now,
|
||||
label: params.label, unit: params.unit, total: params.total,
|
||||
estimate: params.estimate, waitingFor: params.waitingFor,
|
||||
start: now, rateStart: now,
|
||||
}
|
||||
|
||||
// Show the zero state at once: the first item may take minutes,
|
||||
@@ -176,15 +195,20 @@ func (p *progress) draw() {
|
||||
text, withBar = p.line(width-barMargin, false)
|
||||
}
|
||||
|
||||
margin := barMargin
|
||||
if !withBar {
|
||||
text, _ = p.line(width-spinnerMargin, true)
|
||||
margin = spinnerMargin
|
||||
text, _ = p.line(width-margin, true)
|
||||
}
|
||||
|
||||
// The library makes a bar or a spinner. A display that turns from
|
||||
// one to the other, as its line grows or the terminal is resized,
|
||||
// gets a new one.
|
||||
if p.bar == nil || p.withBar != withBar {
|
||||
p.newBar(withBar)
|
||||
// The library makes a bar or a spinner, and before each draw erases
|
||||
// its line with as many spaces as the widest line it has drawn. A
|
||||
// display gets a new one when it turns from one to the other, as its
|
||||
// line grows or the terminal is resized, and when the terminal has
|
||||
// been made too narrow for that erase, which would wrap and leave a
|
||||
// line behind at every draw.
|
||||
if p.bar == nil || p.withBar != withBar || p.widest >= width {
|
||||
p.newBar(withBar, width)
|
||||
}
|
||||
|
||||
if withBar {
|
||||
@@ -192,16 +216,23 @@ func (p *progress) draw() {
|
||||
}
|
||||
|
||||
p.bar.Describe(text)
|
||||
|
||||
// The library drew the text and its margin, less the terminal's last
|
||||
// column, which the margin keeps clear.
|
||||
p.widest = max(p.widest, len(text)+margin-1)
|
||||
}
|
||||
|
||||
// newBar has the library make what draw draws: a bar for the phase's
|
||||
// total when withBar is set, and a spinner when not. Either shows the
|
||||
// text draw gives it, and nothing of its own besides the spinner, or
|
||||
// the bar and its percent. p.mu must be held.
|
||||
func (p *progress) newBar(withBar bool) {
|
||||
func (p *progress) newBar(withBar bool, width int) {
|
||||
if p.bar != nil {
|
||||
// Erase what was drawn before.
|
||||
_ = p.bar.Clear()
|
||||
// Erase what was drawn before, short of the last column of a
|
||||
// terminal that may have been made narrower since: the library's
|
||||
// own erase is as wide as the widest line it drew.
|
||||
blank := strings.Repeat(" ", min(p.widest, width-1))
|
||||
fmt.Fprint(os.Stderr, "\r"+blank+"\r")
|
||||
}
|
||||
|
||||
total := int64(-1) // the library's spinner
|
||||
@@ -220,7 +251,7 @@ func (p *progress) newBar(withBar bool) {
|
||||
progressbar.OptionSetElapsedTime(false),
|
||||
progressbar.OptionSetPredictTime(false),
|
||||
)
|
||||
p.withBar = withBar
|
||||
p.withBar, p.widest = withBar, 0
|
||||
}
|
||||
|
||||
// setTotal gives a phase that started without a total its total, once
|
||||
@@ -273,6 +304,73 @@ func (p *progress) warnf(format string, args ...any) {
|
||||
fmt.Fprintln(os.Stderr, msg)
|
||||
}
|
||||
|
||||
// progressWriter writes a phase's output to out, which may be the file
|
||||
// the phase's display is drawn on, as stdout is in a plain "sfdupes
|
||||
// trees" on a terminal. It holds what is written to it and writes that
|
||||
// to out in pieces of about ioBufSize bytes. When out is the file
|
||||
// stderr is, each piece is written under the display's lock with the
|
||||
// display cleared first, as warnf does for a warning: no draw lands in
|
||||
// the middle of a piece, and the next draw shows the display again, on
|
||||
// the line below. Every write to it must end with a newline, as a row
|
||||
// written with one fmt.Fprintf does, so that each piece ends a line.
|
||||
// Flush writes what it holds once the last row is written.
|
||||
type progressWriter struct {
|
||||
// p is the display to clear, nil unless out is the file stderr is.
|
||||
p *progress
|
||||
out io.Writer
|
||||
buf []byte
|
||||
}
|
||||
|
||||
// writer returns a progressWriter writing to out around p's display.
|
||||
// It clears the display and takes its lock only when out is the file
|
||||
// stderr is: the same device and inode, as os.SameFile compares them,
|
||||
// which is one terminal, or one pipe or file that both are sent to.
|
||||
// Otherwise it writes to out without the lock, so the display keeps
|
||||
// being drawn while a write to out waits for a slow reader.
|
||||
func (p *progress) writer(out io.Writer) *progressWriter {
|
||||
w := &progressWriter{out: out}
|
||||
|
||||
if f, ok := out.(*os.File); ok {
|
||||
outInfo, outErr := f.Stat()
|
||||
errInfo, errErr := os.Stderr.Stat()
|
||||
|
||||
if outErr == nil && errErr == nil && os.SameFile(outInfo, errInfo) {
|
||||
w.p = p
|
||||
}
|
||||
}
|
||||
|
||||
return w
|
||||
}
|
||||
|
||||
// Write holds b, and writes what it holds to out once that comes to
|
||||
// ioBufSize bytes.
|
||||
func (w *progressWriter) Write(b []byte) (int, error) {
|
||||
w.buf = append(w.buf, b...)
|
||||
if len(w.buf) < ioBufSize {
|
||||
return len(b), nil
|
||||
}
|
||||
|
||||
return len(b), w.Flush()
|
||||
}
|
||||
|
||||
// Flush writes what w holds to out. When out is the file stderr is, the
|
||||
// display is cleared first and not drawn again until it is written.
|
||||
func (w *progressWriter) Flush() error {
|
||||
if w.p != nil {
|
||||
w.p.mu.Lock()
|
||||
defer w.p.mu.Unlock()
|
||||
|
||||
if w.p.bar != nil {
|
||||
_ = w.p.bar.Clear()
|
||||
}
|
||||
}
|
||||
|
||||
_, err := w.out.Write(w.buf)
|
||||
w.buf = w.buf[:0]
|
||||
|
||||
return err
|
||||
}
|
||||
|
||||
// finish ends the phase's display, drawn once more as the phase ends.
|
||||
// A phase stopped short of its total, as an interrupted one is, shows
|
||||
// the count it stopped at rather than a full bar.
|
||||
@@ -297,9 +395,9 @@ func (p *progress) finish() {
|
||||
//
|
||||
// [3/5] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s
|
||||
//
|
||||
// without the percent when percent is false, as before the library's
|
||||
// bar, which shows its own. A phase with no total shows the items done
|
||||
// instead, or what it waits for.
|
||||
// where "files" is the phase's unit, without the percent when percent
|
||||
// is false, as before the library's bar, which shows its own. A phase
|
||||
// with no total shows the items done instead, or what it waits for.
|
||||
//
|
||||
// The phase and its place, the progress, the elapsed time and the ETA
|
||||
// are kept whole. A line wider than room leaves out the rate, or what
|
||||
@@ -323,7 +421,7 @@ func (p *progress) line(room int, percent bool) (string, bool) {
|
||||
}
|
||||
|
||||
head = p.label + ": " + p.itemsDone(percent) + ", "
|
||||
extra = fmt.Sprintf("%.0f files/s", rate)
|
||||
extra = fmt.Sprintf("%.0f %s/s", rate, p.unit)
|
||||
tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(rate))
|
||||
}
|
||||
|
||||
@@ -341,11 +439,11 @@ func (p *progress) line(room int, percent bool) (string, bool) {
|
||||
|
||||
// itemsDone formats the items done, as "[1234/5000] 25%", with "~"
|
||||
// before a total that is an estimate and without the percent when
|
||||
// percent is false, or as "1234 files" when the phase has no total.
|
||||
// p.mu must be held.
|
||||
// percent is false, or as "1234 files", in the phase's unit, when the
|
||||
// phase has no total. p.mu must be held.
|
||||
func (p *progress) itemsDone(percent bool) string {
|
||||
if p.total < 0 {
|
||||
return fmt.Sprintf("%d files", p.count)
|
||||
return fmt.Sprintf("%d %s", p.count, p.unit)
|
||||
}
|
||||
|
||||
approx := ""
|
||||
|
||||
+239
-19
@@ -1,6 +1,8 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"math"
|
||||
"os"
|
||||
"path/filepath"
|
||||
@@ -82,34 +84,48 @@ func TestProgressLine(t *testing.T) {
|
||||
}{{
|
||||
p: &progress{
|
||||
label: scanPhase("load"), total: -1,
|
||||
waitingFor: "counting the records",
|
||||
waitingFor: countingRecords,
|
||||
},
|
||||
want: "[1/5] load: counting the records, elapsed 10s, eta unknown",
|
||||
}, {
|
||||
p: &progress{label: scanPhase("hash"), total: 100, count: 20},
|
||||
p: &progress{
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
|
||||
},
|
||||
want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
|
||||
}, {
|
||||
p: &progress{label: scanPhase("hash"), total: 100},
|
||||
p: &progress{label: scanPhase("hash"), unit: scanUnit, total: 100},
|
||||
want: "[3/5] hash: [0/100] 0%, 0 files/s, elapsed 10s, eta unknown",
|
||||
}, {
|
||||
p: &progress{label: scanPhase("update"), total: 0},
|
||||
p: &progress{label: scanPhase("update"), unit: scanUnit, total: 0},
|
||||
want: "[4/5] update: [0/0] 100%, 0 files/s, elapsed 10s, eta 0s",
|
||||
}, {
|
||||
p: &progress{
|
||||
label: scanPhase("walk"), total: 100, estimate: true, count: 20,
|
||||
label: scanPhase("walk"), unit: scanUnit, total: 100,
|
||||
estimate: true, count: 20,
|
||||
},
|
||||
want: "[2/5] walk: [20/~100] 20%, 2 files/s, elapsed 10s, eta 40s",
|
||||
}, {
|
||||
// Past the estimate the walk cannot tell how long it has left.
|
||||
p: &progress{
|
||||
label: scanPhase("walk"), total: 100, estimate: true, count: 120,
|
||||
label: scanPhase("walk"), unit: scanUnit, total: 100,
|
||||
estimate: true, count: 120,
|
||||
},
|
||||
want: "[2/5] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " +
|
||||
"eta unknown",
|
||||
}, {
|
||||
// A first scan's walk has no estimate.
|
||||
p: &progress{label: scanPhase("walk"), total: -1, count: 20},
|
||||
p: &progress{
|
||||
label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20,
|
||||
},
|
||||
want: "[2/5] walk: 20 files, 2 files/s, elapsed 10s, eta unknown",
|
||||
}, {
|
||||
// The rate is in what the phase counts.
|
||||
p: &progress{
|
||||
label: treesPhase("group"), unit: "directories", total: 100,
|
||||
count: 20,
|
||||
},
|
||||
want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " +
|
||||
"eta 40s",
|
||||
}}
|
||||
|
||||
for _, tc := range cases {
|
||||
@@ -129,7 +145,7 @@ func TestLineFitsRoom(t *testing.T) {
|
||||
|
||||
ago := time.Now().Add(-10 * time.Second)
|
||||
hash := &progress{
|
||||
label: scanPhase("hash"), total: 100, count: 20,
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
|
||||
start: ago, rateStart: ago,
|
||||
}
|
||||
content := &progress{
|
||||
@@ -183,7 +199,8 @@ func TestSetTotalStartsRate(t *testing.T) {
|
||||
|
||||
waited := time.Now().Add(-20 * time.Second)
|
||||
p := &progress{
|
||||
label: scanPhase("load"), total: -1, start: waited, rateStart: waited,
|
||||
label: scanPhase("load"), unit: scanUnit, total: -1, start: waited,
|
||||
rateStart: waited,
|
||||
}
|
||||
|
||||
p.setTotal(100)
|
||||
@@ -226,7 +243,9 @@ func awaitOutput(t *testing.T, stderr func() string, want string,
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestPlainProgressPrintsAtStartAndEnd(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
p := newProgress(progressParams{label: scanPhase("hash"), total: 1000})
|
||||
p := newProgress(progressParams{
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 1000,
|
||||
})
|
||||
|
||||
for range 1000 {
|
||||
p.increment()
|
||||
@@ -252,7 +271,7 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
p := startProgress(progressParams{
|
||||
label: scanPhase("load"), total: -1,
|
||||
waitingFor: "counting the records",
|
||||
waitingFor: countingRecords,
|
||||
}, nil, testRedraw)
|
||||
|
||||
awaitOutput(t, stderr,
|
||||
@@ -289,8 +308,14 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) {
|
||||
params progressParams
|
||||
want string // how the line shows a count of 50
|
||||
}{
|
||||
{progressParams{label: scanPhase("walk"), total: -1}, "50 files,"},
|
||||
{progressParams{label: scanPhase("hash"), total: 100}, "[50/100],"},
|
||||
{
|
||||
progressParams{label: scanPhase("walk"), unit: scanUnit, total: -1},
|
||||
"50 files,",
|
||||
},
|
||||
{
|
||||
progressParams{label: scanPhase("hash"), unit: scanUnit, total: 100},
|
||||
"[50/100],",
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
@@ -341,8 +366,9 @@ func drawLate(stderr func() string, params progressParams, width int,
|
||||
}
|
||||
|
||||
// TestTerminalLinesKeepETA checks the line each display shows on an
|
||||
// 80-column terminal at a large scan's counts and times: the rate is
|
||||
// left out, so that the elapsed time and the ETA stay whole.
|
||||
// 80-column terminal at a large scan's counts and times, and the line
|
||||
// of trees' read phase at the same counts: the rate is left out, so
|
||||
// that the elapsed time and the ETA stay whole.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestTerminalLinesKeepETA(t *testing.T) {
|
||||
@@ -354,18 +380,29 @@ func TestTerminalLinesKeepETA(t *testing.T) {
|
||||
}{{
|
||||
// Past its estimate, the walk cannot tell how long it has left.
|
||||
params: progressParams{
|
||||
label: scanPhase("walk"), total: 1200000, estimate: true,
|
||||
label: scanPhase("walk"), unit: scanUnit, total: 1200000,
|
||||
estimate: true,
|
||||
},
|
||||
want: "[2/5] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " +
|
||||
"eta unknown",
|
||||
}, {
|
||||
params: progressParams{label: scanPhase("hash"), total: 9876543},
|
||||
params: progressParams{
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
|
||||
},
|
||||
want: "[3/5] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " +
|
||||
"eta 17h30m0s",
|
||||
}, {
|
||||
params: progressParams{label: scanPhase("content"), total: 9876543},
|
||||
params: progressParams{
|
||||
label: scanPhase("content"), unit: scanUnit, total: 9876543,
|
||||
},
|
||||
want: "[5/5] content: [1234567/9876543] 12%, elapsed 2h30m0s, " +
|
||||
"eta 17h30m0s",
|
||||
}, {
|
||||
params: progressParams{
|
||||
label: treesPhase("read"), unit: "records", total: 9876543,
|
||||
},
|
||||
want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " +
|
||||
"eta 17h30m0s",
|
||||
}}
|
||||
|
||||
for _, tc := range cases {
|
||||
@@ -389,7 +426,9 @@ func TestTerminalLinesFitWidth(t *testing.T) {
|
||||
|
||||
for _, width := range []int{120, 100, 80, 72, 60, 50, 40} {
|
||||
for _, total := range []int64{9876543, 20000} {
|
||||
params := progressParams{label: scanPhase("content"), total: total}
|
||||
params := progressParams{
|
||||
label: scanPhase("content"), unit: scanUnit, total: total,
|
||||
}
|
||||
out := drawLate(stderr, params, width, total/8)
|
||||
|
||||
if width == 120 && !strings.Contains(out, "% |") {
|
||||
@@ -410,6 +449,68 @@ func TestTerminalLinesFitWidth(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestTerminalNarrowedRedrawsInPlace draws a large scan's display, two
|
||||
// and a half hours into the phase, on a terminal 120 columns wide, then
|
||||
// makes the terminal narrower than that line and draws it again a few
|
||||
// times. Nothing written after that, the erase before each redraw
|
||||
// included, may reach the narrower terminal's last column: it would
|
||||
// wrap, and every redraw would leave a line behind. At these widths the
|
||||
// spinner stays a spinner and the bar stays a bar.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
|
||||
cases := []struct {
|
||||
params progressParams
|
||||
narrow int
|
||||
}{
|
||||
{
|
||||
progressParams{label: scanPhase("walk"), unit: scanUnit, total: -1},
|
||||
60,
|
||||
},
|
||||
{
|
||||
progressParams{
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
|
||||
},
|
||||
90,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
width := 120
|
||||
|
||||
// No timed draws: the test draws.
|
||||
p := startProgress(tc.params, func() int { return width }, time.Hour)
|
||||
|
||||
p.mu.Lock()
|
||||
p.count = 1234567
|
||||
p.start = p.start.Add(-150 * time.Minute)
|
||||
p.rateStart = p.start
|
||||
p.mu.Unlock()
|
||||
|
||||
p.draw()
|
||||
|
||||
width = tc.narrow
|
||||
before := len(stderr())
|
||||
|
||||
p.draw()
|
||||
p.draw()
|
||||
p.finish()
|
||||
|
||||
lines := strings.FieldsFuncSeq(stderr()[before:], func(r rune) bool {
|
||||
return r == '\r' || r == '\n'
|
||||
})
|
||||
|
||||
for line := range lines {
|
||||
if n := utf8.RuneCountInString(line); n >= width {
|
||||
t.Errorf("%d columns written on a terminal made %d wide: %q",
|
||||
n, width, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestProgressWarningsOnOwnLines drives the terminal displays, spinner
|
||||
// and bar, through a run of warnings with no items between them, as when
|
||||
// the walk meets many unreadable paths, for many redraws: every warning
|
||||
@@ -459,6 +560,125 @@ func TestProgressWarningsOnOwnLines(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestProgressWriterRowsOnOwnLines drives the terminal displays,
|
||||
// spinner and bar, through a run of rows written through the display's
|
||||
// writer to the same terminal, as stdout is in a plain "sfdupes trees",
|
||||
// each written out at once, for many redraws: the terminal must show
|
||||
// every row whole, in order, on a line of its own.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestProgressWriterRowsOnOwnLines(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
|
||||
cases := []progressParams{
|
||||
{
|
||||
label: treesPhase("open"), total: -1,
|
||||
waitingFor: "opening the database",
|
||||
},
|
||||
{label: treesPhase("write"), unit: "groups", total: 10},
|
||||
}
|
||||
|
||||
var want []string
|
||||
|
||||
for _, params := range cases {
|
||||
p := startProgress(params, terminalWidth, testRedraw)
|
||||
out := p.writer(os.Stderr)
|
||||
|
||||
// No pause between rows: one is garbled only if a redraw lands
|
||||
// on its line.
|
||||
for start := time.Now(); time.Since(start) < 50*testRedraw; {
|
||||
row := fmt.Sprintf("row %d", len(want))
|
||||
want = append(want, row)
|
||||
|
||||
_, err := fmt.Fprintln(out, row)
|
||||
if err == nil {
|
||||
err = out.Flush()
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
p.finish()
|
||||
}
|
||||
|
||||
// A terminal shows each line as the text after its last carriage
|
||||
// return.
|
||||
var shown []string
|
||||
|
||||
for line := range strings.SplitSeq(stderr(), "\n") {
|
||||
if strings.Contains(line, "row ") {
|
||||
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
|
||||
}
|
||||
}
|
||||
|
||||
if len(shown) != len(want) {
|
||||
t.Fatalf("terminal shows %d rows, want %d", len(shown), len(want))
|
||||
}
|
||||
|
||||
for i := range want {
|
||||
if shown[i] != want[i] {
|
||||
t.Fatalf("terminal shows %q, want %q", shown[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestProgressWriterStalledStdoutKeepsLines writes a row larger than a
|
||||
// pipe holds through the display's writer to a pipe nobody reads yet,
|
||||
// as stdout is when its reader has stopped while stderr goes to a file:
|
||||
// while the write waits, the display must go on printing its line.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestProgressWriterStalledStdoutKeepsLines(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
|
||||
r, w, err := os.Pipe()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() {
|
||||
_ = w.Close()
|
||||
_ = r.Close()
|
||||
})
|
||||
|
||||
p := startProgress(progressParams{label: treesPhase("write"), total: 1},
|
||||
nil, testRedraw)
|
||||
out := p.writer(w)
|
||||
written := make(chan error, 1)
|
||||
|
||||
go func() {
|
||||
_, writeErr := fmt.Fprintln(out, strings.Repeat("x", ioBufSize))
|
||||
written <- writeErr
|
||||
}()
|
||||
|
||||
// Pass or fail, the pipe is then read so that the write ends, and
|
||||
// the display ends before stderr is put back.
|
||||
t.Cleanup(func() {
|
||||
go func() {
|
||||
_, _ = io.Copy(io.Discard, r)
|
||||
}()
|
||||
|
||||
writeErr := <-written
|
||||
if writeErr != nil {
|
||||
t.Error(writeErr)
|
||||
}
|
||||
|
||||
p.finish()
|
||||
})
|
||||
|
||||
// The first byte read shows that the write has started; the rest
|
||||
// of the row waits until the pipe is read.
|
||||
_, err = r.Read(make([]byte, 1))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
line := "[4/4] write: "
|
||||
awaitOutput(t, stderr, line, strings.Count(stderr(), line)+3)
|
||||
}
|
||||
|
||||
// TestTerminalStoppedShortKeepsCount checks that the terminal display
|
||||
// of a phase that stops before its total, as an interrupted one does,
|
||||
// is left showing the count it stopped at rather than a full bar.
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
@@ -29,13 +30,25 @@ type scanRec struct {
|
||||
path string
|
||||
}
|
||||
|
||||
// reportPhase names one phase of report with its place among all of
|
||||
// them, as "[2/2] read", for the phase's progress display. The list
|
||||
// here is the one place the phases are counted.
|
||||
func reportPhase(name string) string {
|
||||
return phaseLabel([]string{openPhase, "read"}, name)
|
||||
}
|
||||
|
||||
// runReport implements the report subcommand: it prints the file-level
|
||||
// duplicates report as TSV on stdout. SQLite groups and orders the
|
||||
// records, and each row is written as it is read, so no group is held
|
||||
// in memory. It never touches the scanned filesystem; its only I/O is
|
||||
// the database (with SQLite's temporary sort file), stdout, and stderr.
|
||||
// It names the database on stderr before it opens it. Any database
|
||||
// duplicates report as TSV on stdout. It never touches the scanned
|
||||
// filesystem; its only I/O is the database, stdout, and stderr. It
|
||||
// names the database on stderr before it opens it. Any database
|
||||
// problem, including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in two phases, each under a progress display of its
|
||||
// own on stderr: open (open the database) and read (count the records,
|
||||
// then read them in signature order, writing each size's duplicate
|
||||
// groups to stdout once that size is read; see readReport). Each
|
||||
// display ends before the next phase starts, and before an error is
|
||||
// returned to be printed.
|
||||
func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
@@ -45,65 +58,194 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
fmt.Fprintf(os.Stderr, "report: database %s (%s)\n",
|
||||
escapePath(dbPath), source)
|
||||
|
||||
// Opening can wait on a busy database for as long as the busy
|
||||
// timeout.
|
||||
prog := newProgress(progressParams{
|
||||
label: reportPhase(openPhase), total: -1,
|
||||
waitingFor: openingDatabase,
|
||||
})
|
||||
db, err := openReportDatabase(ctx, dbPath)
|
||||
|
||||
prog.finish()
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
out := bufio.NewWriterSize(stdout, ioBufSize)
|
||||
prog = newProgress(progressParams{
|
||||
label: reportPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords,
|
||||
})
|
||||
report, err := readReport(ctx, db, dbPath, stdout, prog)
|
||||
|
||||
_, err = fmt.Fprintln(out, "first\tdupe\tsize")
|
||||
if err != nil {
|
||||
return fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
var (
|
||||
groups, dupeFiles int
|
||||
reclaimable int64
|
||||
writeErr error
|
||||
)
|
||||
|
||||
records, err := loadDupeRows(ctx, db,
|
||||
func(first, path string, size int64) error {
|
||||
// A group's first path is its first row; every other
|
||||
// path is a dupe.
|
||||
if path == first {
|
||||
groups++
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
_, writeErr = fmt.Fprintf(out, "%s\t%s\t%d\n",
|
||||
escapePath(first), escapePath(path), size)
|
||||
dupeFiles++
|
||||
reclaimable += size
|
||||
|
||||
return writeErr
|
||||
})
|
||||
|
||||
if writeErr != nil {
|
||||
return fmt.Errorf("write stdout: %w", writeErr)
|
||||
}
|
||||
prog.finish()
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("database %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
err = out.Flush()
|
||||
if err != nil {
|
||||
return fmt.Errorf("write stdout: %w", err)
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr,
|
||||
"report: %d records read, %d duplicate groups, %d dupe files, "+
|
||||
"%s reclaimable\n",
|
||||
records, groups, dupeFiles, humanBytes(reclaimable))
|
||||
report.records, report.dupeGroups, report.dupeFiles,
|
||||
humanBytes(report.reclaimable))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readReport gives prog the number of records in db, the database at
|
||||
// dbPath, as its total, then reads every record in signature order,
|
||||
// counting each on prog, and writes the report to stdout: the header
|
||||
// line, then the rows of each size's duplicate groups once that size is
|
||||
// read. stdout may be the terminal prog is drawn on, so the lines go
|
||||
// through prog's writer. It returns what it wrote, with the totals for
|
||||
// the summary line.
|
||||
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
stdout io.Writer, prog *progress,
|
||||
) (*reportWriter, error) {
|
||||
out := prog.writer(stdout)
|
||||
report := &reportWriter{out: out}
|
||||
|
||||
_, err := fmt.Fprintln(out, "first\tdupe\tsize")
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
var writeErr error
|
||||
|
||||
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
|
||||
prog.increment()
|
||||
|
||||
writeErr = report.add(r)
|
||||
|
||||
return writeErr
|
||||
})
|
||||
|
||||
if writeErr != nil {
|
||||
return nil, fmt.Errorf("write stdout: %w", writeErr)
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("database %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
err = report.finish()
|
||||
if err == nil {
|
||||
err = out.Flush()
|
||||
}
|
||||
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
return report, nil
|
||||
}
|
||||
|
||||
// reportWriter writes the report's rows to out from the records, added
|
||||
// in signature order, largest size first, as loadSignatureRows reads
|
||||
// them. The records sharing a signature come together, and so do the
|
||||
// records of one size, so it holds only the duplicate groups of the
|
||||
// size being read, and writes them, ordered by first path, once a
|
||||
// record of a smaller size comes. It counts what the summary line
|
||||
// reports.
|
||||
type reportWriter struct {
|
||||
out io.Writer
|
||||
|
||||
sig scanRec // the signature being read; its path is not used
|
||||
paths []string // the paths read so far with sig
|
||||
groups [][]string // the duplicate groups of sig.size, each sorted
|
||||
|
||||
records, dupeGroups, dupeFiles int
|
||||
reclaimable int64
|
||||
}
|
||||
|
||||
// add takes the next record. One with the signature being read joins
|
||||
// its paths; any other ends them, and one of a smaller size first
|
||||
// writes the groups of the size before.
|
||||
func (w *reportWriter) add(r scanRec) error {
|
||||
w.records++
|
||||
|
||||
// A record without a content hash has unknown content (README
|
||||
// "Database"), so it is never a duplicate.
|
||||
if r.content == "" {
|
||||
return nil
|
||||
}
|
||||
|
||||
if r.size == w.sig.size && r.head == w.sig.head &&
|
||||
r.tail == w.sig.tail && r.content == w.sig.content {
|
||||
w.paths = append(w.paths, r.path)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
w.endGroup()
|
||||
|
||||
if r.size != w.sig.size {
|
||||
err := w.writeGroups()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
w.sig = r
|
||||
w.paths = append(w.paths, r.path)
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// endGroup ends the paths read with w.sig: two or more are a duplicate
|
||||
// group, held with its paths sorted until writeGroups writes it.
|
||||
func (w *reportWriter) endGroup() {
|
||||
if len(w.paths) < minGroupSize {
|
||||
// A path with no duplicate is dropped, and its slice reused.
|
||||
w.paths = w.paths[:0]
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
slices.Sort(w.paths)
|
||||
w.groups = append(w.groups, w.paths)
|
||||
w.paths = nil
|
||||
}
|
||||
|
||||
// writeGroups writes the duplicate groups held, all of size w.sig.size,
|
||||
// ordered by first path: for each group, a row for every path but the
|
||||
// first. No two groups share a first path, since no two records share
|
||||
// a path.
|
||||
func (w *reportWriter) writeGroups() error {
|
||||
slices.SortFunc(w.groups, func(a, b []string) int {
|
||||
return strings.Compare(a[0], b[0])
|
||||
})
|
||||
|
||||
for _, g := range w.groups {
|
||||
for _, path := range g[1:] {
|
||||
_, err := fmt.Fprintf(w.out, "%s\t%s\t%d\n",
|
||||
escapePath(g[0]), escapePath(path), w.sig.size)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
w.dupeFiles++
|
||||
w.reclaimable += w.sig.size
|
||||
}
|
||||
|
||||
w.dupeGroups++
|
||||
}
|
||||
|
||||
w.groups = nil
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// finish writes the groups of the last size read.
|
||||
func (w *reportWriter) finish() error {
|
||||
w.endGroup()
|
||||
|
||||
return w.writeGroups()
|
||||
}
|
||||
|
||||
// escapePath returns a path as it is written in a report column (README
|
||||
// "Report output format"): a backslash, tab, newline or carriage return
|
||||
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
||||
|
||||
+190
-25
@@ -9,6 +9,7 @@ import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -28,7 +29,8 @@ func awkwardPairRecs() []scanRec {
|
||||
}
|
||||
}
|
||||
|
||||
// seedDatabase writes recs into a fresh database and returns its path.
|
||||
// seedDatabase writes recs into a fresh database, closes it as scan
|
||||
// does, out of WAL mode, and returns its path.
|
||||
func seedDatabase(t *testing.T, recs []scanRec) string {
|
||||
t.Helper()
|
||||
|
||||
@@ -44,10 +46,7 @@ func seedDatabase(t *testing.T, recs []scanRec) string {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
err = db.Close()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
closeScanDatabase(t.Context(), db, path)
|
||||
|
||||
return path
|
||||
}
|
||||
@@ -60,27 +59,41 @@ type dupeGroup struct {
|
||||
}
|
||||
|
||||
// dupeGroups returns the duplicate groups report reads from db, in
|
||||
// report order.
|
||||
// report order, from the rows it writes: a group's rows come together,
|
||||
// each naming the group's first path.
|
||||
func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
|
||||
t.Helper()
|
||||
|
||||
var groups []dupeGroup
|
||||
var out bytes.Buffer
|
||||
|
||||
_, err := loadDupeRows(t.Context(), db,
|
||||
func(first, path string, size int64) error {
|
||||
if path == first {
|
||||
groups = append(groups, dupeGroup{size: size})
|
||||
}
|
||||
|
||||
g := &groups[len(groups)-1]
|
||||
g.paths = append(g.paths, path)
|
||||
|
||||
return nil
|
||||
})
|
||||
_, err := readReport(t.Context(), db, "", &out, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var groups []dupeGroup
|
||||
|
||||
rows := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n")
|
||||
for _, row := range rows[1:] { // after the header
|
||||
fields := strings.Split(row, "\t")
|
||||
if len(fields) != 3 {
|
||||
t.Fatalf("row %q, want first, dupe and size", row)
|
||||
}
|
||||
|
||||
size, err := strconv.ParseInt(fields[2], 10, 64)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if len(groups) == 0 || groups[len(groups)-1].paths[0] != fields[0] {
|
||||
groups = append(groups,
|
||||
dupeGroup{size: size, paths: []string{fields[0]}})
|
||||
}
|
||||
|
||||
g := &groups[len(groups)-1]
|
||||
g.paths = append(g.paths, fields[1])
|
||||
}
|
||||
|
||||
return groups
|
||||
}
|
||||
|
||||
@@ -118,26 +131,147 @@ func TestRunReportEscapesPaths(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestReportStdoutFailsWhileReading(t *testing.T) {
|
||||
// Each row holds two paths longer than dir, so the report is more
|
||||
// than twice the stdout buffer and stdout fails while rows are
|
||||
// still being read, not at the final flush.
|
||||
t.Parallel()
|
||||
|
||||
// Every two records are a duplicate pair of a size of their own, and
|
||||
// a pair's row is written once a smaller size is read. Each row
|
||||
// holds two paths longer than dir, so the rows come to more than
|
||||
// twice the stdout buffer, and stdout fails while records are still
|
||||
// being read, not once every record is.
|
||||
dir := "/" + strings.Repeat("d", 4096)
|
||||
|
||||
recs := make([]scanRec, ioBufSize/len(dir))
|
||||
recs := make([]scanRec, 2*ioBufSize/len(dir))
|
||||
for i := range recs {
|
||||
recs[i] = scanRec{
|
||||
size: 1, head: "h", tail: "t", content: "c",
|
||||
size: int64(i/2) + 1, head: "h", tail: "t", content: "c",
|
||||
path: fmt.Sprintf("%s/%d", dir, i),
|
||||
}
|
||||
}
|
||||
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
db := openTestDB(t)
|
||||
|
||||
err := runReport(t.Context(), failingWriter{})
|
||||
err := applyChanges(t.Context(), db, recs, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Counts the records read, and draws nothing.
|
||||
prog := &progress{}
|
||||
|
||||
_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
|
||||
if !errors.Is(err, errWriteFailed) ||
|
||||
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
||||
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
|
||||
}
|
||||
|
||||
if prog.count >= int64(len(recs)) {
|
||||
t.Errorf("stdout failed after all %d records were read, want it "+
|
||||
"to stop the read", prog.count)
|
||||
}
|
||||
}
|
||||
|
||||
// reportOpenLine is the plain line of report's open phase when it opens
|
||||
// the database at once; it prints it as the phase starts and as it ends.
|
||||
const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
|
||||
"eta unknown"
|
||||
|
||||
// smokeReport is report's stdout for the smoke-test records.
|
||||
const smokeReport = "first\tdupe\tsize\n" +
|
||||
"/d/t1/f1\t/d/t2/f1\t3000\n" +
|
||||
"/d/t1/f1\t/d/t3/f1\t3000\n" +
|
||||
"/d/t1/sub/f2\t/d/t2/sub/f2\t100\n" +
|
||||
"/d/t1/sub/f2\t/d/t3/sub/f2renamed\t100\n"
|
||||
|
||||
// TestRunReportProgressNamesEachPhase runs report on the smoke-test
|
||||
// records with stderr not a terminal, and checks every line it prints
|
||||
// there. After the line naming the database, each phase prints a line
|
||||
// naming it with its place among the two as it starts and another as
|
||||
// it ends; read counts the 6 records before it reads them. The summary
|
||||
// comes last, and stdout holds the report alone.
|
||||
func TestRunReportProgressNamesEachPhase(t *testing.T) {
|
||||
path := seedDatabase(t, smokeTreeRecs())
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
|
||||
code := run([]string{cmdReport}, &stdout, os.Stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(report) = %d, want %d; stderr: %s",
|
||||
code, exitOK, stderr())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != smokeReport {
|
||||
t.Errorf("stdout = %q, want %q", got, smokeReport)
|
||||
}
|
||||
|
||||
// How each line starts; the rate at the end of a phase varies.
|
||||
want := []string{
|
||||
"report: database " + path + " (from " + databaseEnv + ")",
|
||||
reportOpenLine,
|
||||
reportOpenLine,
|
||||
"[2/2] read: counting the records, elapsed 0s, eta unknown",
|
||||
"[2/2] read: [6/6] 100%, ",
|
||||
"report: 6 records read, 2 duplicate groups, 4 dupe files, " +
|
||||
"6.1 KiB reclaimable",
|
||||
}
|
||||
|
||||
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
|
||||
if len(lines) != len(want) {
|
||||
t.Fatalf("stderr lines = %q, want %d lines starting %q",
|
||||
lines, len(want), want)
|
||||
}
|
||||
|
||||
for i := range want {
|
||||
if !strings.HasPrefix(lines[i], want[i]) {
|
||||
t.Errorf("stderr line %d = %q, want it to start %q",
|
||||
i+1, lines[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadReportOnTerminal writes the smoke-test report to the terminal
|
||||
// its read phase's display is drawn on, as a plain "sfdupes report"
|
||||
// does: the terminal must show the header and every row whole, each on
|
||||
// a line of its own.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestReadReportOnTerminal(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
db := openTestDB(t)
|
||||
|
||||
err := applyChanges(t.Context(), db, smokeTreeRecs(), nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// No timed draws: the display is drawn as it starts and as it ends.
|
||||
p := startProgress(progressParams{
|
||||
label: reportPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords,
|
||||
}, terminalWidth, time.Hour)
|
||||
|
||||
_, err = readReport(t.Context(), db, "", os.Stderr, p)
|
||||
|
||||
p.finish()
|
||||
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// A terminal shows each line as the text after its last carriage
|
||||
// return.
|
||||
var shown []string
|
||||
|
||||
for line := range strings.SplitSeq(stderr(), "\n") {
|
||||
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
|
||||
}
|
||||
|
||||
want := strings.Split(strings.TrimSuffix(smokeReport, "\n"), "\n")
|
||||
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
|
||||
t.Errorf("terminal shows lines %q, want %q first", shown, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
|
||||
@@ -354,6 +488,37 @@ func TestDupeGroupsTieBreak(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestDupeGroupsOrderedByFirstPath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// report reads the groups of size 50 in signature order, here /a,
|
||||
// /z, /k, which is neither the order of their first paths nor its
|
||||
// reverse, and must write them ordered by first path. The group of
|
||||
// size 7 comes after them all, though its first path sorts first.
|
||||
recs := []scanRec{
|
||||
{size: 50, head: "a", tail: "a", content: "a", path: "/k/2"},
|
||||
{size: 50, head: "a", tail: "a", content: "a", path: "/k/1"},
|
||||
{size: 50, head: "b", tail: "b", content: "b", path: "/z/1"},
|
||||
{size: 50, head: "b", tail: "b", content: "b", path: "/z/2"},
|
||||
{size: 50, head: "c", tail: "c", content: "c", path: "/a/2"},
|
||||
{size: 50, head: "c", tail: "c", content: "c", path: "/a/1"},
|
||||
{size: 7, head: "d", tail: "d", content: "d", path: "/0/2"},
|
||||
{size: 7, head: "d", tail: "d", content: "d", path: "/0/1"},
|
||||
}
|
||||
|
||||
groups := dupeGroupsOf(t, recs)
|
||||
|
||||
got := make([]string, 0, len(groups))
|
||||
for _, g := range groups {
|
||||
got = append(got, g.paths[0])
|
||||
}
|
||||
|
||||
want := []string{"/a/1", "/k/1", "/z/1", "/0/1"}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("groups' first paths = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDupeGroupsDeterministic(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -84,6 +84,10 @@ type fileMeta struct {
|
||||
hashed bool
|
||||
}
|
||||
|
||||
// scanUnit is what every scan phase counts, as its progress display
|
||||
// names it.
|
||||
const scanUnit = "files"
|
||||
|
||||
// scanPhase names one phase of a scan with its place among all of them,
|
||||
// as "[2/5] walk", for the phase's progress display. The list here is
|
||||
// the one place the phases are counted.
|
||||
@@ -441,8 +445,8 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||
// display the scan looks hung before the walk begins. Counting them
|
||||
// first gives the display its total.
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("load"), total: -1,
|
||||
waitingFor: "counting the records",
|
||||
label: scanPhase("load"), unit: scanUnit, total: -1,
|
||||
waitingFor: countingRecords,
|
||||
})
|
||||
defer prog.finish()
|
||||
|
||||
@@ -492,7 +496,8 @@ func (s *scanState) walkPhase(
|
||||
}
|
||||
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("walk"), total: estimate, estimate: true,
|
||||
label: scanPhase("walk"), unit: scanUnit, total: estimate,
|
||||
estimate: true,
|
||||
})
|
||||
|
||||
var (
|
||||
@@ -683,7 +688,7 @@ func (s *scanState) readRuns(ctx context.Context, workers int,
|
||||
defer pool.stop()
|
||||
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase(label), total: int64(len(runs)),
|
||||
label: scanPhase(label), unit: scanUnit, total: int64(len(runs)),
|
||||
})
|
||||
defer prog.finish()
|
||||
|
||||
@@ -772,7 +777,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
||||
|
||||
total := len(s.batch) + len(s.sentinels) + len(deletes)
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("update"), total: int64(total),
|
||||
label: scanPhase("update"), unit: scanUnit, total: int64(total),
|
||||
})
|
||||
|
||||
defer prog.finish()
|
||||
@@ -860,7 +865,7 @@ func (s *scanState) contentCandidates(
|
||||
// record. The checks start there, against a count of the records
|
||||
// the search returns, and so do their rate and ETA.
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("content"), total: -1,
|
||||
label: scanPhase("content"), unit: scanUnit, total: -1,
|
||||
waitingFor: "searching the database",
|
||||
})
|
||||
defer prog.finish()
|
||||
|
||||
+3
-3
@@ -1478,7 +1478,7 @@ func dbRecords(t *testing.T, db *sql.DB) []scanRec {
|
||||
|
||||
var recs []scanRec
|
||||
|
||||
err := loadFileRows(t.Context(), db, func(r scanRec) {
|
||||
err := loadFileRows(t.Context(), db, func(int64) {}, func(r scanRec) {
|
||||
recs = append(recs, r)
|
||||
})
|
||||
if err != nil {
|
||||
@@ -1550,7 +1550,7 @@ func assertSmokeTreeGroups(t *testing.T, dir string, db *sql.DB) {
|
||||
|
||||
super, dirs := dbTree(t, db)
|
||||
|
||||
tg := collectTreeGroups(dirs, super)
|
||||
tg := collectTreeGroups(dirs, super, nil)
|
||||
if len(tg) != 1 {
|
||||
t.Fatalf("len(tree groups) = %d, want 1", len(tg))
|
||||
}
|
||||
@@ -2092,7 +2092,7 @@ func TestTreesUnhashedNeverEqual(t *testing.T) {
|
||||
for name, unknown := range cases {
|
||||
super, dirs := treeOf(t, append(slices.Clone(shared), unknown...))
|
||||
|
||||
if tg := collectTreeGroups(dirs, super); len(tg) != 0 {
|
||||
if tg := collectTreeGroups(dirs, super, nil); len(tg) != 0 {
|
||||
t.Errorf("%s: tree groups = %d, want 0 (the files may differ)",
|
||||
name, len(tg))
|
||||
}
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
@@ -24,6 +24,13 @@ type treeNode struct {
|
||||
totalSize int64
|
||||
}
|
||||
|
||||
// treesPhase names one phase of trees with its place among all of them,
|
||||
// as "[2/4] read", for the phase's progress display. The list here is
|
||||
// the one place the phases are counted.
|
||||
func treesPhase(name string) string {
|
||||
return phaseLabel([]string{openPhase, "read", "group", "write"}, name)
|
||||
}
|
||||
|
||||
// runTrees implements the trees subcommand: it reads every record from
|
||||
// the database in path order, reconstructs the directory hierarchy from
|
||||
// the record paths, computes a Merkle-style digest per directory, and
|
||||
@@ -31,6 +38,13 @@ type treeNode struct {
|
||||
// touches the scanned filesystem; its only I/O is the database, stdout,
|
||||
// and stderr. It names the database on stderr before it opens it. Any
|
||||
// database problem, including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in four phases, each under a progress display of its
|
||||
// own on stderr: open (open the database), read (count the records,
|
||||
// then read them into the hierarchy), group (group the directories by
|
||||
// digest) and write (write the groups to stdout). Each display ends
|
||||
// before the next phase starts, and before an error is returned to be
|
||||
// printed.
|
||||
func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
@@ -40,34 +54,99 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
fmt.Fprintf(os.Stderr, "trees: database %s (%s)\n",
|
||||
escapePath(dbPath), source)
|
||||
|
||||
// Opening can wait on a busy database for as long as the busy
|
||||
// timeout.
|
||||
prog := newProgress(progressParams{
|
||||
label: treesPhase(openPhase), total: -1,
|
||||
waitingFor: openingDatabase,
|
||||
})
|
||||
db, err := openReportDatabase(ctx, dbPath)
|
||||
|
||||
prog.finish()
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
records := 0
|
||||
tree := newTreeBuilder()
|
||||
|
||||
err = loadFileRows(ctx, db, func(r scanRec) {
|
||||
records++
|
||||
|
||||
tree.add(r)
|
||||
prog = newProgress(progressParams{
|
||||
label: treesPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords,
|
||||
})
|
||||
records, super, allDirs, err := readTree(ctx, db, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
if err != nil {
|
||||
return fmt.Errorf("database %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
prog = newProgress(progressParams{
|
||||
label: treesPhase("group"), unit: "directories",
|
||||
total: int64(len(allDirs)),
|
||||
})
|
||||
dupes := collectTreeGroups(allDirs, super, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
prog = newProgress(progressParams{
|
||||
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
|
||||
})
|
||||
dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr,
|
||||
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
|
||||
"%s reclaimable\n",
|
||||
records, len(dupes), dupeTrees, humanBytes(reclaimable))
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// readTree gives prog the number of records in db as its total, then
|
||||
// reads every record in path order into the directory hierarchy,
|
||||
// counting each on prog. It returns the number of records read, the
|
||||
// super-root and every directory.
|
||||
func readTree(ctx context.Context, db *sql.DB,
|
||||
prog *progress,
|
||||
) (int, *treeNode, []*treeNode, error) {
|
||||
records := 0
|
||||
tree := newTreeBuilder()
|
||||
|
||||
err := loadFileRows(ctx, db, prog.setTotal, func(r scanRec) {
|
||||
records++
|
||||
|
||||
prog.increment()
|
||||
tree.add(r)
|
||||
})
|
||||
if err != nil {
|
||||
return 0, nil, nil, err
|
||||
}
|
||||
|
||||
super, allDirs := tree.finish()
|
||||
|
||||
dupes := collectTreeGroups(allDirs, super)
|
||||
return records, super, allDirs, nil
|
||||
}
|
||||
|
||||
out := bufio.NewWriterSize(stdout, ioBufSize)
|
||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||
// a row for every member of each group but its first, counting each
|
||||
// group on prog once its rows are written. stdout may be the terminal
|
||||
// prog is drawn on, so the lines go through prog's writer. It returns
|
||||
// the number of rows (dupe trees) and the bytes they could reclaim.
|
||||
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||
prog *progress,
|
||||
) (int, int64, error) {
|
||||
out := prog.writer(stdout)
|
||||
|
||||
_, err = fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
|
||||
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
|
||||
if err != nil {
|
||||
return fmt.Errorf("write stdout: %w", err)
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
dupeTrees := 0
|
||||
@@ -81,25 +160,22 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
escapePath(first.path), escapePath(n.path),
|
||||
first.fileCount, first.totalSize)
|
||||
if err != nil {
|
||||
return fmt.Errorf("write stdout: %w", err)
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
dupeTrees++
|
||||
reclaimable += first.totalSize
|
||||
}
|
||||
|
||||
prog.increment()
|
||||
}
|
||||
|
||||
err = out.Flush()
|
||||
if err != nil {
|
||||
return fmt.Errorf("write stdout: %w", err)
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr,
|
||||
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
|
||||
"%s reclaimable\n",
|
||||
records, len(dupes), dupeTrees, humanBytes(reclaimable))
|
||||
|
||||
return nil
|
||||
return dupeTrees, reclaimable, nil
|
||||
}
|
||||
|
||||
// treeBuilder reconstructs the directory hierarchy from records added
|
||||
@@ -221,14 +297,18 @@ func fileEntry(name string, r scanRec) string {
|
||||
"\x00" + r.head + "\x00" + r.tail + "\x00" + content
|
||||
}
|
||||
|
||||
// collectTreeGroups groups directories by digest and returns every
|
||||
// maximal group with two or more members, each group's members sorted
|
||||
// by path, groups ordered by tree size descending then by first path
|
||||
// ascending.
|
||||
func collectTreeGroups(allDirs []*treeNode, super *treeNode) [][]*treeNode {
|
||||
// collectTreeGroups groups directories by digest, counting each one on
|
||||
// prog as it is grouped, and returns every maximal group with two or
|
||||
// more members, each group's members sorted by path, groups ordered by
|
||||
// tree size descending then by first path ascending.
|
||||
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||
prog *progress,
|
||||
) [][]*treeNode {
|
||||
groups := make(map[[sha256.Size]byte][]*treeNode)
|
||||
for _, d := range allDirs {
|
||||
groups[d.digest] = append(groups[d.digest], d)
|
||||
|
||||
prog.increment()
|
||||
}
|
||||
|
||||
var dupes [][]*treeNode
|
||||
|
||||
+107
-7
@@ -3,8 +3,11 @@ package main
|
||||
import (
|
||||
"bytes"
|
||||
"database/sql"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Signature hashes shared by the smoke-test records.
|
||||
@@ -17,6 +20,11 @@ const (
|
||||
f2Content = "f2c"
|
||||
)
|
||||
|
||||
// treesOpenLine is the plain line of trees' open phase when it opens
|
||||
// the database at once; it prints it as the phase starts and as it ends.
|
||||
const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
|
||||
"eta unknown"
|
||||
|
||||
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
|
||||
// /d/t2 are identical, /d/t3 differs from them only by one filename.
|
||||
func smokeTreeRecs() []scanRec {
|
||||
@@ -38,7 +46,7 @@ func dbTree(t *testing.T, db *sql.DB) (*treeNode, []*treeNode) {
|
||||
|
||||
tree := newTreeBuilder()
|
||||
|
||||
err := loadFileRows(t.Context(), db, tree.add)
|
||||
err := loadFileRows(t.Context(), db, func(int64) {}, tree.add)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -170,7 +178,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
|
||||
t.Fatalf("directory paths = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
groups := collectTreeGroups(dirs, super)
|
||||
groups := collectTreeGroups(dirs, super, nil)
|
||||
|
||||
gotGroups := groupPaths(groups)
|
||||
wantGroups := [][]string{{"/a", "/c"}}
|
||||
@@ -203,6 +211,98 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
|
||||
// records with stderr not a terminal, and checks every line it prints
|
||||
// there. After the line naming the database, each phase prints a line
|
||||
// naming it with its place among the four as it starts and another as
|
||||
// it ends. read counts the 6 records before it reads them, group counts
|
||||
// the 8 directories, and write the 1 group. The summary comes last, and
|
||||
// stdout holds the report alone.
|
||||
func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
||||
path := seedDatabase(t, smokeTreeRecs())
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
var stdout bytes.Buffer
|
||||
|
||||
code := run([]string{cmdTrees}, &stdout, os.Stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(trees) = %d, want %d; stderr: %s",
|
||||
code, exitOK, stderr())
|
||||
}
|
||||
|
||||
report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
|
||||
if got := stdout.String(); got != report {
|
||||
t.Errorf("stdout = %q, want %q", got, report)
|
||||
}
|
||||
|
||||
// How each line starts; the rate at the end of a phase varies.
|
||||
want := []string{
|
||||
"trees: database " + path + " (from " + databaseEnv + ")",
|
||||
treesOpenLine,
|
||||
treesOpenLine,
|
||||
"[2/4] read: counting the records, elapsed 0s, eta unknown",
|
||||
"[2/4] read: [6/6] 100%, ",
|
||||
"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown",
|
||||
"[3/4] group: [8/8] 100%, ",
|
||||
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown",
|
||||
"[4/4] write: [1/1] 100%, ",
|
||||
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
|
||||
}
|
||||
|
||||
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
|
||||
if len(lines) != len(want) {
|
||||
t.Fatalf("stderr lines = %q, want %d lines starting %q",
|
||||
lines, len(want), want)
|
||||
}
|
||||
|
||||
for i := range want {
|
||||
if !strings.HasPrefix(lines[i], want[i]) {
|
||||
t.Errorf("stderr line %d = %q, want it to start %q",
|
||||
i+1, lines[i], want[i])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestWriteTreeGroupsOnTerminal writes the smoke-test report to the
|
||||
// terminal its write phase's display is drawn on, as a plain "sfdupes
|
||||
// trees" does: the terminal must show the header and the row whole,
|
||||
// each on a line of its own.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
super, dirs := treeOf(t, smokeTreeRecs())
|
||||
dupes := collectTreeGroups(dirs, super, nil)
|
||||
|
||||
// No timed draws: the display is drawn as it starts and as it ends.
|
||||
p := startProgress(progressParams{
|
||||
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
|
||||
}, terminalWidth, time.Hour)
|
||||
|
||||
_, _, err := writeTreeGroups(os.Stderr, dupes, p)
|
||||
|
||||
p.finish()
|
||||
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// A terminal shows each line as the text after its last carriage
|
||||
// return.
|
||||
var shown []string
|
||||
|
||||
for line := range strings.SplitSeq(stderr(), "\n") {
|
||||
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
|
||||
}
|
||||
|
||||
want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
|
||||
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
|
||||
t.Errorf("terminal shows lines %q, want %q first", shown, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTreeDigests(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -254,7 +354,7 @@ func TestCollectTreeGroupsMaximal(t *testing.T) {
|
||||
|
||||
super, dirs := treeOf(t, smokeTreeRecs())
|
||||
|
||||
groups := collectTreeGroups(dirs, super)
|
||||
groups := collectTreeGroups(dirs, super, nil)
|
||||
|
||||
want := [][]string{{"/d/t1", "/d/t2"}}
|
||||
if got := groupPaths(groups); !slices.EqualFunc(got, want,
|
||||
@@ -278,14 +378,14 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
forward := groupPaths(collectTreeGroups(dirs, super))
|
||||
forward := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||
|
||||
reversed := slices.Clone(recs)
|
||||
slices.Reverse(reversed)
|
||||
|
||||
superR, dirsR := treeOf(t, reversed)
|
||||
|
||||
backward := groupPaths(collectTreeGroups(dirsR, superR))
|
||||
backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
|
||||
if !slices.EqualFunc(forward, backward, slices.Equal) {
|
||||
t.Fatalf("output depends on record order: %v vs %v",
|
||||
forward, backward)
|
||||
@@ -304,7 +404,7 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super))
|
||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||
|
||||
want := [][]string{{"/p/x1", "/p/x2"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
@@ -326,7 +426,7 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super))
|
||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||
|
||||
want := [][]string{{"/p/a/x", "/q/b/x"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
|
||||
Reference in New Issue
Block a user