sneak/sfdupes
1
0
Fork 0
Code Issues 4 Pull Requests 2 Actions 12 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/sfdupes:next
Branches Tags
sneak/sfdupes:main sneak/sfdupes:issue-116-install-latest-version sneak/sfdupes:next sneak/sfdupes:issue-19-markdown-fmt
sneak/sfdupes:v0.0.2 sneak/sfdupes:v1.0.0-rc2 sneak/sfdupes:v1.0.0-rc1 sneak/sfdupes:v0.0.1
..
compare: sneak/sfdupes:92d07525c67dc539bd33ceed5d7e92765af76dc7
Branches Tags
sneak/sfdupes:issue-116-install-latest-version sneak/sfdupes:next sneak/sfdupes:main sneak/sfdupes:issue-19-markdown-fmt
sneak/sfdupes:v0.0.2 sneak/sfdupes:v1.0.0-rc2 sneak/sfdupes:v1.0.0-rc1 sneak/sfdupes:v0.0.1
1 Commits
next .. 92d07525c6
Author SHA1 Message Date
clawbot 92d07525c6 scan: show each phase as N of M, with progress, elapsed time and ETA, also while it waits (closes #112)
check / check (push) Waiting to run
Details 
Every scan display names its phase and its place among the five, as
"[2/5] walk", with its progress, elapsed time and ETA. load counts the
records first; the walk estimates its total from the records under the
operands; the content phase adds up the sizes of the matching groups
and says it is searching until the first record returns. Each display
redraws every 100 ms on a terminal and every 5 seconds in plain mode.

On a terminal no line reaches the last column: a line too wide drops
its rate, then its bar. After the terminal is made narrower than the
widest line drawn, the display makes a new library object, whose erase
fits.

Judgement call: load and the content checks draw the bar too, once their totals are known.
Judgement call: the bar is 10 columns wide, not the width the line leaves.

Model: opus-5-5
 2026-10-16 16:36:28 +00:00
21 changed files with 569 additions and 3793 deletions
Expand all files Collapse all files
README.md
+97 -301
View File
@@ -38,9 +38,8 @@ database persists between runs; a rescan only hashes files that are new or
changed, or that may have gained a duplicate since the last scan, and removes
records for files that no longer exist. `report` reads the database and prints
the file-level duplicates report. `trees` reads the same database and prints the
duplicate-tree report. `info` prints the database's details: its path, the sizes
of its files and the number of its records. A missing/invalid subcommand — or a
`scan` invocation with no `PATH` operand — prints a usage message and exits 2.
duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
with no `PATH` operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
@@ -100,11 +99,9 @@ As a file `/etc/cron.d/sfdupes`:
`/var/log/sfdupes.log`, holds a first line naming the database (see
"Database"), plain progress lines naming each phase and its place among the
five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
runs, and one as it ends), a warning for each path it skips, a line for each
default exclude its walk does not enter and for each path it leaves out
because it is not on a persistent filesystem (see "Rules for the walk"), and
the summary line (see "Progress" and "`scan` mode"). The log grows with every
scan; rotate it like any other.
runs, and one as it ends), a warning for each path it skips, and the summary
line (see "Progress" and "`scan` mode"). The log grows with every scan; rotate
it like any other.
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
reason among the last lines of the log; `tail` prints them, and cron mails
@@ -112,9 +109,9 @@ As a file `/etc/cron.d/sfdupes`:
- A scan still running when the next one starts carries on. The new one fails at
once, and the lines cron mails include
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
- `report`, `trees` and `info` need only read access to the database (see
"Database"). Under the usual umask of `022` the first scan creates it readable
by every user, so an unprivileged user can run them against root's database.
- `report` and `trees` need only read access to the database (see "Database").
Under the usual umask of `022` the first scan creates it readable by every
user, so an unprivileged user can run them against root's database.
### Reading the reports
@@ -202,13 +199,12 @@ Goals, in order:
under resilver). Holding one small record (path, size, mtime) per file in
memory during a scan is acceptable; holding every file's hashes is not (they
stay in the database). The reporting commands do not hold every file's
hashes either: `report` reads the records in signature order and holds the
duplicate paths of one size at a time, so its memory grows with the
duplicates of the size that has the most of them, and `trees` reads the
records in path order and keeps each directory's path, digest and totals,
plus the hashes of only the files in the directories holding the record
being read, so its memory grows with the number of directories and with the
size of the largest directory.
hashes either: `report` lets SQLite group and order the records and writes
each row as it reads it, so its memory does not grow with the database, and
`trees` reads the records in path order and keeps each directory's path,
digest and totals, plus the hashes of only the files in the directories
holding the record being read, so its memory grows with the number of
directories and with the size of the largest directory.
3. **Scan incrementally, analyze offline.** The expensive filesystem scan
maintains a persistent database; an unchanged file is never read again on a
rescan, except to compute its content hash once a file of 10 MiB or more
@@ -238,7 +234,7 @@ Goals, in order:
### Subcommands
Four subcommands, all implemented:
Three subcommands, all implemented:
1. `scan` — walk the filesystem and synchronize the database: one signature
record per regular file.
@@ -246,22 +242,18 @@ Four subcommands, all implemented:
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
from the database records, compute a Merkle-style digest per directory, and
report maximal groups of identical trees.
4. `info` — the database's details: its path, the sizes of its files, its
schema version and the counts of its records.
```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes scan [--workers N] [-x] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
sfdupes info
sfdupes --version
sfdupes [command] --help
```
`--workers N` sets the size of each `scan` worker pool (default: the number of
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
operand's filesystem, and `--no-default-excludes` lets the walk enter the
default excludes, a list of paths; all three are described under "`scan` mode".
CPUs), and `-x` (`--one-file-system`) keeps the walk of each operand on that
operand's filesystem; both are described under "`scan` mode".
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
subcommand, prints the help text to stderr and exits 0, writing nothing to
@@ -269,7 +261,7 @@ stdout.
### Database
All four subcommands operate on a single SQLite database file:
All three subcommands operate on a single SQLite database file:
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
@@ -292,13 +284,13 @@ All four subcommands operate on a single SQLite database file:
symlinks resolved. After a `cd` through a symlink to a directory, a path
starting `../` therefore names a file beside the directory the symlink leads
to, as it does for any other program. `scan` says `new database` when no
file is at that path as it starts. `report`, `trees` and `info` print the
line before they open the database, so the error for a missing database
comes after it.
file is at that path as it starts. `report` and `trees` print the line
before they open the database, so the error for a missing database comes
after it.
- `scan` creates the database (and its parent directory) on first use. `report`,
`trees` and `info` require an existing database; a missing database file is a
fatal error (exit 1) telling the user to run `scan` first.
- `scan` creates the database (and its parent directory) on first use. `report`
and `trees` require an existing database; a missing database file is a fatal
error (exit 1) telling the user to run `scan` first.
- Only one `scan` runs against a database at a time. For its whole run, `scan`
holds an exclusive `flock(2)` lock on a lock file beside the database, named
by appending `.lock` to the database's path with symlinks resolved
@@ -314,8 +306,8 @@ All four subcommands operate on a single SQLite database file:
on. The lock file is created on first use, open to its owner only, and left in
place: a leftover file blocks nothing, because the lock ends with the process
holding it however it ends, a fatal error or an interrupt included, and
deleting the file while a scan runs would let a second scan start. `report`,
`trees` and `info` never take the lock, so they run during a scan.
deleting the file while a scan runs would let a second scan start. `report`
and `trees` never take the lock, so they run during a scan.
- While `scan` runs, the database is in WAL journal mode with a busy timeout, so
running a report while a cron `scan` is in progress is safe. The filesystem is
authoritative; the database is an eventually-consistent reflection of it.
@@ -331,9 +323,9 @@ All four subcommands operate on a single SQLite database file:
needs the database to itself: a `scan` that starts while a report is still
reading waits for it up to the 10-second busy timeout, then fails; a `scan`
that ends while a report has the database open warns and leaves the database
in WAL mode until the next scan. `report` writes its rows while it reads the
records, so it is still reading while its output is paused (a pager, a stalled
pipe), and a `scan` started then fails after the busy timeout.
in WAL mode until the next scan. `report` writes each row as it reads it, so
it is still reading while its output is paused (a pager, a stalled pipe), and
a `scan` started then fails after the busy timeout.
- A `scan` writes to the database and the files beside it at these points (its
phases are described under "`scan` mode"):
- As it starts, before the walk: the directory, an empty database file and
@@ -348,10 +340,10 @@ All four subcommands operate on a single SQLite database file:
- In the content phase: content hashes, again committed 10,000 at a time.
- At the end, as it closes the database: the `-wal` and `-shm` files go
away, unless a report still has the database open (see above).
- `report`, `trees` and `info` open the database read-only and need only read
access to the database file, and no write access to its directory. While the
database is in WAL mode they also read the `-wal` and `-shm` files beside it,
which SQLite creates with the database file's permissions.
- `report` and `trees` open the database read-only and need only read access to
the database file, and no write access to its directory. While the database is
in WAL mode they also read the `-wal` and `-shm` files beside it, which SQLite
creates with the database file's permissions.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
with any other version is a fatal error. `scan` creates the schema and sets
the version in one transaction, so a first scan stopped while doing so leaves
@@ -387,8 +379,8 @@ All four subcommands operate on a single SQLite database file:
a file of 10 MiB or more, `content` stays empty until the content phase of a
scan (see "`scan` mode" below) has read the file. A record with an empty
`content` is never part of a duplicate group, though it still defines the
file for tree reconstruction. The `files_signature` index lets `report` read
the records in signature order without sorting the table.
file for tree reconstruction. The `files_signature` index lets SQLite group
the records by signature for `report` without sorting the whole table.
### Duplicate detection
@@ -442,12 +434,11 @@ results do not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared walk worker
pool. Overlapping operands are harmless — an operand that duplicates another or
lies under another is dropped before walking, so every file is reached exactly
once and produces one database record. An operand under another one is kept,
though, and walked on its own when any directory from it (from its directory,
for a regular file) up to but not including the operand covering it is a default
exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
`-x`, is on a different filesystem than that operand, because that operand's
walk does not enter such a directory and so does not reach it.
once and produces one database record. With `-x`, though, an operand under
another one is kept and walked on its own when any directory from it (from its
directory, for a regular file) up to but not including the operand covering it
is on a different filesystem than that operand, because that operand's walk does
not enter such a directory and so does not reach it.
An operand that is a symlink (never followed, not even as an operand), socket,
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -457,10 +448,7 @@ beneath it is still scanned. The records stored beneath it are not deleted: they
are treated like any other record outside the scanned operands, including the
content-phase exception below. If it lies under another operand, they are under
that operand instead, and are treated like any other record there. This is not
an error: a scan whose every operand is dropped walks nothing and exits 0. An
operand that is not on a persistent filesystem (see "Rules for the walk") is
dropped the same way, except that the line `scan` prints for it is not a
warning, and it is not counted as skipped.
an error: a scan whose every operand is dropped walks nothing and exits 0.
`scan` synchronizes the database with the filesystem state under the scanned
operands:
@@ -496,16 +484,14 @@ operands:
can be scanned on different schedules into the same database. The one
exception is the content phase below: a stored file of 10 MiB or more without
a `content` hash is read for one, wherever it lies, once its size, `head`, and
`tail` match another record's. If that file is gone, has changed since its
record was written, or is not on a persistent filesystem, it is not read and
the record is left as it is.
- A record at or below a path the walk left out, because that path is a default
exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
another filesystem than its operand, counts as outside the scanned operands:
it is not deleted, and its size joins the size census. A filesystem mounted
under an operand can therefore be scanned on its own schedule into the same
database. Records below an operand at or inside such a path are not kept this
way, since that operand's own walk verifies them.
`tail` match another record's. If that file is gone or has changed since its
record was written, the record is left as it is.
- With `-x`, a record below a directory the walk did not enter because that
directory is on another filesystem than its operand counts as outside the
scanned operands: it is not deleted, and its size joins the size census. A
filesystem mounted under an operand can therefore be scanned on its own
schedule into the same database. Records below an operand at or inside such a
directory are not kept this way, since that operand's own walk verifies them.
`scan` runs **five sequential phases over the whole scan**. Parallelism lives
inside the walk, hash and content phases; batched database writes begin during
@@ -551,20 +537,19 @@ the hash phase:
be read are kept in memory, never every file's hashes. Every record sharing
their size, `head`, and `tail`, including one that already has a `content`
hash, has its file checked with `lstat` first. A file that is gone, is no
longer a regular file, has changed (a different size, or an mtime newer than
recorded), or is not on a persistent filesystem (see "Rules for the walk")
keeps its record as it is and does not count as a match for the others. Any
other `lstat` error is warned about and counted as skipped, with the same
result. If such a record has no `content` hash, it stays out of duplicate
groups; if it has one, it is still reported until a scan covering its own
tree updates or removes it. The files that pass and have no `content` hash
are read only if at least two of those records pass, so a file whose only
matches are stale costs no read; a file that already has a `content` hash is
never read again. They are read by a worker pool as in the hash phase, in
inode order and once per inode, and their content hashes are committed in
batches. A failed read is warned about and counted as skipped; its record
keeps an empty `content`, so it is not a duplicate, and a later scan tries
again.
longer a regular file, or has changed (a different size, or an mtime newer
than recorded) keeps its record as it is and does not count as a match for
the others. Any other `lstat` error is warned about and counted as skipped,
with the same result. If such a record has no `content` hash, it stays out
of duplicate groups; if it has one, it is still reported until a scan
covering its own tree updates or removes it. The files that pass and have no
`content` hash are read only if at least two of those records pass, so a
file whose only matches are stale costs no read; a file that already has a
`content` hash is never read again. They are read by a worker pool as in the
hash phase, in inode order and once per inode, and their content hashes are
committed in batches. A failed read is warned about and counted as skipped;
its record keeps an empty `content`, so it is not a duplicate, and a later
scan tries again.
Rules for the walk:
@@ -575,15 +560,6 @@ Rules for the walk:
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
them would list every file once per snapshot), not even when it is an operand;
such an operand is dropped the same way.
- Never record the scan's own database. Once the database and its `.lock` file
are open, `scan` notes the device and inode of the database file, its `-wal`
and `-shm` files and its `.lock` file, and the walk passes over a file with
one of them, however it reaches it: as an operand, below one, or through a
symlinked directory. It prints no line for such a file and does not count it
as skipped. A record that an earlier scan left for one of them is deleted,
like the record of any file the scan no longer verifies. There is no
`-journal` file during the walk, because the database is in WAL mode then (see
"Database").
- Filesystem boundaries are crossed by default. With `-x` (long form
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
into a directory on a different filesystem than its `PATH` operand; each
@@ -591,74 +567,6 @@ Rules for the walk:
unless they lie below an operand at or inside that directory, count as outside
the scanned operands, as described in "`scan` mode" above: they are not
deleted, and their sizes join the size census.
- Never record anything that is not on a persistent filesystem. A filesystem is
not persistent when the kernel makes it up or keeps it in memory: one of type
`proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
`debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
`mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
`rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
it names the database and before it takes the lock, `scan` lists the mounted
filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
of mounted filesystems on macOS (on any other system it lists none), and notes
the device ID of each one of those types. On Linux that is the device ID the
list gives, so a filesystem that a later mount hides still gives its own. On
macOS it is read with `lstat` on the mount point, and where one filesystem is
mounted over another on the same mount point, the one mounted last counts. If
`scan` cannot list them, it fails (exit 1). The walk then never enters a
directory, and never records a file, on one of those devices, however it
reaches it: as an operand, below one, or through a bind mount, which keeps the
device ID of what it binds. The content phase never reads a file on one of
them either. No flag turns this off. For each operand, directory or file it
leaves out this way, `scan` prints a line on stderr naming it and the type of
its filesystem, as it prints a warning, so it never lands inside a progress
display:
```
scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
```
That line is not a warning, and the path is not counted as skipped. Records
that an earlier scan left at or below it, unless they lie below an operand
at or inside it, count as outside the scanned operands, as described in
"`scan` mode" above: they are not deleted, and their sizes join the size
census.
- Below an operand, never descend into a default exclude: a path that holds the
kernel's view of the running system, or temporary files, so that
`sfdupes scan /` walks only the files worth comparing. The default excludes
depend on the system, and `sfdupes scan --help` lists those of the system it
runs on:
- On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
- On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
`/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
everything in `/System/Volumes/Data` through the firmlinks, so walking it
as well would record each of those files twice.
- On Windows, relative to the root of the drive an operand lies on: the
page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
`hiberfil.sys`, and the directories `System Volume Information`,
`$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
matches one path component, as it does in any default exclude. sfdupes
does not build for Windows yet, so this list is not used.
- On any other system there are none.
`--no-default-excludes` turns them all off; what is not on a persistent
filesystem stays left out. An operand at or inside a default exclude is
still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
persistent filesystem. For each default exclude its walk does not enter,
`scan` prints a line on stderr naming it, as it prints a warning:
```
scan: skipped /var/tmp (default exclude)
```
That line is not a warning, and the directory is not counted as skipped. A
default exclude that is not on a persistent filesystem, as `/proc` is, gets
the line for its filesystem instead, and one that is itself an operand gets
no line, since it is walked as that operand. The records below a default
exclude the walk does not enter, unless they lie below an operand at or
inside it, count as outside the scanned operands, as described in "`scan`
mode" above: they are not deleted, and their sizes join the size census.
- On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were
@@ -696,8 +604,11 @@ total.)
**`report` must never touch the filesystem being analyzed.** It does not stat,
open, or otherwise access any path that appears in the records; its only I/O is
reading the database and writing stdout/stderr. `report` must produce identical
output whether or not the scanned filesystem is still mounted.
reading the database, writing stdout/stderr, and the temporary file SQLite sorts
in when the duplicate rows do not fit in memory. SQLite puts that file in
`$SQLITE_TMPDIR` or `$TMPDIR` when set, otherwise in `/var/tmp` (or `/tmp`), and
deletes it as soon as it has opened it. `report` must produce identical output
whether or not the scanned filesystem is still mounted.
Processing:
@@ -711,16 +622,6 @@ Processing:
by `first` path ascending. Output must be fully deterministic for a given
database state.
`report` runs in two phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in signature order, largest size first. That is the order of the
`files_signature` index read backwards, so SQLite sorts nothing and returns the
first record at once. The records sharing a signature come together, and so do
the records of one size, so `report` holds only the duplicate groups of the size
it is reading, and writes them, ordered by `first` path, once it reaches a
smaller size. Its memory holds at most the duplicate paths of one size at a
time, never every record.
#### Report output format
TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
@@ -794,12 +695,6 @@ Processing:
- Order groups by total tree size descending, tie-broken by `first` path
ascending. Output must be fully deterministic for a given input.
`trees` runs in four phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in path order, building the hierarchy and computing the digests as it goes;
`group` groups the directories by digest and picks the maximal groups; `write`
writes the groups to stdout.
#### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
@@ -818,60 +713,15 @@ Summary to stderr: records read, number of duplicate-tree groups, number of dupe
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
units.
### `info` mode
`info` takes no positional arguments and prints the database's details on
stdout, one per line: a name, a tab, and the value. It opens the database the
way `report` does, read-only and without the scan lock, so it runs during a scan
and needs only read access (see "Database"). Like `report`, it never touches the
scanned filesystem.
```
database /var/lib/sfdupes/db.sqlite (default)
database file size 2147483648 (2.0 GiB)
-wal file size 4124152 (3.9 MiB)
modified 2026-10-16T12:02:50Z
schema version 1
records 12345678
records with head and tail hashes 1234567
records with content hash 123456
size of recorded files 150000000000000 (136.4 TiB)
```
- `database` is the path and where it came from, as on the line every subcommand
prints first on stderr (see "Database").
- `database file size` and `-wal file size` are in bytes, then in human units.
The `-wal` line is there only when a `-wal` file is beside the database, as
while a scan runs.
- `modified` is the time the database file was last changed, in UTC. While the
database is in WAL mode, the newest changes may be in the `-wal` file alone.
- `schema version` is the database's `PRAGMA user_version`. A database with
another version than this build's is refused, as by `report`.
- `records` is the number of records. `records with head and tail hashes` and
`records with content hash` count the records that have those hashes (see
"Database"), and `size of recorded files` adds up the sizes of all records, in
bytes, then in human units.
`info` runs in two phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `count` reads the schema version and
counts the records with one query, a single pass over the table, both in one
read-only transaction, so the numbers agree with each other while a scan is
committing. `info` writes its lines once the `count` display has ended, so on a
terminal they come after the displays.
### Progress
Use the progress-bar library for all progress of `scan`, `report`, `trees` and
`info`; rendering in the style of `pv` is the model. All progress goes to
stderr.
Use the progress-bar library for all scan progress; rendering in the style of
`pv` is the model. All progress goes to stderr.
`scan`, `report`, `trees` and `info` must always show that they are working and
not hung. Each phase gets its own display from the moment it starts, naming the
phase and its place among the command's phases and showing its progress, its
elapsed time and its estimated time remaining (ETA).
A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
`[4/5] update`, `[5/5] content`):
A scan must always show that it is working and not hung. Each phase gets its own
display from the moment it starts, naming the phase and its place among the five
(`[1/5] load`, `[2/5] walk`, `[3/5] hash`, `[4/5] update`, `[5/5] content`) and
showing its progress, its elapsed time and its estimated time remaining (ETA):
- `load` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
@@ -886,75 +736,27 @@ A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
`head`, and `tail`, without fetching the records. It then says it is searching
the database until SQLite returns the first record; the checks, and their rate
and ETA, start there. The files to read then get a display of their own, with
their count as its total, and so does the writing of the last of their records
to the database, with the count of those records as its total.
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
`[4/4] write`, see "`trees` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `group` has the directories `read` found as its total, and `write` the
duplicate-tree groups.
`report` has two phases (`[1/2] open`, `[2/2] read`, see "`report` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count, writing
the rows to stdout as it goes. While the count runs, the display says it is
counting the records.
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `count` has no total either, because one query counts everything: the display
says it is counting the records until the query returns, and its ETA is
unknown.
their count as its total.
Each display is one line holding:
- the phase and its place among the phases
- a `[m/n] x%` display (items processed / total items, percent); before a phase
has a total, the items processed, or what the phase is waiting for
- current rate (items/s), naming the items the phase counts: files, records,
directories or groups. It covers the items completed over the last 5 seconds,
or, when stderr is not a terminal, since the line before, so it falls to 0
while nothing completes, and a burst of items after a wait shows as a burst
rather than as a steady rate. Once no item has completed for 10 seconds, the
word `stalled` and where the phase waits take the rate's place (see below).
- current rate (items/s)
- elapsed time
- estimated time remaining, or `unknown`. It comes from the phase's average rate
since it started counting, so that it does not swing with each burst.
- estimated time remaining, or `unknown`
Example shape (exact layout is flexible, content is not):
```
[3/5] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
[2/5] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
[3/5] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
```
When no item has completed for 10 seconds, as while one large file is read, a
disk is slow or the database is busy, the line says `stalled` and where the
phase waits in place of the rate, which would read 0: `disk` for files, `db` for
the database, `out` for stdout, and two of them, as `disk/db`, for a phase that
may wait on either. `walk` says `disk`. `hash` and the content phase's reads say
`disk/db`, as they read files and write their records to the database between
them, and so do the content phase's checks, which `lstat` each record's file as
the database search returns it. `load`, `update`, the content phase's last write
and the `read` phase of `trees` say `db`. `trees`' `write` says `out`, and its
`group`, which works in memory, says `cpu`. The `read` phase of `report` writes
the report while it reads the records, so it says `db/out`. A phase still
waiting for its total already says what it waits for, and says nothing more.
Additional requirements:
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
the `open` phase of `report` and `trees`, and both phases of `info`, once it
- On a terminal, a phase with an exact total (every phase but `walk`, once it
has counted what it will do) is drawn as its line followed by the progress-bar
library's bar, 10 columns wide, which shows the percent in place of the line's
own. Any other phase, and a phase whose bar has no room, is drawn as the
@@ -962,20 +764,15 @@ Additional requirements:
times a second, so its elapsed time keeps moving while a phase waits for its
next item.
- On a terminal no line reaches the terminal's last column, however narrow the
terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, at counts like those of
the stalled line in the example above, a stalled line keeps its words whenever
the elapsed time and the ETA are both in minutes, except in the content
phase's checks and reads. Their lines are the widest, and at most times in
minutes they lose the words. Where the words and the ETA do not both fit, as
at millions of files and times in hours, the words give way. Only when those
four alone are too wide, as on a terminal narrower than about 70 columns hours
into a scan of millions of files, is the line cut short at the terminal's
edge, so the end of the ETA is lost first. A terminal made narrower while a
phase runs gets the same: the next draw erases the line within the new width
and fits the line to it, and the display goes on redrawing in place.
terminal is. The phase and its place, the progress, the elapsed time and the
ETA are kept whole: a line too wide for the terminal leaves out its rate first
(or what the phase is waiting for), and a bar with no room beside those four
gives way to the spinner. Only when those four alone are too wide, as on a
terminal narrower than about 70 columns hours into a scan of millions of
files, is the line cut short at the terminal's edge, so the end of the ETA is
lost first. A terminal made narrower while a phase runs gets the same: the
next draw erases the line within the new width and fits the line to it, and
the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once
@@ -984,26 +781,21 @@ Additional requirements:
goroutine counts the items, and each display draws them from a goroutine of
its own.
- A warning printed during a phase always lands on a line of its own, never
inside the progress display. So does every line written to stdout when stdout
and stderr are the same file (the same device and inode): one terminal, as in
a plain `sfdupes report` or `sfdupes trees`, or one pipe or file that both are
sent to. When they are not, stdout is written without waiting on the display,
and the display keeps being drawn while a slow reader of stdout holds up a
write.
inside the progress display.
- A phase that stops short of its total, as an interrupted one does, is left
showing the count it stopped at rather than filled up.
- `report` and `trees` modes need no progress display, only their stderr
summaries.
### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is
already running against the same database, the database cannot be
created/opened/read/written, a missing database for `report`/`trees`/`info`,
stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see
below).
created/opened/read/written, a missing database for `report`/`trees`, stdout
write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see below).
- `2`: usage error (including `scan` with no `PATH` operand, `scan` with
`--workers` below 1, and `report`/`trees`/`info` with any positional
argument).
`--workers` below 1, and `report`/`trees` with any positional argument).
A stdout write failure, such as a full disk, is reported in one line on stderr
and exits 1. Two cases never reach sfdupes as a failed write:
@@ -1170,6 +962,10 @@ All of the following, run in this directory, must pass:
./sfdupes report
```
(The database lives in a temp directory of its own: inside `$d`, the scan
would record it, and its empty lock file would join the `empty1`/`empty2`
group.)
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
group (two dupe rows, `first` is the lexicographically smallest path);
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
TODO.md
-82
View File
@@ -29,88 +29,6 @@
# Completed Steps
- `scan` creates the database's directory where the database path leads: the
path with its last element cut off as text, never cleaned, so a `..` after a
symlinked directory is followed through the symlink, as when the file is
opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
symlink to `/data/x`, it creates `/data/new` and the database in it, where it
used to create `/srv/new` and then fail (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/119)
- `scan` never records its own database: once the database and its `.lock` file
are open, it notes the device and inode of the database file, its `-wal` and
`-shm` files and its `.lock` file, and the walk passes over a file with one of
them wherever the operands reach it, through a symlinked directory included,
printing no line and counting nothing. A record an earlier scan left for one
of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
- `scan` never records or reads anything that is not on a persistent filesystem
(`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
matched by the device ID of each such filesystem, listed as the scan starts
from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
however the walk or the content phase reaches it; no flag turns that off.
Below an operand, its walk does not enter the default excludes either, so
`sfdupes scan /` skips the kernel's view of the running system and the
temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
`/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
`/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
Windows list, not used yet. An operand at or inside a default exclude is still
walked. The scan prints a line for each path it leaves out, without counting
it as skipped, and the records there count as outside the scanned operands, as
for `-x`. `--no-default-excludes` turns off the default excludes only, and
`scan --help` lists them (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/124)
- the rate a progress display shows covers the items completed over the last 5
seconds, or since the line before when stderr is not a terminal, so it falls
to 0 while nothing completes, and a burst after a wait no longer reads as a
steady rate of over 1,000 files a second; the ETA still comes from the phase's
average. Once no item has completed for 10 seconds, the word `stalled` and
where the phase waits take the rate's place, as `stalled disk/db`: `disk`,
`db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
either, each phase giving its word where it is defined. A line too wide for
the terminal loses its bar first, then its rate or those words, so the phase,
progress, elapsed time and ETA stay whole. On an 80-column terminal, at counts
like those of README's example, the content phase's checks and reads lose
those words at most times in minutes, and every other phase keeps them
whenever the elapsed time and the ETA are both in minutes. The content phase's
last partial batch is now written under a display of its own (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/125)
- new subcommand `info` prints the database's details on stdout, one per line as
a name, a tab and the value: its path and where it came from, the sizes of the
database file and of its `-wal` file when there is one, when the database file
was last changed, the schema version, the number of records, how many have
head and tail hashes and how many a content hash, and the sizes of all records
added up. It opens the database as `report` does, and shows two phases on
stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come
from one read-only transaction, the counts from one query (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/117)
- `report` shows its work on stderr in two phases, `[1/2] open` and
`[2/2] read`, with the progress display `scan` and `trees` use, from the
moment it starts until the summary line: `open` says it is opening the
database, also while it waits on a busy one; `read` counts the records, then
reads them against that count in signature order, which SQLite takes from the
`files_signature` index without sorting, so the first record comes at once.
`report` forms the duplicate groups itself, holding only those of the size it
is reading, and writes them ordered by first path before it reads the next
size. stdout and the summary line are unchanged, and SQLite no longer needs a
temporary file to sort in (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/114)
- `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`,
`[3/4] group` and `[4/4] write`, each with the progress display `scan` uses,
from the moment it starts until the summary line: `open` says it is opening
the database, also while it waits on a busy one; `read` counts the records,
then reads them against that count; `group` counts the directories it groups,
and `write` the groups it writes. A display's rate names what its phase
counts: files, records, directories or groups. stdout and the summary line are
unchanged. When stdout and stderr are the same file, as one terminal is, none
of stdout's lines shares a line with the display; when they are not, the
display keeps being drawn while stdout waits on a slow reader (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/113)
- each `scan` phase's progress display names the phase and its place among the
five, as `[2/5] walk`, from the moment it starts, with its progress, elapsed
time and ETA: `load` counts the records first, the walk's total is estimated
cancel_test.go
+10 -15
View File
@@ -196,7 +196,7 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
base := baselineGoroutines(t)
st, err := syncScan(newWalkClock(walkCancelAtDone), db,
[]string{dir}, walkCancelWorkers, false, walkRules{})
[]string{dir}, walkCancelWorkers, false)
assertWalkGuardAborted(t, st, err)
assertRecordsIntact(t, db, before)
@@ -268,8 +268,7 @@ func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
cancel()
st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false,
walkRules{})
st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false)
if !errors.Is(err, context.Canceled) {
t.Fatalf("syncScan on a cancelled context = %v, want %v",
err, context.Canceled)
@@ -302,7 +301,7 @@ func TestSyncScanCancelledMidHashKeepsHashedRecords(t *testing.T) {
db := openTestDB(t)
st, err := syncScan(newWalkClock(hashCancelAtDone), db,
[]string{dir}, walkCancelWorkers, false, walkRules{})
[]string{dir}, walkCancelWorkers, false)
if !errors.Is(err, context.Canceled) {
t.Fatalf("syncScan cancelled mid-hash = %v, want %v",
err, context.Canceled)
@@ -357,8 +356,7 @@ func TestRunScanInterrupted(t *testing.T) {
stderr := captureStderr(t)
dir := buildSmokeTree(t)
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false)
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
if err != nil {
t.Fatal(err)
}
@@ -382,7 +380,7 @@ func TestRunScanInterrupted(t *testing.T) {
defer close(done)
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
false, false)
false)
}()
awaitReturn(t, done, "runScan")
@@ -409,8 +407,7 @@ func TestRunScanInterrupted(t *testing.T) {
got, before)
}
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false)
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
if err != nil {
t.Fatal(err)
}
@@ -436,7 +433,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
dir := buildWalkCancelTree(t)
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
walkCancelWorkers, false, false)
walkCancelWorkers, false)
if !errors.Is(err, errInterrupted) {
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
err, errInterrupted)
@@ -620,8 +617,7 @@ func TestWalkOneDirStopsWhenCancelled(t *testing.T) {
// Unbuffered and unread: on a cancelled scan every send gives up.
events := make(chan walkEvent)
subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false,
walkRules{}, events)
subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false, events)
if len(subs) != 0 {
t.Errorf("cancelled walkOneDir returned %+v to descend into, "+
"want none", subs)
@@ -641,8 +637,7 @@ func TestWalkWorkersDropQueuedDirs(t *testing.T) {
missing := filepath.Join(t.TempDir(), "missing")
jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false,
walkRules{})
jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false)
for range 64 {
jobs <- dirJob{path: missing}
@@ -673,7 +668,7 @@ func TestWalkWorkerAbandonsSubdirHandoff(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
jobs, subdirs, events := startWalkWorkers(ctx, 1, false, walkRules{})
jobs, subdirs, events := startWalkWorkers(ctx, 1, false)
// Fill the hand-back channel to its capacity — one slot per worker
// — so the worker's own hand-back is certain to block.
db.go
+58 -184
View File
@@ -61,8 +61,8 @@ CREATE TABLE files (
) WITHOUT ROWID
`
// createIndexSQL indexes the records by signature, so report can read
// them in signature order without sorting the table.
// createIndexSQL indexes the records by signature, so report can have
// SQLite group them without sorting the whole table.
const createIndexSQL = `
CREATE INDEX files_signature ON files (size, head, tail, content)
`
@@ -79,8 +79,7 @@ ON CONFLICT (path) DO UPDATE SET
content = excluded.content
`
// errNoDatabase reports a missing database file for report, trees and
// info.
// errNoDatabase reports a missing database file for report/trees.
var errNoDatabase = errors.New(
"no database (run \"sfdupes scan\" first, or set " + databaseEnv + ")")
@@ -109,8 +108,10 @@ func databasePath() (string, string, error) {
}
// The working directory as the kernel reports it, with no symlink
// in it, as README "Database" says. os.Getwd may return PWD
// instead, which can name it through a symlink.
// in it. os.Getwd may return PWD instead, which can name it through
// a symlink; a ".." after that symlink then leads somewhere else
// once the path is cleaned, as scan cleans it to create the
// database's directory.
wd, err := unix.Getwd()
if err != nil {
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
@@ -123,24 +124,6 @@ func databasePath() (string, string, error) {
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
}
// createDatabaseDirectory creates the directory the database at path
// goes in, and any directory above it, if missing. path is absolute, as
// databasePath returns it. The directory is path with its last element
// cut off as text, by filepath.Split. filepath.Dir would clean it,
// dropping each ".." together with the directory name before it; when
// that name is a symlink, the cleaned path names another directory than
// the one the system reaches when it opens path.
func createDatabaseDirectory(path string) error {
dir, _ := filepath.Split(path)
err := os.MkdirAll(dir, dbDirPerm)
if err != nil {
return fmt.Errorf("create database directory: %w", err)
}
return nil
}
// scanParams are the connection parameters for scan: read-write, with
// WAL journaling and a busy timeout, so a report can run while a cron
// scan is in progress. closeScanDatabase leaves WAL mode again.
@@ -148,9 +131,9 @@ const scanParams = "_pragma=busy_timeout(10000)" +
"&_pragma=journal_mode(WAL)" +
"&_pragma=synchronous(NORMAL)"
// reportParams are the connection parameters for report, trees and
// info: read-only, with the same busy timeout. They set no journal
// mode, because setting one is a write.
// reportParams are the connection parameters for report and trees:
// read-only, with the same busy timeout. They set no journal mode,
// because setting one is a write.
const reportParams = "mode=ro" +
"&_pragma=busy_timeout(10000)" +
"&_pragma=query_only(1)"
@@ -195,9 +178,9 @@ func openDB(path, params string) (*sql.DB, error) {
// that deleted it would let the next scan lock a new file while
// another still holds the old one.
func lockScanDatabase(path string) (*os.File, error) {
err := createDatabaseDirectory(path)
err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil {
return nil, err
return nil, fmt.Errorf("create database directory: %w", err)
}
// Like SQLite, this follows a symlink at path and creates its
@@ -241,9 +224,9 @@ func lockScanDatabase(path string) (*os.File, error) {
// openScanDatabase opens the database for the scan subcommand, creating
// the file, its parent directory, and the schema as needed.
func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
err := createDatabaseDirectory(path)
err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil {
return nil, err
return nil, fmt.Errorf("create database directory: %w", err)
}
db, err := openDB(path, scanParams)
@@ -261,42 +244,6 @@ func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
return db, nil
}
// databaseFiles returns, by device and inode, the files of the database
// at path that the scan keeps open: the database file, its -wal and
// -shm files and its .lock file, each named from the database's path
// with symlinks resolved, as SQLite and lockScanDatabase name them. The
// scan calls it once the database and the lock file are open, so that
// its walk passes over them however it reaches them, through a
// symlinked directory included. A file that does not exist then is left
// out, since no walk can reach it. There is no -journal file while the
// database is in WAL mode.
func databaseFiles(path string) ([]fileRec, error) {
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return nil, fmt.Errorf("resolve database path: %w", err)
}
var files []fileRec
for _, p := range []string{
resolved, resolved + "-wal", resolved + "-shm", resolved + ".lock",
} {
fi, err := os.Lstat(p)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("database: %w", err)
}
dev, ino := inodeOfInfo(fi)
files = append(files, fileRec{path: p, dev: dev, ino: ino})
}
return files, nil
}
// closeScanDatabase switches the database at path from WAL back to
// rollback-journal mode and closes it. Out of WAL mode the database
// file alone holds the whole database, so a reader needs no -wal or
@@ -315,9 +262,9 @@ func closeScanDatabase(ctx context.Context, db *sql.DB, path string) {
_ = db.Close()
}
// openReportDatabase opens an existing database for the report, trees
// and info subcommands. A missing database file is an error directing
// the user to run scan first; the schema version must match exactly.
// openReportDatabase opens an existing database for the report and
// trees subcommands. A missing database file is an error directing the
// user to run scan first; the schema version must match exactly.
func openReportDatabase(ctx context.Context,
path string,
) (*sql.DB, error) {
@@ -456,32 +403,10 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
return v, nil
}
// loadFileRows gives total the number of records, then streams every
// record to fn in path order: byte order, which is the order of the
// primary key, so SQLite does not sort. The count and the records are
// read in one transaction, so they agree while a scan is committing.
func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
fn func(r scanRec),
) error {
// Everything goes through tx: db has one connection, so a query on
// db would wait for tx forever.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return fmt.Errorf("read records: %w", err)
}
defer func() { _ = tx.Rollback() }()
var n int64
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
if err != nil {
return fmt.Errorf("count records: %w", err)
}
total(n)
rows, err := tx.QueryContext(ctx,
// loadFileRows streams every record to fn in path order: byte order,
// which is the order of the primary key, so SQLite does not sort.
func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
rows, err := db.QueryContext(ctx,
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
"FROM files ORDER BY path")
if err != nil {
@@ -516,74 +441,77 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
return nil
}
// signatureRowsSQL selects every record's signature and path in
// signature order, largest size first. That is the order of the
// files_signature index read backwards, so SQLite reads the index alone
// and returns the first row at once, with nothing to sort.
const signatureRowsSQL = `
SELECT size, head, tail, content, path
FROM files
ORDER BY size DESC, head DESC, tail DESC, content DESC
// dupeRowsSQL selects every record in a duplicate group, with the
// group's first path. A group is the records with a content hash that
// share a size, head, tail, and content, when there are two or more of
// them. The rows come in report order: groups by size descending, then
// by first path, and each group's paths ascending.
const dupeRowsSQL = `
SELECT g.first, f.path, f.size
FROM files AS f
JOIN (
SELECT size, head, tail, content, MIN(path) AS first
FROM files
WHERE content <> ''
GROUP BY size, head, tail, content
HAVING COUNT(*) > 1
) AS g USING (size, head, tail, content)
ORDER BY f.size DESC, g.first, f.path
`
// loadSignatureRows gives total the number of records, then streams
// every record's signature and path to fn, without its mtime, in the
// order of signatureRowsSQL. The count and the records are read in one
// transaction, so they agree while a scan is committing. An error from
// fn stops the reading and is returned as it is.
func loadSignatureRows(ctx context.Context, db *sql.DB,
total func(n int64), fn func(r scanRec) error,
) error {
// loadDupeRows streams the rows of dupeRowsSQL to fn and returns the
// number of records in the database. The count and the rows are read
// in one transaction, so they agree while a scan is committing. An
// error from fn stops the reading and is returned as it is.
func loadDupeRows(ctx context.Context, db *sql.DB,
fn func(first, path string, size int64) error,
) (int, error) {
// Everything goes through tx: the report connection is the only
// one, so a query on db would wait for tx forever.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return fmt.Errorf("read records: %w", err)
return 0, fmt.Errorf("read records: %w", err)
}
defer func() { _ = tx.Rollback() }()
var n int64
var records int
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&records)
if err != nil {
return fmt.Errorf("count records: %w", err)
return 0, fmt.Errorf("read records: %w", err)
}
total(n)
rows, err := tx.QueryContext(ctx, signatureRowsSQL)
rows, err := tx.QueryContext(ctx, dupeRowsSQL)
if err != nil {
return fmt.Errorf("read records: %w", err)
return 0, fmt.Errorf("read records: %w", err)
}
defer func() { _ = rows.Close() }()
for rows.Next() {
var (
path []byte
r scanRec
first, path []byte
size int64
)
err = rows.Scan(&r.size, &r.head, &r.tail, &r.content, &path)
err = rows.Scan(&first, &path, &size)
if err != nil {
return fmt.Errorf("read record: %w", err)
return 0, fmt.Errorf("read record: %w", err)
}
r.path = string(path)
err = fn(r)
err = fn(string(first), string(path), size)
if err != nil {
return err
return 0, err
}
}
err = rows.Err()
if err != nil {
return fmt.Errorf("read records: %w", err)
return 0, fmt.Errorf("read records: %w", err)
}
return nil
return records, nil
}
// countRecords returns the number of records in the database.
@@ -598,60 +526,6 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
return n, nil
}
// databaseCountsSQL counts the records, those with head and tail
// hashes and those with a content hash, and adds up their sizes, in one
// pass over the table. A SUM over no rows is NULL, so on an empty table
// each SUM falls back to 0.
const databaseCountsSQL = `
SELECT COUNT(*),
COALESCE(SUM(head <> '' AND tail <> ''), 0),
COALESCE(SUM(content <> ''), 0),
COALESCE(SUM(size), 0)
FROM files
`
// databaseCounts is what info reads from the database: its schema
// version and the counts of databaseCountsSQL.
type databaseCounts struct {
schemaVersion int
records int64 // every record
headTail int64 // records with head and tail hashes
content int64 // records with a content hash
size int64 // the sizes of all records added up
}
// readDatabaseCounts reads the schema version, then the counts of
// databaseCountsSQL, in one read-only transaction, so both describe the
// database at one moment while a scan is committing.
func readDatabaseCounts(ctx context.Context,
db *sql.DB,
) (databaseCounts, error) {
var c databaseCounts
// Everything goes through tx: the report connection is the only
// one, so a query on db would wait for tx forever.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return c, fmt.Errorf("count records: %w", err)
}
defer func() { _ = tx.Rollback() }()
err = tx.QueryRowContext(ctx, "PRAGMA user_version").
Scan(&c.schemaVersion)
if err != nil {
return c, fmt.Errorf("read schema version: %w", err)
}
err = tx.QueryRowContext(ctx, databaseCountsSQL).
Scan(&c.records, &c.headTail, &c.content, &c.size)
if err != nil {
return c, fmt.Errorf("count records: %w", err)
}
return c, nil
}
// loadFileMeta streams every record's path, size, mtime, and whether
// it carries hashes to fn. Scan change detection needs no hash
// values, and skipping the hash columns keeps the scan's in-memory
db_test.go
+1 -228
View File
@@ -5,7 +5,6 @@ import (
"database/sql"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"slices"
@@ -119,116 +118,6 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
}
}
// TestScanCreatesDatabaseDirectoryAfterSymlink has a database path go
// through a symlink to a directory, then "..", to a missing directory.
// With srv/link a symlink to data/x, srv/link/../new/db.sqlite leads to
// data/new/db.sqlite. lockScanDatabase and openScanDatabase, the two
// places scan creates the database's directory, must each create the
// database there, and nothing at srv/new, where the path leads once
// cleaned as text.
func TestScanCreatesDatabaseDirectoryAfterSymlink(t *testing.T) {
t.Parallel()
creates := map[string]func(ctx context.Context, path string) error{
"lockScanDatabase": func(_ context.Context, path string) error {
lock, err := lockScanDatabase(path)
if err == nil {
_ = lock.Close()
}
return err
},
"openScanDatabase": func(ctx context.Context, path string) error {
db, err := openScanDatabase(ctx, path)
if err == nil {
_ = db.Close()
}
return err
},
}
for name, create := range creates {
t.Run(name, func(t *testing.T) {
t.Parallel()
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
err = os.MkdirAll(filepath.Join(dir, "data", "x"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Mkdir(filepath.Join(dir, "srv"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Symlink(filepath.Join(dir, "data", "x"),
filepath.Join(dir, "srv", "link"))
if err != nil {
t.Fatal(err)
}
err = create(t.Context(), dir+"/srv/link/../new/db.sqlite")
if err != nil {
t.Fatalf("%s: %v", name, err)
}
_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
if err != nil {
t.Errorf("database the path leads to: %v", err)
}
_, err = os.Lstat(filepath.Join(dir, "srv", "new"))
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("srv/new: stat error %v, want it missing", err)
}
})
}
}
// TestDatabaseFiles opens a database through a symlink to its file:
// databaseFiles finds the database file and its -wal and -shm files
// beside the symlink's target, where SQLite keeps them, and leaves out
// the .lock file, which only a scan creates.
func TestDatabaseFiles(t *testing.T) {
t.Parallel()
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
db, err := openScanDatabase(t.Context(), link)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
files, err := databaseFiles(link)
if err != nil {
t.Fatal(err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.path)
}
want := []string{path, path + "-wal", path + "-shm"}
if !slices.Equal(got, want) {
t.Errorf("databaseFiles(%s) = %q, want %q", link, got, want)
}
}
func TestOpenDatabaseUnversionedForeign(t *testing.T) {
t.Parallel()
@@ -395,128 +284,12 @@ func TestCloseScanDatabaseWhileReportOpen(t *testing.T) {
defer func() { _ = reportDB.Close() }()
err = loadFileRows(t.Context(), reportDB, func(int64) {},
func(scanRec) {})
err = loadFileRows(t.Context(), reportDB, func(scanRec) {})
if err != nil {
t.Fatalf("loadFileRows: %v", err)
}
}
// TestLoadRowsCountAgreeDuringScan has a scan commit records after
// loadFileRows, as trees calls it, or loadSignatureRows, as report
// calls it, has counted the records and before it reads them, as when
// trees or report runs during a scan: each must read exactly as many
// records as it counted.
func TestLoadRowsCountAgreeDuringScan(t *testing.T) {
t.Parallel()
loads := map[string]func(context.Context, *sql.DB, func(int64),
func(scanRec)) error{
"loadFileRows": loadFileRows,
"loadSignatureRows": func(ctx context.Context, db *sql.DB,
total func(int64), fn func(scanRec),
) error {
return loadSignatureRows(ctx, db, total, func(r scanRec) error {
fn(r)
return nil
})
},
}
for name, load := range loads {
t.Run(name, func(t *testing.T) {
t.Parallel()
path := testDBPath(t)
scanDB, err := openScanDatabase(t.Context(), path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = scanDB.Close() }()
recs := smokeTreeRecs()
err = applyChanges(t.Context(), scanDB, recs[:3], nil, nil)
if err != nil {
t.Fatal(err)
}
reportDB, err := openReportDatabase(t.Context(), path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = reportDB.Close() }()
var counted, read int64
err = load(t.Context(), reportDB, func(n int64) {
counted = n
err := applyChanges(t.Context(), scanDB, recs[3:], nil, nil)
if err != nil {
t.Error(err)
}
}, func(scanRec) { read++ })
if err != nil {
t.Fatal(err)
}
if counted != 3 || read != counted {
t.Errorf("counted %d records and read %d, want 3 and 3",
counted, read)
}
})
}
}
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
// from the files_signature index alone, with nothing to sort: the
// first record comes at once, so report's read phase counts each
// record as it comes rather than waiting on SQLite for all of them.
func TestSignatureRowsNeedNoSort(t *testing.T) {
t.Parallel()
db := openTestDB(t)
rows, err := db.QueryContext(t.Context(),
"EXPLAIN QUERY PLAN "+signatureRowsSQL)
if err != nil {
t.Fatal(err)
}
defer func() { _ = rows.Close() }()
var plan []string
for rows.Next() {
var (
id, parent, unused int
detail string
)
err = rows.Scan(&id, &parent, &unused, &detail)
if err != nil {
t.Fatal(err)
}
plan = append(plan, detail)
}
err = rows.Err()
if err != nil {
t.Fatal(err)
}
want := []string{"SCAN files USING COVERING INDEX files_signature"}
if !slices.Equal(plan, want) {
t.Errorf("query plan = %q, want %q", plan, want)
}
}
func TestApplyChangesRoundTrip(t *testing.T) {
t.Parallel()
exclude.go
-195
View File
@@ -1,195 +0,0 @@
package main
// This file holds two rules for the walk (README "Rules for the walk"):
// the default excludes, the paths a scan's walk does not enter below an
// operand unless scan is given --no-default-excludes, and the
// filesystems that are not persistent, on which the scan never records
// or reads anything.
import (
"bufio"
"errors"
"fmt"
"io"
"io/fs"
"path/filepath"
"slices"
"strings"
"golang.org/x/sys/unix"
)
// The systems that have default excludes, as runtime.GOOS names them.
const (
goosLinux = "linux"
goosDarwin = "darwin"
goosWindows = "windows"
)
// The fields of a line of mountInfoPath that readMountInfo reads,
// counted from 0 (see proc(5)): the device ID, as major:minor, is the
// third field, and after the sixth come optional fields, then a field
// holding "-", then the filesystem type.
const (
deviceField = 2
optionalFieldsStart = 6
)
// typesPerHelpLine is how many filesystem types the scan command's help
// lists on each line.
const typesPerHelpLine = 6
// errMountInfoLine reports a line of mountInfoPath without the fields
// readMountInfo reads.
var errMountInfoLine = errors.New("unexpected line")
// walkRules holds the rules for the walk that a scan sets up as it
// starts. excludes are the default excludes, or none with
// --no-default-excludes. nonPersistent maps the device ID of every
// mounted filesystem that is not persistent to its type; the scan
// records and reads nothing on one of those, whatever the flags.
// databaseFiles are the scan's own database files, which the walk
// passes over (see databaseFiles in db.go).
type walkRules struct {
excludes []string
nonPersistent map[uint64]string
databaseFiles []fileRec
}
// defaultExcludes returns the default excludes of the system goos
// names, as runtime.GOOS does: a fixed list on Linux, macOS and
// Windows, and none on any other system.
func defaultExcludes(goos string) []string {
switch goos {
case goosLinux:
return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
case goosDarwin:
// A walk of / reaches everything in /System/Volumes/Data through
// the firmlinks, so walking it as well would record each of
// those files twice.
return []string{
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data",
}
case goosWindows:
// Relative to the root of the drive an operand lies on. The
// first three are the page, swap and hibernation files.
return []string{
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
`Users\*\AppData\Local\Temp`,
}
default:
return nil
}
}
// isDefaultExclude reports whether path is one of excludes, in which a
// * matches one path component.
func isDefaultExclude(path string, excludes []string) bool {
for _, pattern := range excludes {
// The patterns are fixed, so none is malformed.
if ok, _ := filepath.Match(pattern, path); ok {
return true
}
}
return false
}
// nonPersistentFilesystemTypes returns the filesystem types that are
// not persistent: filesystems that the kernel makes up or that are kept
// in memory. devfs is the macOS filesystem of /dev; the others are
// Linux names.
func nonPersistentFilesystemTypes() []string {
return []string{
"proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "ramfs",
"cgroup", "cgroup2", "debugfs", "tracefs", "securityfs", "pstore",
"bpf", "configfs", "fusectl", "mqueue", "hugetlbfs", "efivarfs",
"binfmt_misc", "autofs", "nsfs", "rpc_pipefs", "selinuxfs", "devfs",
}
}
// nonPersistentLine returns the line the scan prints for path when fi,
// its lstat, puts it on a filesystem in rules.nonPersistent, and ""
// when it is on any other.
func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
dev, ok := deviceOfInfo(fi)
if !ok {
return ""
}
fsType, found := rules.nonPersistent[dev]
if !found {
return ""
}
return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
path, fsType)
}
// readMountInfo reads mountinfo, laid out as mountInfoPath is, and
// returns the device ID of every mounted filesystem that is not
// persistent, with its type. The device ID is the one each line gives,
// not that of the file now at its mount point, so a mount that a later
// mount hides never adds the device of another filesystem.
func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
types := nonPersistentFilesystemTypes()
devices := make(map[uint64]string)
lines := bufio.NewScanner(mountinfo)
for lines.Scan() {
fields := strings.Fields(lines.Text())
sep := slices.Index(fields, "-")
if sep < optionalFieldsStart || sep+1 >= len(fields) {
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
}
var major, minor uint32
_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
if err != nil {
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
}
fsType := fields[sep+1]
if slices.Contains(types, fsType) {
devices[unix.Mkdev(major, minor)] = fsType
}
}
return devices, lines.Err()
}
// defaultExcludesHelp describes, for the scan command's help, the
// default excludes of the system goos names and, where the scan lists
// the mounted filesystems, the types it never records anything on.
func defaultExcludesHelp(goos string) string {
var help strings.Builder
dirs := defaultExcludes(goos)
if len(dirs) == 0 {
help.WriteString("This system has no default excludes.\n")
} else {
help.WriteString("Below an operand, the walk does not enter these " +
"default excludes, unless\n--no-default-excludes is given:\n\n")
for _, dir := range dirs {
fmt.Fprintf(&help, " %s\n", dir)
}
}
if goos == goosLinux || goos == goosDarwin {
help.WriteString("\nWith or without --no-default-excludes, the scan " +
"records nothing on a\nfilesystem that is not persistent, " +
"one of these types:\n\n")
types := nonPersistentFilesystemTypes()
for line := range slices.Chunk(types, typesPerHelpLine) {
fmt.Fprintf(&help, " %s\n", strings.Join(line, ", "))
}
}
return help.String()
}
exclude_darwin.go
-52
View File
@@ -1,52 +0,0 @@
//go:build darwin
package main
import (
"slices"
"golang.org/x/sys/unix"
)
// nonPersistentFilesystems returns the device ID of every mounted
// filesystem that is not persistent, with its type, from the system's
// list of mounted filesystems. The device ID is read with lstat on the
// mount point. Where one filesystem is mounted over another on the same
// mount point, the later one in the list counts. A mount point that
// lstat cannot reach is left out.
func nonPersistentFilesystems() (map[uint64]string, error) {
// The first call counts the filesystems, the second lists them.
n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
if err != nil {
return nil, err
}
list := make([]unix.Statfs_t, n)
n, err = unix.Getfsstat(list, unix.MNT_NOWAIT)
if err != nil {
return nil, err
}
mounts := make(map[string]string, n)
for _, st := range list[:n] {
mounts[unix.ByteSliceToString(st.Mntonname[:])] =
unix.ByteSliceToString(st.Fstypename[:])
}
types := nonPersistentFilesystemTypes()
devices := make(map[uint64]string)
for mountPoint, fsType := range mounts {
if !slices.Contains(types, fsType) {
continue
}
if dev, ok := deviceOfPath(mountPoint); ok {
devices[dev] = fsType
}
}
return devices, nil
}
exclude_other.go
-36
View File
@@ -1,36 +0,0 @@
//go:build !darwin
package main
import (
"fmt"
"os"
"runtime"
)
// mountInfoPath is the file in which Linux lists the mounts the reading
// process sees.
const mountInfoPath = "/proc/self/mountinfo"
// nonPersistentFilesystems returns the device ID of every mounted
// filesystem that is not persistent, with its type: on Linux read from
// mountInfoPath (see readMountInfo), and on any other system none.
func nonPersistentFilesystems() (map[uint64]string, error) {
if runtime.GOOS != goosLinux {
return map[uint64]string{}, nil
}
f, err := os.Open(mountInfoPath)
if err != nil {
return nil, err
}
defer func() { _ = f.Close() }()
devices, err := readMountInfo(f)
if err != nil {
return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
}
return devices, nil
}
exclude_test.go
-583
View File
@@ -1,583 +0,0 @@
package main
import (
"bytes"
"errors"
"maps"
"os"
"path/filepath"
"runtime"
"slices"
"strings"
"testing"
"golang.org/x/sys/unix"
)
func TestDefaultExcludes(t *testing.T) {
t.Parallel()
// README "Rules for the walk": each system's list.
cases := map[string][]string{
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
goosDarwin: {
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data",
},
goosWindows: {
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
`Users\*\AppData\Local\Temp`,
},
"freebsd": nil,
}
for goos, want := range cases {
if got := defaultExcludes(goos); !slices.Equal(got, want) {
t.Errorf("defaultExcludes(%q) = %q, want %q", goos, got, want)
}
}
}
func TestIsDefaultExclude(t *testing.T) {
t.Parallel()
// A * matches one path component, no more and no fewer.
excludes := []string{"/scratch", "/home/*/cache"}
cases := map[string]bool{
"/scratch": true,
"/scratch/x": false,
"/home/u/cache": true,
"/home/u/v/cache": false,
"/home/cache": false,
}
for path, want := range cases {
if got := isDefaultExclude(path, excludes); got != want {
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
}
}
}
func TestReadMountInfo(t *testing.T) {
t.Parallel()
// The second-to-last line has no optional fields. The tmpfs on
// /data/a/b is hidden by the root filesystem's /srv, mounted over
// /data/a after it: the tmpfs still gives its own device, and the
// root filesystem's device is not listed.
mountinfo := strings.Join([]string{
"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw",
"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw",
"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
}, "\n")
got, err := readMountInfo(strings.NewReader(mountinfo))
if err != nil {
t.Fatal(err)
}
want := map[uint64]string{
unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs",
unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs",
}
if !maps.Equal(got, want) {
t.Errorf("readMountInfo = %v, want %v", got, want)
}
for _, line := range []string{
"22 1 8:1 / / rw,relatime\n",
"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n",
} {
_, err = readMountInfo(strings.NewReader(line))
if !errors.Is(err, errMountInfoLine) {
t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
errMountInfoLine)
}
}
}
func TestNonPersistentFilesystems(t *testing.T) {
t.Parallel()
if runtime.GOOS != goosLinux {
t.Skip("the test knows a mount only on Linux: /proc")
}
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfPath("/proc")
if !ok {
t.Fatal("no device for /proc")
}
if got := nonPersistent[dev]; got != "proc" {
t.Errorf("filesystem of /proc = %q, want %q", got, "proc")
}
}
func TestDefaultExcludesHelp(t *testing.T) {
t.Parallel()
for _, goos := range []string{goosLinux, goosDarwin, goosWindows} {
help := defaultExcludesHelp(goos)
for _, dir := range defaultExcludes(goos) {
if !strings.Contains(help, "\n "+dir+"\n") {
t.Errorf("help for %s does not list %s:\n%s", goos, dir, help)
}
}
// Only Linux and macOS list their mounted filesystems.
for _, fsType := range nonPersistentFilesystemTypes() {
listed := strings.Contains(help, " "+fsType+",") ||
strings.Contains(help, " "+fsType+"\n")
if listed != (goos != goosWindows) {
t.Errorf("help for %s lists %s: %v", goos, fsType, listed)
}
}
}
want := "This system has no default excludes.\n"
if got := defaultExcludesHelp("freebsd"); got != want {
t.Errorf("help for freebsd = %q, want %q", got, want)
}
// scan's help shows those of the system it runs on.
var stdout, stderr bytes.Buffer
run([]string{cmdScan, "-h"}, &stdout, &stderr)
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
if !strings.Contains(stderr.String(), help) {
t.Errorf("scan --help = %q, want it to hold %q", stderr.String(), help)
}
}
// TestScanDoesNotEnterDefaultExclude checks that the walk does not
// enter a default exclude below an operand. It says so on a line of its
// own and does not count it as skipped. The records below it count as
// outside the operands: the record of a file gone from it is kept, and
// its size joins the size census, while a file added to it is not
// recorded.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
db := openTestDB(t)
excluded := filepath.Join(dir, "tmp")
// A scan with no default excludes records the file in excluded.
gone := writeFile(t, excluded, "gone", pattern(1, 100))
syncTree(t, db, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
unseen := writeFile(t, excluded, "new", pattern(2, 10))
// The same size as gone, so it is hashed only if the census holds the
// size of gone's record.
added := writeFile(t, dir, "added", pattern(3, 100))
st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
walkRules{excludes: []string{excluded}})
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
recordByPath(t, recs, gone)
if recordByPath(t, recs, added).head == "" {
t.Errorf("%s not hashed: the census lacks the size of %s",
added, gone)
}
if slices.Contains(recordPaths(recs), unseen) {
t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
}
line := "\nscan: skipped " + excluded + " (default exclude)\n"
if !strings.Contains(stderr(), line) {
t.Errorf("stderr = %q, want %q", stderr(), line)
}
}
// TestScanWalksOperandInDefaultExclude checks that an operand at or
// inside a default exclude is walked, also when it lies under another
// operand, whose walk does not enter the default exclude.
func TestScanWalksOperandInDefaultExclude(t *testing.T) {
t.Parallel()
dir := t.TempDir()
excluded := filepath.Join(dir, "tmp")
f := writeFile(t, excluded, "sub/f", pattern(1, 10))
sub := filepath.Dir(f)
cases := map[string][]string{
"at it": {excluded},
"inside it": {sub},
"inside it under an operand": {dir, sub},
}
for name, roots := range cases {
t.Run(name, func(t *testing.T) {
t.Parallel()
db := openTestDB(t)
st, err := syncScan(t.Context(), db, roots, 1, false,
walkRules{excludes: []string{excluded}})
if err != nil {
t.Fatal(err)
}
if st.added != 1 {
t.Errorf("added = %d, want %s recorded once", st.added, f)
}
recordByPath(t, dbRecords(t, db), f)
})
}
}
// TestScanNamesNoDefaultExcludeThatIsAnOperand checks that a default
// exclude given as an operand under another operand, which the walk of
// that other operand does not enter, gets no line saying it was
// skipped: it is walked as an operand of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
db := openTestDB(t)
excluded := filepath.Join(dir, "tmp")
f := writeFile(t, excluded, "f", pattern(1, 10))
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
walkRules{excludes: []string{excluded}})
if err != nil {
t.Fatal(err)
}
if st.added != 1 {
t.Errorf("added = %d, want %s recorded once", st.added, f)
}
recordByPath(t, dbRecords(t, db), f)
if strings.Contains(stderr(), "scan: skipped") {
t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
}
}
// TestRunScanTempDirOperand scans a tree in the tests' temporary
// directory, which on Linux lies inside the default exclude /tmp unless
// TMPDIR names another place, with the system's own default excludes:
// the scan walks the whole tree and reports no default exclude.
func TestRunScanTempDirOperand(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
t.Skip(dir + " is not inside a default exclude")
}
f := writeFile(t, dir, "sub/f", pattern(1, 10))
stderr := scanOK(t, dir)
if strings.Contains(stderr, "scan: skipped") {
t.Errorf("stderr = %q, want no line saying a path was skipped",
stderr)
}
assertRecorded(t, path, f, true)
}
// TestRunScanNoDefaultExcludes scans /var through the command. Its
// /var/tmp, on a persistent filesystem, is a default exclude on Linux:
// without --no-default-excludes the walk does not enter it and says so,
// and with that flag the walk records a file the test puts there.
func TestRunScanNoDefaultExcludes(t *testing.T) {
if runtime.GOOS != goosLinux {
t.Skip("only Linux has /var/tmp among its default excludes")
}
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
t.Fatal(err)
}
rules := walkRules{nonPersistent: nonPersistent}
for _, dir := range []string{"/var", "/var/tmp"} {
fi, lstatErr := os.Lstat(dir)
if lstatErr != nil || !fi.IsDir() ||
rules.nonPersistentLine(dir, fi) != "" {
t.Skip(dir + " is not a directory on a persistent filesystem here")
}
}
//nolint:usetesting // the directory must be inside /var/tmp
tmp, err := os.MkdirTemp("/var/tmp", "sfdupes-test-")
if err != nil {
t.Skip("cannot write to /var/tmp here: " + err.Error())
}
t.Cleanup(func() { _ = os.RemoveAll(tmp) })
f := writeFile(t, tmp, "f", pattern(1, 10))
path := testDBPath(t)
t.Setenv(databaseEnv, path)
line := "scan: skipped /var/tmp (default exclude)\n"
if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
}
assertRecorded(t, path, f, false)
stderr := scanOK(t, "--no-default-excludes", "/var")
if strings.Contains(stderr, line) {
t.Errorf("scan --no-default-excludes /var: stderr = %q, want no %q",
stderr, line)
}
assertRecorded(t, path, f, true)
}
// TestRunScanSkipsNonPersistentOperand scans, through the command, a
// directory on a filesystem that is not persistent, with and without
// --no-default-excludes: either way it is skipped, with a line saying
// so, and nothing in it is recorded.
func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
shm, fsType, _ := shmTempDir(t)
writeFile(t, shm, "f", pattern(1, 10))
t.Setenv(databaseEnv, testDBPath(t))
line := "scan: skipped " + shm + " (" + fsType +
", not a persistent filesystem)\n"
summary := "scan: 0 files seen"
for _, flags := range [][]string{nil, {"--no-default-excludes"}} {
stderr := scanOK(t, append(flags, shm)...)
if !strings.Contains(stderr, line) ||
!strings.Contains(stderr, summary) {
t.Errorf("scan %q: stderr = %q, want %q and %q", flags, stderr,
line, summary)
}
}
}
// TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem
// that is not persistent beside one on the tests' temporary directory.
// The records an earlier scan left in it count as outside the operands:
// the record of a file gone from it is kept, and its size joins the size
// census, while a file added to it is not recorded, and its size does
// not join the census.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
shm, fsType, rules := shmTempDir(t)
stderr := captureStderr(t)
dir := t.TempDir()
db := openTestDB(t)
// A scan that, without the list of filesystems, records gone.
gone := writeFile(t, shm, "gone", pattern(1, 100))
syncTree(t, db, shm)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
unseen := writeFile(t, shm, "new", pattern(2, 10))
// added has the size of gone, and lonely that of unseen: each is
// hashed only if the census holds that size.
added := writeFile(t, dir, "added", pattern(3, 100))
lonely := writeFile(t, dir, "lonely", pattern(4, 10))
st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules)
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
recordByPath(t, recs, gone)
if recordByPath(t, recs, added).head == "" {
t.Errorf("%s not hashed: the census lacks the size of %s",
added, gone)
}
if recordByPath(t, recs, lonely).head != "" {
t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen)
}
if slices.Contains(recordPaths(recs), unseen) {
t.Errorf("%s recorded", unseen)
}
line := "scan: skipped " + shm + " (" + fsType +
", not a persistent filesystem)\n"
if !strings.Contains(stderr(), line) {
t.Errorf("stderr = %q, want %q", stderr(), line)
}
}
// TestScanContentSkipsNonPersistent checks that the content phase does
// not read a file on a filesystem that is not persistent, though an
// earlier scan left a record for it: it is treated as a changed file,
// so its record is left as it is and its identical copy in another
// tree, having no other match, is not read either.
func TestScanContentSkipsNonPersistent(t *testing.T) {
t.Parallel()
shm, _, rules := shmTempDir(t)
db := openTestDB(t)
// A scan that, without the list of filesystems, records a with a
// head and tail but no content hash.
a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
syncTree(t, db, shm)
before := recordByPath(t, dbRecords(t, db), a)
if before.head == "" || before.content != "" {
t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
}
b := sparseFile(t, t.TempDir(), "b", headTailMin)
st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
rules)
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
if r := recordByPath(t, recs, a); r != before {
t.Errorf("record = %+v, want it left as %+v", r, before)
}
if r := recordByPath(t, recs, b); r.content != "" {
t.Errorf("%s: content = %q, want none: its only match is on %s",
b, r.content, shm)
}
}
// TestWalkLeavesOutNonPersistent starts the walk itself in a directory
// on a filesystem that is not persistent, which a scan never does: it
// drops such an operand first. Below a directory on a persistent
// filesystem, one on another takes a mount, which a test cannot make.
// The walk enters no directory and records no file there, and names
// each.
func TestWalkLeavesOutNonPersistent(t *testing.T) {
t.Parallel()
shm, fsType, rules := shmTempDir(t)
f := writeFile(t, shm, "f", pattern(1, 10))
sub := filepath.Dir(writeFile(t, shm, "sub/g", pattern(1, 10)))
var leftOut []string
for ev := range startWalk(t.Context(), []string{shm}, false, rules, 1) {
if ev.rec.path != "" {
t.Errorf("walk recorded %s", ev.rec.path)
}
if ev.leftOut == "" {
continue
}
leftOut = append(leftOut, ev.leftOut)
want := "scan: skipped " + ev.leftOut + " (" + fsType +
", not a persistent filesystem)"
if ev.warn != want {
t.Errorf("line = %q, want %q", ev.warn, want)
}
}
slices.Sort(leftOut)
if want := []string{f, sub}; !slices.Equal(leftOut, want) {
t.Errorf("walk left out %q, want %q", leftOut, want)
}
}
// shmTempDir returns a new directory in /dev/shm, removed when the test
// ends, the type of its filesystem, and the rules holding the real
// filesystems that are not persistent. It skips the test where /dev/shm
// is on a persistent filesystem, or cannot be written to.
func shmTempDir(t *testing.T) (string, string, walkRules) {
t.Helper()
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfPath("/dev/shm")
fsType, found := nonPersistent[dev]
if !ok || !found {
t.Skip("/dev/shm is not a memory-backed filesystem here")
}
//nolint:usetesting // the directory must be inside /dev/shm
dir, err := os.MkdirTemp("/dev/shm", "sfdupes-test-")
if err != nil {
t.Skip("cannot write to /dev/shm here: " + err.Error())
}
t.Cleanup(func() { _ = os.RemoveAll(dir) })
return dir, fsType, walkRules{nonPersistent: nonPersistent}
}
// assertRecorded checks that the database at dbPath holds a record for
// path when want is set, and none when it is not.
func assertRecorded(t *testing.T, dbPath, path string, want bool) {
t.Helper()
db, err := openDB(dbPath, reportParams)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
if got := slices.Contains(recordPaths(dbRecords(t, db)), path); got != want {
t.Errorf("%s recorded: %v, want %v", path, got, want)
}
}
info.go
-128
View File
@@ -1,128 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"strings"
"time"
)
// infoPhase names one phase of info with its place among all of them,
// as "[2/2] count", for the phase's progress display. The list here is
// the one place the phases are counted.
func infoPhase(name string) string {
return phaseLabel([]string{openPhase, "count"}, name)
}
// runInfo implements the info subcommand: it prints the database's
// details on stdout (see writeInfo). It opens the database as report
// does, read-only and without the scan lock, and never touches the
// scanned filesystem. It names the database on stderr before it opens
// it. Any database problem, including a missing database, is fatal.
//
// The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and count (count the records,
// with one query). Each display ends before the next phase starts, and
// before an error is returned to be printed. The details are written
// once the last display has ended, so no display is drawn while they
// are written.
func runInfo(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr, "info: database %s (%s)\n",
escapePath(dbPath), source)
// Opening can wait on a busy database for as long as the busy
// timeout.
prog := newProgress(progressParams{
label: infoPhase(openPhase), total: -1,
waitingFor: openingDatabase,
})
db, err := openReportDatabase(ctx, dbPath)
prog.finish()
if err != nil {
return err
}
defer func() { _ = db.Close() }()
// One query counts everything, so the phase never has a total.
prog = newProgress(progressParams{
label: infoPhase("count"), total: -1,
waitingFor: countingRecords,
})
counts, err := readDatabaseCounts(ctx, db)
prog.finish()
if err != nil {
return fmt.Errorf("database %s: %w", dbPath, err)
}
return writeInfo(stdout, dbPath, source, counts)
}
// writeInfo writes the details of the database at dbPath to stdout,
// one per line, as a name, a tab and the value (README "`info` mode"):
// the path and source as databasePath returns them, the sizes of the
// database file and of its -wal file when there is one, when the
// database file was last changed, and the schema version and record
// counts that readDatabaseCounts read.
func writeInfo(stdout io.Writer, dbPath, source string,
counts databaseCounts,
) error {
// SQLite keeps the -wal file beside the database file, named from
// its path with symlinks resolved.
file, err := filepath.EvalSymlinks(dbPath)
if err != nil {
return fmt.Errorf("database: %w", err)
}
dbFile, err := os.Stat(file)
if err != nil {
return fmt.Errorf("database: %w", err)
}
walFile, err := os.Stat(file + "-wal")
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return fmt.Errorf("database: %w", err)
}
var b strings.Builder
fmt.Fprintf(&b, "database\t%s (%s)\n", escapePath(dbPath), source)
fmt.Fprintf(&b, "database file size\t%d (%s)\n",
dbFile.Size(), humanBytes(dbFile.Size()))
if walFile != nil {
fmt.Fprintf(&b, "-wal file size\t%d (%s)\n",
walFile.Size(), humanBytes(walFile.Size()))
}
fmt.Fprintf(&b, "modified\t%s\n",
dbFile.ModTime().UTC().Format(time.RFC3339))
fmt.Fprintf(&b, "schema version\t%d\n", counts.schemaVersion)
fmt.Fprintf(&b, "records\t%d\n", counts.records)
fmt.Fprintf(&b, "records with head and tail hashes\t%d\n",
counts.headTail)
fmt.Fprintf(&b, "records with content hash\t%d\n", counts.content)
fmt.Fprintf(&b, "size of recorded files\t%d (%s)\n",
counts.size, humanBytes(counts.size))
_, err = io.WriteString(stdout, b.String())
if err != nil {
return fmt.Errorf("write stdout: %w", err)
}
return nil
}
info_test.go
-168
View File
@@ -1,168 +0,0 @@
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// infoOpenLine is the plain line of info's open phase when it opens the
// database at once; it prints it as the phase starts and as it ends.
const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown"
// infoCountLine is the plain line of info's count phase when the count
// takes under a second; it prints it as the phase starts and as it
// ends.
const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " +
"eta unknown"
// infoRecs are four records: a duplicate pair with every hash, a file
// of 20 MiB with head and tail hashes but no content hash yet, and a
// file never hashed.
func infoRecs() []scanRec {
return []scanRec{
{size: 5, head: "h", tail: "t", content: "c", path: "/d/a"},
{size: 5, head: "h", tail: "t", content: "c", path: "/d/b"},
{size: 20 << 20, head: "bh", tail: "bt", path: "/d/big"},
{size: 7, path: "/d/unique"},
}
}
// TestRunInfo runs info on an empty and on a filled database, with
// stdout and stderr the same file that is not a terminal, and checks
// everything written there: the line naming the database, each phase's
// line as it starts and as it ends, and then the details, each on a
// line of its own. No -wal file is beside a database scan has closed.
func TestRunInfo(t *testing.T) {
cases := map[string]struct {
recs []scanRec
want string // the details after the database file's time
}{
"empty": {
want: "schema version\t1\n" +
"records\t0\n" +
"records with head and tail hashes\t0\n" +
"records with content hash\t0\n" +
"size of recorded files\t0 (0 B)\n",
},
"filled": {
recs: infoRecs(),
want: "schema version\t1\n" +
"records\t4\n" +
"records with head and tail hashes\t3\n" +
"records with content hash\t2\n" +
"size of recorded files\t20971537 (20.0 MiB)\n",
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
path := seedDatabase(t, tc.recs)
t.Setenv(databaseEnv, path)
file, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
stderr := captureStderr(t)
code := run([]string{cmdInfo}, os.Stderr, os.Stderr)
if code != exitOK {
t.Fatalf("run(info) = %d, want %d; stderr: %s",
code, exitOK, stderr())
}
named := path + " (from " + databaseEnv + ")\n"
want := "info: database " + named +
infoOpenLine + "\n" + infoOpenLine + "\n" +
infoCountLine + "\n" + infoCountLine + "\n" +
"database\t" + named +
fmt.Sprintf("database file size\t%d (%s)\n",
file.Size(), humanBytes(file.Size())) +
"modified\t" + file.ModTime().UTC().Format(time.RFC3339) +
"\n" + tc.want
if got := stderr(); got != want {
t.Errorf("stdout and stderr = %q, want %q", got, want)
}
})
}
}
// TestRunInfoDuringScan runs info while a scan holds the lock and has
// the database open in WAL mode, with records committed to the -wal
// file only. info must count those records and show the -wal file's
// size, also when the database path is a symlink: SQLite keeps the
// -wal file beside the symlink's target.
func TestRunInfoDuringScan(t *testing.T) {
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
db, err := openScanDatabase(t.Context(), path)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
err = applyChanges(t.Context(), db, infoRecs(), nil, nil)
if err != nil {
t.Fatal(err)
}
holdScanLock(t, path)
wal, err := os.Stat(path + "-wal")
if err != nil {
t.Fatal(err)
}
for _, named := range []string{path, link} {
t.Run(filepath.Base(named), func(t *testing.T) {
t.Setenv(databaseEnv, named)
stdout := runStdout(t, cmdInfo)
for _, want := range []string{
"database\t" + named + " (from " + databaseEnv + ")\n",
fmt.Sprintf("\n-wal file size\t%d (%s)\n",
wal.Size(), humanBytes(wal.Size())),
"\nrecords\t4\n",
} {
if !strings.Contains(stdout, want) {
t.Errorf("stdout = %q, want it to hold %q", stdout, want)
}
}
})
}
}
func TestRunInfoNeedsOnlyReadAccess(t *testing.T) {
// README "`info` mode": like report, info needs only read access to
// the database file and none to its directory.
path := testDBPath(t)
t.Setenv(databaseEnv, path)
scanFixture(t)
makeReadOnly(t, path)
stdout := runStdout(t, cmdInfo)
if want := "\nrecords\t2\n"; !strings.Contains(stdout, want) {
t.Errorf("stdout = %q, want it to hold %q", stdout, want)
}
if strings.Contains(stdout, "-wal") {
t.Errorf("stdout = %q, want no -wal file between scans", stdout)
}
}
main.go
+29 -55
View File
@@ -8,14 +8,13 @@
// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
// or larger. scan maintains a persistent SQLite database of file
// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
// that the other subcommands read.
// that the reporting subcommands read.
//
// Usage:
//
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
// sfdupes scan [--workers N] [-x] PATH...
// sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv
// sfdupes info
// sfdupes --version
//
// See README.md for the complete specification.
@@ -45,7 +44,6 @@ const (
cmdScan = "scan"
cmdReport = "report"
cmdTrees = "trees"
cmdInfo = "info"
)
// errNoSubcommand is returned by the root command when it is invoked
@@ -75,8 +73,8 @@ func main() {
// exit code. It is the program's single exit point: the subcommands
// return their errors instead of exiting, so every deferred cleanup —
// above all closing the database, which checkpoints the SQLite WAL —
// runs before the process ends. The report, trees and info subcommands
// write their data to stdout.
// runs before the process ends. The report and trees subcommands write
// their data to stdout.
func run(args []string, stdout, stderr io.Writer) int {
// A nil slice makes cobra fall back to os.Args, which would let a
// test binary's own flags reach the command tree.
@@ -150,6 +148,30 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
root.Flags().BoolVarP(&showVersion, "version", "v", false,
"print the version to stdout")
var (
scanWorkers int
scanOneFS bool
)
scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] PATH...",
Short: "Walk trees and synchronize the scan database",
Args: cobra.MinimumNArgs(1),
PreRunE: func(cmd *cobra.Command, _ []string) error {
return checkScanWorkers(cmd, scanWorkers)
},
RunE: runE(func(ctx context.Context, args []string) error {
ctx, stop := interruptContext(ctx)
defer stop()
return runScan(ctx, args, scanWorkers, scanOneFS)
}),
}
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
"concurrent workers for the walk, hash, and content phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
reportCmd := &cobra.Command{
Use: cmdReport,
Short: "Read the scan database and print the file-level duplicates report",
@@ -168,59 +190,11 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}),
}
infoCmd := &cobra.Command{
Use: cmdInfo,
Short: "Read the scan database and print its details",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runInfo(ctx, stdout)
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
root.AddCommand(scanCmd, reportCmd, treesCmd)
return root
}
// newScanCommand builds the scan subcommand, with its --workers, -x and
// --no-default-excludes flags. Its help lists the default excludes of
// the system it runs on, and the filesystem types it never records
// anything on.
func newScanCommand() *cobra.Command {
var (
scanWorkers int
scanOneFS bool
scanNoDefaultExcludes bool
)
short := "Walk trees and synchronize the scan database"
scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
Short: short,
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1),
PreRunE: func(cmd *cobra.Command, _ []string) error {
return checkScanWorkers(cmd, scanWorkers)
},
RunE: runE(func(ctx context.Context, args []string) error {
ctx, stop := interruptContext(ctx)
defer stop()
return runScan(ctx, args, scanWorkers, scanOneFS,
scanNoDefaultExcludes)
}),
}
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
"concurrent workers for the walk, hash, and content phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
return scanCmd
}
// checkScanWorkers rejects a scan --workers value below 1. That is a
// usage error reported in one line: cobra prints the returned message
// without the usage text, and run exits 2.
main_test.go
+20 -81
View File
@@ -4,7 +4,6 @@ import (
"bytes"
"context"
"errors"
"fmt"
"io"
"io/fs"
"os"
@@ -19,36 +18,6 @@ import (
// error prints and a runtime failure must not.
const usageMarker = "Usage:"
// TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail
// without saying why.
func TestMain(m *testing.M) {
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
fmt.Fprintf(os.Stderr, "list the mounted filesystems: %v\n", err)
os.Exit(1)
}
dir := os.TempDir()
fi, err := os.Stat(dir)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
dev, ok := deviceOfInfo(fi)
if fsType, found := nonPersistent[dev]; ok && found {
fmt.Fprintf(os.Stderr, "%s is on %s, not a persistent filesystem, "+
"and scan records nothing there: set TMPDIR to a directory on a "+
"persistent filesystem\n", dir, fsType)
os.Exit(1)
}
os.Exit(m.Run())
}
// walSuffixes are the SQLite sidecar files a WAL-mode database keeps
// while it is open. A clean close checkpoints the WAL and removes
// both; finding either afterwards means the database was never closed.
@@ -315,16 +284,10 @@ func assertFatalOutput(t *testing.T, stderr, stdout string) {
}
func TestRunMissingDatabaseIsFatal(t *testing.T) {
// README §Database: report, trees and info need an existing
// database; a missing one exits 1 with a message telling the user to
// run scan, after the line naming the database.
openLines := map[string]string{
cmdReport: reportOpenLine,
cmdTrees: treesOpenLine,
cmdInfo: infoOpenLine,
}
for name, openLine := range openLines {
// README §Database: report and trees need an existing database; a
// missing one exits 1 with a message telling the user to run scan,
// after the line naming the database.
for _, name := range []string{cmdReport, cmdTrees} {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
@@ -340,10 +303,8 @@ func TestRunMissingDatabaseIsFatal(t *testing.T) {
t.Errorf("run(%s) = %d, want %d", name, code, exitFatal)
}
// Each shows its open phase, as it starts and as it ends,
// before the error.
want := name + ": database " + path + " (from " + databaseEnv +
")\n" + openLine + "\n" + openLine + "\n" +
")\n" +
"sfdupes: " + path + ": no database (run \"sfdupes " +
"scan\" first, or set " + databaseEnv + ")\n"
if got := stderr(); got != want {
@@ -544,9 +505,8 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
// exist at the default path on the machine running the test, so
// nothing here may create or read one: scan gets an operand that
// does not exist, which stops it before it takes the lock, and
// report, trees and info a cancelled context, which stops them
// before they read anything. They show their open phase after the
// line.
// report and trees a cancelled context, which stops them before
// they read anything.
t.Setenv(databaseEnv, "")
// Whether scan calls the database new depends on that machine too.
@@ -566,7 +526,7 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}{
cmdScan: {
run: func() error {
return runScan(t.Context(), []string{missing}, 1, false, false)
return runScan(t.Context(), []string{missing}, 1, false)
},
want: scanWant,
},
@@ -574,22 +534,13 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
run: func() error {
return runReport(cancelledContext(t), io.Discard)
},
want: "report: database " + defaultDatabasePath + " (default)\n" +
reportOpenLine + "\n" + reportOpenLine + "\n",
want: "report: database " + defaultDatabasePath + " (default)\n",
},
cmdTrees: {
run: func() error {
return runTrees(cancelledContext(t), io.Discard)
},
want: "trees: database " + defaultDatabasePath + " (default)\n" +
treesOpenLine + "\n" + treesOpenLine + "\n",
},
cmdInfo: {
run: func() error {
return runInfo(cancelledContext(t), io.Discard)
},
want: "info: database " + defaultDatabasePath + " (default)\n" +
infoOpenLine + "\n" + infoOpenLine + "\n",
want: "trees: database " + defaultDatabasePath + " (default)\n",
},
}
@@ -617,7 +568,6 @@ func TestRunUsageErrors(t *testing.T) {
"scan without paths": {[]string{cmdScan}, usageMarker},
"report with args": {[]string{cmdReport, "x"}, usageMarker},
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
"unknown subcommand": {[]string{"nope"}, "unknown command"},
}
@@ -958,16 +908,11 @@ func TestRunReportSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want)
}
// Its phases come between the two lines; see
// TestRunReportProgressNamesEachPhase.
first := "report: database " + path + " (from " + databaseEnv + ")\n"
last := "\nreport: 2 records read, 1 duplicate groups, 1 dupe files, " +
want = "report: database " + path + " (from " + databaseEnv + ")\n" +
"report: 2 records read, 1 duplicate groups, 1 dupe files, " +
"300 B reclaimable\n"
if got := stderr(); !strings.HasPrefix(got, first) ||
!strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to start %q and end %q",
got, first, last)
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
}
assertNoSidecars(t, path)
@@ -997,16 +942,11 @@ func TestRunTreesSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want)
}
// Its phases come between the two lines; see
// TestRunTreesProgressNamesEachPhase.
first := "trees: database " + path + " (from " + databaseEnv + ")\n"
last := "\ntrees: 2 records read, 1 duplicate tree groups, " +
"1 dupe trees, 300 B reclaimable\n"
if got := stderr(); !strings.HasPrefix(got, first) ||
!strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to start %q and end %q",
got, first, last)
want = "trees: database " + path + " (from " + databaseEnv + ")\n" +
"trees: 2 records read, 1 duplicate tree groups, 1 dupe trees, " +
"300 B reclaimable\n"
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
}
assertNoSidecars(t, path)
@@ -1298,7 +1238,7 @@ func TestRunReportsDuringScan(t *testing.T) {
func TestRunStdoutClosedIsFatal(t *testing.T) {
// README §Error handling: a stdout write failure exits 1, reported
// in one line on stderr.
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
for _, name := range []string{cmdReport, cmdTrees} {
t.Run(name, func(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
@@ -1348,7 +1288,6 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
cases := map[string]func(context.Context, io.Writer) error{
cmdReport: runReport,
cmdTrees: runTrees,
cmdInfo: runInfo,
}
for name, fn := range cases {
progress.go
+48 -194
View File
@@ -2,7 +2,6 @@ package main
import (
"fmt"
"io"
"math"
"os"
"slices"
@@ -22,15 +21,6 @@ const plainInterval = 5 * time.Second
// a TTY.
const redrawInterval = 100 * time.Millisecond
// rateWindow is the time the rate a display shows covers: the items
// completed over the last rateWindow, which is the time since the line
// before when stderr is not a TTY.
const rateWindow = 5 * time.Second
// stallTime is how long a phase goes with no item completed before its
// line says it has stalled, and where it waits, in place of the rate.
const stallTime = 10 * time.Second
// spinnerType selects the progressbar library's spinner style.
const spinnerType = 14
@@ -56,18 +46,6 @@ const spinnerMargin = 5
// around them, and the terminal's last column.
const barMargin = barWidth + 11
// countingRecords is what the display of a phase that reads the records
// says while it counts them, before it has its total.
const countingRecords = "counting the records"
// openPhase is the name of the phase in which report, trees and info
// open the database, and openingDatabase what its display says while
// it does.
const (
openPhase = "open"
openingDatabase = "opening the database"
)
// stderrIsTTY reports whether stderr is attached to a terminal.
func stderrIsTTY() bool {
return term.IsTerminal(int(os.Stderr.Fd()))
@@ -94,18 +72,9 @@ func phaseLabel(phases []string, phase string) string {
// progressParams describe one phase's progress display as it starts.
type progressParams struct {
label string // the phase and its place, from phaseLabel
unit string // what the phase counts, as "files"
total int64 // the items the phase will complete; -1 when not known
estimate bool // total is an estimate, which the count may pass
waitingFor string // what the phase waits for until setTotal is called
stalledOn string // where the phase waits, as "disk/db"; see stallTime
}
// drawnCount is the count of items completed as of one draw, and when
// that draw was.
type drawnCount struct {
at time.Time
count int64
}
// progress renders one phase's progress on stderr. On a TTY the
@@ -121,59 +90,46 @@ type drawnCount struct {
// below it. Every method must be called from the main goroutine. A nil
// *progress is a valid no-display receiver: every method is a no-op, so
// batched database flushes during the hash phase can reuse the update
// phase's helpers without rendering anything, and its writer writes
// with no display to clear.
// phase's helpers without rendering anything.
type progress struct {
now func() time.Time // reads the time
width func() int // reads the terminal's width; nil when not a TTY
stop chan struct{} // closed by finish to end the draws
done chan struct{} // closed once the draws have ended
width func() int // reads the terminal's width; nil when not a TTY
stop chan struct{} // closed by finish to end the draws
done chan struct{} // closed once the draws have ended
mu sync.Mutex
bar *progressbar.ProgressBar // what the library draws; nil until then
withBar bool // bar is the library's bar, not its spinner
widest int // the widest line bar has drawn, in columns
label string
unit string
total int64
estimate bool
waitingFor string
stalledOn string
count int64
start time.Time // when the phase started, for the elapsed time
rateStart time.Time // when the count started, for the ETA
lastItem time.Time // when the last item was completed, or the count started
// drawn holds the count at each draw over the last rateWindow, oldest
// first; the rate counts from the first. It keeps the draw before the
// latest even when that is older, as the line before is when lines
// are plainInterval apart and a tick comes late.
drawn []drawnCount
rateStart time.Time // when the count started, for the rate
}
// newProgress starts the display of one phase on stderr.
func newProgress(params progressParams) *progress {
if stderrIsTTY() {
return startProgress(params, terminalWidth, redrawInterval, time.Now)
return startProgress(params, terminalWidth, redrawInterval)
}
return startProgress(params, nil, plainInterval, time.Now)
return startProgress(params, nil, plainInterval)
}
// startProgress shows the phase params describes at once, and again
// every interval until finish: with the progressbar library on a
// terminal, whose width each draw reads by calling width, or as a plain
// line when width is nil. The display reads the time by calling now.
// line when width is nil.
func startProgress(params progressParams, width func() int,
interval time.Duration, now func() time.Time,
interval time.Duration,
) *progress {
start := now()
now := time.Now()
p := &progress{
now: now, width: width,
stop: make(chan struct{}), done: make(chan struct{}),
label: params.label, unit: params.unit, total: params.total,
estimate: params.estimate, waitingFor: params.waitingFor,
stalledOn: params.stalledOn, start: start, rateStart: start,
lastItem: start, drawn: []drawnCount{{at: start}},
width: width, stop: make(chan struct{}), done: make(chan struct{}),
label: params.label, total: params.total, estimate: params.estimate,
waitingFor: params.waitingFor, start: now, rateStart: now,
}
// Show the zero state at once: the first item may take minutes,
@@ -200,42 +156,32 @@ func startProgress(params progressParams, width func() int,
}
// draw shows the phase's progress: on a TTY in place of the line
// before, and otherwise as a plain line. It first adds the count to
// drawn, then drops from drawn the draws older than rateWindow, except
// the one before this draw.
// before, and otherwise as a plain line.
func (p *progress) draw() {
p.mu.Lock()
defer p.mu.Unlock()
now := p.now()
p.drawn = append(p.drawn, drawnCount{at: now, count: p.count})
for len(p.drawn) > 2 && now.Sub(p.drawn[0].at) > rateWindow {
p.drawn = p.drawn[1:]
}
if p.width == nil {
fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true))
text, _ := p.line(math.MaxInt, true)
fmt.Fprintln(os.Stderr, text)
return
}
width := p.width()
// A phase with an exact total gets the library's bar when its whole
// line, without the percent the bar shows, fits beside the bar: the
// bar gives way before any part of the line. A total of 0 leaves the
// bar nothing to fill.
// A phase with an exact total gets the library's bar when its line,
// without the percent the bar shows, fits whole beside the bar. A
// total of 0 leaves the bar nothing to fill.
text, withBar := "", false
if p.total > 0 && !p.estimate {
text = p.line(now, math.MaxInt, false)
withBar = len(text) <= width-barMargin
text, withBar = p.line(width-barMargin, false)
}
margin := barMargin
if !withBar {
margin = spinnerMargin
text = p.line(now, width-margin, true)
text, _ = p.line(width-margin, true)
}
// The library makes a bar or a spinner, and before each draw erases
@@ -292,22 +238,18 @@ func (p *progress) newBar(withBar bool, width int) {
}
// setTotal gives a phase that started without a total its total, once
// it is known. The rate, the ETA and the time with no item completed
// count from then on: the time spent waiting for the total says nothing
// about how fast items go.
// it is known. The rate, and so the ETA, count from then on: the time
// spent waiting for the total says nothing about how fast items go.
func (p *progress) setTotal(total int64) {
if p == nil {
return
}
now := p.now()
p.mu.Lock()
defer p.mu.Unlock()
p.total = total
p.rateStart, p.lastItem = now, now
p.drawn = []drawnCount{{at: now, count: p.count}}
p.rateStart = time.Now()
}
// increment records one completed item, which the display shows at its
@@ -317,13 +259,10 @@ func (p *progress) increment() {
return
}
now := p.now()
p.mu.Lock()
defer p.mu.Unlock()
p.count++
p.lastItem = now
}
// warnf prints a one-line warning to stderr without corrupting the
@@ -348,73 +287,6 @@ func (p *progress) warnf(format string, args ...any) {
fmt.Fprintln(os.Stderr, msg)
}
// progressWriter writes a phase's output to out, which may be the file
// the phase's display is drawn on, as stdout is in a plain "sfdupes
// trees" on a terminal. It holds what is written to it and writes that
// to out in pieces of about ioBufSize bytes. When out is the file
// stderr is, each piece is written under the display's lock with the
// display cleared first, as warnf does for a warning: no draw lands in
// the middle of a piece, and the next draw shows the display again, on
// the line below. Every write to it must end with a newline, as a row
// written with one fmt.Fprintf does, so that each piece ends a line.
// Flush writes what it holds once the last row is written.
type progressWriter struct {
// p is the display to clear, nil unless out is the file stderr is.
p *progress
out io.Writer
buf []byte
}
// writer returns a progressWriter writing to out around p's display.
// It clears the display and takes its lock only when out is the file
// stderr is: the same device and inode, as os.SameFile compares them,
// which is one terminal, or one pipe or file that both are sent to.
// Otherwise it writes to out without the lock, so the display keeps
// being drawn while a write to out waits for a slow reader.
func (p *progress) writer(out io.Writer) *progressWriter {
w := &progressWriter{out: out}
if f, ok := out.(*os.File); ok {
outInfo, outErr := f.Stat()
errInfo, errErr := os.Stderr.Stat()
if outErr == nil && errErr == nil && os.SameFile(outInfo, errInfo) {
w.p = p
}
}
return w
}
// Write holds b, and writes what it holds to out once that comes to
// ioBufSize bytes.
func (w *progressWriter) Write(b []byte) (int, error) {
w.buf = append(w.buf, b...)
if len(w.buf) < ioBufSize {
return len(b), nil
}
return len(b), w.Flush()
}
// Flush writes what w holds to out. When out is the file stderr is, the
// display is cleared first and not drawn again until it is written.
func (w *progressWriter) Flush() error {
if w.p != nil {
w.p.mu.Lock()
defer w.p.mu.Unlock()
if w.p.bar != nil {
_ = w.p.bar.Clear()
}
}
_, err := w.out.Write(w.buf)
w.buf = w.buf[:0]
return err
}
// finish ends the phase's display, drawn once more as the phase ends.
// A phase stopped short of its total, as an interrupted one is, shows
// the count it stopped at rather than a full bar.
@@ -435,77 +307,59 @@ func (p *progress) finish() {
}
}
// line formats the phase's progress as of now in at most room columns,
// as
// line formats the phase's progress in at most room columns, as
//
// [3/5] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s
//
// where "files" is the phase's unit, without the percent when percent
// is false, as before the library's bar, which shows its own. A phase
// with no total shows the items done instead, or what it waits for. The
// rate is that since the first draw in drawn, and the ETA comes from
// the rate since the count started. Once no item has been completed for
// stallTime, the rate, which would read 0, gives way to the word
// "stalled" and where the phase waits, as "stalled disk/db".
// without the percent when percent is false, as before the library's
// bar, which shows its own. A phase with no total shows the items done
// instead, or what it waits for.
//
// The phase and its place, the progress, the elapsed time and the ETA
// are kept whole. A line wider than room leaves out the rate, the words
// of a stalled phase, or what the phase waits for; only if it is still
// wider is it cut at room. Every part is ASCII, so its length is its
// width in columns. p.mu must be held.
func (p *progress) line(now time.Time, room int, percent bool) string {
elapsed := now.Sub(p.start).Round(time.Second)
// are kept whole. A line wider than room leaves out the rate, or what
// the phase waits for; only if it is still wider is it cut at room, and
// line reports that it does not fit. Every part is ASCII, so its length
// is its width in columns. p.mu must be held.
func (p *progress) line(room int, percent bool) (string, bool) {
elapsed := time.Since(p.start).Round(time.Second)
// head and tail hold the parts kept whole; extra, between them, is
// the part left out.
var head, extra, tail string
if p.total < 0 && p.waitingFor != "" {
head, extra = p.label+": ", p.waitingFor+", "
head, extra = p.label+": ", p.waitingFor
tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed)
} else {
since := p.drawn[0]
rate := perSecond(p.count-since.count, now.Sub(since.at))
average := perSecond(p.count, now.Sub(p.rateStart))
rate := 0.0
if s := time.Since(p.rateStart).Seconds(); s > 0 {
rate = float64(p.count) / s
}
head = p.label + ": " + p.itemsDone(percent) + ", "
extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit)
tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average))
if now.Sub(p.lastItem) >= stallTime {
extra = "stalled " + p.stalledOn + ", "
}
extra = fmt.Sprintf("%.0f files/s", rate)
tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(rate))
}
full, kept := head+extra+tail, head+tail
full, kept := head+extra+", "+tail, head+tail
switch {
case len(full) <= room:
return full
return full, true
case len(kept) <= room:
return kept
return kept, true
}
return kept[:max(room, 0)]
}
// perSecond returns count items over d as items a second, or 0 when no
// time has passed.
func perSecond(count int64, d time.Duration) float64 {
if d <= 0 {
return 0
}
return float64(count) / d.Seconds()
return kept[:max(room, 0)], false
}
// itemsDone formats the items done, as "[1234/5000] 25%", with "~"
// before a total that is an estimate and without the percent when
// percent is false, or as "1234 files", in the phase's unit, when the
// phase has no total. p.mu must be held.
// percent is false, or as "1234 files" when the phase has no total.
// p.mu must be held.
func (p *progress) itemsDone(percent bool) string {
if p.total < 0 {
return fmt.Sprintf("%d %s", p.count, p.unit)
return fmt.Sprintf("%d files", p.count)
}
approx := ""
progress_test.go
+65 -499
View File
@@ -1,14 +1,10 @@
package main
import (
"fmt"
"io"
"math"
"os"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"unicode/utf8"
@@ -22,33 +18,6 @@ const testRedraw = time.Millisecond
// it expects.
const testDeadline = 5 * time.Second
// hashStalledOn is where the hash phase says it waits once it has
// stalled, as readRuns gives it.
const hashStalledOn = "disk/db"
// testClock is a clock a test moves by hand, for a display that reads
// it from its own goroutine too.
type testClock struct {
mu sync.Mutex
now time.Time
}
// Now returns the clock's time.
func (c *testClock) Now() time.Time {
c.mu.Lock()
defer c.mu.Unlock()
return c.now
}
// Add moves the clock on by d.
func (c *testClock) Add(d time.Duration) {
c.mu.Lock()
defer c.mu.Unlock()
c.now = c.now.Add(d)
}
//nolint:paralleltest // replaces the process-wide os.Stderr
func TestStderrIsTTYFalseForNonTerminals(t *testing.T) {
r, pipe, err := os.Pipe()
@@ -101,13 +70,11 @@ func TestPhaseLabel(t *testing.T) {
}
// TestProgressLine checks the line a display shows in each state, ten
// seconds into the phase, with no draw since the first and the last
// item just completed.
// seconds into the phase.
func TestProgressLine(t *testing.T) {
t.Parallel()
now := time.Now()
ago := now.Add(-10 * time.Second)
ago := time.Now().Add(-10 * time.Second)
cases := []struct {
p *progress
@@ -115,165 +82,120 @@ func TestProgressLine(t *testing.T) {
}{{
p: &progress{
label: scanPhase("load"), total: -1,
waitingFor: countingRecords,
waitingFor: "counting the records",
},
want: "[1/5] load: counting the records, elapsed 10s, eta unknown",
}, {
p: &progress{
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
},
p: &progress{label: scanPhase("hash"), total: 100, count: 20},
want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
}, {
p: &progress{label: scanPhase("hash"), unit: scanUnit, total: 100},
p: &progress{label: scanPhase("hash"), total: 100},
want: "[3/5] hash: [0/100] 0%, 0 files/s, elapsed 10s, eta unknown",
}, {
p: &progress{label: scanPhase("update"), unit: scanUnit, total: 0},
p: &progress{label: scanPhase("update"), total: 0},
want: "[4/5] update: [0/0] 100%, 0 files/s, elapsed 10s, eta 0s",
}, {
p: &progress{
label: scanPhase("walk"), unit: scanUnit, total: 100,
estimate: true, count: 20,
label: scanPhase("walk"), total: 100, estimate: true, count: 20,
},
want: "[2/5] walk: [20/~100] 20%, 2 files/s, elapsed 10s, eta 40s",
}, {
// Past the estimate the walk cannot tell how long it has left.
p: &progress{
label: scanPhase("walk"), unit: scanUnit, total: 100,
estimate: true, count: 120,
label: scanPhase("walk"), total: 100, estimate: true, count: 120,
},
want: "[2/5] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " +
"eta unknown",
}, {
// A first scan's walk has no estimate.
p: &progress{
label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20,
},
p: &progress{label: scanPhase("walk"), total: -1, count: 20},
want: "[2/5] walk: 20 files, 2 files/s, elapsed 10s, eta unknown",
}, {
// The rate is in what the phase counts.
p: &progress{
label: treesPhase("group"), unit: "directories", total: 100,
count: 20,
},
want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " +
"eta 40s",
}}
for _, tc := range cases {
tc.p.start, tc.p.rateStart, tc.p.lastItem = ago, ago, now
tc.p.drawn = []drawnCount{{at: ago}}
tc.p.start, tc.p.rateStart = ago, ago
if got := tc.p.line(now, math.MaxInt, true); got != tc.want {
if got, _ := tc.p.line(math.MaxInt, true); got != tc.want {
t.Errorf("line = %q, want %q", got, tc.want)
}
}
}
// TestLineFitsRoom checks how a line is fitted to the room it has, ten
// seconds into the phase, and fifty into a phase with no file read for
// the last 42: it leaves out the rate, what the phase waits for, or the
// words of a stalled phase, and is cut only when what is left is still
// too wide.
// seconds into the phase: it leaves out the rate, or what the phase
// waits for, and is cut only when what is left is still too wide.
func TestLineFitsRoom(t *testing.T) {
t.Parallel()
now := time.Now()
ago := now.Add(-10 * time.Second)
ago := time.Now().Add(-10 * time.Second)
hash := &progress{
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
start: ago, rateStart: ago, lastItem: now,
drawn: []drawnCount{{at: ago}},
label: scanPhase("hash"), total: 100, count: 20,
start: ago, rateStart: ago,
}
content := &progress{
label: scanPhase("content"), total: -1,
waitingFor: "searching the database", start: ago, rateStart: ago,
}
started := now.Add(-50 * time.Second)
stalled := &progress{
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
stalledOn: hashStalledOn, start: started, rateStart: started,
lastItem: now.Add(-42 * time.Second),
drawn: []drawnCount{{at: now.Add(-rateWindow), count: 20}},
}
cases := []struct {
p *progress
room int
percent bool
want string
fits bool
}{{
p: hash, room: 57, percent: true,
want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
fits: true,
}, {
p: hash, room: 56, percent: true,
want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40s",
fits: true,
}, {
p: hash, room: 45, percent: true,
want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40",
fits: false,
}, {
// Before the library's bar, which shows the percent.
p: hash, room: 57, percent: false,
want: "[3/5] hash: [20/100], 2 files/s, elapsed 10s, eta 40s",
fits: true,
}, {
p: content, room: 40, percent: true,
want: "[5/5] content: elapsed 10s, eta unknown",
}, {
p: stalled, room: 65, percent: true,
want: "[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " +
"eta 3m20s",
}, {
p: stalled, room: 64, percent: true,
want: "[3/5] hash: [20/100] 20%, elapsed 50s, eta 3m20s",
fits: true,
}}
for _, tc := range cases {
if got := tc.p.line(now, tc.room, tc.percent); got != tc.want {
t.Errorf("line(%d, %t) = %q, want %q",
tc.room, tc.percent, got, tc.want)
got, fits := tc.p.line(tc.room, tc.percent)
if got != tc.want || fits != tc.fits {
t.Errorf("line(%d, %t) = %q, %t, want %q, %t",
tc.room, tc.percent, got, fits, tc.want, tc.fits)
}
}
}
// TestSetTotalStartsRate checks that setTotal starts a phase's rate,
// its ETA and its time with no record read from when the total becomes
// known, not from when the phase started waiting for it: after 20
// seconds of waiting for the total, the phase has not stalled 5 seconds
// after it, and has 10 seconds after it with no record read.
// and so its ETA, from when the total becomes known, not from when the
// phase started waiting for it.
func TestSetTotalStartsRate(t *testing.T) {
t.Parallel()
clock := &testClock{now: time.Now()}
waited := clock.Now().Add(-20 * time.Second)
waited := time.Now().Add(-20 * time.Second)
p := &progress{
now: clock.Now, label: scanPhase("load"), unit: scanUnit, total: -1,
stalledOn: "db", start: waited, rateStart: waited,
lastItem: waited, drawn: []drawnCount{{at: waited}},
label: scanPhase("load"), total: -1, start: waited, rateStart: waited,
}
p.setTotal(100)
clock.Add(5 * time.Second)
// Ten seconds after the total came, without waiting ten seconds,
// 20 records are read.
p.start = p.start.Add(-10 * time.Second)
p.rateStart = p.rateStart.Add(-10 * time.Second)
p.count = 20
want := "[1/5] load: [0/100] 0%, 0 files/s, elapsed 25s, eta unknown"
if got := p.line(clock.Now(), math.MaxInt, true); got != want {
t.Errorf("line = %q, want %q", got, want)
}
clock.Add(5 * time.Second)
want = "[1/5] load: [0/100] 0%, stalled db, elapsed 30s, eta unknown"
if got := p.line(clock.Now(), math.MaxInt, true); got != want {
t.Errorf("line = %q, want %q", got, want)
}
// Then 20 records are read.
for range 20 {
p.increment()
}
want = "[1/5] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s"
if got := p.line(clock.Now(), math.MaxInt, true); got != want {
want := "[1/5] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s"
if got, _ := p.line(math.MaxInt, true); got != want {
t.Errorf("line = %q, want %q", got, want)
}
}
@@ -304,9 +226,7 @@ func awaitOutput(t *testing.T, stderr func() string, want string,
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestPlainProgressPrintsAtStartAndEnd(t *testing.T) {
stderr := captureStderr(t)
p := newProgress(progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 1000,
})
p := newProgress(progressParams{label: scanPhase("hash"), total: 1000})
for range 1000 {
p.increment()
@@ -332,138 +252,14 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) {
stderr := captureStderr(t)
p := startProgress(progressParams{
label: scanPhase("load"), total: -1,
waitingFor: countingRecords,
}, nil, testRedraw, time.Now)
waitingFor: "counting the records",
}, nil, testRedraw)
awaitOutput(t, stderr,
"[1/5] load: counting the records, elapsed 0s, eta unknown\n", 3)
p.finish()
}
// TestPlainProgressStalledProducer checks that a phase whose files stop
// coming, as when one large file is being read or a disk is slow, goes
// on printing its line every interval: the rate falls to 0 once no file
// has been read for rateWindow, and once none has for stallTime the
// line says the phase has stalled, and where it waits, in place of the
// rate. The test moves the display's clock by hand.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestPlainProgressStalledProducer(t *testing.T) {
stderr := captureStderr(t)
clock := &testClock{now: time.Now()}
p := startProgress(progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 100,
stalledOn: hashStalledOn,
}, nil, testRedraw, clock.Now)
// Pass or fail, the display ends before stderr is put back.
t.Cleanup(p.finish)
// 20 files read as the phase starts, then none.
for range 20 {
p.increment()
}
clock.Add(5 * time.Second)
awaitOutput(t, stderr,
"[3/5] hash: [20/100] 20%, 4 files/s, elapsed 5s, eta 20s\n", 3)
clock.Add(time.Second)
awaitOutput(t, stderr,
"[3/5] hash: [20/100] 20%, 0 files/s, elapsed 6s, eta 24s\n", 3)
clock.Add(4 * time.Second)
awaitOutput(t, stderr,
"[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 10s, eta 40s\n",
3)
}
// TestPlainProgressRateAfterStall draws a phase's plain line by hand, a
// little over plainInterval after the line before, as a late tick
// does, through a stall and a burst of files after it: the rate covers
// the files read since the line before, while the ETA comes from the
// phase's average.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestPlainProgressRateAfterStall(t *testing.T) {
stderr := captureStderr(t)
clock := &testClock{now: time.Now()}
// No timed draws: the test draws.
p := startProgress(progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 1000,
stalledOn: hashStalledOn,
}, nil, time.Hour, clock.Now)
late := plainInterval + time.Millisecond
clock.Add(late)
p.draw()
clock.Add(late)
p.draw()
for range 100 {
p.increment()
}
clock.Add(late)
p.finish()
want := []string{
"[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown",
"[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 5s, eta unknown",
"[3/5] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, eta unknown",
"[3/5] hash: [100/1000] 10%, 20 files/s, elapsed 15s, eta 2m15s",
}
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
if !slices.Equal(lines, want) {
t.Errorf("stderr lines = %q, want %q", lines, want)
}
}
// TestProgressRateAfterStallRedrawn draws a phase by hand every
// redrawInterval, as a terminal does, through a stall of stallTime and
// a burst of files after it: the rate covers the files read over the
// last rateWindow, and falls to 0 once the burst is older than that,
// while the ETA comes from the phase's average. Plain lines show each
// draw.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressRateAfterStallRedrawn(t *testing.T) {
stderr := captureStderr(t)
clock := &testClock{now: time.Now()}
// No timed draws: the test draws.
p := startProgress(progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 1000,
stalledOn: hashStalledOn,
}, nil, time.Hour, clock.Now)
// redraw draws the display every redrawInterval for d.
redraw := func(d time.Duration) {
for range d / redrawInterval {
clock.Add(redrawInterval)
p.draw()
}
}
redraw(stallTime)
for range 50 {
p.increment()
}
redraw(rateWindow + redrawInterval)
p.finish()
want := "[3/5] hash: [50/1000] 5%, 10 files/s, elapsed 15s, eta 4m45s\n" +
"[3/5] hash: [50/1000] 5%, 0 files/s, elapsed 15s, eta 4m47s\n"
if !strings.Contains(stderr(), want) {
t.Errorf("stderr = %q, want the lines %q", stderr(), want)
}
}
// lastFrame returns what a terminal shows of the frames a display drew:
// the last one. The library starts each frame with a carriage return
// and erases the previous one with spaces first.
@@ -493,20 +289,14 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) {
params progressParams
want string // how the line shows a count of 50
}{
{
progressParams{label: scanPhase("walk"), unit: scanUnit, total: -1},
"50 files,",
},
{
progressParams{label: scanPhase("hash"), unit: scanUnit, total: 100},
"[50/100],",
},
{progressParams{label: scanPhase("walk"), total: -1}, "50 files,"},
{progressParams{label: scanPhase("hash"), total: 100}, "[50/100],"},
}
for _, tc := range cases {
before := len(stderr())
since := func() string { return stderr()[before:] }
p := startProgress(tc.params, terminalWidth, testRedraw, time.Now)
p := startProgress(tc.params, terminalWidth, testRedraw)
for range 50 {
p.increment()
@@ -528,23 +318,21 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) {
// drawLate draws the display params describes on a terminal width
// columns wide as it starts and once more as it ends, with count items
// done two and a half hours into the phase, the last just now, and
// returns what it wrote to stderr.
// done two and a half hours into the phase, and returns what it wrote
// to stderr.
func drawLate(stderr func() string, params progressParams, width int,
count int64,
) string {
before := len(stderr())
clock := &testClock{now: time.Now()}
// No timed draws come between the two.
p := startProgress(params, func() int { return width }, time.Hour,
clock.Now)
p := startProgress(params, func() int { return width }, time.Hour)
// Without counting the items one by one or waiting the hours.
clock.Add(150 * time.Minute)
p.mu.Lock()
p.count, p.lastItem = count, clock.Now()
p.count = count
p.start = p.start.Add(-150 * time.Minute)
p.rateStart = p.start
p.mu.Unlock()
p.finish()
@@ -553,9 +341,8 @@ func drawLate(stderr func() string, params progressParams, width int,
}
// TestTerminalLinesKeepETA checks the line each display shows on an
// 80-column terminal at a large scan's counts and times, and the line
// of trees' read phase at the same counts: the rate is left out, so
// that the elapsed time and the ETA stay whole.
// 80-column terminal at a large scan's counts and times: the rate is
// left out, so that the elapsed time and the ETA stay whole.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalLinesKeepETA(t *testing.T) {
@@ -567,29 +354,18 @@ func TestTerminalLinesKeepETA(t *testing.T) {
}{{
// Past its estimate, the walk cannot tell how long it has left.
params: progressParams{
label: scanPhase("walk"), unit: scanUnit, total: 1200000,
estimate: true,
label: scanPhase("walk"), total: 1200000, estimate: true,
},
want: "[2/5] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " +
"eta unknown",
}, {
params: progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
},
params: progressParams{label: scanPhase("hash"), total: 9876543},
want: "[3/5] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s",
}, {
params: progressParams{
label: scanPhase("content"), unit: scanUnit, total: 9876543,
},
params: progressParams{label: scanPhase("content"), total: 9876543},
want: "[5/5] content: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s",
}, {
params: progressParams{
label: treesPhase("read"), unit: "records", total: 9876543,
},
want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s",
}}
for _, tc := range cases {
@@ -600,85 +376,6 @@ func TestTerminalLinesKeepETA(t *testing.T) {
}
}
// TestTerminalStalledLines draws every phase's line on an 80-column
// terminal once the phase has stalled, its last item done 42 seconds
// before. At an ordinary scan's counts and times, 12345 of 45678 items
// done four and a half minutes in, the line shows where the phase waits
// whole, beside the phase, progress, elapsed time and ETA. At a large
// scan's, 1234567 of 9876543 two and a half hours in, those four still
// show whole, as the words give way first where they do not fit.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalStalledLines(t *testing.T) {
stderr := captureStderr(t)
// drawStalled draws the display params describes as it starts and
// once more as it ends, with count of total items done elapsed into
// the phase, and returns what the terminal shows.
drawStalled := func(params progressParams, total, count int64,
elapsed time.Duration,
) string {
before := len(stderr())
clock := &testClock{now: time.Now()}
params.total = total
// No timed draws come between the two.
p := startProgress(params, func() int { return 80 }, time.Hour,
clock.Now)
clock.Add(elapsed - 42*time.Second)
p.mu.Lock()
p.count, p.lastItem = count, clock.Now()
p.mu.Unlock()
clock.Add(42 * time.Second)
p.finish()
return lastFrame(stderr()[before:])
}
phases := []progressParams{
{label: scanPhase("load"), stalledOn: "db"},
{label: scanPhase("walk"), estimate: true, stalledOn: "disk"},
{label: scanPhase("hash"), stalledOn: hashStalledOn},
{label: scanPhase("update"), stalledOn: "db"},
// The content phase's checks and reads, then its last write.
{label: scanPhase("content"), stalledOn: "disk/db"},
{label: scanPhase("content"), stalledOn: "db"},
{label: treesPhase("read"), stalledOn: "db"},
{label: treesPhase("group"), stalledOn: "cpu"},
{label: treesPhase("write"), stalledOn: "out"},
{label: reportPhase("read"), stalledOn: "db/out"},
}
for _, params := range phases {
approx := ""
if params.estimate {
approx = "~"
}
shown := drawStalled(params, 45678, 12345,
4*time.Minute+30*time.Second)
want := params.label + ": [12345/" + approx + "45678] 27%, stalled " +
params.stalledOn + ", elapsed 4m30s, eta 12m9s"
if !strings.Contains(shown, want) {
t.Errorf("terminal shows %q, want %q", shown, want)
}
shown = drawStalled(params, 9876543, 1234567, 150*time.Minute)
for _, want := range []string{
params.label + ": [1234567/" + approx + "9876543] 12%, ",
", elapsed 2h30m0s, eta 17h30m0s",
} {
if !strings.Contains(shown, want) {
t.Errorf("terminal shows %q, want %q", shown, want)
}
}
}
}
// TestTerminalLinesFitWidth draws a large scan's bar display and a small
// scan's, two and a half hours into the phase, on terminals from 120
// columns down to 40, and checks that every line stays clear of the
@@ -692,9 +389,7 @@ func TestTerminalLinesFitWidth(t *testing.T) {
for _, width := range []int{120, 100, 80, 72, 60, 50, 40} {
for _, total := range []int64{9876543, 20000} {
params := progressParams{
label: scanPhase("content"), unit: scanUnit, total: total,
}
params := progressParams{label: scanPhase("content"), total: total}
out := drawLate(stderr, params, width, total/8)
if width == 120 && !strings.Contains(out, "% |") {
@@ -721,7 +416,7 @@ func TestTerminalLinesFitWidth(t *testing.T) {
// times. Nothing written after that, the erase before each redraw
// included, may reach the narrower terminal's last column: it would
// wrap, and every redraw would leave a line behind. At these widths the
// spinner stays a spinner and the bar gives way to the spinner.
// spinner stays a spinner and the bar stays a bar.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
@@ -731,30 +426,20 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
params progressParams
narrow int
}{
{
progressParams{label: scanPhase("walk"), unit: scanUnit, total: -1},
60,
},
{
progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
},
90,
},
{progressParams{label: scanPhase("walk"), total: -1}, 60},
{progressParams{label: scanPhase("hash"), total: 9876543}, 90},
}
for _, tc := range cases {
width := 120
clock := &testClock{now: time.Now()}
// No timed draws: the test draws.
p := startProgress(tc.params, func() int { return width }, time.Hour,
clock.Now)
clock.Add(150 * time.Minute)
p := startProgress(tc.params, func() int { return width }, time.Hour)
p.mu.Lock()
p.count, p.lastItem = 1234567, clock.Now()
p.count = 1234567
p.start = p.start.Add(-150 * time.Minute)
p.rateStart = p.start
p.mu.Unlock()
p.draw()
@@ -796,7 +481,7 @@ func TestProgressWarningsOnOwnLines(t *testing.T) {
issued := 0
for _, params := range cases {
p := startProgress(params, terminalWidth, testRedraw, time.Now)
p := startProgress(params, terminalWidth, testRedraw)
// No pause between warnings: one written straight to stderr is
// garbled only if a redraw lands while it is being written.
@@ -828,125 +513,6 @@ func TestProgressWarningsOnOwnLines(t *testing.T) {
}
}
// TestProgressWriterRowsOnOwnLines drives the terminal displays,
// spinner and bar, through a run of rows written through the display's
// writer to the same terminal, as stdout is in a plain "sfdupes trees",
// each written out at once, for many redraws: the terminal must show
// every row whole, in order, on a line of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressWriterRowsOnOwnLines(t *testing.T) {
stderr := captureStderr(t)
cases := []progressParams{
{
label: treesPhase("open"), total: -1,
waitingFor: "opening the database",
},
{label: treesPhase("write"), unit: "groups", total: 10},
}
var want []string
for _, params := range cases {
p := startProgress(params, terminalWidth, testRedraw, time.Now)
out := p.writer(os.Stderr)
// No pause between rows: one is garbled only if a redraw lands
// on its line.
for start := time.Now(); time.Since(start) < 50*testRedraw; {
row := fmt.Sprintf("row %d", len(want))
want = append(want, row)
_, err := fmt.Fprintln(out, row)
if err == nil {
err = out.Flush()
}
if err != nil {
t.Fatal(err)
}
}
p.finish()
}
// A terminal shows each line as the text after its last carriage
// return.
var shown []string
for line := range strings.SplitSeq(stderr(), "\n") {
if strings.Contains(line, "row ") {
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
}
}
if len(shown) != len(want) {
t.Fatalf("terminal shows %d rows, want %d", len(shown), len(want))
}
for i := range want {
if shown[i] != want[i] {
t.Fatalf("terminal shows %q, want %q", shown[i], want[i])
}
}
}
// TestProgressWriterStalledStdoutKeepsLines writes a row larger than a
// pipe holds through the display's writer to a pipe nobody reads yet,
// as stdout is when its reader has stopped while stderr goes to a file:
// while the write waits, the display must go on printing its line.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressWriterStalledStdoutKeepsLines(t *testing.T) {
stderr := captureStderr(t)
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() {
_ = w.Close()
_ = r.Close()
})
p := startProgress(progressParams{label: treesPhase("write"), total: 1},
nil, testRedraw, time.Now)
out := p.writer(w)
written := make(chan error, 1)
go func() {
_, writeErr := fmt.Fprintln(out, strings.Repeat("x", ioBufSize))
written <- writeErr
}()
// Pass or fail, the pipe is then read so that the write ends, and
// the display ends before stderr is put back.
t.Cleanup(func() {
go func() {
_, _ = io.Copy(io.Discard, r)
}()
writeErr := <-written
if writeErr != nil {
t.Error(writeErr)
}
p.finish()
})
// The first byte read shows that the write has started; the rest
// of the row waits until the pipe is read.
_, err = r.Read(make([]byte, 1))
if err != nil {
t.Fatal(err)
}
line := "[4/4] write: "
awaitOutput(t, stderr, line, strings.Count(stderr(), line)+3)
}
// TestTerminalStoppedShortKeepsCount checks that the terminal display
// of a phase that stops before its total, as an interrupted one does,
// is left showing the count it stopped at rather than a full bar.
@@ -955,7 +521,7 @@ func TestProgressWriterStalledStdoutKeepsLines(t *testing.T) {
func TestTerminalStoppedShortKeepsCount(t *testing.T) {
stderr := captureStderr(t)
p := startProgress(progressParams{label: scanPhase("hash"), total: 10},
terminalWidth, redrawInterval, time.Now)
terminalWidth, redrawInterval)
p.increment()
p.increment()
@@ -974,7 +540,7 @@ func TestTerminalStoppedShortKeepsCount(t *testing.T) {
func TestWarningAfterLastItemKeepsBar(t *testing.T) {
stderr := captureStderr(t)
p := startProgress(progressParams{label: scanPhase("hash"), total: 2},
terminalWidth, testRedraw, time.Now)
terminalWidth, testRedraw)
p.increment()
p.increment()
report.go
+46 -190
View File
@@ -1,12 +1,11 @@
package main
import (
"bufio"
"context"
"database/sql"
"fmt"
"io"
"os"
"slices"
"strings"
"time"
)
@@ -30,25 +29,13 @@ type scanRec struct {
path string
}
// reportPhase names one phase of report with its place among all of
// them, as "[2/2] read", for the phase's progress display. The list
// here is the one place the phases are counted.
func reportPhase(name string) string {
return phaseLabel([]string{openPhase, "read"}, name)
}
// runReport implements the report subcommand: it prints the file-level
// duplicates report as TSV on stdout. It never touches the scanned
// filesystem; its only I/O is the database, stdout, and stderr. It
// names the database on stderr before it opens it. Any database
// duplicates report as TSV on stdout. SQLite groups and orders the
// records, and each row is written as it is read, so no group is held
// in memory. It never touches the scanned filesystem; its only I/O is
// the database (with SQLite's temporary sort file), stdout, and stderr.
// It names the database on stderr before it opens it. Any database
// problem, including a missing database, is fatal.
//
// The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and read (count the records,
// then read them in signature order, writing each size's duplicate
// groups to stdout once that size is read; see readReport). Each
// display ends before the next phase starts, and before an error is
// returned to be printed.
func runReport(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
@@ -58,196 +45,65 @@ func runReport(ctx context.Context, stdout io.Writer) error {
fmt.Fprintf(os.Stderr, "report: database %s (%s)\n",
escapePath(dbPath), source)
// Opening can wait on a busy database for as long as the busy
// timeout.
prog := newProgress(progressParams{
label: reportPhase(openPhase), total: -1,
waitingFor: openingDatabase,
})
db, err := openReportDatabase(ctx, dbPath)
prog.finish()
if err != nil {
return err
}
defer func() { _ = db.Close() }()
// The read phase writes the report to stdout while it reads the
// records, so it may wait on either.
prog = newProgress(progressParams{
label: reportPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords, stalledOn: "db/out",
})
report, err := readReport(ctx, db, dbPath, stdout, prog)
out := bufio.NewWriterSize(stdout, ioBufSize)
prog.finish()
_, err = fmt.Fprintln(out, "first\tdupe\tsize")
if err != nil {
return fmt.Errorf("write stdout: %w", err)
}
var (
groups, dupeFiles int
reclaimable int64
writeErr error
)
records, err := loadDupeRows(ctx, db,
func(first, path string, size int64) error {
// A group's first path is its first row; every other
// path is a dupe.
if path == first {
groups++
return nil
}
_, writeErr = fmt.Fprintf(out, "%s\t%s\t%d\n",
escapePath(first), escapePath(path), size)
dupeFiles++
reclaimable += size
return writeErr
})
if writeErr != nil {
return fmt.Errorf("write stdout: %w", writeErr)
}
if err != nil {
return err
return fmt.Errorf("database %s: %w", dbPath, err)
}
err = out.Flush()
if err != nil {
return fmt.Errorf("write stdout: %w", err)
}
fmt.Fprintf(os.Stderr,
"report: %d records read, %d duplicate groups, %d dupe files, "+
"%s reclaimable\n",
report.records, report.dupeGroups, report.dupeFiles,
humanBytes(report.reclaimable))
records, groups, dupeFiles, humanBytes(reclaimable))
return nil
}
// readReport gives prog the number of records in db, the database at
// dbPath, as its total, then reads every record in signature order,
// counting each on prog, and writes the report to stdout: the header
// line, then the rows of each size's duplicate groups once that size is
// read. stdout may be the terminal prog is drawn on, so the lines go
// through prog's writer. It returns what it wrote, with the totals for
// the summary line.
func readReport(ctx context.Context, db *sql.DB, dbPath string,
stdout io.Writer, prog *progress,
) (*reportWriter, error) {
out := prog.writer(stdout)
report := &reportWriter{out: out}
_, err := fmt.Fprintln(out, "first\tdupe\tsize")
if err != nil {
return nil, fmt.Errorf("write stdout: %w", err)
}
var writeErr error
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
prog.increment()
writeErr = report.add(r)
return writeErr
})
if writeErr != nil {
return nil, fmt.Errorf("write stdout: %w", writeErr)
}
if err != nil {
return nil, fmt.Errorf("database %s: %w", dbPath, err)
}
err = report.finish()
if err == nil {
err = out.Flush()
}
if err != nil {
return nil, fmt.Errorf("write stdout: %w", err)
}
return report, nil
}
// reportWriter writes the report's rows to out from the records, added
// in signature order, largest size first, as loadSignatureRows reads
// them. The records sharing a signature come together, and so do the
// records of one size, so it holds only the duplicate groups of the
// size being read, and writes them, ordered by first path, once a
// record of a smaller size comes. It counts what the summary line
// reports.
type reportWriter struct {
out io.Writer
sig scanRec // the signature being read; its path is not used
paths []string // the paths read so far with sig
groups [][]string // the duplicate groups of sig.size, each sorted
records, dupeGroups, dupeFiles int
reclaimable int64
}
// add takes the next record. One with the signature being read joins
// its paths; any other ends them, and one of a smaller size first
// writes the groups of the size before.
func (w *reportWriter) add(r scanRec) error {
w.records++
// A record without a content hash has unknown content (README
// "Database"), so it is never a duplicate.
if r.content == "" {
return nil
}
if r.size == w.sig.size && r.head == w.sig.head &&
r.tail == w.sig.tail && r.content == w.sig.content {
w.paths = append(w.paths, r.path)
return nil
}
w.endGroup()
if r.size != w.sig.size {
err := w.writeGroups()
if err != nil {
return err
}
}
w.sig = r
w.paths = append(w.paths, r.path)
return nil
}
// endGroup ends the paths read with w.sig: two or more are a duplicate
// group, held with its paths sorted until writeGroups writes it.
func (w *reportWriter) endGroup() {
if len(w.paths) < minGroupSize {
// A path with no duplicate is dropped, and its slice reused.
w.paths = w.paths[:0]
return
}
slices.Sort(w.paths)
w.groups = append(w.groups, w.paths)
w.paths = nil
}
// writeGroups writes the duplicate groups held, all of size w.sig.size,
// ordered by first path: for each group, a row for every path but the
// first. No two groups share a first path, since no two records share
// a path.
func (w *reportWriter) writeGroups() error {
slices.SortFunc(w.groups, func(a, b []string) int {
return strings.Compare(a[0], b[0])
})
for _, g := range w.groups {
for _, path := range g[1:] {
_, err := fmt.Fprintf(w.out, "%s\t%s\t%d\n",
escapePath(g[0]), escapePath(path), w.sig.size)
if err != nil {
return err
}
w.dupeFiles++
w.reclaimable += w.sig.size
}
w.dupeGroups++
}
w.groups = nil
return nil
}
// finish writes the groups of the last size read.
func (w *reportWriter) finish() error {
w.endGroup()
return w.writeGroups()
}
// escapePath returns a path as it is written in a report column (README
// "Report output format"): a backslash, tab, newline or carriage return
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
report_test.go
+24 -189
View File
@@ -9,7 +9,6 @@ import (
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"testing"
"time"
@@ -29,8 +28,7 @@ func awkwardPairRecs() []scanRec {
}
}
// seedDatabase writes recs into a fresh database, closes it as scan
// does, out of WAL mode, and returns its path.
// seedDatabase writes recs into a fresh database and returns its path.
func seedDatabase(t *testing.T, recs []scanRec) string {
t.Helper()
@@ -46,7 +44,10 @@ func seedDatabase(t *testing.T, recs []scanRec) string {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, path)
err = db.Close()
if err != nil {
t.Fatal(err)
}
return path
}
@@ -59,39 +60,25 @@ type dupeGroup struct {
}
// dupeGroups returns the duplicate groups report reads from db, in
// report order, from the rows it writes: a group's rows come together,
// each naming the group's first path.
// report order.
func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
t.Helper()
var out bytes.Buffer
_, err := readReport(t.Context(), db, "", &out, nil)
if err != nil {
t.Fatal(err)
}
var groups []dupeGroup
rows := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n")
for _, row := range rows[1:] { // after the header
fields := strings.Split(row, "\t")
if len(fields) != 3 {
t.Fatalf("row %q, want first, dupe and size", row)
}
_, err := loadDupeRows(t.Context(), db,
func(first, path string, size int64) error {
if path == first {
groups = append(groups, dupeGroup{size: size})
}
size, err := strconv.ParseInt(fields[2], 10, 64)
if err != nil {
t.Fatal(err)
}
g := &groups[len(groups)-1]
g.paths = append(g.paths, path)
if len(groups) == 0 || groups[len(groups)-1].paths[0] != fields[0] {
groups = append(groups,
dupeGroup{size: size, paths: []string{fields[0]}})
}
g := &groups[len(groups)-1]
g.paths = append(g.paths, fields[1])
return nil
})
if err != nil {
t.Fatal(err)
}
return groups
@@ -131,147 +118,26 @@ func TestRunReportEscapesPaths(t *testing.T) {
}
func TestReportStdoutFailsWhileReading(t *testing.T) {
t.Parallel()
// Every two records are a duplicate pair of a size of their own, and
// a pair's row is written once a smaller size is read. Each row
// holds two paths longer than dir, so the rows come to more than
// twice the stdout buffer, and stdout fails while records are still
// being read, not once every record is.
// Each row holds two paths longer than dir, so the report is more
// than twice the stdout buffer and stdout fails while rows are
// still being read, not at the final flush.
dir := "/" + strings.Repeat("d", 4096)
recs := make([]scanRec, 2*ioBufSize/len(dir))
recs := make([]scanRec, ioBufSize/len(dir))
for i := range recs {
recs[i] = scanRec{
size: int64(i/2) + 1, head: "h", tail: "t", content: "c",
size: 1, head: "h", tail: "t", content: "c",
path: fmt.Sprintf("%s/%d", dir, i),
}
}
db := openTestDB(t)
t.Setenv(databaseEnv, seedDatabase(t, recs))
err := applyChanges(t.Context(), db, recs, nil, nil)
if err != nil {
t.Fatal(err)
}
// Counts the records read, and draws nothing.
prog := &progress{now: time.Now}
_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
err := runReport(t.Context(), failingWriter{})
if !errors.Is(err, errWriteFailed) ||
!strings.HasPrefix(err.Error(), "write stdout: ") {
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
}
if prog.count >= int64(len(recs)) {
t.Errorf("stdout failed after all %d records were read, want it "+
"to stop the read", prog.count)
}
}
// reportOpenLine is the plain line of report's open phase when it opens
// the database at once; it prints it as the phase starts and as it ends.
const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown"
// smokeReport is report's stdout for the smoke-test records.
const smokeReport = "first\tdupe\tsize\n" +
"/d/t1/f1\t/d/t2/f1\t3000\n" +
"/d/t1/f1\t/d/t3/f1\t3000\n" +
"/d/t1/sub/f2\t/d/t2/sub/f2\t100\n" +
"/d/t1/sub/f2\t/d/t3/sub/f2renamed\t100\n"
// TestRunReportProgressNamesEachPhase runs report on the smoke-test
// records with stderr not a terminal, and checks every line it prints
// there. After the line naming the database, each phase prints a line
// naming it with its place among the two as it starts and another as
// it ends; read counts the 6 records before it reads them. The summary
// comes last, and stdout holds the report alone.
func TestRunReportProgressNamesEachPhase(t *testing.T) {
path := seedDatabase(t, smokeTreeRecs())
t.Setenv(databaseEnv, path)
stderr := captureStderr(t)
var stdout bytes.Buffer
code := run([]string{cmdReport}, &stdout, os.Stderr)
if code != exitOK {
t.Fatalf("run(report) = %d, want %d; stderr: %s",
code, exitOK, stderr())
}
if got := stdout.String(); got != smokeReport {
t.Errorf("stdout = %q, want %q", got, smokeReport)
}
// How each line starts; the rate at the end of a phase varies.
want := []string{
"report: database " + path + " (from " + databaseEnv + ")",
reportOpenLine,
reportOpenLine,
"[2/2] read: counting the records, elapsed 0s, eta unknown",
"[2/2] read: [6/6] 100%, ",
"report: 6 records read, 2 duplicate groups, 4 dupe files, " +
"6.1 KiB reclaimable",
}
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
if len(lines) != len(want) {
t.Fatalf("stderr lines = %q, want %d lines starting %q",
lines, len(want), want)
}
for i := range want {
if !strings.HasPrefix(lines[i], want[i]) {
t.Errorf("stderr line %d = %q, want it to start %q",
i+1, lines[i], want[i])
}
}
}
// TestReadReportOnTerminal writes the smoke-test report to the terminal
// its read phase's display is drawn on, as a plain "sfdupes report"
// does: the terminal must show the header and every row whole, each on
// a line of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestReadReportOnTerminal(t *testing.T) {
stderr := captureStderr(t)
db := openTestDB(t)
err := applyChanges(t.Context(), db, smokeTreeRecs(), nil, nil)
if err != nil {
t.Fatal(err)
}
// No timed draws: the display is drawn as it starts and as it ends.
p := startProgress(progressParams{
label: reportPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords,
}, terminalWidth, time.Hour, time.Now)
_, err = readReport(t.Context(), db, "", os.Stderr, p)
p.finish()
if err != nil {
t.Fatal(err)
}
// A terminal shows each line as the text after its last carriage
// return.
var shown []string
for line := range strings.SplitSeq(stderr(), "\n") {
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
}
want := strings.Split(strings.TrimSuffix(smokeReport, "\n"), "\n")
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
t.Errorf("terminal shows lines %q, want %q first", shown, want)
}
}
func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
@@ -488,37 +354,6 @@ func TestDupeGroupsTieBreak(t *testing.T) {
}
}
func TestDupeGroupsOrderedByFirstPath(t *testing.T) {
t.Parallel()
// report reads the groups of size 50 in signature order, here /a,
// /z, /k, which is neither the order of their first paths nor its
// reverse, and must write them ordered by first path. The group of
// size 7 comes after them all, though its first path sorts first.
recs := []scanRec{
{size: 50, head: "a", tail: "a", content: "a", path: "/k/2"},
{size: 50, head: "a", tail: "a", content: "a", path: "/k/1"},
{size: 50, head: "b", tail: "b", content: "b", path: "/z/1"},
{size: 50, head: "b", tail: "b", content: "b", path: "/z/2"},
{size: 50, head: "c", tail: "c", content: "c", path: "/a/2"},
{size: 50, head: "c", tail: "c", content: "c", path: "/a/1"},
{size: 7, head: "d", tail: "d", content: "d", path: "/0/2"},
{size: 7, head: "d", tail: "d", content: "d", path: "/0/1"},
}
groups := dupeGroupsOf(t, recs)
got := make([]string, 0, len(groups))
for _, g := range groups {
got = append(got, g.paths[0])
}
want := []string{"/a/1", "/k/1", "/z/1", "/0/1"}
if !slices.Equal(got, want) {
t.Errorf("groups' first paths = %q, want %q", got, want)
}
}
func TestDupeGroupsDeterministic(t *testing.T) {
t.Parallel()
scan.go
+127 -269
View File
@@ -13,7 +13,6 @@ import (
"os"
"os/signal"
"path/filepath"
"runtime"
"slices"
"strings"
"sync"
@@ -85,10 +84,6 @@ type fileMeta struct {
hashed bool
}
// scanUnit is what every scan phase counts, as its progress display
// names it.
const scanUnit = "files"
// scanPhase names one phase of a scan with its place among all of them,
// as "[2/5] walk", for the phase's progress display. The list here is
// the one place the phases are counted.
@@ -115,13 +110,9 @@ func scanPhase(name string) string {
// SIGTERM that interruptContext catches, the scan keeps what it has
// hashed (see syncScan), prints how many files its walk reached, and
// returns errInterrupted. workers must be at least 1; the scan command
// rejects anything less. Before the lock is taken, the scan lists the
// filesystems that are not persistent, on which the walk records
// nothing; unless noDefaultExcludes is set, the walk does not enter the
// default excludes either. Once the database is open, the scan notes
// its files, which the walk passes over.
// rejects anything less.
func runScan(ctx context.Context, roots []string, workers int,
oneFS, noDefaultExcludes bool,
oneFS bool,
) error {
dbPath, source, err := databasePath()
if err != nil {
@@ -144,16 +135,6 @@ func runScan(ctx context.Context, roots []string, workers int,
return err
}
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
return fmt.Errorf("list the mounted filesystems: %w", err)
}
rules := walkRules{nonPersistent: nonPersistent}
if !noDefaultExcludes {
rules.excludes = defaultExcludes(runtime.GOOS)
}
lock, err := lockScanDatabase(dbPath)
if err != nil {
return err
@@ -174,12 +155,7 @@ func runScan(ctx context.Context, roots []string, workers int,
defer closeScanDatabase(ctx, db, dbPath)
rules.databaseFiles, err = databaseFiles(dbPath)
if err != nil {
return err
}
st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
st, err := syncScan(ctx, db, roots, workers, oneFS)
if errors.Is(err, context.Canceled) {
return interrupted(st.walked)
}
@@ -251,12 +227,14 @@ func resolveRoots(roots []string) ([]string, error) {
}
// pruneRoots drops operands already covered by another operand:
// duplicates and any operand lying under another one. An operand under
// another one is kept, though, when the walk of the deepest kept operand
// above it does not reach it (see walkReaches). Every remaining file is
// then reachable through exactly one root, so walked paths are unique
// without keeping a scan-wide set of every path seen.
func pruneRoots(roots []string, oneFS bool, rules walkRules,
// duplicates and any operand lying under another one. With -x, an
// operand under another one is kept when the walk of the deepest kept
// operand above it cannot reach it: when device puts any directory from
// it (from its directory, for a regular file) up to but not including
// that operand on a different filesystem than that operand. Every
// remaining file is then reachable through exactly one root, so walked
// paths are unique without keeping a scan-wide set of every path seen.
func pruneRoots(roots []string, oneFS bool,
device func(path string) (uint64, bool),
) []string {
// Shorter-first ordering guarantees an ancestor is kept before any
@@ -280,57 +258,40 @@ func pruneRoots(roots []string, oneFS bool, rules walkRules,
continue
}
// A duplicate is dropped, and so is an operand the walk of above
// reaches.
if above != r && !walkReaches(above, r, oneFS, rules, device) {
kept = append(kept, r)
// Without -x, and always for a duplicate, r is dropped.
if !oneFS || above == r {
continue
}
aboveDev, aboveOK := device(above)
if !aboveOK {
continue
}
// The walk of above enters a directory only when it is on
// above's filesystem, and then reaches every regular file in it
// whatever device the file reports, so for a regular-file
// operand the directories checked start at its directory.
dir := r
fi, err := os.Lstat(r)
if err == nil && fi.Mode().IsRegular() {
dir = filepath.Dir(r)
}
for ; dir != above; dir = filepath.Dir(dir) {
dev, ok := device(dir)
if ok && dev != aboveDev {
kept = append(kept, r)
break
}
}
}
return kept
}
// walkReaches reports whether the walk of operand above reaches r, an
// operand under it. That walk does not enter a default exclude in
// rules, nor a directory that device puts on a filesystem in rules that
// is not persistent, nor, with -x, one that device puts on a different
// filesystem than above, when device knows above's. It reaches every
// regular file in a directory it enters, so for a regular-file operand
// the directories checked start at its directory; they end below above.
func walkReaches(above, r string, oneFS bool, rules walkRules,
device func(path string) (uint64, bool),
) bool {
aboveDev, aboveDevOK := device(above)
dir := r
fi, err := os.Lstat(r)
if err == nil && fi.Mode().IsRegular() {
dir = filepath.Dir(r)
}
for ; dir != above; dir = filepath.Dir(dir) {
if isDefaultExclude(dir, rules.excludes) {
return false
}
dev, ok := device(dir)
if !ok {
continue
}
if _, found := rules.nonPersistent[dev]; found {
return false
}
if oneFS && aboveDevOK && dev != aboveDev {
return false
}
}
return true
}
// deviceOfPath returns the filesystem device ID of the file at path,
// when lstat succeeds and the platform exposes one.
func deviceOfPath(path string) (uint64, bool) {
@@ -372,11 +333,11 @@ type scanState struct {
// the hashed records still waiting in the batch, starts no other write
// or deletion, and returns the cancellation.
func syncScan(ctx context.Context, db *sql.DB, roots []string,
workers int, oneFS bool, rules walkRules,
workers int, oneFS bool,
) (scanStats, error) {
s := &scanState{db: db}
err := s.runPhases(ctx, roots, workers, oneFS, rules)
err := s.runPhases(ctx, roots, workers, oneFS)
if err == nil || ctx.Err() == nil {
return s.st, err
}
@@ -400,18 +361,15 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
// and content (fill in the content hash of every record of headTailMin
// or more whose size, head, and tail match another record's). Records
// outside the roots are never touched, except that the content phase
// fills in their content hash. Operands the walk cannot start from, or
// that are not on a persistent filesystem, are dropped first, so the
// records beneath them are treated as if those operands had not been
// given. Below an operand, the walk follows rules.
// fills in their content hash. Operands the walk cannot start from are
// dropped first, so the records beneath them are treated as if those
// operands had not been given.
func (s *scanState) runPhases(ctx context.Context, roots []string,
workers int, oneFS bool, rules walkRules,
workers int, oneFS bool,
) error {
// Types and filesystems are checked before pruning so that an
// operand under a dropped one is still scanned, not dropped as lying
// under it.
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
deviceOfPath)
// Types are checked before pruning so that an operand under a
// dropped one is still scanned, not dropped as lying under it.
roots = pruneRoots(s.walkableRoots(roots), oneFS, deviceOfPath)
err := s.loadIndex(ctx, roots)
if err != nil {
@@ -419,7 +377,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
}
changed, unhashed := s.walkPhase(
startWalk(ctx, roots, oneFS, rules, workers), roots)
startWalk(ctx, roots, oneFS, workers), roots)
// A cancelled walk stops early, so its size census covers only part
// of the roots, and every file it never reached would look vanished
@@ -441,7 +399,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
return err
}
return s.contentPhase(ctx, workers, rules)
return s.contentPhase(ctx, workers)
}
// walkableRoots returns the operands the walk can start from: regular
@@ -449,11 +407,9 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
// about, counted as skipped, and dropped. A dropped operand is no
// longer a root, so the records stored beneath it are treated as if it
// had not been given: those under no other root count as outside the
// roots and are not deleted as unverified. An operand on a filesystem
// in rules that is not persistent is dropped the same way, with a line
// saying so that is not a warning, and is not counted as skipped. An
// operand that fails lstat here is kept, and the walk warns about it.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
// roots and are not deleted as unverified. An operand that fails lstat
// here is kept, and the walk warns about it.
func (s *scanState) walkableRoots(roots []string) []string {
kept := make([]string, 0, len(roots))
for _, root := range roots {
@@ -467,13 +423,6 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
continue
}
line := rules.nonPersistentLine(root, fi)
if line != "" {
fmt.Fprintln(os.Stderr, escapePath(line))
continue
}
}
kept = append(kept, root)
@@ -492,8 +441,8 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// display the scan looks hung before the walk begins. Counting them
// first gives the display its total.
prog := newProgress(progressParams{
label: scanPhase("load"), unit: scanUnit, total: -1,
waitingFor: countingRecords, stalledOn: "db",
label: scanPhase("load"), total: -1,
waitingFor: "counting the records",
})
defer prog.finish()
@@ -528,12 +477,10 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// (the content phase may still fill in its content hash). It returns
// the new-or-changed files and the unchanged files whose records lack
// hashes; both remain candidates until the census decides whether
// their sizes are shared. A record at or below a path the walk left
// out, because -x stopped at it, it is a default exclude, or it is not
// on a persistent filesystem, counts as outside the roots: it is not
// deleted, and its size joins the census. That does not hold below a
// root at or inside that path, since that root's own walk verifies the
// records below it; nor is a line printed for such a root.
// their sizes are shared. A record below a directory -x did not enter
// counts as outside the roots: it is not deleted, and its size joins
// the census. That does not hold below a root at or inside that
// directory, since that root's own walk verifies the records below it.
func (s *scanState) walkPhase(
events <-chan walkEvent, roots []string,
) ([]fileRec, []fileRec) {
@@ -545,31 +492,17 @@ func (s *scanState) walkPhase(
}
prog := newProgress(progressParams{
label: scanPhase("walk"), unit: scanUnit, total: estimate,
estimate: true, stalledOn: "disk",
label: scanPhase("walk"), total: estimate, estimate: true,
})
var (
changed, unhashed []fileRec
leftOut []string
otherFSDirs []string
)
for ev := range events {
if ev.otherFSDir != "" {
leftOut = append(leftOut, ev.otherFSDir)
continue
}
// The line saying so is printed as a warning is, but what the
// walk leaves out is not counted as skipped. A default exclude
// that is itself a root is walked as one, so it gets no line.
if ev.leftOut != "" {
leftOut = append(leftOut, ev.leftOut)
if !slices.Contains(roots, ev.leftOut) {
prog.warnf("%s", ev.warn)
}
otherFSDirs = append(otherFSDirs, ev.otherFSDir)
continue
}
@@ -604,13 +537,12 @@ func (s *scanState) walkPhase(
prog.finish()
for path, old := range s.existing {
// Kept only when a path the walk left out lies below the deepest
// root above the record, or is the record's own. A root at or
// below that path was walked on its own and did not find the
// file. Both contain the record, so the longer path is the
// deeper.
left := deepestRoot(path, leftOut)
if len(left) > len(deepestRoot(path, roots)) {
// Kept only when a directory -x did not enter lies below the
// deepest root above the record. A root at or below that
// directory was walked on its own and did not find the file.
// Both contain the record, so the longer path is the deeper.
otherFSDir := deepestRoot(path, otherFSDirs)
if len(otherFSDir) > len(deepestRoot(path, roots)) {
delete(s.existing, path)
s.sizes = append(s.sizes, old.size)
@@ -716,15 +648,6 @@ func sameInode(a, b fileRec) bool {
return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
}
// isDatabaseFile reports whether f, a file the walk reached, is one of
// the scan's own database files in rules, which the walk passes over
// without a line and without counting it.
func isDatabaseFile(f fileRec, rules walkRules) bool {
return slices.ContainsFunc(rules.databaseFiles, func(d fileRec) bool {
return sameInode(d, f)
})
}
// hashPhase hashes every queued file with hashSignature — the head and
// tail of a file of headTailMin or more, the whole file below that —
// committing completed records to the database in batches as results
@@ -759,11 +682,8 @@ func (s *scanState) readRuns(ctx context.Context, workers int,
pool := startHashPool(ctx, runs, workers, hash)
defer pool.stop()
// Between results the phase may wait on a file being read or on a
// batch of records being written.
prog := newProgress(progressParams{
label: scanPhase(label), unit: scanUnit, total: int64(len(runs)),
stalledOn: "disk/db",
label: scanPhase(label), total: int64(len(runs)),
})
defer prog.finish()
@@ -852,8 +772,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
total := len(s.batch) + len(s.sentinels) + len(deletes)
prog := newProgress(progressParams{
label: scanPhase("update"), unit: scanUnit, total: int64(total),
stalledOn: "db",
label: scanPhase("update"), total: int64(total),
})
defer prog.finish()
@@ -893,12 +812,9 @@ func (s *scanState) updatePhase(ctx context.Context) error {
// hash phase's worker pool and their records written back in batches. A
// failed read is warned about and counted as skipped; the record keeps
// its empty content, so it is never grouped, and a later scan tries
// again. No file on a filesystem in rules that is not persistent is
// read.
func (s *scanState) contentPhase(ctx context.Context, workers int,
rules walkRules,
) error {
toRead, recs, err := s.contentCandidates(ctx, rules)
// again.
func (s *scanState) contentPhase(ctx context.Context, workers int) error {
toRead, recs, err := s.contentCandidates(ctx)
if err != nil {
return err
}
@@ -919,43 +835,33 @@ func (s *scanState) contentPhase(ctx context.Context, workers int,
return err
}
// The last partial batch is written under a display of its own, as
// the update phase writes the hash phase's.
prog := newProgress(progressParams{
label: scanPhase("content"), unit: scanUnit,
total: int64(len(s.batch)), stalledOn: "db",
})
defer prog.finish()
return applyChanges(ctx, s.db, s.batch, nil, prog)
return applyChanges(ctx, s.db, s.batch, nil, nil)
}
// contentCandidates returns the files the content phase reads, and
// their records by path. Every record contentCandidatesSQL returns has
// its file checked with lstat, whether or not it already has a content
// hash: a file that is gone, is no longer a regular file, has changed
// by the walk's rule, or is on a filesystem in rules that is not
// persistent keeps its record as it is and does not count as a match
// for the others, and any other lstat error is warned about and counted
// as skipped, with the same result. If such a record has no content
// hash, it stays out of duplicate groups; if it has one, it is still
// reported until a scan covering its own tree updates or removes it.
// The files of a group that pass and have no content hash are read only
// if at least minGroupSize of the group's files pass, so a group whose
// other members are all stale costs no reads. Only the records to be
// read are kept.
// hash: a file that is gone, is no longer a regular file, or has
// changed by the walk's rule keeps its record as it is and does not
// count as a match for the others, and any other lstat error is warned
// about and counted as skipped, with the same result. If such a record
// has no content hash, it stays out of duplicate groups; if it has one,
// it is still reported until a scan covering its own tree updates or
// removes it. The files of a group that pass and have no content hash
// are read only if at least minGroupSize of the group's files pass, so
// a group whose other members are all stale costs no reads. Only the
// records to be read are kept.
func (s *scanState) contentCandidates(
ctx context.Context, rules walkRules,
ctx context.Context,
) ([]fileRec, map[string]scanRec, error) {
// The search and the checks take real time on a large database;
// without a display the scan looks hung before the reads begin.
// The display says it is searching until SQLite returns the first
// record. The checks start there, against a count of the records
// the search returns, and so do their rate and ETA. Between records
// they may wait on a file's lstat or on the search.
// the search returns, and so do their rate and ETA.
prog := newProgress(progressParams{
label: scanPhase("content"), unit: scanUnit, total: -1,
waitingFor: "searching the database", stalledOn: "disk/db",
label: scanPhase("content"), total: -1,
waitingFor: "searching the database",
})
defer prog.finish()
@@ -1003,7 +909,7 @@ func (s *scanState) contentCandidates(
first = r
}
f, ok, err := unchangedFile(r, rules)
f, ok, err := unchangedFile(r)
if err != nil {
s.st.skipped++
@@ -1037,10 +943,9 @@ func (s *scanState) contentCandidates(
// unchangedFile lstats the file r names and returns it for reading if
// it is still the regular file r records: the same size, and an mtime
// no newer than recorded (the walk's change rule). A file that is gone,
// has changed, or is on a filesystem in rules that is not persistent
// reports false; any other lstat error is returned.
func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
// no newer than recorded (the walk's change rule). A file that is gone
// or has changed reports false; any other lstat error is returned.
func unchangedFile(r scanRec) (fileRec, bool, error) {
fi, err := os.Lstat(r.path)
if errors.Is(err, fs.ErrNotExist) {
return fileRec{}, false, nil
@@ -1057,12 +962,6 @@ func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
dev, ino := inodeOfInfo(fi)
// The scan reads nothing on a filesystem that is not persistent,
// whatever record an earlier scan left for the file.
if _, found := rules.nonPersistent[dev]; found {
return fileRec{}, false, nil
}
return fileRec{
path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
}, true, nil
@@ -1129,36 +1028,32 @@ type dirJob struct {
}
// walkEvent is one walk result delivered to the walk phase: a regular
// file's statted record, a warning when fail is set, or a path the walk
// left out: when otherFSDir is set, a directory -x did not enter
// because it is on another filesystem than its operand, and when
// leftOut is set, a default exclude or a directory or file on a
// filesystem that is not persistent, with warn holding the line that
// says so.
// file's statted record, a warning when fail is set, or, when
// otherFSDir is set, a directory -x did not enter because it is on
// another filesystem than its operand.
type walkEvent struct {
rec fileRec
warn string
fail bool
otherFSDir string
leftOut string
}
// startWalk seeds every root into the shared walk worker pool and
// returns the event stream: one record per regular file, one warning
// event per per-path error, and one event per path -x or rules kept the
// walk from entering or recording. The channel is closed when the walk
// completes, and also when ctx is cancelled — every goroutine in the
// pool abandons its blocking send in that case, so the consumer sees a
// truncated but properly terminated stream instead of a stalled one.
// event per per-path error, and one event per directory -x did not
// enter. The channel is closed when the walk completes, and also when
// ctx is cancelled — every goroutine in the pool abandons its blocking
// send in that case, so the consumer sees a truncated but properly
// terminated stream instead of a stalled one.
func startWalk(ctx context.Context, roots []string, oneFS bool,
rules walkRules, workers int,
workers int,
) <-chan walkEvent {
jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS, rules)
jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS)
go func() {
initial := make([]dirJob, 0, len(roots))
for _, root := range roots {
initial = append(initial, seedRoot(ctx, root, rules, events)...)
initial = append(initial, seedRoot(ctx, root, events)...)
}
dispatchDirs(ctx, initial, jobs, subdirs)
@@ -1212,14 +1107,13 @@ func operandWarning(root string, fi fs.FileInfo) string {
}
// seedRoot turns one PATH operand into the walk's starting state: a
// regular-file operand is statted and emitted directly, unless it is
// one of the scan's own database files in rules, and a directory
// regular-file operand is statted and emitted directly, and a directory
// operand becomes an initial job. walkableRoots has already dropped
// every other operand. One that has changed into something else since
// is warned about and skipped here; it is still a root, so the records
// stored beneath it, other than those under another root, are deleted
// as unverified.
func seedRoot(ctx context.Context, root string, rules walkRules,
func seedRoot(ctx context.Context, root string,
events chan<- walkEvent,
) []dirJob {
fi, err := os.Lstat(root)
@@ -1246,17 +1140,14 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
}
dev, ino := inodeOfInfo(fi)
rec := fileRec{
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: root,
size: fi.Size(),
mtime: fi.ModTime(),
dev: dev,
ino: ino,
}
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
}})
return nil
}
@@ -1268,7 +1159,7 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
// workers drop the directories still queued rather than stop reading
// jobs, so the range always runs out and the pool always tears down.
func startWalkWorkers(ctx context.Context, workers int,
oneFS bool, rules walkRules,
oneFS bool,
) (chan dirJob, chan []dirJob, chan walkEvent) {
jobs := make(chan dirJob, workQueueDepth)
subdirs := make(chan []dirJob, workers)
@@ -1284,7 +1175,7 @@ func startWalkWorkers(ctx context.Context, workers int,
}
select {
case subdirs <- walkOneDir(ctx, job, oneFS, rules, events):
case subdirs <- walkOneDir(ctx, job, oneFS, events):
case <-ctx.Done():
}
}
@@ -1342,7 +1233,7 @@ func dispatchDirs(ctx context.Context, initial []dirJob,
// entry and a warning event per unreadable one, and returns the
// subdirectories to descend into.
func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
rules walkRules, events chan<- walkEvent,
events chan<- walkEvent,
) []dirJob {
entries, err := os.ReadDir(job.path)
if err != nil {
@@ -1366,8 +1257,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
p := filepath.Join(job.path, e.Name())
if e.IsDir() {
sub, ok := subdirJob(ctx, p, e, job, oneFS, rules, events)
if ok {
if sub, ok := subdirJob(ctx, p, e, job, oneFS, events); ok {
subs = append(subs, sub)
}
@@ -1379,7 +1269,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
continue
}
emitFile(ctx, p, e, rules, events)
emitFile(ctx, p, e, events)
}
return subs
@@ -1389,12 +1279,9 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
// The lstat happens here in the walk worker, while the directory's
// metadata is still hot; a path that fails to stat (or stops being a
// regular file) between the directory read and the lstat is warned
// about and skipped. A file on a filesystem in rules that is not
// persistent, as through a bind mount of one file, is not recorded: it
// gets an event naming it instead. One of the scan's own database files
// in rules gets no event at all.
// about and skipped.
func emitFile(ctx context.Context, p string, e fs.DirEntry,
rules walkRules, events chan<- walkEvent,
events chan<- walkEvent,
) {
info, err := e.Info()
if err != nil {
@@ -1410,43 +1297,35 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
return
}
line := rules.nonPersistentLine(p, info)
if line != "" {
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
return
}
dev, ino := inodeOfInfo(info)
rec := fileRec{
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: p,
size: info.Size(),
mtime: info.ModTime(),
dev: dev,
ino: ino,
}
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
}})
}
// subdirJob applies the descent rules to directory p: never enter
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
// snapshot), never enter a directory on a filesystem in rules that is
// not persistent, nor a default exclude in rules, and with -x never
// enter a directory on a different filesystem than its operand. For the
// last three it sends an event that names the directory, so the scan
// keeps the records below it, except those below an operand at or
// snapshot), and with -x never enter a directory on a different
// filesystem than its operand, sending an event that names it so the
// scan keeps the records below it, except those below an operand at or
// inside it.
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
parent dirJob, oneFS bool, rules walkRules,
events chan<- walkEvent,
parent dirJob, oneFS bool, events chan<- walkEvent,
) (dirJob, bool) {
if e.Name() == ".zfs" {
return dirJob{}, false
}
job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
if !oneFS || !parent.rootDevOK {
return job, true
}
info, err := e.Info()
if err != nil {
sendEvent(ctx, events, walkEvent{
@@ -1457,27 +1336,6 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
return dirJob{}, false
}
line := rules.nonPersistentLine(p, info)
if line != "" {
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
return dirJob{}, false
}
if isDefaultExclude(p, rules.excludes) {
sendEvent(ctx, events, walkEvent{
leftOut: p,
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
})
return dirJob{}, false
}
job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
if !oneFS || !parent.rootDevOK {
return job, true
}
if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
sendEvent(ctx, events, walkEvent{otherFSDir: p})
scan_test.go
+12 -129
View File
@@ -782,8 +782,7 @@ func collectWalk(t *testing.T, roots []string, oneFS bool,
errs int
)
for ev := range startWalk(t.Context(), roots, oneFS, walkRules{},
workers) {
for ev := range startWalk(t.Context(), roots, oneFS, workers) {
if ev.fail {
errs++
@@ -1005,8 +1004,7 @@ func callSubdirJob(t *testing.T, p string, e fs.DirEntry,
t.Helper()
events := make(chan walkEvent, 1)
job, ok := subdirJob(t.Context(), p, e, parent, oneFS, walkRules{},
events)
job, ok := subdirJob(t.Context(), p, e, parent, oneFS, events)
close(events)
var evs []walkEvent
@@ -1154,7 +1152,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
return dev, ok
}
roots = pruneRoots(roots, true, walkRules{}, device)
roots = pruneRoots(roots, true, device)
ctx := t.Context()
s := &scanState{db: db}
@@ -1164,14 +1162,14 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
t.Fatal(err)
}
jobs, subdirs, events := startWalkWorkers(ctx, 2, true, walkRules{})
jobs, subdirs, events := startWalkWorkers(ctx, 2, true)
// seedRoot emits a regular-file operand itself and returns a job
// only for a directory operand.
var initial []dirJob
for _, root := range roots {
for _, job := range seedRoot(ctx, root, walkRules{}, events) {
for _, job := range seedRoot(ctx, root, events) {
job.rootDev = devices[root]
initial = append(initial, job)
}
@@ -1465,7 +1463,7 @@ const smokeTreeFiles = 15
func syncTree(t *testing.T, db *sql.DB, roots ...string) scanStats {
t.Helper()
st, err := syncScan(t.Context(), db, roots, 4, false, walkRules{})
st, err := syncScan(t.Context(), db, roots, 4, false)
if err != nil {
t.Fatalf("syncScan: %v", err)
}
@@ -1480,7 +1478,7 @@ func dbRecords(t *testing.T, db *sql.DB) []scanRec {
var recs []scanRec
err := loadFileRows(t.Context(), db, func(int64) {}, func(r scanRec) {
err := loadFileRows(t.Context(), db, func(r scanRec) {
recs = append(recs, r)
})
if err != nil {
@@ -1552,7 +1550,7 @@ func assertSmokeTreeGroups(t *testing.T, dir string, db *sql.DB) {
super, dirs := dbTree(t, db)
tg := collectTreeGroups(dirs, super, nil)
tg := collectTreeGroups(dirs, super)
if len(tg) != 1 {
t.Fatalf("len(tree groups) = %d, want 1", len(tg))
}
@@ -2094,7 +2092,7 @@ func TestTreesUnhashedNeverEqual(t *testing.T) {
for name, unknown := range cases {
super, dirs := treeOf(t, append(slices.Clone(shared), unknown...))
if tg := collectTreeGroups(dirs, super, nil); len(tg) != 0 {
if tg := collectTreeGroups(dirs, super); len(tg) != 0 {
t.Errorf("%s: tree groups = %d, want 0 (the files may differ)",
name, len(tg))
}
@@ -2414,7 +2412,7 @@ func TestPruneRoots(t *testing.T) {
// Duplicates and operands under other operands are dropped; /cc is
// not under /c (sibling with a shared prefix).
got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
false, walkRules{}, device)
false, device)
want := []string{"/a", "/c", "/cc"}
if !slices.Equal(got, want) {
@@ -2423,7 +2421,7 @@ func TestPruneRoots(t *testing.T) {
nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
got = pruneRoots(nested, false, walkRules{}, device)
got = pruneRoots(nested, false, device)
want = []string{"/p"}
if !slices.Equal(got, want) {
@@ -2431,31 +2429,12 @@ func TestPruneRoots(t *testing.T) {
}
// With -x, /p/q, on another filesystem than /p, is kept as well.
got = pruneRoots(nested, true, walkRules{}, device)
got = pruneRoots(nested, true, device)
want = []string{"/p", pqRoot}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots with -x = %q, want %q", got, want)
}
// The walk of /p does not enter /p/q when it is a default exclude, so
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
device)
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
}
// Nor does it enter /p/q when the filesystem of /p/q is not
// persistent, so /p/q/r/m, of no known device, is kept.
got = pruneRoots([]string{"/p", "/p/q/r/m"}, false,
walkRules{nonPersistent: map[uint64]string{2: "tmpfs"}}, device)
want = []string{"/p", "/p/q/r/m"}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots with %s not persistent = %q, want %q",
pqRoot, got, want)
}
}
func TestReportsNeverTouchFilesystem(t *testing.T) {
@@ -2501,99 +2480,3 @@ func TestUnderRoot(t *testing.T) {
}
}
}
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
// database, reached directly and through a symlinked directory. The
// walk passes over the database file, its -wal and -shm files and its
// .lock file with no line and no count, and the record an earlier scan
// left for the .lock file is deleted. A rescan of the unchanged tree
// updates nothing, and report lists none of the database's files.
func TestRunScanPassesOverDatabase(t *testing.T) {
for _, viaSymlink := range []bool{false, true} {
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
base := t.TempDir()
tree := filepath.Join(base, "real", "tree")
operand := tree
writeFile(t, tree, "a", pattern(1, 100))
writeFile(t, tree, "b", pattern(1, 100))
writeFile(t, tree, "empty1", nil)
writeFile(t, tree, "empty2", nil)
if viaSymlink {
err := os.Symlink(filepath.Join(base, "real"),
filepath.Join(base, "link"))
if err != nil {
t.Fatal(err)
}
operand = filepath.Join(base, "link", "tree")
}
dbPath := filepath.Join(tree, "db.sqlite")
t.Setenv(databaseEnv, dbPath)
db, err := openScanDatabase(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
err = applyChanges(t.Context(), db, []scanRec{
{path: filepath.Join(operand, "db.sqlite.lock")},
}, nil, nil)
if err != nil {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, dbPath)
assertScanSummary(t,
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
operand)
assertScanSummary(t,
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
operand)
report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") ||
!strings.Contains(report, filepath.Join(operand, "empty2")) {
t.Errorf("report = %q, want the empty files and none of "+
"the database's files", report)
}
})
}
}
// TestRunScanPassesOverDatabaseOperands scans the database file and its
// .lock file given as operands themselves: neither is recorded or
// counted.
func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
dbPath := testDBPath(t)
t.Setenv(databaseEnv, dbPath)
// The first scan creates both files.
scanOK(t, t.TempDir())
assertScanSummary(t,
"0 files seen (0 added, 0 updated, 0 removed, 0 unchanged)",
dbPath, dbPath+".lock")
}
// assertScanSummary scans operands and fails the test unless the
// summary line gives counts with nothing skipped, and the line naming
// the database is the only one that names one of its files.
func assertScanSummary(t *testing.T, counts string, operands ...string) {
t.Helper()
stderr := scanOK(t, operands...)
want := "scan: " + counts + ", 0 skipped\n"
if !strings.Contains(stderr, want) {
t.Errorf("scan %q: stderr = %q, want %q", operands, stderr, want)
}
if strings.Count(stderr, "db.sqlite") != 1 {
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
"database to name one of its files", operands, stderr)
}
}
trees.go
+25 -108
View File
@@ -1,9 +1,9 @@
package main
import (
"bufio"
"context"
"crypto/sha256"
"database/sql"
"fmt"
"io"
"os"
@@ -24,13 +24,6 @@ type treeNode struct {
totalSize int64
}
// treesPhase names one phase of trees with its place among all of them,
// as "[2/4] read", for the phase's progress display. The list here is
// the one place the phases are counted.
func treesPhase(name string) string {
return phaseLabel([]string{openPhase, "read", "group", "write"}, name)
}
// runTrees implements the trees subcommand: it reads every record from
// the database in path order, reconstructs the directory hierarchy from
// the record paths, computes a Merkle-style digest per directory, and
@@ -38,13 +31,6 @@ func treesPhase(name string) string {
// touches the scanned filesystem; its only I/O is the database, stdout,
// and stderr. It names the database on stderr before it opens it. Any
// database problem, including a missing database, is fatal.
//
// The work runs in four phases, each under a progress display of its
// own on stderr: open (open the database), read (count the records,
// then read them into the hierarchy), group (group the directories by
// digest) and write (write the groups to stdout). Each display ends
// before the next phase starts, and before an error is returned to be
// printed.
func runTrees(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
@@ -54,102 +40,34 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
fmt.Fprintf(os.Stderr, "trees: database %s (%s)\n",
escapePath(dbPath), source)
// Opening can wait on a busy database for as long as the busy
// timeout.
prog := newProgress(progressParams{
label: treesPhase(openPhase), total: -1,
waitingFor: openingDatabase,
})
db, err := openReportDatabase(ctx, dbPath)
prog.finish()
if err != nil {
return err
}
defer func() { _ = db.Close() }()
prog = newProgress(progressParams{
label: treesPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords, stalledOn: "db",
records := 0
tree := newTreeBuilder()
err = loadFileRows(ctx, db, func(r scanRec) {
records++
tree.add(r)
})
records, super, allDirs, err := readTree(ctx, db, prog)
prog.finish()
if err != nil {
return fmt.Errorf("database %s: %w", dbPath, err)
}
// Grouping works in memory, so it waits on nothing but the
// processor.
prog = newProgress(progressParams{
label: treesPhase("group"), unit: "directories",
total: int64(len(allDirs)), stalledOn: "cpu",
})
dupes := collectTreeGroups(allDirs, super, prog)
prog.finish()
prog = newProgress(progressParams{
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
stalledOn: "out",
})
dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
prog.finish()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr,
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
"%s reclaimable\n",
records, len(dupes), dupeTrees, humanBytes(reclaimable))
return nil
}
// readTree gives prog the number of records in db as its total, then
// reads every record in path order into the directory hierarchy,
// counting each on prog. It returns the number of records read, the
// super-root and every directory.
func readTree(ctx context.Context, db *sql.DB,
prog *progress,
) (int, *treeNode, []*treeNode, error) {
records := 0
tree := newTreeBuilder()
err := loadFileRows(ctx, db, prog.setTotal, func(r scanRec) {
records++
prog.increment()
tree.add(r)
})
if err != nil {
return 0, nil, nil, err
}
super, allDirs := tree.finish()
return records, super, allDirs, nil
}
dupes := collectTreeGroups(allDirs, super)
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
// a row for every member of each group but its first, counting each
// group on prog once its rows are written. stdout may be the terminal
// prog is drawn on, so the lines go through prog's writer. It returns
// the number of rows (dupe trees) and the bytes they could reclaim.
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
prog *progress,
) (int, int64, error) {
out := prog.writer(stdout)
out := bufio.NewWriterSize(stdout, ioBufSize)
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
_, err = fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
if err != nil {
return 0, 0, fmt.Errorf("write stdout: %w", err)
return fmt.Errorf("write stdout: %w", err)
}
dupeTrees := 0
@@ -163,22 +81,25 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
escapePath(first.path), escapePath(n.path),
first.fileCount, first.totalSize)
if err != nil {
return 0, 0, fmt.Errorf("write stdout: %w", err)
return fmt.Errorf("write stdout: %w", err)
}
dupeTrees++
reclaimable += first.totalSize
}
prog.increment()
}
err = out.Flush()
if err != nil {
return 0, 0, fmt.Errorf("write stdout: %w", err)
return fmt.Errorf("write stdout: %w", err)
}
return dupeTrees, reclaimable, nil
fmt.Fprintf(os.Stderr,
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
"%s reclaimable\n",
records, len(dupes), dupeTrees, humanBytes(reclaimable))
return nil
}
// treeBuilder reconstructs the directory hierarchy from records added
@@ -300,18 +221,14 @@ func fileEntry(name string, r scanRec) string {
"\x00" + r.head + "\x00" + r.tail + "\x00" + content
}
// collectTreeGroups groups directories by digest, counting each one on
// prog as it is grouped, and returns every maximal group with two or
// more members, each group's members sorted by path, groups ordered by
// tree size descending then by first path ascending.
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
prog *progress,
) [][]*treeNode {
// collectTreeGroups groups directories by digest and returns every
// maximal group with two or more members, each group's members sorted
// by path, groups ordered by tree size descending then by first path
// ascending.
func collectTreeGroups(allDirs []*treeNode, super *treeNode) [][]*treeNode {
groups := make(map[[sha256.Size]byte][]*treeNode)
for _, d := range allDirs {
groups[d.digest] = append(groups[d.digest], d)
prog.increment()
}
var dupes [][]*treeNode
trees_test.go
+7 -107
View File
@@ -3,11 +3,8 @@ package main
import (
"bytes"
"database/sql"
"os"
"slices"
"strings"
"testing"
"time"
)
// Signature hashes shared by the smoke-test records.
@@ -20,11 +17,6 @@ const (
f2Content = "f2c"
)
// treesOpenLine is the plain line of trees' open phase when it opens
// the database at once; it prints it as the phase starts and as it ends.
const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
"eta unknown"
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
// /d/t2 are identical, /d/t3 differs from them only by one filename.
func smokeTreeRecs() []scanRec {
@@ -46,7 +38,7 @@ func dbTree(t *testing.T, db *sql.DB) (*treeNode, []*treeNode) {
tree := newTreeBuilder()
err := loadFileRows(t.Context(), db, func(int64) {}, tree.add)
err := loadFileRows(t.Context(), db, tree.add)
if err != nil {
t.Fatal(err)
}
@@ -178,7 +170,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
t.Fatalf("directory paths = %q, want %q", got, want)
}
groups := collectTreeGroups(dirs, super, nil)
groups := collectTreeGroups(dirs, super)
gotGroups := groupPaths(groups)
wantGroups := [][]string{{"/a", "/c"}}
@@ -211,98 +203,6 @@ func TestRunTreesEscapesPaths(t *testing.T) {
}
}
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
// records with stderr not a terminal, and checks every line it prints
// there. After the line naming the database, each phase prints a line
// naming it with its place among the four as it starts and another as
// it ends. read counts the 6 records before it reads them, group counts
// the 8 directories, and write the 1 group. The summary comes last, and
// stdout holds the report alone.
func TestRunTreesProgressNamesEachPhase(t *testing.T) {
path := seedDatabase(t, smokeTreeRecs())
t.Setenv(databaseEnv, path)
stderr := captureStderr(t)
var stdout bytes.Buffer
code := run([]string{cmdTrees}, &stdout, os.Stderr)
if code != exitOK {
t.Fatalf("run(trees) = %d, want %d; stderr: %s",
code, exitOK, stderr())
}
report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
if got := stdout.String(); got != report {
t.Errorf("stdout = %q, want %q", got, report)
}
// How each line starts; the rate at the end of a phase varies.
want := []string{
"trees: database " + path + " (from " + databaseEnv + ")",
treesOpenLine,
treesOpenLine,
"[2/4] read: counting the records, elapsed 0s, eta unknown",
"[2/4] read: [6/6] 100%, ",
"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown",
"[3/4] group: [8/8] 100%, ",
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown",
"[4/4] write: [1/1] 100%, ",
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
}
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
if len(lines) != len(want) {
t.Fatalf("stderr lines = %q, want %d lines starting %q",
lines, len(want), want)
}
for i := range want {
if !strings.HasPrefix(lines[i], want[i]) {
t.Errorf("stderr line %d = %q, want it to start %q",
i+1, lines[i], want[i])
}
}
}
// TestWriteTreeGroupsOnTerminal writes the smoke-test report to the
// terminal its write phase's display is drawn on, as a plain "sfdupes
// trees" does: the terminal must show the header and the row whole,
// each on a line of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestWriteTreeGroupsOnTerminal(t *testing.T) {
stderr := captureStderr(t)
super, dirs := treeOf(t, smokeTreeRecs())
dupes := collectTreeGroups(dirs, super, nil)
// No timed draws: the display is drawn as it starts and as it ends.
p := startProgress(progressParams{
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
}, terminalWidth, time.Hour, time.Now)
_, _, err := writeTreeGroups(os.Stderr, dupes, p)
p.finish()
if err != nil {
t.Fatal(err)
}
// A terminal shows each line as the text after its last carriage
// return.
var shown []string
for line := range strings.SplitSeq(stderr(), "\n") {
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
}
want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
t.Errorf("terminal shows lines %q, want %q first", shown, want)
}
}
func TestTreeDigests(t *testing.T) {
t.Parallel()
@@ -354,7 +254,7 @@ func TestCollectTreeGroupsMaximal(t *testing.T) {
super, dirs := treeOf(t, smokeTreeRecs())
groups := collectTreeGroups(dirs, super, nil)
groups := collectTreeGroups(dirs, super)
want := [][]string{{"/d/t1", "/d/t2"}}
if got := groupPaths(groups); !slices.EqualFunc(got, want,
@@ -378,14 +278,14 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
super, dirs := treeOf(t, recs)
forward := groupPaths(collectTreeGroups(dirs, super, nil))
forward := groupPaths(collectTreeGroups(dirs, super))
reversed := slices.Clone(recs)
slices.Reverse(reversed)
superR, dirsR := treeOf(t, reversed)
backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
backward := groupPaths(collectTreeGroups(dirsR, superR))
if !slices.EqualFunc(forward, backward, slices.Equal) {
t.Fatalf("output depends on record order: %v vs %v",
forward, backward)
@@ -404,7 +304,7 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super, nil))
got := groupPaths(collectTreeGroups(dirs, super))
want := [][]string{{"/p/x1", "/p/x2"}}
if !slices.EqualFunc(got, want, slices.Equal) {
@@ -426,7 +326,7 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super, nil))
got := groupPaths(collectTreeGroups(dirs, super))
want := [][]string{{"/p/a/x", "/q/b/x"}}
if !slices.EqualFunc(got, want, slices.Equal) {