sneak/sfdupes
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions 12 Packages Projects Releases Wiki Activity

Compare commits

base: sneak/sfdupes:326584173325bbd3dbceeb9bf87acd163ec77d97
Branches Tags
sneak/sfdupes:main sneak/sfdupes:next sneak/sfdupes:issue-19-markdown-fmt
sneak/sfdupes:v0.0.2 sneak/sfdupes:v1.0.0-rc2 sneak/sfdupes:v1.0.0-rc1 sneak/sfdupes:v0.0.1
..
compare: sneak/sfdupes:next
Branches Tags
sneak/sfdupes:next sneak/sfdupes:main sneak/sfdupes:issue-19-markdown-fmt
sneak/sfdupes:v0.0.2 sneak/sfdupes:v1.0.0-rc2 sneak/sfdupes:v1.0.0-rc1 sneak/sfdupes:v0.0.1
10 Commits
3265841733 .. next
Author SHA1 Message Date
clawbot 6346319701 Install from @latest, retract v0.0.1, print a go-installed binary's version (closes #116)
check / check (push) Waiting to run
Details 
README "Install" now leads with go install ...@latest: until 1.0 each
release is a v0.0.N tag on main's tip, so @latest gives that tip once
it has passed the build. It also says what @main and v0.0.1 give and
how to check an installed binary. go.mod retracts v0.0.1, the early
build without a database.

sfdupes --version printed "dev" for every go-installed binary, since
only the linker set Version. When the linker sets none, it now prints
the main module's version from runtime/debug.ReadBuildInfo; "dev" is
left only when neither is known. Version is now empty by default.

The retraction takes effect only once a release carrying it is the
newest.

Model: opus-5-5
 2026-10-17 05:44:49 +02:00
clawbot 4e4a94db2b scan: create the database's directory where its path leads (closes #119)
check / check (push) Canceled after 0s
Details 
scan created a missing database directory from filepath.Dir of the
path, which cleans it as text and drops each ".." together with the
name before it. After a symlinked directory that is another directory
than the one the system reaches when it opens the file, so scan left an
empty directory beside the symlink and then failed to create the
database. lockScanDatabase and openScanDatabase now both call
createDatabaseDirectory, beside databasePath, which cuts the last
element off the path as text and creates what is missing along it. A
comment in databasePath now says why the working directory has its
symlinks resolved.

Unverified: a missing directory before a ".." in the path is created along the way; no test covers it.

Model: opus-5-5
 2026-10-17 04:44:53 +02:00
clawbot dcfe0ea080 scan: never record its own database files (closes #115)
check / check (push) Canceled after 0s
Details 
When the database lay under a scanned operand, the walk recorded the
database file, its -wal and -shm files and its .lock file like any other
file, so every rescan updated them and report paired them with empty
files. Once the database and the lock file are open, the scan now notes
their device and inode, and the walk passes over a file with one of
them, as an operand or below one, printing no line and counting
nothing. Matching by device and inode holds through a symlinked
directory, where comparing paths would not. A record an earlier scan
left for one of them is deleted as unverified. README's smoke-test note
on keeping the database out of the tree is gone, as it no longer holds.

Judgement call: walkRules, in exclude.go, carries the noted files to the walk.

Model: opus-5-5
 2026-10-17 04:10:50 +02:00
clawbot 003e184f71 scan: skip OS default excludes, and record nothing on filesystems that are not persistent (closes #124)
check / check (push) Canceled after 0s
Details 
scan never records or reads anything on a filesystem that is not
persistent (tmpfs, proc, sysfs and the like), however it reaches it; no
flag turns this off. It notes each such filesystem's device ID as it
starts: on Linux from /proc/self/mountinfo, on macOS by lstat of each
listed mount point. Below an operand the walk also skips the default
excludes, a path list per system (Linux, macOS, and Windows, not used
yet); --no-default-excludes turns off only those. A default exclude that
is itself an operand gets no skip line. The tests stop at once, saying
why, when TMPDIR is not on a persistent filesystem.

Judgement call: an unreadable mount list fails the scan, and no flag gets round it.
Judgement call: a file on such a filesystem below an operand gets the same line as a directory.
Judgement call: such an operand that another operand's walk also reaches is named twice.
Unverified: whether macOS's mount list gives the device lstat reports, so macOS keeps lstat, and a hidden mount there can still add another filesystem's device.
Unverified: the macOS code is never built or run here; bind mounts need root to test.

Model: opus-5-5
 2026-10-17 03:44:52 +02:00
clawbot 68f6bde5ee Show the progress rate over the last 5 seconds, and where a phase waits after 10 seconds with no item done (closes #125)
check / check (push) Canceled after 0s
Details 
The rate was the phase's average, so a burst after a long wait read as
over 1,000 files a second. It now covers the last 5 seconds, or the
time since the previous line off a terminal; the ETA still comes from
the average. Once no item has completed for 10 seconds, the rate gives
way to "stalled" and where the phase waits: disk, db, out (stdout) or
cpu, or two of them, as "stalled disk/db". A line too wide for the
terminal loses its bar first, then its rate or those words, so the
phase, progress, elapsed time and ETA stay whole. The content phase's
last partial batch is now written under a display of its own.

Deviation: the stall words hold no time and name where the phase waits, not what it does.
Judgement call: trees' group phase, which works in memory, says "cpu".

Model: opus-5-5
 2026-10-17 01:10:57 +02:00
clawbot e9e09effc6 Add sfdupes info: show the database's details (closes #117)
check / check (push) Canceled after 0s
Details 
info prints the database's path and source, the sizes of the database
file and its -wal file, when the database file last changed, the schema
version and the record counts, one per line as a name, a tab and the
value. It opens the database as report does and shows an open and a
count phase on stderr. The schema version and the counts come from one
read-only transaction, the counts from one query.

seedDatabase now closes test databases out of WAL mode, as scan does;
otherwise info's own connection creates an empty -wal file. The open
phase's name and text became constants for the linter, and the scan
command moved to newScanCommand to keep newRootCommand short enough.

Judgement call: the lines are written after the last display ends, so they skip the display's stdout writer.

Model: opus-5-5
 2026-10-16 22:41:16 +02:00
clawbot ebe4d4dbeb report: show its phases as N of M, with progress, elapsed time and ETA (closes #114)
check / check (push) Canceled after 0s
Details 
report now runs in two phases, open and read, under the progress
display scan and trees use, until the summary line. open covers a wait
on a busy database. read counts the records, then reads them in the
same read-only transaction in signature order, the files_signature
index read backwards, so SQLite sorts nothing and the read ends at
exactly its count. The duplicate groups are formed in Go from runs of
equal signatures; only those of the size being read are held, and they
are written, ordered by first path, before the next size. The rows go
through the display's stdout writer. stdout and the summary line are
unchanged.

Judgement call: writing is part of read, not a phase of its own, since each size's rows are written while the read goes on.

Model: opus-5-5
 2026-10-16 21:44:55 +02:00
clawbot 8e0facdcd3 trees: show its phases as N of M, with progress, elapsed time and ETA (closes #113)
check / check (push) Canceled after 0s
Details 
trees now runs in four phases, open, read, group and write, each under
the progress display scan uses, until the summary line. open covers a
wait on a busy database. read counts the records and reads them in one
read-only transaction, so the two agree during a scan; group counts the
directories, and write the groups. The rows reach stdout through a
writer. When stdout and stderr are the same file, it clears the display
and holds its lock while writing, so no row shares a line with the
display on a shared terminal; otherwise it writes without the lock, so
a slow reader of stdout cannot stop the display. Each rate names what
its phase counts.

Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches.

Model: opus-5-5
 2026-10-16 21:00:14 +02:00
clawbot 8dc73aa3f5 scan: show each phase as N of M, with progress, elapsed time and ETA, also while it waits (closes #112)
check / check (push) Canceled after 0s
Details 
Every scan display names its phase and its place among the five, as
"[2/5] walk", with its progress, elapsed time and ETA. load counts the
records first; the walk estimates its total from the records under the
operands; the content phase adds up the sizes of the matching groups
and says it is searching until the first record returns. Each display
redraws every 100 ms on a terminal and every 5 seconds in plain mode.

On a terminal no line reaches the last column: a line too wide drops
its rate, then its bar. After the terminal is made narrower than the
widest line drawn, the display makes a new library object, whose erase
fits.

Judgement call: load and the content checks draw the bar too, once their totals are known.
Judgement call: the bar is 10 columns wide, not the width the line leaves.

Model: opus-5-5
 2026-10-16 19:11:10 +02:00
clawbot 264817c413 Name the database each command uses on stderr (closes #111)
check / check (push) Canceled after 0s
Details 
scan, report and trees now print one line on stderr before anything
else, naming the database and whether its path is the default or comes
from SFDUPES_DATABASE; scan also says when the file is new. report and
trees print it before opening the database, so a missing-database error
follows it. A relative path gets the working directory put in front of
it, with its symlinks resolved as the kernel reports it, and is not
cleaned: a ".." after a symlink names another file than the cleaned
path does. README "Database" lists when scan writes each file.

Unverified: the line on a terminal has no test of its own.

Model: opus-5-5
 2026-10-16 15:28:02 +02:00
22 changed files with 4872 additions and 739 deletions
Expand all files Collapse all files
README.md
+386 -118
View File
@@ -38,8 +38,9 @@ database persists between runs; a rescan only hashes files that are new or
changed, or that may have gained a duplicate since the last scan, and removes
records for files that no longer exist. `report` reads the database and prints
the file-level duplicates report. `trees` reads the same database and prints the
duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
with no `PATH` operand — prints a usage message and exits 2.
duplicate-tree report. `info` prints the database's details: its path, the sizes
of its files and the number of its records. A missing/invalid subcommand — or a
`scan` invocation with no `PATH` operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
@@ -48,15 +49,40 @@ results are as fresh as the last completed scan.
### Install
With Go installed, this builds and installs the current `main` branch:
With Go installed, this builds and installs the newest release:
```sh
go install sneak.berlin/go/sfdupes@latest
```
Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time
`main` moves and its tip passes `script/cibuild`, that tip is tagged with the
next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build.
`@main` installs the tip before it is tagged:
```sh
go install sneak.berlin/go/sfdupes@main
```
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. A
binary installed this way reports its version as `dev`; one built from a clone
or into the Docker image carries the git tag or commit it was built from.
`v0.0.1` is an early build without a database. It does not work as this README
describes, and `go.mod` retracts it.
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To
see which version an installed binary is:
```sh
sfdupes --version
go version -m "$(command -v sfdupes)"
```
`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`,
`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`,
or for `@main` a pseudo-version ending in the first 12 characters of the
commit's hash. A binary built from a clone or into the Docker image carries the
git tag or commit it was built from instead, and `VERSION` is `dev` only when
the binary carries no version at all. `go version -m` prints what Go recorded in
the binary; its `mod` line names the module's version. Binaries from `v0.0.2`
and earlier print `dev`, so check those with `go version -m`.
From a clone, `make build` writes the binary to `./sfdupes`:
@@ -97,10 +123,13 @@ As a file `/etc/cron.d/sfdupes`:
`SFDUPES_DATABASE=/path/to/db.sqlite` before the command on the same line.
- `scan` writes nothing to stdout. Its stderr, appended here to
`/var/log/sfdupes.log`, holds a first line naming the database (see
"Database"), a plain progress line as each phase starts and then at most every
5 seconds, a warning for each path it skips, and the summary line (see
"Progress" and "`scan` mode"). The log grows with every scan; rotate it like
any other.
"Database"), plain progress lines naming each phase and its place among the
five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
runs, and one as it ends), a warning for each path it skips, a line for each
default exclude its walk does not enter and for each path it leaves out
because it is not on a persistent filesystem (see "Rules for the walk"), and
the summary line (see "Progress" and "`scan` mode"). The log grows with every
scan; rotate it like any other.
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
reason among the last lines of the log; `tail` prints them, and cron mails
@@ -108,9 +137,9 @@ As a file `/etc/cron.d/sfdupes`:
- A scan still running when the next one starts carries on. The new one fails at
once, and the lines cron mails include
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
- `report` and `trees` need only read access to the database (see "Database").
Under the usual umask of `022` the first scan creates it readable by every
user, so an unprivileged user can run them against root's database.
- `report`, `trees` and `info` need only read access to the database (see
"Database"). Under the usual umask of `022` the first scan creates it readable
by every user, so an unprivileged user can run them against root's database.
### Reading the reports
@@ -198,12 +227,13 @@ Goals, in order:
under resilver). Holding one small record (path, size, mtime) per file in
memory during a scan is acceptable; holding every file's hashes is not (they
stay in the database). The reporting commands do not hold every file's
hashes either: `report` lets SQLite group and order the records and writes
each row as it reads it, so its memory does not grow with the database, and
`trees` reads the records in path order and keeps each directory's path,
digest and totals, plus the hashes of only the files in the directories
holding the record being read, so its memory grows with the number of
directories and with the size of the largest directory.
hashes either: `report` reads the records in signature order and holds the
duplicate paths of one size at a time, so its memory grows with the
duplicates of the size that has the most of them, and `trees` reads the
records in path order and keeps each directory's path, digest and totals,
plus the hashes of only the files in the directories holding the record
being read, so its memory grows with the number of directories and with the
size of the largest directory.
3. **Scan incrementally, analyze offline.** The expensive filesystem scan
maintains a persistent database; an unchanged file is never read again on a
rescan, except to compute its content hash once a file of 10 MiB or more
@@ -233,7 +263,7 @@ Goals, in order:
### Subcommands
Three subcommands, all implemented:
Four subcommands, all implemented:
1. `scan` — walk the filesystem and synchronize the database: one signature
record per regular file.
@@ -241,26 +271,30 @@ Three subcommands, all implemented:
3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy
from the database records, compute a Merkle-style digest per directory, and
report maximal groups of identical trees.
4. `info` — the database's details: its path, the sizes of its files, its
schema version and the counts of its records.
```
sfdupes scan [--workers N] [-x] PATH...
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
sfdupes info
sfdupes --version
sfdupes [command] --help
```
`--workers N` sets the size of each `scan` worker pool (default: the number of
CPUs), and `-x` (`--one-file-system`) keeps the walk of each operand on that
operand's filesystem; both are described under "`scan` mode".
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
subcommand, prints the help text to stderr and exits 0, writing nothing to
stdout.
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
operand's filesystem, and `--no-default-excludes` lets the walk enter the
default excludes, a list of paths; all three are described under "`scan` mode".
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see
"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`,
alone or after a subcommand, prints the help text to stderr and exits 0, writing
nothing to stdout.
### Database
All three subcommands operate on a single SQLite database file:
All four subcommands operate on a single SQLite database file:
- Location: the value of the `SFDUPES_DATABASE` environment variable when set
and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
@@ -283,13 +317,13 @@ All three subcommands operate on a single SQLite database file:
symlinks resolved. After a `cd` through a symlink to a directory, a path
starting `../` therefore names a file beside the directory the symlink leads
to, as it does for any other program. `scan` says `new database` when no
file is at that path as it starts. `report` and `trees` print the line
before they open the database, so the error for a missing database comes
after it.
file is at that path as it starts. `report`, `trees` and `info` print the
line before they open the database, so the error for a missing database
comes after it.
- `scan` creates the database (and its parent directory) on first use. `report`
and `trees` require an existing database; a missing database file is a fatal
error (exit 1) telling the user to run `scan` first.
- `scan` creates the database (and its parent directory) on first use. `report`,
`trees` and `info` require an existing database; a missing database file is a
fatal error (exit 1) telling the user to run `scan` first.
- Only one `scan` runs against a database at a time. For its whole run, `scan`
holds an exclusive `flock(2)` lock on a lock file beside the database, named
by appending `.lock` to the database's path with symlinks resolved
@@ -305,8 +339,8 @@ All three subcommands operate on a single SQLite database file:
on. The lock file is created on first use, open to its owner only, and left in
place: a leftover file blocks nothing, because the lock ends with the process
holding it however it ends, a fatal error or an interrupt included, and
deleting the file while a scan runs would let a second scan start. `report`
and `trees` never take the lock, so they run during a scan.
deleting the file while a scan runs would let a second scan start. `report`,
`trees` and `info` never take the lock, so they run during a scan.
- While `scan` runs, the database is in WAL journal mode with a busy timeout, so
running a report while a cron `scan` is in progress is safe. The filesystem is
authoritative; the database is an eventually-consistent reflection of it.
@@ -322,9 +356,9 @@ All three subcommands operate on a single SQLite database file:
needs the database to itself: a `scan` that starts while a report is still
reading waits for it up to the 10-second busy timeout, then fails; a `scan`
that ends while a report has the database open warns and leaves the database
in WAL mode until the next scan. `report` writes each row as it reads it, so
it is still reading while its output is paused (a pager, a stalled pipe), and
a `scan` started then fails after the busy timeout.
in WAL mode until the next scan. `report` writes its rows while it reads the
records, so it is still reading while its output is paused (a pager, a stalled
pipe), and a `scan` started then fails after the busy timeout.
- A `scan` writes to the database and the files beside it at these points (its
phases are described under "`scan` mode"):
- As it starts, before the walk: the directory, an empty database file and
@@ -339,10 +373,10 @@ All three subcommands operate on a single SQLite database file:
- In the content phase: content hashes, again committed 10,000 at a time.
- At the end, as it closes the database: the `-wal` and `-shm` files go
away, unless a report still has the database open (see above).
- `report` and `trees` open the database read-only and need only read access to
the database file, and no write access to its directory. While the database is
in WAL mode they also read the `-wal` and `-shm` files beside it, which SQLite
creates with the database file's permissions.
- `report`, `trees` and `info` open the database read-only and need only read
access to the database file, and no write access to its directory. While the
database is in WAL mode they also read the `-wal` and `-shm` files beside it,
which SQLite creates with the database file's permissions.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
with any other version is a fatal error. `scan` creates the schema and sets
the version in one transaction, so a first scan stopped while doing so leaves
@@ -378,8 +412,8 @@ All three subcommands operate on a single SQLite database file:
a file of 10 MiB or more, `content` stays empty until the content phase of a
scan (see "`scan` mode" below) has read the file. A record with an empty
`content` is never part of a duplicate group, though it still defines the
file for tree reconstruction. The `files_signature` index lets SQLite group
the records by signature for `report` without sorting the whole table.
file for tree reconstruction. The `files_signature` index lets `report` read
the records in signature order without sorting the table.
### Duplicate detection
@@ -433,11 +467,12 @@ results do not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared walk worker
pool. Overlapping operands are harmless — an operand that duplicates another or
lies under another is dropped before walking, so every file is reached exactly
once and produces one database record. With `-x`, though, an operand under
another one is kept and walked on its own when any directory from it (from its
directory, for a regular file) up to but not including the operand covering it
is on a different filesystem than that operand, because that operand's walk does
not enter such a directory and so does not reach it.
once and produces one database record. An operand under another one is kept,
though, and walked on its own when any directory from it (from its directory,
for a regular file) up to but not including the operand covering it is a default
exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
`-x`, is on a different filesystem than that operand, because that operand's
walk does not enter such a directory and so does not reach it.
An operand that is a symlink (never followed, not even as an operand), socket,
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -447,7 +482,10 @@ beneath it is still scanned. The records stored beneath it are not deleted: they
are treated like any other record outside the scanned operands, including the
content-phase exception below. If it lies under another operand, they are under
that operand instead, and are treated like any other record there. This is not
an error: a scan whose every operand is dropped walks nothing and exits 0.
an error: a scan whose every operand is dropped walks nothing and exits 0. An
operand that is not on a persistent filesystem (see "Rules for the walk") is
dropped the same way, except that the line `scan` prints for it is not a
warning, and it is not counted as skipped.
`scan` synchronizes the database with the filesystem state under the scanned
operands:
@@ -483,19 +521,26 @@ operands:
can be scanned on different schedules into the same database. The one
exception is the content phase below: a stored file of 10 MiB or more without
a `content` hash is read for one, wherever it lies, once its size, `head`, and
`tail` match another record's. If that file is gone or has changed since its
record was written, the record is left as it is.
- With `-x`, a record below a directory the walk did not enter because that
directory is on another filesystem than its operand counts as outside the
scanned operands: it is not deleted, and its size joins the size census. A
filesystem mounted under an operand can therefore be scanned on its own
schedule into the same database. Records below an operand at or inside such a
directory are not kept this way, since that operand's own walk verifies them.
`tail` match another record's. If that file is gone, has changed since its
record was written, or is not on a persistent filesystem, it is not read and
the record is left as it is.
- A record at or below a path the walk left out, because that path is a default
exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
another filesystem than its operand, counts as outside the scanned operands:
it is not deleted, and its size joins the size census. A filesystem mounted
under an operand can therefore be scanned on its own schedule into the same
database. Records below an operand at or inside such a path are not kept this
way, since that operand's own walk verifies them.
`scan` runs **four sequential phases over the whole scan**. Parallelism lives
inside each phase; batched database writes begin during the hash phase:
`scan` runs **five sequential phases over the whole scan**. Parallelism lives
inside the walk, hash and content phases; batched database writes begin during
the hash phase:
1. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
1. **load** — read the existing records. Each record under the operands is kept
in memory, by path, with its size, its mtime and whether it has hashes, for
the walk to compare each file with; the size of every other record joins the
size census.
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
with the walk worker pool: every operand seeds the shared queue, and each
worker reads one directory at a time, handing discovered subdirectories back
to the queue and running `lstat` on each regular file as it is discovered
@@ -505,7 +550,7 @@ inside each phase; batched database writes begin during the hash phase:
or busy pools. The walk builds the size census and resolves unchanged
already-hashed files on the fly; every other file is carried to the hash
phase as a (path, size, mtime) record.
2. **hash** — with the census complete, each carried file's size decides its
3. **hash** — with the census complete, each carried file's size decides its
fate. Size-unique files are never read: new or changed ones are recorded
without hashes in the update phase, unchanged unhashed ones simply keep
their records. Every file with a shared size is hashed by the worker pool as
@@ -521,29 +566,30 @@ inside each phase; batched database writes begin during the hash phase:
**while hashing runs**, so a scan interrupted after hours keeps everything
hashed so far and the next scan resumes cheaply, skipping records already
written.
3. **update** — commit the final partial batch, the hash-less records for
4. **update** — commit the final partial batch, the hash-less records for
size-unique new and changed files, and the deletions for records the scan
did not verify (vanished files, plus paths that failed to stat or hash).
4. **content** — find every record of 10 MiB or more without a `content` hash
5. **content** — find every record of 10 MiB or more without a `content` hash
whose size, `head`, and `tail` equal another record's, anywhere in the
database: records from this scan and records stored by earlier scans, inside
or outside the scanned operands. SQLite finds them, so only the records to
be read are kept in memory, never every file's hashes. Every record sharing
their size, `head`, and `tail`, including one that already has a `content`
hash, has its file checked with `lstat` first. A file that is gone, is no
longer a regular file, or has changed (a different size, or an mtime newer
than recorded) keeps its record as it is and does not count as a match for
the others. Any other `lstat` error is warned about and counted as skipped,
with the same result. If such a record has no `content` hash, it stays out
of duplicate groups; if it has one, it is still reported until a scan
covering its own tree updates or removes it. The files that pass and have no
`content` hash are read only if at least two of those records pass, so a
file whose only matches are stale costs no read; a file that already has a
`content` hash is never read again. They are read by a worker pool as in the
hash phase, in inode order and once per inode, and their content hashes are
committed in batches. A failed read is warned about and counted as skipped;
its record keeps an empty `content`, so it is not a duplicate, and a later
scan tries again.
longer a regular file, has changed (a different size, or an mtime newer than
recorded), or is not on a persistent filesystem (see "Rules for the walk")
keeps its record as it is and does not count as a match for the others. Any
other `lstat` error is warned about and counted as skipped, with the same
result. If such a record has no `content` hash, it stays out of duplicate
groups; if it has one, it is still reported until a scan covering its own
tree updates or removes it. The files that pass and have no `content` hash
are read only if at least two of those records pass, so a file whose only
matches are stale costs no read; a file that already has a `content` hash is
never read again. They are read by a worker pool as in the hash phase, in
inode order and once per inode, and their content hashes are committed in
batches. A failed read is warned about and counted as skipped; its record
keeps an empty `content`, so it is not a duplicate, and a later scan tries
again.
Rules for the walk:
@@ -554,6 +600,15 @@ Rules for the walk:
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
them would list every file once per snapshot), not even when it is an operand;
such an operand is dropped the same way.
- Never record the scan's own database. Once the database and its `.lock` file
are open, `scan` notes the device and inode of the database file, its `-wal`
and `-shm` files and its `.lock` file, and the walk passes over a file with
one of them, however it reaches it: as an operand, below one, or through a
symlinked directory. It prints no line for such a file and does not count it
as skipped. A record that an earlier scan left for one of them is deleted,
like the record of any file the scan no longer verifies. There is no
`-journal` file during the walk, because the database is in WAL mode then (see
"Database").
- Filesystem boundaries are crossed by default. With `-x` (long form
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
into a directory on a different filesystem than its `PATH` operand; each
@@ -561,6 +616,74 @@ Rules for the walk:
unless they lie below an operand at or inside that directory, count as outside
the scanned operands, as described in "`scan` mode" above: they are not
deleted, and their sizes join the size census.
- Never record anything that is not on a persistent filesystem. A filesystem is
not persistent when the kernel makes it up or keeps it in memory: one of type
`proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
`debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
`mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
`rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
it names the database and before it takes the lock, `scan` lists the mounted
filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
of mounted filesystems on macOS (on any other system it lists none), and notes
the device ID of each one of those types. On Linux that is the device ID the
list gives, so a filesystem that a later mount hides still gives its own. On
macOS it is read with `lstat` on the mount point, and where one filesystem is
mounted over another on the same mount point, the one mounted last counts. If
`scan` cannot list them, it fails (exit 1). The walk then never enters a
directory, and never records a file, on one of those devices, however it
reaches it: as an operand, below one, or through a bind mount, which keeps the
device ID of what it binds. The content phase never reads a file on one of
them either. No flag turns this off. For each operand, directory or file it
leaves out this way, `scan` prints a line on stderr naming it and the type of
its filesystem, as it prints a warning, so it never lands inside a progress
display:
```
scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
```
That line is not a warning, and the path is not counted as skipped. Records
that an earlier scan left at or below it, unless they lie below an operand
at or inside it, count as outside the scanned operands, as described in
"`scan` mode" above: they are not deleted, and their sizes join the size
census.
- Below an operand, never descend into a default exclude: a path that holds the
kernel's view of the running system, or temporary files, so that
`sfdupes scan /` walks only the files worth comparing. The default excludes
depend on the system, and `sfdupes scan --help` lists those of the system it
runs on:
- On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
- On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
`/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
everything in `/System/Volumes/Data` through the firmlinks, so walking it
as well would record each of those files twice.
- On Windows, relative to the root of the drive an operand lies on: the
page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
`hiberfil.sys`, and the directories `System Volume Information`,
`$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
matches one path component, as it does in any default exclude. sfdupes
does not build for Windows yet, so this list is not used.
- On any other system there are none.
`--no-default-excludes` turns them all off; what is not on a persistent
filesystem stays left out. An operand at or inside a default exclude is
still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
persistent filesystem. For each default exclude its walk does not enter,
`scan` prints a line on stderr naming it, as it prints a warning:
```
scan: skipped /var/tmp (default exclude)
```
That line is not a warning, and the directory is not counted as skipped. A
default exclude that is not on a persistent filesystem, as `/proc` is, gets
the line for its filesystem instead, and one that is itself an operand gets
no line, since it is walked as that operand. The records below a default
exclude the walk does not enter, unless they lie below an operand at or
inside it, count as outside the scanned operands, as described in "`scan`
mode" above: they are not deleted, and their sizes join the size census.
- On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were
@@ -578,8 +701,9 @@ files. `--workers` must be at least 1: a smaller value is a usage error,
reported in one line on stderr with exit 2 before anything is scanned. All three
phases are seek-bound on spinning disks, so raising `--workers` well past the
core count can help on pools with many spindles. The main goroutine owns
partitioning, database writes, and progress rendering; progress display must
never block the workers.
partitioning, database writes, and the progress counts, which each progress
display draws from a goroutine of its own; progress display must never block the
workers.
`scan` writes nothing to stdout. The summary line on stderr reports the files
seen this run broken down by disposition, plus skips:
@@ -597,11 +721,8 @@ total.)
**`report` must never touch the filesystem being analyzed.** It does not stat,
open, or otherwise access any path that appears in the records; its only I/O is
reading the database, writing stdout/stderr, and the temporary file SQLite sorts
in when the duplicate rows do not fit in memory. SQLite puts that file in
`$SQLITE_TMPDIR` or `$TMPDIR` when set, otherwise in `/var/tmp` (or `/tmp`), and
deletes it as soon as it has opened it. `report` must produce identical output
whether or not the scanned filesystem is still mounted.
reading the database and writing stdout/stderr. `report` must produce identical
output whether or not the scanned filesystem is still mounted.
Processing:
@@ -615,6 +736,16 @@ Processing:
by `first` path ascending. Output must be fully deterministic for a given
database state.
`report` runs in two phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in signature order, largest size first. That is the order of the
`files_signature` index read backwards, so SQLite sorts nothing and returns the
first record at once. The records sharing a signature come together, and so do
the records of one size, so `report` holds only the duplicate groups of the size
it is reading, and writes them, ordered by `first` path, once it reaches a
smaller size. Its memory holds at most the duplicate paths of one size at a
time, never every record.
#### Report output format
TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
@@ -688,6 +819,12 @@ Processing:
- Order groups by total tree size descending, tie-broken by `first` path
ascending. Output must be fully deterministic for a given input.
`trees` runs in four phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in path order, building the hierarchy and computing the digests as it goes;
`group` groups the directories by digest and picks the maximal groups; `write`
writes the groups to stdout.
#### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
@@ -706,57 +843,192 @@ Summary to stderr: records read, number of duplicate-tree groups, number of dupe
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
units.
### `info` mode
`info` takes no positional arguments and prints the database's details on
stdout, one per line: a name, a tab, and the value. It opens the database the
way `report` does, read-only and without the scan lock, so it runs during a scan
and needs only read access (see "Database"). Like `report`, it never touches the
scanned filesystem.
```
database /var/lib/sfdupes/db.sqlite (default)
database file size 2147483648 (2.0 GiB)
-wal file size 4124152 (3.9 MiB)
modified 2026-10-16T12:02:50Z
schema version 1
records 12345678
records with head and tail hashes 1234567
records with content hash 123456
size of recorded files 150000000000000 (136.4 TiB)
```
- `database` is the path and where it came from, as on the line every subcommand
prints first on stderr (see "Database").
- `database file size` and `-wal file size` are in bytes, then in human units.
The `-wal` line is there only when a `-wal` file is beside the database, as
while a scan runs.
- `modified` is the time the database file was last changed, in UTC. While the
database is in WAL mode, the newest changes may be in the `-wal` file alone.
- `schema version` is the database's `PRAGMA user_version`. A database with
another version than this build's is refused, as by `report`.
- `records` is the number of records. `records with head and tail hashes` and
`records with content hash` count the records that have those hashes (see
"Database"), and `size of recorded files` adds up the sizes of all records, in
bytes, then in human units.
`info` runs in two phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `count` reads the schema version and
counts the records with one query, a single pass over the table, both in one
read-only transaction, so the numbers agree with each other while a scan is
committing. `info` writes its lines once the `count` display has ended, so on a
terminal they come after the displays.
### Progress
Use the progress-bar library for all scan progress; rendering in the style of
`pv` is the model. All progress goes to stderr.
Use the progress-bar library for all progress of `scan`, `report`, `trees` and
`info`; rendering in the style of `pv` is the model. All progress goes to
stderr.
Each phase gets its own display, rendered the moment the phase starts — a scan
must never look hung. Loading the existing-record index (`load`) and the walk
have no known totals while running: show a live count, rate, and elapsed time
(spinner-style, no percentage or ETA). The content phase's display (`content`)
starts the same way, counting the records checked while SQLite finds the files
to read and `lstat` checks them, then shows a bar once reading starts. The hash
and update phases, and the content phase's reads, have exact totals — only files
that actually need hashing appear in the hash and content totals, so their ETAs
are meaningful. Required elements for the bars with known totals:
`scan`, `report`, `trees` and `info` must always show that they are working and
not hung. Each phase gets its own display from the moment it starts, naming the
phase and its place among the command's phases and showing its progress, its
elapsed time and its estimated time remaining (ETA).
A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
`[4/5] update`, `[5/5] content`):
- `load` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `walk` has no exact total. Its estimate is the number of records the last scan
left under the operands, known once `load` is done and shown with a `~`. Once
the walk passes the estimate, or when there are no such records, as on a first
scan, the ETA is unknown.
- `hash` and `update` have exact totals. Only files that actually need hashing
appear in the hash total, so its ETA is meaningful.
- `content` first counts the records whose files it checks with `lstat`, which
SQLite does by adding up the number of records with each matching size,
`head`, and `tail`, without fetching the records. It then says it is searching
the database until SQLite returns the first record; the checks, and their rate
and ETA, start there. The files to read then get a display of their own, with
their count as its total, and so does the writing of the last of their records
to the database, with the count of those records as its total.
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
`[4/4] write`, see "`trees` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `group` has the directories `read` found as its total, and `write` the
duplicate-tree groups.
`report` has two phases (`[1/2] open`, `[2/2] read`, see "`report` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count, writing
the rows to stdout as it goes. While the count runs, the display says it is
counting the records.
`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `count` has no total either, because one query counts everything: the display
says it is counting the records until the query returns, and its ETA is
unknown.
Each display is one line holding:
- the phase and its place among the phases
- a `[m/n] x%` display (items processed / total items, percent); before a phase
has a total, the items processed, or what the phase is waiting for
- current rate (items/s), naming the items the phase counts: files, records,
directories or groups. It covers the items completed over the last 5 seconds,
or, when stderr is not a terminal, since the line before, so it falls to 0
while nothing completes, and a burst of items after a wait shows as a burst
rather than as a steady rate. Once no item has completed for 10 seconds, the
word `stalled` and where the phase waits take the rate's place (see below).
- elapsed time
- estimated time remaining
- a `[m/n] x%` display (items processed / total items, percent)
- current rate (items/s)
- estimated time remaining, or `unknown`. It comes from the phase's average rate
since it started counting, so that it does not swing with each burst.
Example shape (exact layout is flexible, content is not):
```
hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
[3/5] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
[2/5] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
[3/5] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
```
When no item has completed for 10 seconds, as while one large file is read, a
disk is slow or the database is busy, the line says `stalled` and where the
phase waits in place of the rate, which would read 0: `disk` for files, `db` for
the database, `out` for stdout, and two of them, as `disk/db`, for a phase that
may wait on either. `walk` says `disk`. `hash` and the content phase's reads say
`disk/db`, as they read files and write their records to the database between
them, and so do the content phase's checks, which `lstat` each record's file as
the database search returns it. `load`, `update`, the content phase's last write
and the `read` phase of `trees` say `db`. `trees`' `write` says `out`, and its
`group`, which works in memory, says `cpu`. The `read` phase of `report` writes
the report while it reads the records, so it says `db/out`. A phase still
waiting for its total already says what it waits for, and says nothing more.
Additional requirements:
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
the `open` phase of `report` and `trees`, and both phases of `info`, once it
has counted what it will do) is drawn as its line followed by the progress-bar
library's bar, 10 columns wide, which shows the percent in place of the line's
own. Any other phase, and a phase whose bar has no room, is drawn as the
library's spinner followed by its line. Each display is drawn again several
times a second, so its elapsed time keeps moving while a phase waits for its
next item.
- On a terminal no line reaches the terminal's last column, however narrow the
terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, at counts like those of
the stalled line in the example above, a stalled line keeps its words whenever
the elapsed time and the ETA are both in minutes, except in the content
phase's checks and reads. Their lines are the widest, and at most times in
minutes they lose the words. Where the words and the ETA do not both fit, as
at millions of files and times in hours, the words give way. Only when those
four alone are too wide, as on a terminal narrower than about 70 columns hours
into a scan of millions of files, is the line cut short at the terminal's
edge, so the end of the ETA is lost first. A terminal made narrower while a
phase runs gets the same: the next draw erases the line within the new width
and fits the line to it, and the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print a plain one-line progress update the moment each phase starts,
then no more often than every 5 seconds.
- Progress updates are driven from the main goroutine and must be non-blocking
with respect to the worker pool. On a terminal the spinner-style displays also
redraw on their own several times a second, so their count and elapsed time
stay current while a phase waits for its next item.
redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once
more as the phase ends.
- Progress must be non-blocking with respect to the worker pool: the main
goroutine counts the items, and each display draws them from a goroutine of
its own.
- A warning printed during a phase always lands on a line of its own, never
inside the progress display.
- A bar whose phase stops short of its total, as an interrupted one does, is
left as last drawn rather than filled up.
- `report` and `trees` modes need no progress display, only their stderr
summaries.
inside the progress display. So does every line written to stdout when stdout
and stderr are the same file (the same device and inode): one terminal, as in
a plain `sfdupes report` or `sfdupes trees`, or one pipe or file that both are
sent to. When they are not, stdout is written without waiting on the display,
and the display keeps being drawn while a slow reader of stdout holds up a
write.
- A phase that stops short of its total, as an interrupted one does, is left
showing the count it stopped at rather than filled up.
### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is
already running against the same database, the database cannot be
created/opened/read/written, a missing database for `report`/`trees`, stdout
write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see below).
created/opened/read/written, a missing database for `report`/`trees`/`info`,
stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see
below).
- `2`: usage error (including `scan` with no `PATH` operand, `scan` with
`--workers` below 1, and `report`/`trees` with any positional argument).
`--workers` below 1, and `report`/`trees`/`info` with any positional
argument).
A stdout write failure, such as a full disk, is reported in one line on stderr
and exits 1. Two cases never reach sfdupes as a failed write:
@@ -923,10 +1195,6 @@ All of the following, run in this directory, must pass:
./sfdupes report
```
(The database lives in a temp directory of its own: inside `$d`, the scan
would record it, and its empty lock file would join the `empty1`/`empty2`
group.)
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
group (two dupe rows, `first` is the lexicographically smallest path);
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
TODO.md
+107
View File
@@ -29,6 +29,113 @@
# Completed Steps
- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so
`go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has
passed the build; README "Install" says so, says what `@main` and `v0.0.1`
give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the
early build without a database. When the linker set no version,
`sfdupes --version` prints the main module's version that Go recorded in the
binary, the tag or the pseudo-version of a `go install` build, and `dev` only
when neither is known (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/116)
- `scan` creates the database's directory where the database path leads: the
path with its last element cut off as text, never cleaned, so a `..` after a
symlinked directory is followed through the symlink, as when the file is
opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
symlink to `/data/x`, it creates `/data/new` and the database in it, where it
used to create `/srv/new` and then fail (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/119)
- `scan` never records its own database: once the database and its `.lock` file
are open, it notes the device and inode of the database file, its `-wal` and
`-shm` files and its `.lock` file, and the walk passes over a file with one of
them wherever the operands reach it, through a symlinked directory included,
printing no line and counting nothing. A record an earlier scan left for one
of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
- `scan` never records or reads anything that is not on a persistent filesystem
(`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
matched by the device ID of each such filesystem, listed as the scan starts
from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
however the walk or the content phase reaches it; no flag turns that off.
Below an operand, its walk does not enter the default excludes either, so
`sfdupes scan /` skips the kernel's view of the running system and the
temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
`/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
`/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
Windows list, not used yet. An operand at or inside a default exclude is still
walked. The scan prints a line for each path it leaves out, without counting
it as skipped, and the records there count as outside the scanned operands, as
for `-x`. `--no-default-excludes` turns off the default excludes only, and
`scan --help` lists them (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/124)
- the rate a progress display shows covers the items completed over the last 5
seconds, or since the line before when stderr is not a terminal, so it falls
to 0 while nothing completes, and a burst after a wait no longer reads as a
steady rate of over 1,000 files a second; the ETA still comes from the phase's
average. Once no item has completed for 10 seconds, the word `stalled` and
where the phase waits take the rate's place, as `stalled disk/db`: `disk`,
`db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
either, each phase giving its word where it is defined. A line too wide for
the terminal loses its bar first, then its rate or those words, so the phase,
progress, elapsed time and ETA stay whole. On an 80-column terminal, at counts
like those of README's example, the content phase's checks and reads lose
those words at most times in minutes, and every other phase keeps them
whenever the elapsed time and the ETA are both in minutes. The content phase's
last partial batch is now written under a display of its own (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/125)
- new subcommand `info` prints the database's details on stdout, one per line as
a name, a tab and the value: its path and where it came from, the sizes of the
database file and of its `-wal` file when there is one, when the database file
was last changed, the schema version, the number of records, how many have
head and tail hashes and how many a content hash, and the sizes of all records
added up. It opens the database as `report` does, and shows two phases on
stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come
from one read-only transaction, the counts from one query (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/117)
- `report` shows its work on stderr in two phases, `[1/2] open` and
`[2/2] read`, with the progress display `scan` and `trees` use, from the
moment it starts until the summary line: `open` says it is opening the
database, also while it waits on a busy one; `read` counts the records, then
reads them against that count in signature order, which SQLite takes from the
`files_signature` index without sorting, so the first record comes at once.
`report` forms the duplicate groups itself, holding only those of the size it
is reading, and writes them ordered by first path before it reads the next
size. stdout and the summary line are unchanged, and SQLite no longer needs a
temporary file to sort in (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/114)
- `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`,
`[3/4] group` and `[4/4] write`, each with the progress display `scan` uses,
from the moment it starts until the summary line: `open` says it is opening
the database, also while it waits on a busy one; `read` counts the records,
then reads them against that count; `group` counts the directories it groups,
and `write` the groups it writes. A display's rate names what its phase
counts: files, records, directories or groups. stdout and the summary line are
unchanged. When stdout and stderr are the same file, as one terminal is, none
of stdout's lines shares a line with the display; when they are not, the
display keeps being drawn while stdout waits on a slow reader (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/113)
- each `scan` phase's progress display names the phase and its place among the
five, as `[2/5] walk`, from the moment it starts, with its progress, elapsed
time and ETA: `load` counts the records first, the walk's total is estimated
from the records under the operands, and the content phase counts the records
it checks from SQLite's grouping alone, then says it is searching the database
until the first one is returned. On a terminal a phase with an exact total
draws its line followed by the library's bar, and any other a spinner followed
by the line; every display is drawn again several times a second. No line
reaches the terminal's last column, however narrow, also after the terminal is
made narrower during a phase: a line too wide leaves out its rate, then its
bar, keeping the phase, progress, elapsed time and ETA whole, and is cut short
only when those alone do not fit, as below about 70 columns late in a large
scan. In plain mode the line is printed every 5 seconds whether or not an item
completed (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/112)
- `scan`, `report` and `trees` first print a line on stderr naming the database
they use, as an absolute path, and whether that path is the default or comes
from `SFDUPES_DATABASE`; `scan` says when the database is new. README
cancel_test.go
+15 -10
View File
@@ -196,7 +196,7 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
base := baselineGoroutines(t)
st, err := syncScan(newWalkClock(walkCancelAtDone), db,
[]string{dir}, walkCancelWorkers, false)
[]string{dir}, walkCancelWorkers, false, walkRules{})
assertWalkGuardAborted(t, st, err)
assertRecordsIntact(t, db, before)
@@ -268,7 +268,8 @@ func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
cancel()
st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false)
st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false,
walkRules{})
if !errors.Is(err, context.Canceled) {
t.Fatalf("syncScan on a cancelled context = %v, want %v",
err, context.Canceled)
@@ -301,7 +302,7 @@ func TestSyncScanCancelledMidHashKeepsHashedRecords(t *testing.T) {
db := openTestDB(t)
st, err := syncScan(newWalkClock(hashCancelAtDone), db,
[]string{dir}, walkCancelWorkers, false)
[]string{dir}, walkCancelWorkers, false, walkRules{})
if !errors.Is(err, context.Canceled) {
t.Fatalf("syncScan cancelled mid-hash = %v, want %v",
err, context.Canceled)
@@ -356,7 +357,8 @@ func TestRunScanInterrupted(t *testing.T) {
stderr := captureStderr(t)
dir := buildSmokeTree(t)
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false)
if err != nil {
t.Fatal(err)
}
@@ -380,7 +382,7 @@ func TestRunScanInterrupted(t *testing.T) {
defer close(done)
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
false)
false, false)
}()
awaitReturn(t, done, "runScan")
@@ -407,7 +409,8 @@ func TestRunScanInterrupted(t *testing.T) {
got, before)
}
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false)
if err != nil {
t.Fatal(err)
}
@@ -433,7 +436,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
dir := buildWalkCancelTree(t)
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
walkCancelWorkers, false)
walkCancelWorkers, false, false)
if !errors.Is(err, errInterrupted) {
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
err, errInterrupted)
@@ -617,7 +620,8 @@ func TestWalkOneDirStopsWhenCancelled(t *testing.T) {
// Unbuffered and unread: on a cancelled scan every send gives up.
events := make(chan walkEvent)
subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false, events)
subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false,
walkRules{}, events)
if len(subs) != 0 {
t.Errorf("cancelled walkOneDir returned %+v to descend into, "+
"want none", subs)
@@ -637,7 +641,8 @@ func TestWalkWorkersDropQueuedDirs(t *testing.T) {
missing := filepath.Join(t.TempDir(), "missing")
jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false)
jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false,
walkRules{})
for range 64 {
jobs <- dirJob{path: missing}
@@ -668,7 +673,7 @@ func TestWalkWorkerAbandonsSubdirHandoff(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context())
defer cancel()
jobs, subdirs, events := startWalkWorkers(ctx, 1, false)
jobs, subdirs, events := startWalkWorkers(ctx, 1, false, walkRules{})
// Fill the hand-back channel to its capacity — one slot per worker
// — so the worker's own hand-back is certain to block.
db.go
+230 -72
View File
@@ -61,8 +61,8 @@ CREATE TABLE files (
) WITHOUT ROWID
`
// createIndexSQL indexes the records by signature, so report can have
// SQLite group them without sorting the whole table.
// createIndexSQL indexes the records by signature, so report can read
// them in signature order without sorting the table.
const createIndexSQL = `
CREATE INDEX files_signature ON files (size, head, tail, content)
`
@@ -79,7 +79,8 @@ ON CONFLICT (path) DO UPDATE SET
content = excluded.content
`
// errNoDatabase reports a missing database file for report/trees.
// errNoDatabase reports a missing database file for report, trees and
// info.
var errNoDatabase = errors.New(
"no database (run \"sfdupes scan\" first, or set " + databaseEnv + ")")
@@ -108,10 +109,8 @@ func databasePath() (string, string, error) {
}
// The working directory as the kernel reports it, with no symlink
// in it. os.Getwd may return PWD instead, which can name it through
// a symlink; a ".." after that symlink then leads somewhere else
// once the path is cleaned, as scan cleans it to create the
// database's directory.
// in it, as README "Database" says. os.Getwd may return PWD
// instead, which can name it through a symlink.
wd, err := unix.Getwd()
if err != nil {
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
@@ -124,6 +123,24 @@ func databasePath() (string, string, error) {
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
}
// createDatabaseDirectory creates the directory the database at path
// goes in, and any directory above it, if missing. path is absolute, as
// databasePath returns it. The directory is path with its last element
// cut off as text, by filepath.Split. filepath.Dir would clean it,
// dropping each ".." together with the directory name before it; when
// that name is a symlink, the cleaned path names another directory than
// the one the system reaches when it opens path.
func createDatabaseDirectory(path string) error {
dir, _ := filepath.Split(path)
err := os.MkdirAll(dir, dbDirPerm)
if err != nil {
return fmt.Errorf("create database directory: %w", err)
}
return nil
}
// scanParams are the connection parameters for scan: read-write, with
// WAL journaling and a busy timeout, so a report can run while a cron
// scan is in progress. closeScanDatabase leaves WAL mode again.
@@ -131,9 +148,9 @@ const scanParams = "_pragma=busy_timeout(10000)" +
"&_pragma=journal_mode(WAL)" +
"&_pragma=synchronous(NORMAL)"
// reportParams are the connection parameters for report and trees:
// read-only, with the same busy timeout. They set no journal mode,
// because setting one is a write.
// reportParams are the connection parameters for report, trees and
// info: read-only, with the same busy timeout. They set no journal
// mode, because setting one is a write.
const reportParams = "mode=ro" +
"&_pragma=busy_timeout(10000)" +
"&_pragma=query_only(1)"
@@ -178,9 +195,9 @@ func openDB(path, params string) (*sql.DB, error) {
// that deleted it would let the next scan lock a new file while
// another still holds the old one.
func lockScanDatabase(path string) (*os.File, error) {
err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
err := createDatabaseDirectory(path)
if err != nil {
return nil, fmt.Errorf("create database directory: %w", err)
return nil, err
}
// Like SQLite, this follows a symlink at path and creates its
@@ -224,9 +241,9 @@ func lockScanDatabase(path string) (*os.File, error) {
// openScanDatabase opens the database for the scan subcommand, creating
// the file, its parent directory, and the schema as needed.
func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
err := createDatabaseDirectory(path)
if err != nil {
return nil, fmt.Errorf("create database directory: %w", err)
return nil, err
}
db, err := openDB(path, scanParams)
@@ -244,6 +261,42 @@ func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
return db, nil
}
// databaseFiles returns, by device and inode, the files of the database
// at path that the scan keeps open: the database file, its -wal and
// -shm files and its .lock file, each named from the database's path
// with symlinks resolved, as SQLite and lockScanDatabase name them. The
// scan calls it once the database and the lock file are open, so that
// its walk passes over them however it reaches them, through a
// symlinked directory included. A file that does not exist then is left
// out, since no walk can reach it. There is no -journal file while the
// database is in WAL mode.
func databaseFiles(path string) ([]fileRec, error) {
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return nil, fmt.Errorf("resolve database path: %w", err)
}
var files []fileRec
for _, p := range []string{
resolved, resolved + "-wal", resolved + "-shm", resolved + ".lock",
} {
fi, err := os.Lstat(p)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("database: %w", err)
}
dev, ino := inodeOfInfo(fi)
files = append(files, fileRec{path: p, dev: dev, ino: ino})
}
return files, nil
}
// closeScanDatabase switches the database at path from WAL back to
// rollback-journal mode and closes it. Out of WAL mode the database
// file alone holds the whole database, so a reader needs no -wal or
@@ -262,9 +315,9 @@ func closeScanDatabase(ctx context.Context, db *sql.DB, path string) {
_ = db.Close()
}
// openReportDatabase opens an existing database for the report and
// trees subcommands. A missing database file is an error directing the
// user to run scan first; the schema version must match exactly.
// openReportDatabase opens an existing database for the report, trees
// and info subcommands. A missing database file is an error directing
// the user to run scan first; the schema version must match exactly.
func openReportDatabase(ctx context.Context,
path string,
) (*sql.DB, error) {
@@ -403,10 +456,32 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
return v, nil
}
// loadFileRows streams every record to fn in path order: byte order,
// which is the order of the primary key, so SQLite does not sort.
func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
rows, err := db.QueryContext(ctx,
// loadFileRows gives total the number of records, then streams every
// record to fn in path order: byte order, which is the order of the
// primary key, so SQLite does not sort. The count and the records are
// read in one transaction, so they agree while a scan is committing.
func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
fn func(r scanRec),
) error {
// Everything goes through tx: db has one connection, so a query on
// db would wait for tx forever.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return fmt.Errorf("read records: %w", err)
}
defer func() { _ = tx.Rollback() }()
var n int64
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
if err != nil {
return fmt.Errorf("count records: %w", err)
}
total(n)
rows, err := tx.QueryContext(ctx,
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
"FROM files ORDER BY path")
if err != nil {
@@ -441,77 +516,140 @@ func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
return nil
}
// dupeRowsSQL selects every record in a duplicate group, with the
// group's first path. A group is the records with a content hash that
// share a size, head, tail, and content, when there are two or more of
// them. The rows come in report order: groups by size descending, then
// by first path, and each group's paths ascending.
const dupeRowsSQL = `
SELECT g.first, f.path, f.size
FROM files AS f
JOIN (
SELECT size, head, tail, content, MIN(path) AS first
FROM files
WHERE content <> ''
GROUP BY size, head, tail, content
HAVING COUNT(*) > 1
) AS g USING (size, head, tail, content)
ORDER BY f.size DESC, g.first, f.path
// signatureRowsSQL selects every record's signature and path in
// signature order, largest size first. That is the order of the
// files_signature index read backwards, so SQLite reads the index alone
// and returns the first row at once, with nothing to sort.
const signatureRowsSQL = `
SELECT size, head, tail, content, path
FROM files
ORDER BY size DESC, head DESC, tail DESC, content DESC
`
// loadDupeRows streams the rows of dupeRowsSQL to fn and returns the
// number of records in the database. The count and the rows are read
// in one transaction, so they agree while a scan is committing. An
// error from fn stops the reading and is returned as it is.
func loadDupeRows(ctx context.Context, db *sql.DB,
fn func(first, path string, size int64) error,
) (int, error) {
// loadSignatureRows gives total the number of records, then streams
// every record's signature and path to fn, without its mtime, in the
// order of signatureRowsSQL. The count and the records are read in one
// transaction, so they agree while a scan is committing. An error from
// fn stops the reading and is returned as it is.
func loadSignatureRows(ctx context.Context, db *sql.DB,
total func(n int64), fn func(r scanRec) error,
) error {
// Everything goes through tx: the report connection is the only
// one, so a query on db would wait for tx forever.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return 0, fmt.Errorf("read records: %w", err)
return fmt.Errorf("read records: %w", err)
}
defer func() { _ = tx.Rollback() }()
var records int
var n int64
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&records)
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
if err != nil {
return 0, fmt.Errorf("read records: %w", err)
return fmt.Errorf("count records: %w", err)
}
rows, err := tx.QueryContext(ctx, dupeRowsSQL)
total(n)
rows, err := tx.QueryContext(ctx, signatureRowsSQL)
if err != nil {
return 0, fmt.Errorf("read records: %w", err)
return fmt.Errorf("read records: %w", err)
}
defer func() { _ = rows.Close() }()
for rows.Next() {
var (
first, path []byte
size int64
path []byte
r scanRec
)
err = rows.Scan(&first, &path, &size)
err = rows.Scan(&r.size, &r.head, &r.tail, &r.content, &path)
if err != nil {
return 0, fmt.Errorf("read record: %w", err)
return fmt.Errorf("read record: %w", err)
}
err = fn(string(first), string(path), size)
r.path = string(path)
err = fn(r)
if err != nil {
return 0, err
return err
}
}
err = rows.Err()
if err != nil {
return 0, fmt.Errorf("read records: %w", err)
return fmt.Errorf("read records: %w", err)
}
return records, nil
return nil
}
// countRecords returns the number of records in the database.
func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
var n int64
err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
if err != nil {
return 0, fmt.Errorf("count records: %w", err)
}
return n, nil
}
// databaseCountsSQL counts the records, those with head and tail
// hashes and those with a content hash, and adds up their sizes, in one
// pass over the table. A SUM over no rows is NULL, so on an empty table
// each SUM falls back to 0.
const databaseCountsSQL = `
SELECT COUNT(*),
COALESCE(SUM(head <> '' AND tail <> ''), 0),
COALESCE(SUM(content <> ''), 0),
COALESCE(SUM(size), 0)
FROM files
`
// databaseCounts is what info reads from the database: its schema
// version and the counts of databaseCountsSQL.
type databaseCounts struct {
schemaVersion int
records int64 // every record
headTail int64 // records with head and tail hashes
content int64 // records with a content hash
size int64 // the sizes of all records added up
}
// readDatabaseCounts reads the schema version, then the counts of
// databaseCountsSQL, in one read-only transaction, so both describe the
// database at one moment while a scan is committing.
func readDatabaseCounts(ctx context.Context,
db *sql.DB,
) (databaseCounts, error) {
var c databaseCounts
// Everything goes through tx: the report connection is the only
// one, so a query on db would wait for tx forever.
tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
if err != nil {
return c, fmt.Errorf("count records: %w", err)
}
defer func() { _ = tx.Rollback() }()
err = tx.QueryRowContext(ctx, "PRAGMA user_version").
Scan(&c.schemaVersion)
if err != nil {
return c, fmt.Errorf("read schema version: %w", err)
}
err = tx.QueryRowContext(ctx, databaseCountsSQL).
Scan(&c.records, &c.headTail, &c.content, &c.size)
if err != nil {
return c, fmt.Errorf("count records: %w", err)
}
return c, nil
}
// loadFileMeta streams every record's path, size, mtime, and whether
@@ -552,26 +690,46 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
return nil
}
// contentCandidatesSQL selects every record of at least headTailMin
// bytes whose size, head, and tail equal another record's, in each
// group (the records sharing a size, head, and tail) where at least one
// record has no content hash, with whether each record has one. SQLite
// does the grouping, so no other record's hashes are loaded into
// memory; the rows come ordered by size, head, and tail, so each
// group's rows arrive together.
// contentGroupsSQL selects each group (the records sharing a size,
// head, and tail) of records of at least headTailMin bytes that has more
// than one record and at least one record without a content hash, with
// the number of its records.
const contentGroupsSQL = `
SELECT size, head, tail, COUNT(*) AS records
FROM files
WHERE size >= ? AND head <> ''
GROUP BY size, head, tail
HAVING COUNT(*) > 1 AND SUM(content = '') > 0
`
// contentCandidatesSQL selects every record of the groups
// contentGroupsSQL selects, with whether each record has a content
// hash. SQLite does the grouping, so no other record's hashes are
// loaded into memory; the rows come ordered by size, head, and tail, so
// each group's rows arrive together.
const contentCandidatesSQL = `
SELECT f.path, f.size, f.mtime, f.mtime_nsec, f.head, f.tail, f.content <> ''
FROM files AS f
JOIN (
SELECT size, head, tail
FROM files
WHERE size >= ? AND head <> ''
GROUP BY size, head, tail
HAVING COUNT(*) > 1 AND SUM(content = '') > 0
) AS g USING (size, head, tail)
JOIN (` + contentGroupsSQL + `) AS g USING (size, head, tail)
ORDER BY size, head, tail
`
// countContentCandidates returns the number of rows of
// contentCandidatesSQL, adding up the records of the groups
// contentGroupsSQL selects, so it neither joins nor sorts.
func countContentCandidates(ctx context.Context, db *sql.DB) (int64, error) {
var n int64
err := db.QueryRowContext(ctx,
"SELECT COALESCE(SUM(records), 0) FROM ("+contentGroupsSQL+")",
headTailMin).Scan(&n)
if err != nil {
return 0, fmt.Errorf("count content candidates: %w", err)
}
return n, nil
}
// loadContentCandidates streams the rows of contentCandidatesSQL to fn:
// each record, without its content hash, and whether it has one.
func loadContentCandidates(ctx context.Context, db *sql.DB,
db_test.go
+297 -8
View File
@@ -5,6 +5,7 @@ import (
"database/sql"
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"slices"
@@ -118,6 +119,116 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
}
}
// TestScanCreatesDatabaseDirectoryAfterSymlink has a database path go
// through a symlink to a directory, then "..", to a missing directory.
// With srv/link a symlink to data/x, srv/link/../new/db.sqlite leads to
// data/new/db.sqlite. lockScanDatabase and openScanDatabase, the two
// places scan creates the database's directory, must each create the
// database there, and nothing at srv/new, where the path leads once
// cleaned as text.
func TestScanCreatesDatabaseDirectoryAfterSymlink(t *testing.T) {
t.Parallel()
creates := map[string]func(ctx context.Context, path string) error{
"lockScanDatabase": func(_ context.Context, path string) error {
lock, err := lockScanDatabase(path)
if err == nil {
_ = lock.Close()
}
return err
},
"openScanDatabase": func(ctx context.Context, path string) error {
db, err := openScanDatabase(ctx, path)
if err == nil {
_ = db.Close()
}
return err
},
}
for name, create := range creates {
t.Run(name, func(t *testing.T) {
t.Parallel()
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
err = os.MkdirAll(filepath.Join(dir, "data", "x"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Mkdir(filepath.Join(dir, "srv"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Symlink(filepath.Join(dir, "data", "x"),
filepath.Join(dir, "srv", "link"))
if err != nil {
t.Fatal(err)
}
err = create(t.Context(), dir+"/srv/link/../new/db.sqlite")
if err != nil {
t.Fatalf("%s: %v", name, err)
}
_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
if err != nil {
t.Errorf("database the path leads to: %v", err)
}
_, err = os.Lstat(filepath.Join(dir, "srv", "new"))
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("srv/new: stat error %v, want it missing", err)
}
})
}
}
// TestDatabaseFiles opens a database through a symlink to its file:
// databaseFiles finds the database file and its -wal and -shm files
// beside the symlink's target, where SQLite keeps them, and leaves out
// the .lock file, which only a scan creates.
func TestDatabaseFiles(t *testing.T) {
t.Parallel()
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
db, err := openScanDatabase(t.Context(), link)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
files, err := databaseFiles(link)
if err != nil {
t.Fatal(err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.path)
}
want := []string{path, path + "-wal", path + "-shm"}
if !slices.Equal(got, want) {
t.Errorf("databaseFiles(%s) = %q, want %q", link, got, want)
}
}
func TestOpenDatabaseUnversionedForeign(t *testing.T) {
t.Parallel()
@@ -284,12 +395,128 @@ func TestCloseScanDatabaseWhileReportOpen(t *testing.T) {
defer func() { _ = reportDB.Close() }()
err = loadFileRows(t.Context(), reportDB, func(scanRec) {})
err = loadFileRows(t.Context(), reportDB, func(int64) {},
func(scanRec) {})
if err != nil {
t.Fatalf("loadFileRows: %v", err)
}
}
// TestLoadRowsCountAgreeDuringScan has a scan commit records after
// loadFileRows, as trees calls it, or loadSignatureRows, as report
// calls it, has counted the records and before it reads them, as when
// trees or report runs during a scan: each must read exactly as many
// records as it counted.
func TestLoadRowsCountAgreeDuringScan(t *testing.T) {
t.Parallel()
loads := map[string]func(context.Context, *sql.DB, func(int64),
func(scanRec)) error{
"loadFileRows": loadFileRows,
"loadSignatureRows": func(ctx context.Context, db *sql.DB,
total func(int64), fn func(scanRec),
) error {
return loadSignatureRows(ctx, db, total, func(r scanRec) error {
fn(r)
return nil
})
},
}
for name, load := range loads {
t.Run(name, func(t *testing.T) {
t.Parallel()
path := testDBPath(t)
scanDB, err := openScanDatabase(t.Context(), path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = scanDB.Close() }()
recs := smokeTreeRecs()
err = applyChanges(t.Context(), scanDB, recs[:3], nil, nil)
if err != nil {
t.Fatal(err)
}
reportDB, err := openReportDatabase(t.Context(), path)
if err != nil {
t.Fatal(err)
}
defer func() { _ = reportDB.Close() }()
var counted, read int64
err = load(t.Context(), reportDB, func(n int64) {
counted = n
err := applyChanges(t.Context(), scanDB, recs[3:], nil, nil)
if err != nil {
t.Error(err)
}
}, func(scanRec) { read++ })
if err != nil {
t.Fatal(err)
}
if counted != 3 || read != counted {
t.Errorf("counted %d records and read %d, want 3 and 3",
counted, read)
}
})
}
}
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
// from the files_signature index alone, with nothing to sort: the
// first record comes at once, so report's read phase counts each
// record as it comes rather than waiting on SQLite for all of them.
func TestSignatureRowsNeedNoSort(t *testing.T) {
t.Parallel()
db := openTestDB(t)
rows, err := db.QueryContext(t.Context(),
"EXPLAIN QUERY PLAN "+signatureRowsSQL)
if err != nil {
t.Fatal(err)
}
defer func() { _ = rows.Close() }()
var plan []string
for rows.Next() {
var (
id, parent, unused int
detail string
)
err = rows.Scan(&id, &parent, &unused, &detail)
if err != nil {
t.Fatal(err)
}
plan = append(plan, detail)
}
err = rows.Err()
if err != nil {
t.Fatal(err)
}
want := []string{"SCAN files USING COVERING INDEX files_signature"}
if !slices.Equal(plan, want) {
t.Errorf("query plan = %q, want %q", plan, want)
}
}
func TestApplyChangesRoundTrip(t *testing.T) {
t.Parallel()
@@ -309,8 +536,10 @@ func TestApplyChangesRoundTrip(t *testing.T) {
},
}
err := applyChanges(t.Context(), db, recs, nil,
newProgress("update", 2))
prog := newProgress(progressParams{label: "update", total: 4})
defer prog.finish()
err := applyChanges(t.Context(), db, recs, nil, prog)
if err != nil {
t.Fatalf("applyChanges: %v", err)
}
@@ -329,7 +558,7 @@ func TestApplyChangesRoundTrip(t *testing.T) {
}
err = applyChanges(t.Context(), db, []scanRec{upd},
[]string{"/a/tab\tnew\nline"}, newProgress("update", 2))
[]string{"/a/tab\tnew\nline"}, prog)
if err != nil {
t.Fatalf("applyChanges: %v", err)
}
@@ -357,8 +586,10 @@ func TestApplyChangesBatching(t *testing.T) {
})
}
err := applyChanges(t.Context(), db, recs, nil,
newProgress("update", int64(n)))
prog := newProgress(progressParams{label: "update", total: int64(2 * n)})
defer prog.finish()
err := applyChanges(t.Context(), db, recs, nil, prog)
if err != nil {
t.Fatalf("applyChanges: %v", err)
}
@@ -372,8 +603,7 @@ func TestApplyChangesBatching(t *testing.T) {
deletes = append(deletes, r.path)
}
err = applyChanges(t.Context(), db, nil, deletes,
newProgress("update", int64(n)))
err = applyChanges(t.Context(), db, nil, deletes, prog)
if err != nil {
t.Fatalf("applyChanges deletes: %v", err)
}
@@ -382,3 +612,62 @@ func TestApplyChangesBatching(t *testing.T) {
t.Fatalf("records = %d, want 0", len(got))
}
}
// TestCountContentCandidates checks that countContentCandidates counts
// the records the content phase's search returns: those of each group
// of two or more records of headTailMin or more with a head, where at
// least one has no content hash.
func TestCountContentCandidates(t *testing.T) {
t.Parallel()
db := openTestDB(t)
// Each group's records: size, head, and content, one per record.
groups := []struct {
size int64
head string
contents []string
}{
{headTailMin, "h", []string{"", "c", "c"}}, // returned
{headTailMin + 1, "h", []string{"", ""}}, // returned
{headTailMin + 2, "h", []string{"c", "c"}}, // all hashed
{headTailMin + 3, "h", []string{""}}, // one record
{headTailMin - 1, "h", []string{"", ""}}, // too small
{headTailMin + 4, "", []string{"", ""}}, // no head
}
var recs []scanRec
for _, g := range groups {
for _, content := range g.contents {
recs = append(recs, scanRec{
size: g.size, mtime: time.Unix(1, 0), head: g.head,
tail: "t", content: content,
path: fmt.Sprintf("/r/%d", len(recs)),
})
}
}
err := applyChanges(t.Context(), db, recs, nil, nil)
if err != nil {
t.Fatalf("applyChanges: %v", err)
}
got, err := countContentCandidates(t.Context(), db)
if err != nil {
t.Fatalf("countContentCandidates: %v", err)
}
var returned int64
err = loadContentCandidates(t.Context(), db,
func(scanRec, bool) { returned++ })
if err != nil {
t.Fatalf("loadContentCandidates: %v", err)
}
if got != 5 || returned != 5 {
t.Errorf("count = %d, records returned = %d, want 5 and 5",
got, returned)
}
}
exclude.go
+195
View File
@@ -0,0 +1,195 @@
package main
// This file holds two rules for the walk (README "Rules for the walk"):
// the default excludes, the paths a scan's walk does not enter below an
// operand unless scan is given --no-default-excludes, and the
// filesystems that are not persistent, on which the scan never records
// or reads anything.
import (
"bufio"
"errors"
"fmt"
"io"
"io/fs"
"path/filepath"
"slices"
"strings"
"golang.org/x/sys/unix"
)
// The systems that have default excludes, as runtime.GOOS names them.
const (
goosLinux = "linux"
goosDarwin = "darwin"
goosWindows = "windows"
)
// The fields of a line of mountInfoPath that readMountInfo reads,
// counted from 0 (see proc(5)): the device ID, as major:minor, is the
// third field, and after the sixth come optional fields, then a field
// holding "-", then the filesystem type.
const (
deviceField = 2
optionalFieldsStart = 6
)
// typesPerHelpLine is how many filesystem types the scan command's help
// lists on each line.
const typesPerHelpLine = 6
// errMountInfoLine reports a line of mountInfoPath without the fields
// readMountInfo reads.
var errMountInfoLine = errors.New("unexpected line")
// walkRules holds the rules for the walk that a scan sets up as it
// starts. excludes are the default excludes, or none with
// --no-default-excludes. nonPersistent maps the device ID of every
// mounted filesystem that is not persistent to its type; the scan
// records and reads nothing on one of those, whatever the flags.
// databaseFiles are the scan's own database files, which the walk
// passes over (see databaseFiles in db.go).
type walkRules struct {
excludes []string
nonPersistent map[uint64]string
databaseFiles []fileRec
}
// defaultExcludes returns the default excludes of the system goos
// names, as runtime.GOOS does: a fixed list on Linux, macOS and
// Windows, and none on any other system.
func defaultExcludes(goos string) []string {
switch goos {
case goosLinux:
return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
case goosDarwin:
// A walk of / reaches everything in /System/Volumes/Data through
// the firmlinks, so walking it as well would record each of
// those files twice.
return []string{
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data",
}
case goosWindows:
// Relative to the root of the drive an operand lies on. The
// first three are the page, swap and hibernation files.
return []string{
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
`Users\*\AppData\Local\Temp`,
}
default:
return nil
}
}
// isDefaultExclude reports whether path is one of excludes, in which a
// * matches one path component.
func isDefaultExclude(path string, excludes []string) bool {
for _, pattern := range excludes {
// The patterns are fixed, so none is malformed.
if ok, _ := filepath.Match(pattern, path); ok {
return true
}
}
return false
}
// nonPersistentFilesystemTypes returns the filesystem types that are
// not persistent: filesystems that the kernel makes up or that are kept
// in memory. devfs is the macOS filesystem of /dev; the others are
// Linux names.
func nonPersistentFilesystemTypes() []string {
return []string{
"proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "ramfs",
"cgroup", "cgroup2", "debugfs", "tracefs", "securityfs", "pstore",
"bpf", "configfs", "fusectl", "mqueue", "hugetlbfs", "efivarfs",
"binfmt_misc", "autofs", "nsfs", "rpc_pipefs", "selinuxfs", "devfs",
}
}
// nonPersistentLine returns the line the scan prints for path when fi,
// its lstat, puts it on a filesystem in rules.nonPersistent, and ""
// when it is on any other.
func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
dev, ok := deviceOfInfo(fi)
if !ok {
return ""
}
fsType, found := rules.nonPersistent[dev]
if !found {
return ""
}
return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
path, fsType)
}
// readMountInfo reads mountinfo, laid out as mountInfoPath is, and
// returns the device ID of every mounted filesystem that is not
// persistent, with its type. The device ID is the one each line gives,
// not that of the file now at its mount point, so a mount that a later
// mount hides never adds the device of another filesystem.
func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
types := nonPersistentFilesystemTypes()
devices := make(map[uint64]string)
lines := bufio.NewScanner(mountinfo)
for lines.Scan() {
fields := strings.Fields(lines.Text())
sep := slices.Index(fields, "-")
if sep < optionalFieldsStart || sep+1 >= len(fields) {
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
}
var major, minor uint32
_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
if err != nil {
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
}
fsType := fields[sep+1]
if slices.Contains(types, fsType) {
devices[unix.Mkdev(major, minor)] = fsType
}
}
return devices, lines.Err()
}
// defaultExcludesHelp describes, for the scan command's help, the
// default excludes of the system goos names and, where the scan lists
// the mounted filesystems, the types it never records anything on.
func defaultExcludesHelp(goos string) string {
var help strings.Builder
dirs := defaultExcludes(goos)
if len(dirs) == 0 {
help.WriteString("This system has no default excludes.\n")
} else {
help.WriteString("Below an operand, the walk does not enter these " +
"default excludes, unless\n--no-default-excludes is given:\n\n")
for _, dir := range dirs {
fmt.Fprintf(&help, " %s\n", dir)
}
}
if goos == goosLinux || goos == goosDarwin {
help.WriteString("\nWith or without --no-default-excludes, the scan " +
"records nothing on a\nfilesystem that is not persistent, " +
"one of these types:\n\n")
types := nonPersistentFilesystemTypes()
for line := range slices.Chunk(types, typesPerHelpLine) {
fmt.Fprintf(&help, " %s\n", strings.Join(line, ", "))
}
}
return help.String()
}
exclude_darwin.go
+52
View File
@@ -0,0 +1,52 @@
//go:build darwin
package main
import (
"slices"
"golang.org/x/sys/unix"
)
// nonPersistentFilesystems returns the device ID of every mounted
// filesystem that is not persistent, with its type, from the system's
// list of mounted filesystems. The device ID is read with lstat on the
// mount point. Where one filesystem is mounted over another on the same
// mount point, the later one in the list counts. A mount point that
// lstat cannot reach is left out.
func nonPersistentFilesystems() (map[uint64]string, error) {
// The first call counts the filesystems, the second lists them.
n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
if err != nil {
return nil, err
}
list := make([]unix.Statfs_t, n)
n, err = unix.Getfsstat(list, unix.MNT_NOWAIT)
if err != nil {
return nil, err
}
mounts := make(map[string]string, n)
for _, st := range list[:n] {
mounts[unix.ByteSliceToString(st.Mntonname[:])] =
unix.ByteSliceToString(st.Fstypename[:])
}
types := nonPersistentFilesystemTypes()
devices := make(map[uint64]string)
for mountPoint, fsType := range mounts {
if !slices.Contains(types, fsType) {
continue
}
if dev, ok := deviceOfPath(mountPoint); ok {
devices[dev] = fsType
}
}
return devices, nil
}
exclude_other.go
+36
View File
@@ -0,0 +1,36 @@
//go:build !darwin
package main
import (
"fmt"
"os"
"runtime"
)
// mountInfoPath is the file in which Linux lists the mounts the reading
// process sees.
const mountInfoPath = "/proc/self/mountinfo"
// nonPersistentFilesystems returns the device ID of every mounted
// filesystem that is not persistent, with its type: on Linux read from
// mountInfoPath (see readMountInfo), and on any other system none.
func nonPersistentFilesystems() (map[uint64]string, error) {
if runtime.GOOS != goosLinux {
return map[uint64]string{}, nil
}
f, err := os.Open(mountInfoPath)
if err != nil {
return nil, err
}
defer func() { _ = f.Close() }()
devices, err := readMountInfo(f)
if err != nil {
return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
}
return devices, nil
}
exclude_test.go
+583
View File
@@ -0,0 +1,583 @@
package main
import (
"bytes"
"errors"
"maps"
"os"
"path/filepath"
"runtime"
"slices"
"strings"
"testing"
"golang.org/x/sys/unix"
)
func TestDefaultExcludes(t *testing.T) {
t.Parallel()
// README "Rules for the walk": each system's list.
cases := map[string][]string{
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
goosDarwin: {
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data",
},
goosWindows: {
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
`Users\*\AppData\Local\Temp`,
},
"freebsd": nil,
}
for goos, want := range cases {
if got := defaultExcludes(goos); !slices.Equal(got, want) {
t.Errorf("defaultExcludes(%q) = %q, want %q", goos, got, want)
}
}
}
func TestIsDefaultExclude(t *testing.T) {
t.Parallel()
// A * matches one path component, no more and no fewer.
excludes := []string{"/scratch", "/home/*/cache"}
cases := map[string]bool{
"/scratch": true,
"/scratch/x": false,
"/home/u/cache": true,
"/home/u/v/cache": false,
"/home/cache": false,
}
for path, want := range cases {
if got := isDefaultExclude(path, excludes); got != want {
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
}
}
}
func TestReadMountInfo(t *testing.T) {
t.Parallel()
// The second-to-last line has no optional fields. The tmpfs on
// /data/a/b is hidden by the root filesystem's /srv, mounted over
// /data/a after it: the tmpfs still gives its own device, and the
// root filesystem's device is not listed.
mountinfo := strings.Join([]string{
"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw",
"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw",
"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
}, "\n")
got, err := readMountInfo(strings.NewReader(mountinfo))
if err != nil {
t.Fatal(err)
}
want := map[uint64]string{
unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs",
unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs",
}
if !maps.Equal(got, want) {
t.Errorf("readMountInfo = %v, want %v", got, want)
}
for _, line := range []string{
"22 1 8:1 / / rw,relatime\n",
"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n",
} {
_, err = readMountInfo(strings.NewReader(line))
if !errors.Is(err, errMountInfoLine) {
t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
errMountInfoLine)
}
}
}
func TestNonPersistentFilesystems(t *testing.T) {
t.Parallel()
if runtime.GOOS != goosLinux {
t.Skip("the test knows a mount only on Linux: /proc")
}
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfPath("/proc")
if !ok {
t.Fatal("no device for /proc")
}
if got := nonPersistent[dev]; got != "proc" {
t.Errorf("filesystem of /proc = %q, want %q", got, "proc")
}
}
func TestDefaultExcludesHelp(t *testing.T) {
t.Parallel()
for _, goos := range []string{goosLinux, goosDarwin, goosWindows} {
help := defaultExcludesHelp(goos)
for _, dir := range defaultExcludes(goos) {
if !strings.Contains(help, "\n "+dir+"\n") {
t.Errorf("help for %s does not list %s:\n%s", goos, dir, help)
}
}
// Only Linux and macOS list their mounted filesystems.
for _, fsType := range nonPersistentFilesystemTypes() {
listed := strings.Contains(help, " "+fsType+",") ||
strings.Contains(help, " "+fsType+"\n")
if listed != (goos != goosWindows) {
t.Errorf("help for %s lists %s: %v", goos, fsType, listed)
}
}
}
want := "This system has no default excludes.\n"
if got := defaultExcludesHelp("freebsd"); got != want {
t.Errorf("help for freebsd = %q, want %q", got, want)
}
// scan's help shows those of the system it runs on.
var stdout, stderr bytes.Buffer
run([]string{cmdScan, "-h"}, &stdout, &stderr)
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
if !strings.Contains(stderr.String(), help) {
t.Errorf("scan --help = %q, want it to hold %q", stderr.String(), help)
}
}
// TestScanDoesNotEnterDefaultExclude checks that the walk does not
// enter a default exclude below an operand. It says so on a line of its
// own and does not count it as skipped. The records below it count as
// outside the operands: the record of a file gone from it is kept, and
// its size joins the size census, while a file added to it is not
// recorded.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
db := openTestDB(t)
excluded := filepath.Join(dir, "tmp")
// A scan with no default excludes records the file in excluded.
gone := writeFile(t, excluded, "gone", pattern(1, 100))
syncTree(t, db, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
unseen := writeFile(t, excluded, "new", pattern(2, 10))
// The same size as gone, so it is hashed only if the census holds the
// size of gone's record.
added := writeFile(t, dir, "added", pattern(3, 100))
st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
walkRules{excludes: []string{excluded}})
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
recordByPath(t, recs, gone)
if recordByPath(t, recs, added).head == "" {
t.Errorf("%s not hashed: the census lacks the size of %s",
added, gone)
}
if slices.Contains(recordPaths(recs), unseen) {
t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
}
line := "\nscan: skipped " + excluded + " (default exclude)\n"
if !strings.Contains(stderr(), line) {
t.Errorf("stderr = %q, want %q", stderr(), line)
}
}
// TestScanWalksOperandInDefaultExclude checks that an operand at or
// inside a default exclude is walked, also when it lies under another
// operand, whose walk does not enter the default exclude.
func TestScanWalksOperandInDefaultExclude(t *testing.T) {
t.Parallel()
dir := t.TempDir()
excluded := filepath.Join(dir, "tmp")
f := writeFile(t, excluded, "sub/f", pattern(1, 10))
sub := filepath.Dir(f)
cases := map[string][]string{
"at it": {excluded},
"inside it": {sub},
"inside it under an operand": {dir, sub},
}
for name, roots := range cases {
t.Run(name, func(t *testing.T) {
t.Parallel()
db := openTestDB(t)
st, err := syncScan(t.Context(), db, roots, 1, false,
walkRules{excludes: []string{excluded}})
if err != nil {
t.Fatal(err)
}
if st.added != 1 {
t.Errorf("added = %d, want %s recorded once", st.added, f)
}
recordByPath(t, dbRecords(t, db), f)
})
}
}
// TestScanNamesNoDefaultExcludeThatIsAnOperand checks that a default
// exclude given as an operand under another operand, which the walk of
// that other operand does not enter, gets no line saying it was
// skipped: it is walked as an operand of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
db := openTestDB(t)
excluded := filepath.Join(dir, "tmp")
f := writeFile(t, excluded, "f", pattern(1, 10))
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
walkRules{excludes: []string{excluded}})
if err != nil {
t.Fatal(err)
}
if st.added != 1 {
t.Errorf("added = %d, want %s recorded once", st.added, f)
}
recordByPath(t, dbRecords(t, db), f)
if strings.Contains(stderr(), "scan: skipped") {
t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
}
}
// TestRunScanTempDirOperand scans a tree in the tests' temporary
// directory, which on Linux lies inside the default exclude /tmp unless
// TMPDIR names another place, with the system's own default excludes:
// the scan walks the whole tree and reports no default exclude.
func TestRunScanTempDirOperand(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
t.Skip(dir + " is not inside a default exclude")
}
f := writeFile(t, dir, "sub/f", pattern(1, 10))
stderr := scanOK(t, dir)
if strings.Contains(stderr, "scan: skipped") {
t.Errorf("stderr = %q, want no line saying a path was skipped",
stderr)
}
assertRecorded(t, path, f, true)
}
// TestRunScanNoDefaultExcludes scans /var through the command. Its
// /var/tmp, on a persistent filesystem, is a default exclude on Linux:
// without --no-default-excludes the walk does not enter it and says so,
// and with that flag the walk records a file the test puts there.
func TestRunScanNoDefaultExcludes(t *testing.T) {
if runtime.GOOS != goosLinux {
t.Skip("only Linux has /var/tmp among its default excludes")
}
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
t.Fatal(err)
}
rules := walkRules{nonPersistent: nonPersistent}
for _, dir := range []string{"/var", "/var/tmp"} {
fi, lstatErr := os.Lstat(dir)
if lstatErr != nil || !fi.IsDir() ||
rules.nonPersistentLine(dir, fi) != "" {
t.Skip(dir + " is not a directory on a persistent filesystem here")
}
}
//nolint:usetesting // the directory must be inside /var/tmp
tmp, err := os.MkdirTemp("/var/tmp", "sfdupes-test-")
if err != nil {
t.Skip("cannot write to /var/tmp here: " + err.Error())
}
t.Cleanup(func() { _ = os.RemoveAll(tmp) })
f := writeFile(t, tmp, "f", pattern(1, 10))
path := testDBPath(t)
t.Setenv(databaseEnv, path)
line := "scan: skipped /var/tmp (default exclude)\n"
if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
}
assertRecorded(t, path, f, false)
stderr := scanOK(t, "--no-default-excludes", "/var")
if strings.Contains(stderr, line) {
t.Errorf("scan --no-default-excludes /var: stderr = %q, want no %q",
stderr, line)
}
assertRecorded(t, path, f, true)
}
// TestRunScanSkipsNonPersistentOperand scans, through the command, a
// directory on a filesystem that is not persistent, with and without
// --no-default-excludes: either way it is skipped, with a line saying
// so, and nothing in it is recorded.
func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
shm, fsType, _ := shmTempDir(t)
writeFile(t, shm, "f", pattern(1, 10))
t.Setenv(databaseEnv, testDBPath(t))
line := "scan: skipped " + shm + " (" + fsType +
", not a persistent filesystem)\n"
summary := "scan: 0 files seen"
for _, flags := range [][]string{nil, {"--no-default-excludes"}} {
stderr := scanOK(t, append(flags, shm)...)
if !strings.Contains(stderr, line) ||
!strings.Contains(stderr, summary) {
t.Errorf("scan %q: stderr = %q, want %q and %q", flags, stderr,
line, summary)
}
}
}
// TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem
// that is not persistent beside one on the tests' temporary directory.
// The records an earlier scan left in it count as outside the operands:
// the record of a file gone from it is kept, and its size joins the size
// census, while a file added to it is not recorded, and its size does
// not join the census.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
shm, fsType, rules := shmTempDir(t)
stderr := captureStderr(t)
dir := t.TempDir()
db := openTestDB(t)
// A scan that, without the list of filesystems, records gone.
gone := writeFile(t, shm, "gone", pattern(1, 100))
syncTree(t, db, shm)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
unseen := writeFile(t, shm, "new", pattern(2, 10))
// added has the size of gone, and lonely that of unseen: each is
// hashed only if the census holds that size.
added := writeFile(t, dir, "added", pattern(3, 100))
lonely := writeFile(t, dir, "lonely", pattern(4, 10))
st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules)
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
recordByPath(t, recs, gone)
if recordByPath(t, recs, added).head == "" {
t.Errorf("%s not hashed: the census lacks the size of %s",
added, gone)
}
if recordByPath(t, recs, lonely).head != "" {
t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen)
}
if slices.Contains(recordPaths(recs), unseen) {
t.Errorf("%s recorded", unseen)
}
line := "scan: skipped " + shm + " (" + fsType +
", not a persistent filesystem)\n"
if !strings.Contains(stderr(), line) {
t.Errorf("stderr = %q, want %q", stderr(), line)
}
}
// TestScanContentSkipsNonPersistent checks that the content phase does
// not read a file on a filesystem that is not persistent, though an
// earlier scan left a record for it: it is treated as a changed file,
// so its record is left as it is and its identical copy in another
// tree, having no other match, is not read either.
func TestScanContentSkipsNonPersistent(t *testing.T) {
t.Parallel()
shm, _, rules := shmTempDir(t)
db := openTestDB(t)
// A scan that, without the list of filesystems, records a with a
// head and tail but no content hash.
a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
syncTree(t, db, shm)
before := recordByPath(t, dbRecords(t, db), a)
if before.head == "" || before.content != "" {
t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
}
b := sparseFile(t, t.TempDir(), "b", headTailMin)
st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
rules)
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
if r := recordByPath(t, recs, a); r != before {
t.Errorf("record = %+v, want it left as %+v", r, before)
}
if r := recordByPath(t, recs, b); r.content != "" {
t.Errorf("%s: content = %q, want none: its only match is on %s",
b, r.content, shm)
}
}
// TestWalkLeavesOutNonPersistent starts the walk itself in a directory
// on a filesystem that is not persistent, which a scan never does: it
// drops such an operand first. Below a directory on a persistent
// filesystem, one on another takes a mount, which a test cannot make.
// The walk enters no directory and records no file there, and names
// each.
func TestWalkLeavesOutNonPersistent(t *testing.T) {
t.Parallel()
shm, fsType, rules := shmTempDir(t)
f := writeFile(t, shm, "f", pattern(1, 10))
sub := filepath.Dir(writeFile(t, shm, "sub/g", pattern(1, 10)))
var leftOut []string
for ev := range startWalk(t.Context(), []string{shm}, false, rules, 1) {
if ev.rec.path != "" {
t.Errorf("walk recorded %s", ev.rec.path)
}
if ev.leftOut == "" {
continue
}
leftOut = append(leftOut, ev.leftOut)
want := "scan: skipped " + ev.leftOut + " (" + fsType +
", not a persistent filesystem)"
if ev.warn != want {
t.Errorf("line = %q, want %q", ev.warn, want)
}
}
slices.Sort(leftOut)
if want := []string{f, sub}; !slices.Equal(leftOut, want) {
t.Errorf("walk left out %q, want %q", leftOut, want)
}
}
// shmTempDir returns a new directory in /dev/shm, removed when the test
// ends, the type of its filesystem, and the rules holding the real
// filesystems that are not persistent. It skips the test where /dev/shm
// is on a persistent filesystem, or cannot be written to.
func shmTempDir(t *testing.T) (string, string, walkRules) {
t.Helper()
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfPath("/dev/shm")
fsType, found := nonPersistent[dev]
if !ok || !found {
t.Skip("/dev/shm is not a memory-backed filesystem here")
}
//nolint:usetesting // the directory must be inside /dev/shm
dir, err := os.MkdirTemp("/dev/shm", "sfdupes-test-")
if err != nil {
t.Skip("cannot write to /dev/shm here: " + err.Error())
}
t.Cleanup(func() { _ = os.RemoveAll(dir) })
return dir, fsType, walkRules{nonPersistent: nonPersistent}
}
// assertRecorded checks that the database at dbPath holds a record for
// path when want is set, and none when it is not.
func assertRecorded(t *testing.T, dbPath, path string, want bool) {
t.Helper()
db, err := openDB(dbPath, reportParams)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
if got := slices.Contains(recordPaths(dbRecords(t, db)), path); got != want {
t.Errorf("%s recorded: %v, want %v", path, got, want)
}
}
go.mod
+2
View File
@@ -24,3 +24,5 @@ require (
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
)
retract v0.0.1 // early build without a database; does not work as the README describes
info.go
+128
View File
@@ -0,0 +1,128 @@
package main
import (
"context"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"strings"
"time"
)
// infoPhase names one phase of info with its place among all of them,
// as "[2/2] count", for the phase's progress display. The list here is
// the one place the phases are counted.
func infoPhase(name string) string {
return phaseLabel([]string{openPhase, "count"}, name)
}
// runInfo implements the info subcommand: it prints the database's
// details on stdout (see writeInfo). It opens the database as report
// does, read-only and without the scan lock, and never touches the
// scanned filesystem. It names the database on stderr before it opens
// it. Any database problem, including a missing database, is fatal.
//
// The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and count (count the records,
// with one query). Each display ends before the next phase starts, and
// before an error is returned to be printed. The details are written
// once the last display has ended, so no display is drawn while they
// are written.
func runInfo(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr, "info: database %s (%s)\n",
escapePath(dbPath), source)
// Opening can wait on a busy database for as long as the busy
// timeout.
prog := newProgress(progressParams{
label: infoPhase(openPhase), total: -1,
waitingFor: openingDatabase,
})
db, err := openReportDatabase(ctx, dbPath)
prog.finish()
if err != nil {
return err
}
defer func() { _ = db.Close() }()
// One query counts everything, so the phase never has a total.
prog = newProgress(progressParams{
label: infoPhase("count"), total: -1,
waitingFor: countingRecords,
})
counts, err := readDatabaseCounts(ctx, db)
prog.finish()
if err != nil {
return fmt.Errorf("database %s: %w", dbPath, err)
}
return writeInfo(stdout, dbPath, source, counts)
}
// writeInfo writes the details of the database at dbPath to stdout,
// one per line, as a name, a tab and the value (README "`info` mode"):
// the path and source as databasePath returns them, the sizes of the
// database file and of its -wal file when there is one, when the
// database file was last changed, and the schema version and record
// counts that readDatabaseCounts read.
func writeInfo(stdout io.Writer, dbPath, source string,
counts databaseCounts,
) error {
// SQLite keeps the -wal file beside the database file, named from
// its path with symlinks resolved.
file, err := filepath.EvalSymlinks(dbPath)
if err != nil {
return fmt.Errorf("database: %w", err)
}
dbFile, err := os.Stat(file)
if err != nil {
return fmt.Errorf("database: %w", err)
}
walFile, err := os.Stat(file + "-wal")
if err != nil && !errors.Is(err, fs.ErrNotExist) {
return fmt.Errorf("database: %w", err)
}
var b strings.Builder
fmt.Fprintf(&b, "database\t%s (%s)\n", escapePath(dbPath), source)
fmt.Fprintf(&b, "database file size\t%d (%s)\n",
dbFile.Size(), humanBytes(dbFile.Size()))
if walFile != nil {
fmt.Fprintf(&b, "-wal file size\t%d (%s)\n",
walFile.Size(), humanBytes(walFile.Size()))
}
fmt.Fprintf(&b, "modified\t%s\n",
dbFile.ModTime().UTC().Format(time.RFC3339))
fmt.Fprintf(&b, "schema version\t%d\n", counts.schemaVersion)
fmt.Fprintf(&b, "records\t%d\n", counts.records)
fmt.Fprintf(&b, "records with head and tail hashes\t%d\n",
counts.headTail)
fmt.Fprintf(&b, "records with content hash\t%d\n", counts.content)
fmt.Fprintf(&b, "size of recorded files\t%d (%s)\n",
counts.size, humanBytes(counts.size))
_, err = io.WriteString(stdout, b.String())
if err != nil {
return fmt.Errorf("write stdout: %w", err)
}
return nil
}
info_test.go
+168
View File
@@ -0,0 +1,168 @@
package main
import (
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
// infoOpenLine is the plain line of info's open phase when it opens the
// database at once; it prints it as the phase starts and as it ends.
const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown"
// infoCountLine is the plain line of info's count phase when the count
// takes under a second; it prints it as the phase starts and as it
// ends.
const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " +
"eta unknown"
// infoRecs are four records: a duplicate pair with every hash, a file
// of 20 MiB with head and tail hashes but no content hash yet, and a
// file never hashed.
func infoRecs() []scanRec {
return []scanRec{
{size: 5, head: "h", tail: "t", content: "c", path: "/d/a"},
{size: 5, head: "h", tail: "t", content: "c", path: "/d/b"},
{size: 20 << 20, head: "bh", tail: "bt", path: "/d/big"},
{size: 7, path: "/d/unique"},
}
}
// TestRunInfo runs info on an empty and on a filled database, with
// stdout and stderr the same file that is not a terminal, and checks
// everything written there: the line naming the database, each phase's
// line as it starts and as it ends, and then the details, each on a
// line of its own. No -wal file is beside a database scan has closed.
func TestRunInfo(t *testing.T) {
cases := map[string]struct {
recs []scanRec
want string // the details after the database file's time
}{
"empty": {
want: "schema version\t1\n" +
"records\t0\n" +
"records with head and tail hashes\t0\n" +
"records with content hash\t0\n" +
"size of recorded files\t0 (0 B)\n",
},
"filled": {
recs: infoRecs(),
want: "schema version\t1\n" +
"records\t4\n" +
"records with head and tail hashes\t3\n" +
"records with content hash\t2\n" +
"size of recorded files\t20971537 (20.0 MiB)\n",
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
path := seedDatabase(t, tc.recs)
t.Setenv(databaseEnv, path)
file, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
stderr := captureStderr(t)
code := run([]string{cmdInfo}, os.Stderr, os.Stderr)
if code != exitOK {
t.Fatalf("run(info) = %d, want %d; stderr: %s",
code, exitOK, stderr())
}
named := path + " (from " + databaseEnv + ")\n"
want := "info: database " + named +
infoOpenLine + "\n" + infoOpenLine + "\n" +
infoCountLine + "\n" + infoCountLine + "\n" +
"database\t" + named +
fmt.Sprintf("database file size\t%d (%s)\n",
file.Size(), humanBytes(file.Size())) +
"modified\t" + file.ModTime().UTC().Format(time.RFC3339) +
"\n" + tc.want
if got := stderr(); got != want {
t.Errorf("stdout and stderr = %q, want %q", got, want)
}
})
}
}
// TestRunInfoDuringScan runs info while a scan holds the lock and has
// the database open in WAL mode, with records committed to the -wal
// file only. info must count those records and show the -wal file's
// size, also when the database path is a symlink: SQLite keeps the
// -wal file beside the symlink's target.
func TestRunInfoDuringScan(t *testing.T) {
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
db, err := openScanDatabase(t.Context(), path)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
err = applyChanges(t.Context(), db, infoRecs(), nil, nil)
if err != nil {
t.Fatal(err)
}
holdScanLock(t, path)
wal, err := os.Stat(path + "-wal")
if err != nil {
t.Fatal(err)
}
for _, named := range []string{path, link} {
t.Run(filepath.Base(named), func(t *testing.T) {
t.Setenv(databaseEnv, named)
stdout := runStdout(t, cmdInfo)
for _, want := range []string{
"database\t" + named + " (from " + databaseEnv + ")\n",
fmt.Sprintf("\n-wal file size\t%d (%s)\n",
wal.Size(), humanBytes(wal.Size())),
"\nrecords\t4\n",
} {
if !strings.Contains(stdout, want) {
t.Errorf("stdout = %q, want it to hold %q", stdout, want)
}
}
})
}
}
func TestRunInfoNeedsOnlyReadAccess(t *testing.T) {
// README "`info` mode": like report, info needs only read access to
// the database file and none to its directory.
path := testDBPath(t)
t.Setenv(databaseEnv, path)
scanFixture(t)
makeReadOnly(t, path)
stdout := runStdout(t, cmdInfo)
if want := "\nrecords\t2\n"; !strings.Contains(stdout, want) {
t.Errorf("stdout = %q, want it to hold %q", stdout, want)
}
if strings.Contains(stdout, "-wal") {
t.Errorf("stdout = %q, want no -wal file between scans", stdout)
}
}
main.go
+84 -32
View File
@@ -8,13 +8,14 @@
// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
// or larger. scan maintains a persistent SQLite database of file
// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
// that the reporting subcommands read.
// that the other subcommands read.
//
// Usage:
//
// sfdupes scan [--workers N] [-x] PATH...
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
// sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv
// sfdupes info
// sfdupes --version
//
// See README.md for the complete specification.
@@ -27,6 +28,7 @@ import (
"io"
"os"
"runtime"
"runtime/debug"
"github.com/spf13/cobra"
)
@@ -44,6 +46,7 @@ const (
cmdScan = "scan"
cmdReport = "report"
cmdTrees = "trees"
cmdInfo = "info"
)
// errNoSubcommand is returned by the root command when it is invoked
@@ -56,10 +59,32 @@ var errNoSubcommand = errors.New("no subcommand")
var errWorkersBelowOne = errors.New("--workers must be at least 1")
// Version is the build version, injected at link time via -ldflags
// (see the Makefile); "dev" for a plain go build.
// (see the Makefile); empty when the linker set none, as for a binary
// built by go install. buildVersion turns it into what --version prints.
//
//nolint:gochecknoglobals // written only by the linker
var Version = "dev"
var Version string
// buildVersion returns the version that sfdupes --version prints
// (README "Install"): the version the linker set, when it set one;
// otherwise the main module's version that Go recorded in the binary,
// a tag such as v0.0.2 or the pseudo-version of an @main build; and
// "dev" when neither is known. info is what debug.ReadBuildInfo
// returns, nil when the binary carries none. Go records "(devel)" for
// a build that has no version, such as one inside a copy of the source
// without git.
func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
if linkerVersion != "" {
return linkerVersion
}
if info == nil || info.Main.Version == "" ||
info.Main.Version == "(devel)" {
return "dev"
}
return info.Main.Version
}
func main() {
// Once the reader of a stdout pipe has gone, as in "sfdupes report |
@@ -73,8 +98,8 @@ func main() {
// exit code. It is the program's single exit point: the subcommands
// return their errors instead of exiting, so every deferred cleanup —
// above all closing the database, which checkpoints the SQLite WAL —
// runs before the process ends. The report and trees subcommands write
// their data to stdout.
// runs before the process ends. The report, trees and info subcommands
// write their data to stdout.
func run(args []string, stdout, stderr io.Writer) int {
// A nil slice makes cobra fall back to os.Args, which would let a
// test binary's own flags reach the command tree.
@@ -114,7 +139,10 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
var showVersion bool
printVersion := runE(func(context.Context, []string) error {
_, err := fmt.Fprintf(stdout, "sfdupes %s\n", Version)
info, _ := debug.ReadBuildInfo()
_, err := fmt.Fprintf(stdout, "sfdupes %s\n",
buildVersion(Version, info))
if err != nil {
return fmt.Errorf("write stdout: %w", err)
}
@@ -148,30 +176,6 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
root.Flags().BoolVarP(&showVersion, "version", "v", false,
"print the version to stdout")
var (
scanWorkers int
scanOneFS bool
)
scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] PATH...",
Short: "Walk trees and synchronize the scan database",
Args: cobra.MinimumNArgs(1),
PreRunE: func(cmd *cobra.Command, _ []string) error {
return checkScanWorkers(cmd, scanWorkers)
},
RunE: runE(func(ctx context.Context, args []string) error {
ctx, stop := interruptContext(ctx)
defer stop()
return runScan(ctx, args, scanWorkers, scanOneFS)
}),
}
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
"concurrent workers for the walk, hash, and content phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
reportCmd := &cobra.Command{
Use: cmdReport,
Short: "Read the scan database and print the file-level duplicates report",
@@ -190,11 +194,59 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}),
}
root.AddCommand(scanCmd, reportCmd, treesCmd)
infoCmd := &cobra.Command{
Use: cmdInfo,
Short: "Read the scan database and print its details",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runInfo(ctx, stdout)
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
return root
}
// newScanCommand builds the scan subcommand, with its --workers, -x and
// --no-default-excludes flags. Its help lists the default excludes of
// the system it runs on, and the filesystem types it never records
// anything on.
func newScanCommand() *cobra.Command {
var (
scanWorkers int
scanOneFS bool
scanNoDefaultExcludes bool
)
short := "Walk trees and synchronize the scan database"
scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
Short: short,
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1),
PreRunE: func(cmd *cobra.Command, _ []string) error {
return checkScanWorkers(cmd, scanWorkers)
},
RunE: runE(func(ctx context.Context, args []string) error {
ctx, stop := interruptContext(ctx)
defer stop()
return runScan(ctx, args, scanWorkers, scanOneFS,
scanNoDefaultExcludes)
}),
}
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
"concurrent workers for the walk, hash, and content phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
return scanCmd
}
// checkScanWorkers rejects a scan --workers value below 1. That is a
// usage error reported in one line: cobra prints the returned message
// without the usage text, and run exits 2.
main_test.go
+119 -21
View File
@@ -4,10 +4,12 @@ import (
"bytes"
"context"
"errors"
"fmt"
"io"
"io/fs"
"os"
"path/filepath"
"runtime/debug"
"slices"
"strconv"
"strings"
@@ -18,6 +20,36 @@ import (
// error prints and a runtime failure must not.
const usageMarker = "Usage:"
// TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail
// without saying why.
func TestMain(m *testing.M) {
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
fmt.Fprintf(os.Stderr, "list the mounted filesystems: %v\n", err)
os.Exit(1)
}
dir := os.TempDir()
fi, err := os.Stat(dir)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
dev, ok := deviceOfInfo(fi)
if fsType, found := nonPersistent[dev]; ok && found {
fmt.Fprintf(os.Stderr, "%s is on %s, not a persistent filesystem, "+
"and scan records nothing there: set TMPDIR to a directory on a "+
"persistent filesystem\n", dir, fsType)
os.Exit(1)
}
os.Exit(m.Run())
}
// walSuffixes are the SQLite sidecar files a WAL-mode database keeps
// while it is open. A clean close checkpoints the WAL and removes
// both; finding either afterwards means the database was never closed.
@@ -284,10 +316,16 @@ func assertFatalOutput(t *testing.T, stderr, stdout string) {
}
func TestRunMissingDatabaseIsFatal(t *testing.T) {
// README §Database: report and trees need an existing database; a
// missing one exits 1 with a message telling the user to run scan,
// after the line naming the database.
for _, name := range []string{cmdReport, cmdTrees} {
// README §Database: report, trees and info need an existing
// database; a missing one exits 1 with a message telling the user to
// run scan, after the line naming the database.
openLines := map[string]string{
cmdReport: reportOpenLine,
cmdTrees: treesOpenLine,
cmdInfo: infoOpenLine,
}
for name, openLine := range openLines {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
@@ -303,8 +341,10 @@ func TestRunMissingDatabaseIsFatal(t *testing.T) {
t.Errorf("run(%s) = %d, want %d", name, code, exitFatal)
}
// Each shows its open phase, as it starts and as it ends,
// before the error.
want := name + ": database " + path + " (from " + databaseEnv +
")\n" +
")\n" + openLine + "\n" + openLine + "\n" +
"sfdupes: " + path + ": no database (run \"sfdupes " +
"scan\" first, or set " + databaseEnv + ")\n"
if got := stderr(); got != want {
@@ -505,8 +545,9 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
// exist at the default path on the machine running the test, so
// nothing here may create or read one: scan gets an operand that
// does not exist, which stops it before it takes the lock, and
// report and trees a cancelled context, which stops them before
// they read anything.
// report, trees and info a cancelled context, which stops them
// before they read anything. They show their open phase after the
// line.
t.Setenv(databaseEnv, "")
// Whether scan calls the database new depends on that machine too.
@@ -526,7 +567,7 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}{
cmdScan: {
run: func() error {
return runScan(t.Context(), []string{missing}, 1, false)
return runScan(t.Context(), []string{missing}, 1, false, false)
},
want: scanWant,
},
@@ -534,13 +575,22 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
run: func() error {
return runReport(cancelledContext(t), io.Discard)
},
want: "report: database " + defaultDatabasePath + " (default)\n",
want: "report: database " + defaultDatabasePath + " (default)\n" +
reportOpenLine + "\n" + reportOpenLine + "\n",
},
cmdTrees: {
run: func() error {
return runTrees(cancelledContext(t), io.Discard)
},
want: "trees: database " + defaultDatabasePath + " (default)\n",
want: "trees: database " + defaultDatabasePath + " (default)\n" +
treesOpenLine + "\n" + treesOpenLine + "\n",
},
cmdInfo: {
run: func() error {
return runInfo(cancelledContext(t), io.Discard)
},
want: "info: database " + defaultDatabasePath + " (default)\n" +
infoOpenLine + "\n" + infoOpenLine + "\n",
},
}
@@ -568,6 +618,7 @@ func TestRunUsageErrors(t *testing.T) {
"scan without paths": {[]string{cmdScan}, usageMarker},
"report with args": {[]string{cmdReport, "x"}, usageMarker},
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
"unknown subcommand": {[]string{"nope"}, "unknown command"},
}
@@ -664,6 +715,8 @@ func TestRunVersion(t *testing.T) {
// README §Subcommands: the version is one line on stdout, with
// nothing on stderr, and exits 0.
info, _ := debug.ReadBuildInfo()
for _, arg := range []string{"--version", "-v"} {
var stdout, stderr bytes.Buffer
@@ -672,7 +725,7 @@ func TestRunVersion(t *testing.T) {
t.Errorf("run(%s) = %d, want %d", arg, code, exitOK)
}
want := "sfdupes " + Version + "\n"
want := "sfdupes " + buildVersion(Version, info) + "\n"
if got := stdout.String(); got != want {
t.Errorf("run(%s) stdout = %q, want %q", arg, got, want)
}
@@ -683,6 +736,40 @@ func TestRunVersion(t *testing.T) {
}
}
func TestBuildVersion(t *testing.T) {
t.Parallel()
// README "Install": the linker's version wins; otherwise the main
// module's version that Go recorded; "dev" when neither is known.
tagged := &debug.BuildInfo{Main: debug.Module{Version: "v0.0.2"}}
pseudo := &debug.BuildInfo{Main: debug.Module{
Version: "v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1",
}}
devel := &debug.BuildInfo{Main: debug.Module{Version: "(devel)"}}
cases := []struct {
name string
linker string
info *debug.BuildInfo
want string
}{
{"linker set", "v0.0.2-3-g4e4a94d", tagged, "v0.0.2-3-g4e4a94d"},
{"linker set, no build info", "4e4a94d", nil, "4e4a94d"},
{"build info with a tag", "", tagged, "v0.0.2"},
{"build info with a pseudo-version", "", pseudo,
"v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1"},
{"(devel)", "", devel, "dev"},
{"no build info", "", nil, "dev"},
}
for _, c := range cases {
if got := buildVersion(c.linker, c.info); got != c.want {
t.Errorf("%s: buildVersion(%q, ...) = %q, want %q",
c.name, c.linker, got, c.want)
}
}
}
func TestRunVersionWriteFailureIsFatal(t *testing.T) {
t.Parallel()
@@ -908,11 +995,16 @@ func TestRunReportSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want)
}
want = "report: database " + path + " (from " + databaseEnv + ")\n" +
"report: 2 records read, 1 duplicate groups, 1 dupe files, " +
// Its phases come between the two lines; see
// TestRunReportProgressNamesEachPhase.
first := "report: database " + path + " (from " + databaseEnv + ")\n"
last := "\nreport: 2 records read, 1 duplicate groups, 1 dupe files, " +
"300 B reclaimable\n"
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
if got := stderr(); !strings.HasPrefix(got, first) ||
!strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to start %q and end %q",
got, first, last)
}
assertNoSidecars(t, path)
@@ -942,11 +1034,16 @@ func TestRunTreesSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want)
}
want = "trees: database " + path + " (from " + databaseEnv + ")\n" +
"trees: 2 records read, 1 duplicate tree groups, 1 dupe trees, " +
"300 B reclaimable\n"
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
// Its phases come between the two lines; see
// TestRunTreesProgressNamesEachPhase.
first := "trees: database " + path + " (from " + databaseEnv + ")\n"
last := "\ntrees: 2 records read, 1 duplicate tree groups, " +
"1 dupe trees, 300 B reclaimable\n"
if got := stderr(); !strings.HasPrefix(got, first) ||
!strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to start %q and end %q",
got, first, last)
}
assertNoSidecars(t, path)
@@ -1238,7 +1335,7 @@ func TestRunReportsDuringScan(t *testing.T) {
func TestRunStdoutClosedIsFatal(t *testing.T) {
// README §Error handling: a stdout write failure exits 1, reported
// in one line on stderr.
for _, name := range []string{cmdReport, cmdTrees} {
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
t.Run(name, func(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
@@ -1288,6 +1385,7 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
cases := map[string]func(context.Context, io.Writer) error{
cmdReport: runReport,
cmdTrees: runTrees,
cmdInfo: runInfo,
}
for name, fn := range cases {
progress.go
+468 -124
View File
@@ -2,122 +2,333 @@ package main
import (
"fmt"
"io"
"math"
"os"
"slices"
"strings"
"sync"
"time"
"github.com/schollz/progressbar/v3"
"golang.org/x/term"
)
// plainInterval is the minimum time between progress lines when stderr
// is not a TTY.
// plainInterval is the time between progress lines while a phase runs,
// when stderr is not a TTY.
const plainInterval = 5 * time.Second
// barThrottle caps how often the TTY progress bar redraws.
const barThrottle = 100 * time.Millisecond
// redrawInterval is the time between redraws of the progress display on
// a TTY.
const redrawInterval = 100 * time.Millisecond
// walkSpinnerType selects the progressbar library's spinner style used
// for the walk pass, which has no known total.
const walkSpinnerType = 14
// rateWindow is the time the rate a display shows covers: the items
// completed over the last rateWindow, which is the time since the line
// before when stderr is not a TTY.
const rateWindow = 5 * time.Second
// stallTime is how long a phase goes with no item completed before its
// line says it has stalled, and where it waits, in place of the rate.
const stallTime = 10 * time.Second
// spinnerType selects the progressbar library's spinner style.
const spinnerType = 14
// barWidth is the width of the progressbar library's bar between its
// two ends, as in the example line in README "Progress".
const barWidth = 10
// percentScale converts a fraction to a percentage.
const percentScale = 100
// fallbackWidth is the width taken for a terminal whose width cannot be
// read, as the progressbar library takes it.
const fallbackWidth = 80
// spinnerMargin is the columns of a spinner's line besides its text:
// the spinner and the three spaces the library draws around the text,
// and the terminal's last column, which every line stays clear of so
// that no terminal wraps it.
const spinnerMargin = 5
// barMargin is the columns of a bar's line besides its text: the
// percent and the bar the library draws after the text, the spaces
// around them, and the terminal's last column.
const barMargin = barWidth + 11
// countingRecords is what the display of a phase that reads the records
// says while it counts them, before it has its total.
const countingRecords = "counting the records"
// openPhase is the name of the phase in which report, trees and info
// open the database, and openingDatabase what its display says while
// it does.
const (
openPhase = "open"
openingDatabase = "opening the database"
)
// stderrIsTTY reports whether stderr is attached to a terminal.
func stderrIsTTY() bool {
return term.IsTerminal(int(os.Stderr.Fd()))
}
// progress renders one scan pass's progress on stderr. On a TTY it
// delegates to the progressbar library (spinner style when the total is
// unknown, full bar with count/percent/rate/elapsed/ETA otherwise). When
// stderr is not a TTY it emits no ANSI redraws: it prints a plain
// one-line update as the pass starts, then no more often than every
// plainInterval.
//
// All methods must be called from the main goroutine only. On a TTY
// the library also redraws a spinner from its own goroutine, several
// times a second, so its count and elapsed time stay current while a
// pass waits for its next item. A nil *progress is a valid
// no-display receiver: every method is a no-op, so batched database
// flushes during the streaming pass can reuse the update-pass helpers
// without rendering anything.
type progress struct {
label string
total int64 // -1 when unknown (walk pass)
bar *progressbar.ProgressBar
count int64
start time.Time
last time.Time
}
func newProgress(label string, total int64) *progress {
p := &progress{label: label, total: total, start: time.Now()}
if stderrIsTTY() {
p.bar = newBar(label, total)
return p
// terminalWidth returns the width of the terminal on stderr, or
// fallbackWidth, as the progressbar library does, when it cannot read
// one.
func terminalWidth() int {
width, _, err := term.GetSize(int(os.Stderr.Fd()))
if err != nil || width <= 0 {
return fallbackWidth
}
// Print the zero state at once: the first item may take minutes,
// and a pass must never look hung.
p.last = p.start
fmt.Fprintln(os.Stderr, p.plainLine())
return width
}
// phaseLabel names phase with its place among phases, as "[2/5] walk".
func phaseLabel(phases []string, phase string) string {
return fmt.Sprintf("[%d/%d] %s",
slices.Index(phases, phase)+1, len(phases), phase)
}
// progressParams describe one phase's progress display as it starts.
type progressParams struct {
label string // the phase and its place, from phaseLabel
unit string // what the phase counts, as "files"
total int64 // the items the phase will complete; -1 when not known
estimate bool // total is an estimate, which the count may pass
waitingFor string // what the phase waits for until setTotal is called
stalledOn string // where the phase waits, as "disk/db"; see stallTime
}
// drawnCount is the count of items completed as of one draw, and when
// that draw was.
type drawnCount struct {
at time.Time
count int64
}
// progress renders one phase's progress on stderr. On a TTY the
// progressbar library draws line, fitted to the terminal's width: after
// the library's spinner, or, for a phase with an exact total, before the
// library's bar when the terminal has room for the bar beside it. It is
// drawn again every redrawInterval, so its elapsed time keeps moving
// while the phase waits for its next item. When stderr is not a TTY it
// emits no ANSI redraws: it prints line as the phase starts, every
// plainInterval while it runs, and as it ends.
//
// The draws come from a goroutine of their own, so mu guards the fields
// below it. Every method must be called from the main goroutine. A nil
// *progress is a valid no-display receiver: every method is a no-op, so
// batched database flushes during the hash phase can reuse the update
// phase's helpers without rendering anything, and its writer writes
// with no display to clear.
type progress struct {
now func() time.Time // reads the time
width func() int // reads the terminal's width; nil when not a TTY
stop chan struct{} // closed by finish to end the draws
done chan struct{} // closed once the draws have ended
mu sync.Mutex
bar *progressbar.ProgressBar // what the library draws; nil until then
withBar bool // bar is the library's bar, not its spinner
widest int // the widest line bar has drawn, in columns
label string
unit string
total int64
estimate bool
waitingFor string
stalledOn string
count int64
start time.Time // when the phase started, for the elapsed time
rateStart time.Time // when the count started, for the ETA
lastItem time.Time // when the last item was completed, or the count started
// drawn holds the count at each draw over the last rateWindow, oldest
// first; the rate counts from the first. It keeps the draw before the
// latest even when that is older, as the line before is when lines
// are plainInterval apart and a tick comes late.
drawn []drawnCount
}
// newProgress starts the display of one phase on stderr.
func newProgress(params progressParams) *progress {
if stderrIsTTY() {
return startProgress(params, terminalWidth, redrawInterval, time.Now)
}
return startProgress(params, nil, plainInterval, time.Now)
}
// startProgress shows the phase params describes at once, and again
// every interval until finish: with the progressbar library on a
// terminal, whose width each draw reads by calling width, or as a plain
// line when width is nil. The display reads the time by calling now.
func startProgress(params progressParams, width func() int,
interval time.Duration, now func() time.Time,
) *progress {
start := now()
p := &progress{
now: now, width: width,
stop: make(chan struct{}), done: make(chan struct{}),
label: params.label, unit: params.unit, total: params.total,
estimate: params.estimate, waitingFor: params.waitingFor,
stalledOn: params.stalledOn, start: start, rateStart: start,
lastItem: start, drawn: []drawnCount{{at: start}},
}
// Show the zero state at once: the first item may take minutes,
// and a phase must never look hung.
p.draw()
go func() {
defer close(p.done)
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-p.stop:
return
case <-ticker.C:
p.draw()
}
}
}()
return p
}
// newBar builds the TTY display for newProgress.
func newBar(label string, total int64) *progressbar.ProgressBar {
opts := []progressbar.Option{
progressbar.OptionSetWriter(os.Stderr),
progressbar.OptionSetDescription(label),
progressbar.OptionShowCount(),
progressbar.OptionShowIts(),
progressbar.OptionSetItsString("files"),
progressbar.OptionSetElapsedTime(true),
progressbar.OptionThrottle(barThrottle),
// Render at zero immediately: a phase must be visible the
// moment it starts, even before its first item completes.
progressbar.OptionSetRenderBlankState(true),
}
if total >= 0 {
opts = append(opts,
progressbar.OptionSetPredictTime(true),
progressbar.OptionShowElapsedTimeOnFinish(),
)
} else {
opts = append(opts,
progressbar.OptionSetPredictTime(false),
progressbar.OptionSpinnerType(walkSpinnerType),
)
// draw shows the phase's progress: on a TTY in place of the line
// before, and otherwise as a plain line. It first adds the count to
// drawn, then drops from drawn the draws older than rateWindow, except
// the one before this draw.
func (p *progress) draw() {
p.mu.Lock()
defer p.mu.Unlock()
now := p.now()
p.drawn = append(p.drawn, drawnCount{at: now, count: p.count})
for len(p.drawn) > 2 && now.Sub(p.drawn[0].at) > rateWindow {
p.drawn = p.drawn[1:]
}
return progressbar.NewOptions64(total, opts...)
if p.width == nil {
fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true))
return
}
width := p.width()
// A phase with an exact total gets the library's bar when its whole
// line, without the percent the bar shows, fits beside the bar: the
// bar gives way before any part of the line. A total of 0 leaves the
// bar nothing to fill.
text, withBar := "", false
if p.total > 0 && !p.estimate {
text = p.line(now, math.MaxInt, false)
withBar = len(text) <= width-barMargin
}
margin := barMargin
if !withBar {
margin = spinnerMargin
text = p.line(now, width-margin, true)
}
// The library makes a bar or a spinner, and before each draw erases
// its line with as many spaces as the widest line it has drawn. A
// display gets a new one when it turns from one to the other, as its
// line grows or the terminal is resized, and when the terminal has
// been made too narrow for that erase, which would wrap and leave a
// line behind at every draw.
if p.bar == nil || p.withBar != withBar || p.widest >= width {
p.newBar(withBar, width)
}
if withBar {
_ = p.bar.Set64(p.count)
}
p.bar.Describe(text)
// The library drew the text and its margin, less the terminal's last
// column, which the margin keeps clear.
p.widest = max(p.widest, len(text)+margin-1)
}
// increment records one completed item and refreshes the display.
// newBar has the library make what draw draws: a bar for the phase's
// total when withBar is set, and a spinner when not. Either shows the
// text draw gives it, and nothing of its own besides the spinner, or
// the bar and its percent. p.mu must be held.
func (p *progress) newBar(withBar bool, width int) {
if p.bar != nil {
// Erase what was drawn before, short of the last column of a
// terminal that may have been made narrower since: the library's
// own erase is as wide as the widest line it drew.
blank := strings.Repeat(" ", min(p.widest, width-1))
fmt.Fprint(os.Stderr, "\r"+blank+"\r")
}
total := int64(-1) // the library's spinner
if withBar {
total = p.total
}
p.bar = progressbar.NewOptions64(total,
progressbar.OptionSetWriter(os.Stderr),
progressbar.OptionSetWidth(barWidth),
progressbar.OptionSpinnerType(spinnerType),
// The spinner turns at each draw, not on a timer of the
// library's own.
progressbar.OptionSetSpinnerChangeInterval(0),
// The text carries the elapsed time and the ETA.
progressbar.OptionSetElapsedTime(false),
progressbar.OptionSetPredictTime(false),
)
p.withBar, p.widest = withBar, 0
}
// setTotal gives a phase that started without a total its total, once
// it is known. The rate, the ETA and the time with no item completed
// count from then on: the time spent waiting for the total says nothing
// about how fast items go.
func (p *progress) setTotal(total int64) {
if p == nil {
return
}
now := p.now()
p.mu.Lock()
defer p.mu.Unlock()
p.total = total
p.rateStart, p.lastItem = now, now
p.drawn = []drawnCount{{at: now, count: p.count}}
}
// increment records one completed item, which the display shows at its
// next draw.
func (p *progress) increment() {
if p == nil {
return
}
now := p.now()
p.mu.Lock()
defer p.mu.Unlock()
p.count++
if p.bar != nil {
_ = p.bar.Add(1)
return
}
if time.Since(p.last) >= plainInterval {
p.last = time.Now()
fmt.Fprintln(os.Stderr, p.plainLine())
}
p.lastItem = now
}
// warnf prints a one-line warning to stderr without corrupting the bar.
// The whole message is escaped like a report's path columns, so a path
// holding a newline cannot split the warning.
// warnf prints a one-line warning to stderr without corrupting the
// display. The whole message is escaped like a report's path columns,
// so a path holding a newline cannot split the warning.
func (p *progress) warnf(format string, args ...any) {
if p == nil {
return
@@ -125,58 +336,186 @@ func (p *progress) warnf(format string, args ...any) {
msg := escapePath(fmt.Sprintf(format, args...))
if p.bar != nil && p.total < 0 {
// The library also redraws a spinner from its own goroutine, so
// a direct write could land inside a redraw. The bar prints the
// warning itself, just before its next redraw.
_, _ = progressbar.Bprintln(p.bar, msg)
return
}
p.mu.Lock()
defer p.mu.Unlock()
if p.bar != nil {
// The warning takes the display's line; the next draw shows
// the display again, on the line below.
_ = p.bar.Clear()
}
fmt.Fprintln(os.Stderr, msg)
// The library never draws a finished bar again, so the warning for
// a pass's last item would erase it for good: draw its last frame
// again below the warning.
if p.bar != nil && p.bar.IsFinished() {
fmt.Fprint(os.Stderr, p.bar.String())
}
}
// finish terminates the pass's display. A bar whose pass stopped short
// of its total, as an interrupted one does, is left as last drawn; the
// library's Finish would fill it up.
// progressWriter writes a phase's output to out, which may be the file
// the phase's display is drawn on, as stdout is in a plain "sfdupes
// trees" on a terminal. It holds what is written to it and writes that
// to out in pieces of about ioBufSize bytes. When out is the file
// stderr is, each piece is written under the display's lock with the
// display cleared first, as warnf does for a warning: no draw lands in
// the middle of a piece, and the next draw shows the display again, on
// the line below. Every write to it must end with a newline, as a row
// written with one fmt.Fprintf does, so that each piece ends a line.
// Flush writes what it holds once the last row is written.
type progressWriter struct {
// p is the display to clear, nil unless out is the file stderr is.
p *progress
out io.Writer
buf []byte
}
// writer returns a progressWriter writing to out around p's display.
// It clears the display and takes its lock only when out is the file
// stderr is: the same device and inode, as os.SameFile compares them,
// which is one terminal, or one pipe or file that both are sent to.
// Otherwise it writes to out without the lock, so the display keeps
// being drawn while a write to out waits for a slow reader.
func (p *progress) writer(out io.Writer) *progressWriter {
w := &progressWriter{out: out}
if f, ok := out.(*os.File); ok {
outInfo, outErr := f.Stat()
errInfo, errErr := os.Stderr.Stat()
if outErr == nil && errErr == nil && os.SameFile(outInfo, errInfo) {
w.p = p
}
}
return w
}
// Write holds b, and writes what it holds to out once that comes to
// ioBufSize bytes.
func (w *progressWriter) Write(b []byte) (int, error) {
w.buf = append(w.buf, b...)
if len(w.buf) < ioBufSize {
return len(b), nil
}
return len(b), w.Flush()
}
// Flush writes what w holds to out. When out is the file stderr is, the
// display is cleared first and not drawn again until it is written.
func (w *progressWriter) Flush() error {
if w.p != nil {
w.p.mu.Lock()
defer w.p.mu.Unlock()
if w.p.bar != nil {
_ = w.p.bar.Clear()
}
}
_, err := w.out.Write(w.buf)
w.buf = w.buf[:0]
return err
}
// finish ends the phase's display, drawn once more as the phase ends.
// A phase stopped short of its total, as an interrupted one is, shows
// the count it stopped at rather than a full bar.
func (p *progress) finish() {
if p == nil {
return
}
if p.bar != nil {
if p.total >= 0 && p.count < p.total {
_ = p.bar.Exit()
} else {
_ = p.bar.Finish()
}
close(p.stop)
<-p.done
p.draw()
if p.width != nil {
// The display stays on the TTY as drawn; what follows starts
// on a line of its own.
fmt.Fprintln(os.Stderr)
return
}
fmt.Fprintln(os.Stderr, p.plainLine())
}
// plainLine formats a non-TTY progress line.
func (p *progress) plainLine() string {
elapsed := time.Since(p.start)
// line formats the phase's progress as of now in at most room columns,
// as
//
// [3/5] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s
//
// where "files" is the phase's unit, without the percent when percent
// is false, as before the library's bar, which shows its own. A phase
// with no total shows the items done instead, or what it waits for. The
// rate is that since the first draw in drawn, and the ETA comes from
// the rate since the count started. Once no item has been completed for
// stallTime, the rate, which would read 0, gives way to the word
// "stalled" and where the phase waits, as "stalled disk/db".
//
// The phase and its place, the progress, the elapsed time and the ETA
// are kept whole. A line wider than room leaves out the rate, the words
// of a stalled phase, or what the phase waits for; only if it is still
// wider is it cut at room. Every part is ASCII, so its length is its
// width in columns. p.mu must be held.
func (p *progress) line(now time.Time, room int, percent bool) string {
elapsed := now.Sub(p.start).Round(time.Second)
// head and tail hold the parts kept whole; extra, between them, is
// the part left out.
var head, extra, tail string
if p.total < 0 && p.waitingFor != "" {
head, extra = p.label+": ", p.waitingFor+", "
tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed)
} else {
since := p.drawn[0]
rate := perSecond(p.count-since.count, now.Sub(since.at))
average := perSecond(p.count, now.Sub(p.rateStart))
head = p.label + ": " + p.itemsDone(percent) + ", "
extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit)
tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average))
if now.Sub(p.lastItem) >= stallTime {
extra = "stalled " + p.stalledOn + ", "
}
}
full, kept := head+extra+tail, head+tail
switch {
case len(full) <= room:
return full
case len(kept) <= room:
return kept
}
return kept[:max(room, 0)]
}
// perSecond returns count items over d as items a second, or 0 when no
// time has passed.
func perSecond(count int64, d time.Duration) float64 {
if d <= 0 {
return 0
}
return float64(count) / d.Seconds()
}
// itemsDone formats the items done, as "[1234/5000] 25%", with "~"
// before a total that is an estimate and without the percent when
// percent is false, or as "1234 files", in the phase's unit, when the
// phase has no total. p.mu must be held.
func (p *progress) itemsDone(percent bool) string {
if p.total < 0 {
return fmt.Sprintf("%s: %d files, elapsed %s",
p.label, p.count, elapsed.Round(time.Second))
return fmt.Sprintf("%d %s", p.count, p.unit)
}
approx := ""
if p.estimate {
approx = "~"
}
done := fmt.Sprintf("[%d/%s%d]", p.count, approx, p.total)
if !percent {
return done
}
pct := float64(percentScale)
@@ -184,19 +523,24 @@ func (p *progress) plainLine() string {
pct = percentScale * float64(p.count) / float64(p.total)
}
rate := 0.0
if s := elapsed.Seconds(); s > 0 {
rate = float64(p.count) / s
return fmt.Sprintf("%s %.0f%%", done, pct)
}
// eta returns the time the phase has left at rate items a second, or
// "unknown": with no total, before the first item, and once the count
// passes an estimate. p.mu must be held.
func (p *progress) eta(rate float64) string {
switch {
case p.total < 0, p.estimate && p.count > p.total:
return "unknown"
case p.count >= p.total:
return "0s"
case rate <= 0:
return "unknown"
}
eta := "?"
if rate > 0 && p.count <= p.total {
remaining := time.Duration(float64(p.total-p.count) / rate * float64(time.Second))
eta = remaining.Round(time.Second).String()
}
remaining := time.Duration(
float64(p.total-p.count) / rate * float64(time.Second))
return fmt.Sprintf("%s: [%d/%d] %.0f%% %.0f files/s elapsed %s eta %s",
p.label, p.count, p.total, pct, rate,
elapsed.Round(time.Second), eta)
return remaining.Round(time.Second).String()
}
progress_test.go
+861 -83
View File
File diff suppressed because it is too large Load Diff
report.go
+190 -46
View File
@@ -1,11 +1,12 @@
package main
import (
"bufio"
"context"
"database/sql"
"fmt"
"io"
"os"
"slices"
"strings"
"time"
)
@@ -29,13 +30,25 @@ type scanRec struct {
path string
}
// reportPhase names one phase of report with its place among all of
// them, as "[2/2] read", for the phase's progress display. The list
// here is the one place the phases are counted.
func reportPhase(name string) string {
return phaseLabel([]string{openPhase, "read"}, name)
}
// runReport implements the report subcommand: it prints the file-level
// duplicates report as TSV on stdout. SQLite groups and orders the
// records, and each row is written as it is read, so no group is held
// in memory. It never touches the scanned filesystem; its only I/O is
// the database (with SQLite's temporary sort file), stdout, and stderr.
// It names the database on stderr before it opens it. Any database
// duplicates report as TSV on stdout. It never touches the scanned
// filesystem; its only I/O is the database, stdout, and stderr. It
// names the database on stderr before it opens it. Any database
// problem, including a missing database, is fatal.
//
// The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and read (count the records,
// then read them in signature order, writing each size's duplicate
// groups to stdout once that size is read; see readReport). Each
// display ends before the next phase starts, and before an error is
// returned to be printed.
func runReport(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
@@ -45,65 +58,196 @@ func runReport(ctx context.Context, stdout io.Writer) error {
fmt.Fprintf(os.Stderr, "report: database %s (%s)\n",
escapePath(dbPath), source)
// Opening can wait on a busy database for as long as the busy
// timeout.
prog := newProgress(progressParams{
label: reportPhase(openPhase), total: -1,
waitingFor: openingDatabase,
})
db, err := openReportDatabase(ctx, dbPath)
prog.finish()
if err != nil {
return err
}
defer func() { _ = db.Close() }()
out := bufio.NewWriterSize(stdout, ioBufSize)
// The read phase writes the report to stdout while it reads the
// records, so it may wait on either.
prog = newProgress(progressParams{
label: reportPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords, stalledOn: "db/out",
})
report, err := readReport(ctx, db, dbPath, stdout, prog)
_, err = fmt.Fprintln(out, "first\tdupe\tsize")
if err != nil {
return fmt.Errorf("write stdout: %w", err)
}
var (
groups, dupeFiles int
reclaimable int64
writeErr error
)
records, err := loadDupeRows(ctx, db,
func(first, path string, size int64) error {
// A group's first path is its first row; every other
// path is a dupe.
if path == first {
groups++
return nil
}
_, writeErr = fmt.Fprintf(out, "%s\t%s\t%d\n",
escapePath(first), escapePath(path), size)
dupeFiles++
reclaimable += size
return writeErr
})
if writeErr != nil {
return fmt.Errorf("write stdout: %w", writeErr)
}
prog.finish()
if err != nil {
return fmt.Errorf("database %s: %w", dbPath, err)
}
err = out.Flush()
if err != nil {
return fmt.Errorf("write stdout: %w", err)
return err
}
fmt.Fprintf(os.Stderr,
"report: %d records read, %d duplicate groups, %d dupe files, "+
"%s reclaimable\n",
records, groups, dupeFiles, humanBytes(reclaimable))
report.records, report.dupeGroups, report.dupeFiles,
humanBytes(report.reclaimable))
return nil
}
// readReport gives prog the number of records in db, the database at
// dbPath, as its total, then reads every record in signature order,
// counting each on prog, and writes the report to stdout: the header
// line, then the rows of each size's duplicate groups once that size is
// read. stdout may be the terminal prog is drawn on, so the lines go
// through prog's writer. It returns what it wrote, with the totals for
// the summary line.
func readReport(ctx context.Context, db *sql.DB, dbPath string,
stdout io.Writer, prog *progress,
) (*reportWriter, error) {
out := prog.writer(stdout)
report := &reportWriter{out: out}
_, err := fmt.Fprintln(out, "first\tdupe\tsize")
if err != nil {
return nil, fmt.Errorf("write stdout: %w", err)
}
var writeErr error
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
prog.increment()
writeErr = report.add(r)
return writeErr
})
if writeErr != nil {
return nil, fmt.Errorf("write stdout: %w", writeErr)
}
if err != nil {
return nil, fmt.Errorf("database %s: %w", dbPath, err)
}
err = report.finish()
if err == nil {
err = out.Flush()
}
if err != nil {
return nil, fmt.Errorf("write stdout: %w", err)
}
return report, nil
}
// reportWriter writes the report's rows to out from the records, added
// in signature order, largest size first, as loadSignatureRows reads
// them. The records sharing a signature come together, and so do the
// records of one size, so it holds only the duplicate groups of the
// size being read, and writes them, ordered by first path, once a
// record of a smaller size comes. It counts what the summary line
// reports.
type reportWriter struct {
out io.Writer
sig scanRec // the signature being read; its path is not used
paths []string // the paths read so far with sig
groups [][]string // the duplicate groups of sig.size, each sorted
records, dupeGroups, dupeFiles int
reclaimable int64
}
// add takes the next record. One with the signature being read joins
// its paths; any other ends them, and one of a smaller size first
// writes the groups of the size before.
func (w *reportWriter) add(r scanRec) error {
w.records++
// A record without a content hash has unknown content (README
// "Database"), so it is never a duplicate.
if r.content == "" {
return nil
}
if r.size == w.sig.size && r.head == w.sig.head &&
r.tail == w.sig.tail && r.content == w.sig.content {
w.paths = append(w.paths, r.path)
return nil
}
w.endGroup()
if r.size != w.sig.size {
err := w.writeGroups()
if err != nil {
return err
}
}
w.sig = r
w.paths = append(w.paths, r.path)
return nil
}
// endGroup ends the paths read with w.sig: two or more are a duplicate
// group, held with its paths sorted until writeGroups writes it.
func (w *reportWriter) endGroup() {
if len(w.paths) < minGroupSize {
// A path with no duplicate is dropped, and its slice reused.
w.paths = w.paths[:0]
return
}
slices.Sort(w.paths)
w.groups = append(w.groups, w.paths)
w.paths = nil
}
// writeGroups writes the duplicate groups held, all of size w.sig.size,
// ordered by first path: for each group, a row for every path but the
// first. No two groups share a first path, since no two records share
// a path.
func (w *reportWriter) writeGroups() error {
slices.SortFunc(w.groups, func(a, b []string) int {
return strings.Compare(a[0], b[0])
})
for _, g := range w.groups {
for _, path := range g[1:] {
_, err := fmt.Fprintf(w.out, "%s\t%s\t%d\n",
escapePath(g[0]), escapePath(path), w.sig.size)
if err != nil {
return err
}
w.dupeFiles++
w.reclaimable += w.sig.size
}
w.dupeGroups++
}
w.groups = nil
return nil
}
// finish writes the groups of the last size read.
func (w *reportWriter) finish() error {
w.endGroup()
return w.writeGroups()
}
// escapePath returns a path as it is written in a report column (README
// "Report output format"): a backslash, tab, newline or carriage return
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
report_test.go
+190 -25
View File
@@ -9,6 +9,7 @@ import (
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"testing"
"time"
@@ -28,7 +29,8 @@ func awkwardPairRecs() []scanRec {
}
}
// seedDatabase writes recs into a fresh database and returns its path.
// seedDatabase writes recs into a fresh database, closes it as scan
// does, out of WAL mode, and returns its path.
func seedDatabase(t *testing.T, recs []scanRec) string {
t.Helper()
@@ -44,10 +46,7 @@ func seedDatabase(t *testing.T, recs []scanRec) string {
t.Fatal(err)
}
err = db.Close()
if err != nil {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, path)
return path
}
@@ -60,27 +59,41 @@ type dupeGroup struct {
}
// dupeGroups returns the duplicate groups report reads from db, in
// report order.
// report order, from the rows it writes: a group's rows come together,
// each naming the group's first path.
func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
t.Helper()
var groups []dupeGroup
var out bytes.Buffer
_, err := loadDupeRows(t.Context(), db,
func(first, path string, size int64) error {
if path == first {
groups = append(groups, dupeGroup{size: size})
}
g := &groups[len(groups)-1]
g.paths = append(g.paths, path)
return nil
})
_, err := readReport(t.Context(), db, "", &out, nil)
if err != nil {
t.Fatal(err)
}
var groups []dupeGroup
rows := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n")
for _, row := range rows[1:] { // after the header
fields := strings.Split(row, "\t")
if len(fields) != 3 {
t.Fatalf("row %q, want first, dupe and size", row)
}
size, err := strconv.ParseInt(fields[2], 10, 64)
if err != nil {
t.Fatal(err)
}
if len(groups) == 0 || groups[len(groups)-1].paths[0] != fields[0] {
groups = append(groups,
dupeGroup{size: size, paths: []string{fields[0]}})
}
g := &groups[len(groups)-1]
g.paths = append(g.paths, fields[1])
}
return groups
}
@@ -118,26 +131,147 @@ func TestRunReportEscapesPaths(t *testing.T) {
}
func TestReportStdoutFailsWhileReading(t *testing.T) {
// Each row holds two paths longer than dir, so the report is more
// than twice the stdout buffer and stdout fails while rows are
// still being read, not at the final flush.
t.Parallel()
// Every two records are a duplicate pair of a size of their own, and
// a pair's row is written once a smaller size is read. Each row
// holds two paths longer than dir, so the rows come to more than
// twice the stdout buffer, and stdout fails while records are still
// being read, not once every record is.
dir := "/" + strings.Repeat("d", 4096)
recs := make([]scanRec, ioBufSize/len(dir))
recs := make([]scanRec, 2*ioBufSize/len(dir))
for i := range recs {
recs[i] = scanRec{
size: 1, head: "h", tail: "t", content: "c",
size: int64(i/2) + 1, head: "h", tail: "t", content: "c",
path: fmt.Sprintf("%s/%d", dir, i),
}
}
t.Setenv(databaseEnv, seedDatabase(t, recs))
db := openTestDB(t)
err := runReport(t.Context(), failingWriter{})
err := applyChanges(t.Context(), db, recs, nil, nil)
if err != nil {
t.Fatal(err)
}
// Counts the records read, and draws nothing.
prog := &progress{now: time.Now}
_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
if !errors.Is(err, errWriteFailed) ||
!strings.HasPrefix(err.Error(), "write stdout: ") {
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
}
if prog.count >= int64(len(recs)) {
t.Errorf("stdout failed after all %d records were read, want it "+
"to stop the read", prog.count)
}
}
// reportOpenLine is the plain line of report's open phase when it opens
// the database at once; it prints it as the phase starts and as it ends.
const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown"
// smokeReport is report's stdout for the smoke-test records.
const smokeReport = "first\tdupe\tsize\n" +
"/d/t1/f1\t/d/t2/f1\t3000\n" +
"/d/t1/f1\t/d/t3/f1\t3000\n" +
"/d/t1/sub/f2\t/d/t2/sub/f2\t100\n" +
"/d/t1/sub/f2\t/d/t3/sub/f2renamed\t100\n"
// TestRunReportProgressNamesEachPhase runs report on the smoke-test
// records with stderr not a terminal, and checks every line it prints
// there. After the line naming the database, each phase prints a line
// naming it with its place among the two as it starts and another as
// it ends; read counts the 6 records before it reads them. The summary
// comes last, and stdout holds the report alone.
func TestRunReportProgressNamesEachPhase(t *testing.T) {
path := seedDatabase(t, smokeTreeRecs())
t.Setenv(databaseEnv, path)
stderr := captureStderr(t)
var stdout bytes.Buffer
code := run([]string{cmdReport}, &stdout, os.Stderr)
if code != exitOK {
t.Fatalf("run(report) = %d, want %d; stderr: %s",
code, exitOK, stderr())
}
if got := stdout.String(); got != smokeReport {
t.Errorf("stdout = %q, want %q", got, smokeReport)
}
// How each line starts; the rate at the end of a phase varies.
want := []string{
"report: database " + path + " (from " + databaseEnv + ")",
reportOpenLine,
reportOpenLine,
"[2/2] read: counting the records, elapsed 0s, eta unknown",
"[2/2] read: [6/6] 100%, ",
"report: 6 records read, 2 duplicate groups, 4 dupe files, " +
"6.1 KiB reclaimable",
}
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
if len(lines) != len(want) {
t.Fatalf("stderr lines = %q, want %d lines starting %q",
lines, len(want), want)
}
for i := range want {
if !strings.HasPrefix(lines[i], want[i]) {
t.Errorf("stderr line %d = %q, want it to start %q",
i+1, lines[i], want[i])
}
}
}
// TestReadReportOnTerminal writes the smoke-test report to the terminal
// its read phase's display is drawn on, as a plain "sfdupes report"
// does: the terminal must show the header and every row whole, each on
// a line of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestReadReportOnTerminal(t *testing.T) {
stderr := captureStderr(t)
db := openTestDB(t)
err := applyChanges(t.Context(), db, smokeTreeRecs(), nil, nil)
if err != nil {
t.Fatal(err)
}
// No timed draws: the display is drawn as it starts and as it ends.
p := startProgress(progressParams{
label: reportPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords,
}, terminalWidth, time.Hour, time.Now)
_, err = readReport(t.Context(), db, "", os.Stderr, p)
p.finish()
if err != nil {
t.Fatal(err)
}
// A terminal shows each line as the text after its last carriage
// return.
var shown []string
for line := range strings.SplitSeq(stderr(), "\n") {
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
}
want := strings.Split(strings.TrimSuffix(smokeReport, "\n"), "\n")
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
t.Errorf("terminal shows lines %q, want %q first", shown, want)
}
}
func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
@@ -354,6 +488,37 @@ func TestDupeGroupsTieBreak(t *testing.T) {
}
}
func TestDupeGroupsOrderedByFirstPath(t *testing.T) {
t.Parallel()
// report reads the groups of size 50 in signature order, here /a,
// /z, /k, which is neither the order of their first paths nor its
// reverse, and must write them ordered by first path. The group of
// size 7 comes after them all, though its first path sorts first.
recs := []scanRec{
{size: 50, head: "a", tail: "a", content: "a", path: "/k/2"},
{size: 50, head: "a", tail: "a", content: "a", path: "/k/1"},
{size: 50, head: "b", tail: "b", content: "b", path: "/z/1"},
{size: 50, head: "b", tail: "b", content: "b", path: "/z/2"},
{size: 50, head: "c", tail: "c", content: "c", path: "/a/2"},
{size: 50, head: "c", tail: "c", content: "c", path: "/a/1"},
{size: 7, head: "d", tail: "d", content: "d", path: "/0/2"},
{size: 7, head: "d", tail: "d", content: "d", path: "/0/1"},
}
groups := dupeGroupsOf(t, recs)
got := make([]string, 0, len(groups))
for _, g := range groups {
got = append(got, g.paths[0])
}
want := []string{"/a/1", "/k/1", "/z/1", "/0/1"}
if !slices.Equal(got, want) {
t.Errorf("groups' first paths = %q, want %q", got, want)
}
}
func TestDupeGroupsDeterministic(t *testing.T) {
t.Parallel()
scan.go
+354 -156
View File
@@ -13,6 +13,7 @@ import (
"os"
"os/signal"
"path/filepath"
"runtime"
"slices"
"strings"
"sync"
@@ -84,27 +85,43 @@ type fileMeta struct {
hashed bool
}
// runScan implements the scan subcommand: four sequential phases —
// walk (which stats each file as it is discovered), hash, update,
// content — that synchronize the persistent database with the
// filesystem state under the PATH operands. Only files whose size at
// least one other file shares are ever hashed: a size-unique file
// cannot be a duplicate. A file of headTailMin or more gets its content
// hash only when its size, head, and tail match another file's. Flag
// parsing and the at-least-one-operand check are done by cobra. Before
// anything else, the scan names the database on stderr, saying whether
// it is new. The scan holds the lock on the database for its whole
// run, so a second scan fails before it walks the filesystem or opens
// the database.
// scanUnit is what every scan phase counts, as its progress display
// names it.
const scanUnit = "files"
// scanPhase names one phase of a scan with its place among all of them,
// as "[2/5] walk", for the phase's progress display. The list here is
// the one place the phases are counted.
func scanPhase(name string) string {
return phaseLabel(
[]string{"load", "walk", "hash", "update", "content"}, name)
}
// runScan implements the scan subcommand: five sequential phases —
// load (which reads the existing records), walk (which stats each file
// as it is discovered), hash, update, content — that synchronize the
// persistent database with the filesystem state under the PATH
// operands. Only files whose size at least one other file shares are
// ever hashed: a size-unique file cannot be a duplicate. A file of
// headTailMin or more gets its content hash only when its size, head,
// and tail match another file's. Flag parsing and the
// at-least-one-operand check are done by cobra. Before anything else,
// the scan names the database on stderr, saying whether it is new. The
// scan holds the lock on the database for its whole run, so a second
// scan fails before it walks the filesystem or opens the database.
// Errors are returned rather than exiting, so that the deferred close —
// which takes the database out of WAL mode — always runs, and the lock
// is released after it. When ctx is cancelled, as by the SIGINT or
// SIGTERM that interruptContext catches, the scan keeps what it has
// hashed (see syncScan), prints how many files its walk reached, and
// returns errInterrupted. workers must be at least 1; the scan command
// rejects anything less.
// rejects anything less. Before the lock is taken, the scan lists the
// filesystems that are not persistent, on which the walk records
// nothing; unless noDefaultExcludes is set, the walk does not enter the
// default excludes either. Once the database is open, the scan notes
// its files, which the walk passes over.
func runScan(ctx context.Context, roots []string, workers int,
oneFS bool,
oneFS, noDefaultExcludes bool,
) error {
dbPath, source, err := databasePath()
if err != nil {
@@ -127,6 +144,16 @@ func runScan(ctx context.Context, roots []string, workers int,
return err
}
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
return fmt.Errorf("list the mounted filesystems: %w", err)
}
rules := walkRules{nonPersistent: nonPersistent}
if !noDefaultExcludes {
rules.excludes = defaultExcludes(runtime.GOOS)
}
lock, err := lockScanDatabase(dbPath)
if err != nil {
return err
@@ -147,7 +174,12 @@ func runScan(ctx context.Context, roots []string, workers int,
defer closeScanDatabase(ctx, db, dbPath)
st, err := syncScan(ctx, db, roots, workers, oneFS)
rules.databaseFiles, err = databaseFiles(dbPath)
if err != nil {
return err
}
st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
if errors.Is(err, context.Canceled) {
return interrupted(st.walked)
}
@@ -219,14 +251,12 @@ func resolveRoots(roots []string) ([]string, error) {
}
// pruneRoots drops operands already covered by another operand:
// duplicates and any operand lying under another one. With -x, an
// operand under another one is kept when the walk of the deepest kept
// operand above it cannot reach it: when device puts any directory from
// it (from its directory, for a regular file) up to but not including
// that operand on a different filesystem than that operand. Every
// remaining file is then reachable through exactly one root, so walked
// paths are unique without keeping a scan-wide set of every path seen.
func pruneRoots(roots []string, oneFS bool,
// duplicates and any operand lying under another one. An operand under
// another one is kept, though, when the walk of the deepest kept operand
// above it does not reach it (see walkReaches). Every remaining file is
// then reachable through exactly one root, so walked paths are unique
// without keeping a scan-wide set of every path seen.
func pruneRoots(roots []string, oneFS bool, rules walkRules,
device func(path string) (uint64, bool),
) []string {
// Shorter-first ordering guarantees an ancestor is kept before any
@@ -250,40 +280,57 @@ func pruneRoots(roots []string, oneFS bool,
continue
}
// Without -x, and always for a duplicate, r is dropped.
if !oneFS || above == r {
continue
}
aboveDev, aboveOK := device(above)
if !aboveOK {
continue
}
// The walk of above enters a directory only when it is on
// above's filesystem, and then reaches every regular file in it
// whatever device the file reports, so for a regular-file
// operand the directories checked start at its directory.
dir := r
fi, err := os.Lstat(r)
if err == nil && fi.Mode().IsRegular() {
dir = filepath.Dir(r)
}
for ; dir != above; dir = filepath.Dir(dir) {
dev, ok := device(dir)
if ok && dev != aboveDev {
kept = append(kept, r)
break
}
// A duplicate is dropped, and so is an operand the walk of above
// reaches.
if above != r && !walkReaches(above, r, oneFS, rules, device) {
kept = append(kept, r)
}
}
return kept
}
// walkReaches reports whether the walk of operand above reaches r, an
// operand under it. That walk does not enter a default exclude in
// rules, nor a directory that device puts on a filesystem in rules that
// is not persistent, nor, with -x, one that device puts on a different
// filesystem than above, when device knows above's. It reaches every
// regular file in a directory it enters, so for a regular-file operand
// the directories checked start at its directory; they end below above.
func walkReaches(above, r string, oneFS bool, rules walkRules,
device func(path string) (uint64, bool),
) bool {
aboveDev, aboveDevOK := device(above)
dir := r
fi, err := os.Lstat(r)
if err == nil && fi.Mode().IsRegular() {
dir = filepath.Dir(r)
}
for ; dir != above; dir = filepath.Dir(dir) {
if isDefaultExclude(dir, rules.excludes) {
return false
}
dev, ok := device(dir)
if !ok {
continue
}
if _, found := rules.nonPersistent[dev]; found {
return false
}
if oneFS && aboveDevOK && dev != aboveDev {
return false
}
}
return true
}
// deviceOfPath returns the filesystem device ID of the file at path,
// when lstat succeeds and the platform exposes one.
func deviceOfPath(path string) (uint64, bool) {
@@ -325,11 +372,11 @@ type scanState struct {
// the hashed records still waiting in the batch, starts no other write
// or deletion, and returns the cancellation.
func syncScan(ctx context.Context, db *sql.DB, roots []string,
workers int, oneFS bool,
workers int, oneFS bool, rules walkRules,
) (scanStats, error) {
s := &scanState{db: db}
err := s.runPhases(ctx, roots, workers, oneFS)
err := s.runPhases(ctx, roots, workers, oneFS, rules)
if err == nil || ctx.Err() == nil {
return s.st, err
}
@@ -344,24 +391,27 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
}
// runPhases synchronizes the database with the filesystem under roots
// in four sequential phases: walk (enumerate and stat every file,
// building a complete size census), hash (read only the new or
// changed — or previously unhashed — files whose size at least one
// other file shares, committing results in batches as they arrive),
// update (record the size-unique files without reading them, and
// delete the records the scan no longer verifies), and content (fill
// in the content hash of every record of headTailMin or more whose
// size, head, and tail match another record's). Records outside the
// roots are never touched, except that the content phase fills in
// their content hash. Operands the walk cannot start from are dropped
// first, so the records beneath them are treated as if those operands
// had not been given.
// in five sequential phases: load (read the existing records), walk
// (enumerate and stat every file, building a complete size census),
// hash (read only the new or changed — or previously unhashed — files
// whose size at least one other file shares, committing results in
// batches as they arrive), update (record the size-unique files without
// reading them, and delete the records the scan no longer verifies),
// and content (fill in the content hash of every record of headTailMin
// or more whose size, head, and tail match another record's). Records
// outside the roots are never touched, except that the content phase
// fills in their content hash. Operands the walk cannot start from, or
// that are not on a persistent filesystem, are dropped first, so the
// records beneath them are treated as if those operands had not been
// given. Below an operand, the walk follows rules.
func (s *scanState) runPhases(ctx context.Context, roots []string,
workers int, oneFS bool,
workers int, oneFS bool, rules walkRules,
) error {
// Types are checked before pruning so that an operand under a
// dropped one is still scanned, not dropped as lying under it.
roots = pruneRoots(s.walkableRoots(roots), oneFS, deviceOfPath)
// Types and filesystems are checked before pruning so that an
// operand under a dropped one is still scanned, not dropped as lying
// under it.
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
deviceOfPath)
err := s.loadIndex(ctx, roots)
if err != nil {
@@ -369,7 +419,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
}
changed, unhashed := s.walkPhase(
startWalk(ctx, roots, oneFS, workers), roots)
startWalk(ctx, roots, oneFS, rules, workers), roots)
// A cancelled walk stops early, so its size census covers only part
// of the roots, and every file it never reached would look vanished
@@ -391,7 +441,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
return err
}
return s.contentPhase(ctx, workers)
return s.contentPhase(ctx, workers, rules)
}
// walkableRoots returns the operands the walk can start from: regular
@@ -399,9 +449,11 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
// about, counted as skipped, and dropped. A dropped operand is no
// longer a root, so the records stored beneath it are treated as if it
// had not been given: those under no other root count as outside the
// roots and are not deleted as unverified. An operand that fails lstat
// here is kept, and the walk warns about it.
func (s *scanState) walkableRoots(roots []string) []string {
// roots and are not deleted as unverified. An operand on a filesystem
// in rules that is not persistent is dropped the same way, with a line
// saying so that is not a warning, and is not counted as skipped. An
// operand that fails lstat here is kept, and the walk warns about it.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
kept := make([]string, 0, len(roots))
for _, root := range roots {
@@ -415,6 +467,13 @@ func (s *scanState) walkableRoots(roots []string) []string {
continue
}
line := rules.nonPersistentLine(root, fi)
if line != "" {
fmt.Fprintln(os.Stderr, escapePath(line))
continue
}
}
kept = append(kept, root)
@@ -430,10 +489,21 @@ func (s *scanState) walkableRoots(roots []string) []string {
// separately into the same database.
func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// Indexing tens of millions of records takes real time; without a
// display the scan looks hung before the walk begins.
prog := newProgress("load", -1)
// display the scan looks hung before the walk begins. Counting them
// first gives the display its total.
prog := newProgress(progressParams{
label: scanPhase("load"), unit: scanUnit, total: -1,
waitingFor: countingRecords, stalledOn: "db",
})
defer prog.finish()
total, err := countRecords(ctx, s.db)
if err != nil {
return err
}
prog.setTotal(total)
s.existing = make(map[string]fileMeta)
return loadFileMeta(ctx, s.db,
@@ -458,23 +528,48 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// (the content phase may still fill in its content hash). It returns
// the new-or-changed files and the unchanged files whose records lack
// hashes; both remain candidates until the census decides whether
// their sizes are shared. A record below a directory -x did not enter
// counts as outside the roots: it is not deleted, and its size joins
// the census. That does not hold below a root at or inside that
// directory, since that root's own walk verifies the records below it.
// their sizes are shared. A record at or below a path the walk left
// out, because -x stopped at it, it is a default exclude, or it is not
// on a persistent filesystem, counts as outside the roots: it is not
// deleted, and its size joins the census. That does not hold below a
// root at or inside that path, since that root's own walk verifies the
// records below it; nor is a line printed for such a root.
func (s *scanState) walkPhase(
events <-chan walkEvent, roots []string,
) ([]fileRec, []fileRec) {
prog := newProgress("walk", -1)
// The records the last scan left under the roots estimate how many
// files the walk will find. A first scan has none, and no estimate.
estimate := int64(len(s.existing))
if estimate == 0 {
estimate = -1
}
prog := newProgress(progressParams{
label: scanPhase("walk"), unit: scanUnit, total: estimate,
estimate: true, stalledOn: "disk",
})
var (
changed, unhashed []fileRec
otherFSDirs []string
leftOut []string
)
for ev := range events {
if ev.otherFSDir != "" {
otherFSDirs = append(otherFSDirs, ev.otherFSDir)
leftOut = append(leftOut, ev.otherFSDir)
continue
}
// The line saying so is printed as a warning is, but what the
// walk leaves out is not counted as skipped. A default exclude
// that is itself a root is walked as one, so it gets no line.
if ev.leftOut != "" {
leftOut = append(leftOut, ev.leftOut)
if !slices.Contains(roots, ev.leftOut) {
prog.warnf("%s", ev.warn)
}
continue
}
@@ -509,12 +604,13 @@ func (s *scanState) walkPhase(
prog.finish()
for path, old := range s.existing {
// Kept only when a directory -x did not enter lies below the
// deepest root above the record. A root at or below that
// directory was walked on its own and did not find the file.
// Both contain the record, so the longer path is the deeper.
otherFSDir := deepestRoot(path, otherFSDirs)
if len(otherFSDir) > len(deepestRoot(path, roots)) {
// Kept only when a path the walk left out lies below the deepest
// root above the record, or is the record's own. A root at or
// below that path was walked on its own and did not find the
// file. Both contain the record, so the longer path is the
// deeper.
left := deepestRoot(path, leftOut)
if len(left) > len(deepestRoot(path, roots)) {
delete(s.existing, path)
s.sizes = append(s.sizes, old.size)
@@ -620,6 +716,15 @@ func sameInode(a, b fileRec) bool {
return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
}
// isDatabaseFile reports whether f, a file the walk reached, is one of
// the scan's own database files in rules, which the walk passes over
// without a line and without counting it.
func isDatabaseFile(f fileRec, rules walkRules) bool {
return slices.ContainsFunc(rules.databaseFiles, func(d fileRec) bool {
return sameInode(d, f)
})
}
// hashPhase hashes every queued file with hashSignature — the head and
// tail of a file of headTailMin or more, the whole file below that —
// committing completed records to the database in batches as results
@@ -636,10 +741,11 @@ func (s *scanState) hashPhase(ctx context.Context, workers int) error {
}
// readRuns reads runs with the worker pool, one read per inode run, in
// the order given, under a progress display named label. The workers
// compute each run's hashes with hash, and each result goes to record;
// a run that fails to read is warned about and counted as skipped
// instead. The total counts actual reads, so the bar shows a real ETA.
// the order given, under the progress display of the phase named label.
// The workers compute each run's hashes with hash, and each result goes
// to record; a run that fails to read is warned about and counted as
// skipped instead. The total counts actual reads, so the display shows
// a real ETA.
//
// Returning early — a failed database write, or a cancelled scan — must
// not strand the pool: the feeder would park forever on a full jobs
@@ -653,7 +759,12 @@ func (s *scanState) readRuns(ctx context.Context, workers int,
pool := startHashPool(ctx, runs, workers, hash)
defer pool.stop()
prog := newProgress(label, int64(len(runs)))
// Between results the phase may wait on a file being read or on a
// batch of records being written.
prog := newProgress(progressParams{
label: scanPhase(label), unit: scanUnit, total: int64(len(runs)),
stalledOn: "disk/db",
})
defer prog.finish()
for range runs {
@@ -740,7 +851,10 @@ func (s *scanState) updatePhase(ctx context.Context) error {
s.st.removed = len(deletes)
total := len(s.batch) + len(s.sentinels) + len(deletes)
prog := newProgress("update", int64(total))
prog := newProgress(progressParams{
label: scanPhase("update"), unit: scanUnit, total: int64(total),
stalledOn: "db",
})
defer prog.finish()
@@ -779,9 +893,12 @@ func (s *scanState) updatePhase(ctx context.Context) error {
// hash phase's worker pool and their records written back in batches. A
// failed read is warned about and counted as skipped; the record keeps
// its empty content, so it is never grouped, and a later scan tries
// again.
func (s *scanState) contentPhase(ctx context.Context, workers int) error {
toRead, recs, err := s.contentCandidates(ctx)
// again. No file on a filesystem in rules that is not persistent is
// read.
func (s *scanState) contentPhase(ctx context.Context, workers int,
rules walkRules,
) error {
toRead, recs, err := s.contentCandidates(ctx, rules)
if err != nil {
return err
}
@@ -802,35 +919,57 @@ func (s *scanState) contentPhase(ctx context.Context, workers int) error {
return err
}
return applyChanges(ctx, s.db, s.batch, nil, nil)
// The last partial batch is written under a display of its own, as
// the update phase writes the hash phase's.
prog := newProgress(progressParams{
label: scanPhase("content"), unit: scanUnit,
total: int64(len(s.batch)), stalledOn: "db",
})
defer prog.finish()
return applyChanges(ctx, s.db, s.batch, nil, prog)
}
// contentCandidates returns the files the content phase reads, and
// their records by path. Every record contentCandidatesSQL returns has
// its file checked with lstat, whether or not it already has a content
// hash: a file that is gone, is no longer a regular file, or has
// changed by the walk's rule keeps its record as it is and does not
// count as a match for the others, and any other lstat error is warned
// about and counted as skipped, with the same result. If such a record
// has no content hash, it stays out of duplicate groups; if it has one,
// it is still reported until a scan covering its own tree updates or
// removes it. The files of a group that pass and have no content hash
// are read only if at least minGroupSize of the group's files pass, so
// a group whose other members are all stale costs no reads. Only the
// records to be read are kept.
// hash: a file that is gone, is no longer a regular file, has changed
// by the walk's rule, or is on a filesystem in rules that is not
// persistent keeps its record as it is and does not count as a match
// for the others, and any other lstat error is warned about and counted
// as skipped, with the same result. If such a record has no content
// hash, it stays out of duplicate groups; if it has one, it is still
// reported until a scan covering its own tree updates or removes it.
// The files of a group that pass and have no content hash are read only
// if at least minGroupSize of the group's files pass, so a group whose
// other members are all stale costs no reads. Only the records to be
// read are kept.
func (s *scanState) contentCandidates(
ctx context.Context,
ctx context.Context, rules walkRules,
) ([]fileRec, map[string]scanRec, error) {
// The query and the checks take real time on a large database;
// The search and the checks take real time on a large database;
// without a display the scan looks hung before the reads begin.
prog := newProgress("content", -1)
// The display says it is searching until SQLite returns the first
// record. The checks start there, against a count of the records
// the search returns, and so do their rate and ETA. Between records
// they may wait on a file's lstat or on the search.
prog := newProgress(progressParams{
label: scanPhase("content"), unit: scanUnit, total: -1,
waitingFor: "searching the database", stalledOn: "disk/db",
})
defer prog.finish()
total, err := countContentCandidates(ctx, s.db)
if err != nil {
return nil, nil, err
}
var (
toRead []fileRec
first scanRec // the current group's first record
passed int // the current group's files that passed the check
unread []fileRec // those of them without a content hash
searching = true // no record has been returned yet
toRead []fileRec
first scanRec // the current group's first record
passed int // the current group's files that passed the check
unread []fileRec // those of them without a content hash
)
recs := make(map[string]scanRec)
@@ -849,7 +988,13 @@ func (s *scanState) contentCandidates(
passed, unread = 0, nil
}
err := loadContentCandidates(ctx, s.db, func(r scanRec, hashed bool) {
err = loadContentCandidates(ctx, s.db, func(r scanRec, hashed bool) {
if searching {
searching = false
prog.setTotal(total)
}
prog.increment()
if r.size != first.size || r.head != first.head || r.tail != first.tail {
@@ -858,7 +1003,7 @@ func (s *scanState) contentCandidates(
first = r
}
f, ok, err := unchangedFile(r)
f, ok, err := unchangedFile(r, rules)
if err != nil {
s.st.skipped++
@@ -880,6 +1025,11 @@ func (s *scanState) contentCandidates(
return nil, nil, err
}
// A search that returned no record still ends showing its count.
if searching {
prog.setTotal(total)
}
endGroup()
return toRead, recs, nil
@@ -887,9 +1037,10 @@ func (s *scanState) contentCandidates(
// unchangedFile lstats the file r names and returns it for reading if
// it is still the regular file r records: the same size, and an mtime
// no newer than recorded (the walk's change rule). A file that is gone
// or has changed reports false; any other lstat error is returned.
func unchangedFile(r scanRec) (fileRec, bool, error) {
// no newer than recorded (the walk's change rule). A file that is gone,
// has changed, or is on a filesystem in rules that is not persistent
// reports false; any other lstat error is returned.
func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
fi, err := os.Lstat(r.path)
if errors.Is(err, fs.ErrNotExist) {
return fileRec{}, false, nil
@@ -906,6 +1057,12 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
dev, ino := inodeOfInfo(fi)
// The scan reads nothing on a filesystem that is not persistent,
// whatever record an earlier scan left for the file.
if _, found := rules.nonPersistent[dev]; found {
return fileRec{}, false, nil
}
return fileRec{
path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
}, true, nil
@@ -972,32 +1129,36 @@ type dirJob struct {
}
// walkEvent is one walk result delivered to the walk phase: a regular
// file's statted record, a warning when fail is set, or, when
// otherFSDir is set, a directory -x did not enter because it is on
// another filesystem than its operand.
// file's statted record, a warning when fail is set, or a path the walk
// left out: when otherFSDir is set, a directory -x did not enter
// because it is on another filesystem than its operand, and when
// leftOut is set, a default exclude or a directory or file on a
// filesystem that is not persistent, with warn holding the line that
// says so.
type walkEvent struct {
rec fileRec
warn string
fail bool
otherFSDir string
leftOut string
}
// startWalk seeds every root into the shared walk worker pool and
// returns the event stream: one record per regular file, one warning
// event per per-path error, and one event per directory -x did not
// enter. The channel is closed when the walk completes, and also when
// ctx is cancelled — every goroutine in the pool abandons its blocking
// send in that case, so the consumer sees a truncated but properly
// terminated stream instead of a stalled one.
// event per per-path error, and one event per path -x or rules kept the
// walk from entering or recording. The channel is closed when the walk
// completes, and also when ctx is cancelled — every goroutine in the
// pool abandons its blocking send in that case, so the consumer sees a
// truncated but properly terminated stream instead of a stalled one.
func startWalk(ctx context.Context, roots []string, oneFS bool,
workers int,
rules walkRules, workers int,
) <-chan walkEvent {
jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS)
jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS, rules)
go func() {
initial := make([]dirJob, 0, len(roots))
for _, root := range roots {
initial = append(initial, seedRoot(ctx, root, events)...)
initial = append(initial, seedRoot(ctx, root, rules, events)...)
}
dispatchDirs(ctx, initial, jobs, subdirs)
@@ -1051,13 +1212,14 @@ func operandWarning(root string, fi fs.FileInfo) string {
}
// seedRoot turns one PATH operand into the walk's starting state: a
// regular-file operand is statted and emitted directly, and a directory
// regular-file operand is statted and emitted directly, unless it is
// one of the scan's own database files in rules, and a directory
// operand becomes an initial job. walkableRoots has already dropped
// every other operand. One that has changed into something else since
// is warned about and skipped here; it is still a root, so the records
// stored beneath it, other than those under another root, are deleted
// as unverified.
func seedRoot(ctx context.Context, root string,
func seedRoot(ctx context.Context, root string, rules walkRules,
events chan<- walkEvent,
) []dirJob {
fi, err := os.Lstat(root)
@@ -1084,14 +1246,17 @@ func seedRoot(ctx context.Context, root string,
}
dev, ino := inodeOfInfo(fi)
sendEvent(ctx, events, walkEvent{rec: fileRec{
rec := fileRec{
path: root,
size: fi.Size(),
mtime: fi.ModTime(),
dev: dev,
ino: ino,
}})
}
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
return nil
}
@@ -1103,7 +1268,7 @@ func seedRoot(ctx context.Context, root string,
// workers drop the directories still queued rather than stop reading
// jobs, so the range always runs out and the pool always tears down.
func startWalkWorkers(ctx context.Context, workers int,
oneFS bool,
oneFS bool, rules walkRules,
) (chan dirJob, chan []dirJob, chan walkEvent) {
jobs := make(chan dirJob, workQueueDepth)
subdirs := make(chan []dirJob, workers)
@@ -1119,7 +1284,7 @@ func startWalkWorkers(ctx context.Context, workers int,
}
select {
case subdirs <- walkOneDir(ctx, job, oneFS, events):
case subdirs <- walkOneDir(ctx, job, oneFS, rules, events):
case <-ctx.Done():
}
}
@@ -1177,7 +1342,7 @@ func dispatchDirs(ctx context.Context, initial []dirJob,
// entry and a warning event per unreadable one, and returns the
// subdirectories to descend into.
func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
events chan<- walkEvent,
rules walkRules, events chan<- walkEvent,
) []dirJob {
entries, err := os.ReadDir(job.path)
if err != nil {
@@ -1201,7 +1366,8 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
p := filepath.Join(job.path, e.Name())
if e.IsDir() {
if sub, ok := subdirJob(ctx, p, e, job, oneFS, events); ok {
sub, ok := subdirJob(ctx, p, e, job, oneFS, rules, events)
if ok {
subs = append(subs, sub)
}
@@ -1213,7 +1379,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
continue
}
emitFile(ctx, p, e, events)
emitFile(ctx, p, e, rules, events)
}
return subs
@@ -1223,9 +1389,12 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
// The lstat happens here in the walk worker, while the directory's
// metadata is still hot; a path that fails to stat (or stops being a
// regular file) between the directory read and the lstat is warned
// about and skipped.
// about and skipped. A file on a filesystem in rules that is not
// persistent, as through a bind mount of one file, is not recorded: it
// gets an event naming it instead. One of the scan's own database files
// in rules gets no event at all.
func emitFile(ctx context.Context, p string, e fs.DirEntry,
events chan<- walkEvent,
rules walkRules, events chan<- walkEvent,
) {
info, err := e.Info()
if err != nil {
@@ -1241,35 +1410,43 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
return
}
dev, ino := inodeOfInfo(info)
line := rules.nonPersistentLine(p, info)
if line != "" {
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
sendEvent(ctx, events, walkEvent{rec: fileRec{
return
}
dev, ino := inodeOfInfo(info)
rec := fileRec{
path: p,
size: info.Size(),
mtime: info.ModTime(),
dev: dev,
ino: ino,
}})
}
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
}
// subdirJob applies the descent rules to directory p: never enter
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
// snapshot), and with -x never enter a directory on a different
// filesystem than its operand, sending an event that names it so the
// scan keeps the records below it, except those below an operand at or
// snapshot), never enter a directory on a filesystem in rules that is
// not persistent, nor a default exclude in rules, and with -x never
// enter a directory on a different filesystem than its operand. For the
// last three it sends an event that names the directory, so the scan
// keeps the records below it, except those below an operand at or
// inside it.
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
parent dirJob, oneFS bool, events chan<- walkEvent,
parent dirJob, oneFS bool, rules walkRules,
events chan<- walkEvent,
) (dirJob, bool) {
if e.Name() == ".zfs" {
return dirJob{}, false
}
job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
if !oneFS || !parent.rootDevOK {
return job, true
}
info, err := e.Info()
if err != nil {
sendEvent(ctx, events, walkEvent{
@@ -1280,6 +1457,27 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
return dirJob{}, false
}
line := rules.nonPersistentLine(p, info)
if line != "" {
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
return dirJob{}, false
}
if isDefaultExclude(p, rules.excludes) {
sendEvent(ctx, events, walkEvent{
leftOut: p,
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
})
return dirJob{}, false
}
job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
if !oneFS || !parent.rootDevOK {
return job, true
}
if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
sendEvent(ctx, events, walkEvent{otherFSDir: p})
scan_test.go
+192 -12
View File
@@ -782,7 +782,8 @@ func collectWalk(t *testing.T, roots []string, oneFS bool,
errs int
)
for ev := range startWalk(t.Context(), roots, oneFS, workers) {
for ev := range startWalk(t.Context(), roots, oneFS, walkRules{},
workers) {
if ev.fail {
errs++
@@ -1004,7 +1005,8 @@ func callSubdirJob(t *testing.T, p string, e fs.DirEntry,
t.Helper()
events := make(chan walkEvent, 1)
job, ok := subdirJob(t.Context(), p, e, parent, oneFS, events)
job, ok := subdirJob(t.Context(), p, e, parent, oneFS, walkRules{},
events)
close(events)
var evs []walkEvent
@@ -1152,7 +1154,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
return dev, ok
}
roots = pruneRoots(roots, true, device)
roots = pruneRoots(roots, true, walkRules{}, device)
ctx := t.Context()
s := &scanState{db: db}
@@ -1162,14 +1164,14 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
t.Fatal(err)
}
jobs, subdirs, events := startWalkWorkers(ctx, 2, true)
jobs, subdirs, events := startWalkWorkers(ctx, 2, true, walkRules{})
// seedRoot emits a regular-file operand itself and returns a job
// only for a directory operand.
var initial []dirJob
for _, root := range roots {
for _, job := range seedRoot(ctx, root, events) {
for _, job := range seedRoot(ctx, root, walkRules{}, events) {
job.rootDev = devices[root]
initial = append(initial, job)
}
@@ -1463,7 +1465,7 @@ const smokeTreeFiles = 15
func syncTree(t *testing.T, db *sql.DB, roots ...string) scanStats {
t.Helper()
st, err := syncScan(t.Context(), db, roots, 4, false)
st, err := syncScan(t.Context(), db, roots, 4, false, walkRules{})
if err != nil {
t.Fatalf("syncScan: %v", err)
}
@@ -1478,7 +1480,7 @@ func dbRecords(t *testing.T, db *sql.DB) []scanRec {
var recs []scanRec
err := loadFileRows(t.Context(), db, func(r scanRec) {
err := loadFileRows(t.Context(), db, func(int64) {}, func(r scanRec) {
recs = append(recs, r)
})
if err != nil {
@@ -1550,7 +1552,7 @@ func assertSmokeTreeGroups(t *testing.T, dir string, db *sql.DB) {
super, dirs := dbTree(t, db)
tg := collectTreeGroups(dirs, super)
tg := collectTreeGroups(dirs, super, nil)
if len(tg) != 1 {
t.Fatalf("len(tree groups) = %d, want 1", len(tg))
}
@@ -1586,6 +1588,69 @@ func TestScanPipeline(t *testing.T) {
assertSmokeTreeGroups(t, dir, db)
}
// TestScanProgressNamesEachPhase checks the plain progress lines of a
// first scan, and of a rescan of one operand holding two of the
// database's records. Each phase's first line names it with its place
// among the five: load counts the records before reading them, the
// content phase searches the database, and the walk estimates its total
// from the records under its operand, or says its ETA is unknown when
// there are none.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanProgressNamesEachPhase(t *testing.T) {
stderr := captureStderr(t)
dir := buildSmokeTree(t)
db := openTestDB(t)
syncTree(t, db, dir)
first := stderr()
syncTree(t, db, filepath.Join(dir, "a"))
rescan := stderr()[len(first):]
load := "[1/5] load: counting the records, elapsed 0s, eta unknown"
content := "[5/5] content: searching the database, elapsed 0s, eta unknown"
cases := map[string][]string{
first: {
load, "[2/5] walk: 0 files, 0 files/s, elapsed 0s, eta unknown",
"[3/5] hash: [0/", "[4/5] update: [0/", content,
},
rescan: {
load, "[2/5] walk: [0/~2] 0%, 0 files/s, elapsed 0s, eta unknown",
"[3/5] hash: [0/", "[4/5] update: [0/", content,
},
}
for out, want := range cases {
var got []string
for line := range strings.Lines(out) {
label, _, _ := strings.Cut(line, ":")
if len(got) == 0 || !strings.HasPrefix(got[len(got)-1], label) {
got = append(got, line)
}
}
if len(got) != len(want) {
t.Fatalf("first lines of the phases = %q, want %q", got, want)
}
for i := range want {
if !strings.HasPrefix(got[i], want[i]) {
t.Errorf("phase %d starts %q, want %q", i+1, got[i], want[i])
}
}
}
// The load phase's total is the count of the records.
loaded := fmt.Sprintf("\n[1/5] load: [%d/%d] 100%%, ",
smokeTreeFiles, smokeTreeFiles)
if !strings.Contains(rescan, loaded) {
t.Errorf("rescan stderr = %q, want %q", rescan, loaded)
}
}
func TestSyncScanUnchangedReuse(t *testing.T) {
t.Parallel()
@@ -2029,7 +2094,7 @@ func TestTreesUnhashedNeverEqual(t *testing.T) {
for name, unknown := range cases {
super, dirs := treeOf(t, append(slices.Clone(shared), unknown...))
if tg := collectTreeGroups(dirs, super); len(tg) != 0 {
if tg := collectTreeGroups(dirs, super, nil); len(tg) != 0 {
t.Errorf("%s: tree groups = %d, want 0 (the files may differ)",
name, len(tg))
}
@@ -2349,7 +2414,7 @@ func TestPruneRoots(t *testing.T) {
// Duplicates and operands under other operands are dropped; /cc is
// not under /c (sibling with a shared prefix).
got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
false, device)
false, walkRules{}, device)
want := []string{"/a", "/c", "/cc"}
if !slices.Equal(got, want) {
@@ -2358,7 +2423,7 @@ func TestPruneRoots(t *testing.T) {
nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
got = pruneRoots(nested, false, device)
got = pruneRoots(nested, false, walkRules{}, device)
want = []string{"/p"}
if !slices.Equal(got, want) {
@@ -2366,12 +2431,31 @@ func TestPruneRoots(t *testing.T) {
}
// With -x, /p/q, on another filesystem than /p, is kept as well.
got = pruneRoots(nested, true, device)
got = pruneRoots(nested, true, walkRules{}, device)
want = []string{"/p", pqRoot}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots with -x = %q, want %q", got, want)
}
// The walk of /p does not enter /p/q when it is a default exclude, so
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
device)
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
}
// Nor does it enter /p/q when the filesystem of /p/q is not
// persistent, so /p/q/r/m, of no known device, is kept.
got = pruneRoots([]string{"/p", "/p/q/r/m"}, false,
walkRules{nonPersistent: map[uint64]string{2: "tmpfs"}}, device)
want = []string{"/p", "/p/q/r/m"}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots with %s not persistent = %q, want %q",
pqRoot, got, want)
}
}
func TestReportsNeverTouchFilesystem(t *testing.T) {
@@ -2417,3 +2501,99 @@ func TestUnderRoot(t *testing.T) {
}
}
}
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
// database, reached directly and through a symlinked directory. The
// walk passes over the database file, its -wal and -shm files and its
// .lock file with no line and no count, and the record an earlier scan
// left for the .lock file is deleted. A rescan of the unchanged tree
// updates nothing, and report lists none of the database's files.
func TestRunScanPassesOverDatabase(t *testing.T) {
for _, viaSymlink := range []bool{false, true} {
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
base := t.TempDir()
tree := filepath.Join(base, "real", "tree")
operand := tree
writeFile(t, tree, "a", pattern(1, 100))
writeFile(t, tree, "b", pattern(1, 100))
writeFile(t, tree, "empty1", nil)
writeFile(t, tree, "empty2", nil)
if viaSymlink {
err := os.Symlink(filepath.Join(base, "real"),
filepath.Join(base, "link"))
if err != nil {
t.Fatal(err)
}
operand = filepath.Join(base, "link", "tree")
}
dbPath := filepath.Join(tree, "db.sqlite")
t.Setenv(databaseEnv, dbPath)
db, err := openScanDatabase(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
err = applyChanges(t.Context(), db, []scanRec{
{path: filepath.Join(operand, "db.sqlite.lock")},
}, nil, nil)
if err != nil {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, dbPath)
assertScanSummary(t,
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
operand)
assertScanSummary(t,
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
operand)
report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") ||
!strings.Contains(report, filepath.Join(operand, "empty2")) {
t.Errorf("report = %q, want the empty files and none of "+
"the database's files", report)
}
})
}
}
// TestRunScanPassesOverDatabaseOperands scans the database file and its
// .lock file given as operands themselves: neither is recorded or
// counted.
func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
dbPath := testDBPath(t)
t.Setenv(databaseEnv, dbPath)
// The first scan creates both files.
scanOK(t, t.TempDir())
assertScanSummary(t,
"0 files seen (0 added, 0 updated, 0 removed, 0 unchanged)",
dbPath, dbPath+".lock")
}
// assertScanSummary scans operands and fails the test unless the
// summary line gives counts with nothing skipped, and the line naming
// the database is the only one that names one of its files.
func assertScanSummary(t *testing.T, counts string, operands ...string) {
t.Helper()
stderr := scanOK(t, operands...)
want := "scan: " + counts + ", 0 skipped\n"
if !strings.Contains(stderr, want) {
t.Errorf("scan %q: stderr = %q, want %q", operands, stderr, want)
}
if strings.Count(stderr, "db.sqlite") != 1 {
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
"database to name one of its files", operands, stderr)
}
}
trees.go
+108 -25
View File
@@ -1,9 +1,9 @@
package main
import (
"bufio"
"context"
"crypto/sha256"
"database/sql"
"fmt"
"io"
"os"
@@ -24,6 +24,13 @@ type treeNode struct {
totalSize int64
}
// treesPhase names one phase of trees with its place among all of them,
// as "[2/4] read", for the phase's progress display. The list here is
// the one place the phases are counted.
func treesPhase(name string) string {
return phaseLabel([]string{openPhase, "read", "group", "write"}, name)
}
// runTrees implements the trees subcommand: it reads every record from
// the database in path order, reconstructs the directory hierarchy from
// the record paths, computes a Merkle-style digest per directory, and
@@ -31,6 +38,13 @@ type treeNode struct {
// touches the scanned filesystem; its only I/O is the database, stdout,
// and stderr. It names the database on stderr before it opens it. Any
// database problem, including a missing database, is fatal.
//
// The work runs in four phases, each under a progress display of its
// own on stderr: open (open the database), read (count the records,
// then read them into the hierarchy), group (group the directories by
// digest) and write (write the groups to stdout). Each display ends
// before the next phase starts, and before an error is returned to be
// printed.
func runTrees(ctx context.Context, stdout io.Writer) error {
dbPath, source, err := databasePath()
if err != nil {
@@ -40,34 +54,102 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
fmt.Fprintf(os.Stderr, "trees: database %s (%s)\n",
escapePath(dbPath), source)
// Opening can wait on a busy database for as long as the busy
// timeout.
prog := newProgress(progressParams{
label: treesPhase(openPhase), total: -1,
waitingFor: openingDatabase,
})
db, err := openReportDatabase(ctx, dbPath)
prog.finish()
if err != nil {
return err
}
defer func() { _ = db.Close() }()
records := 0
tree := newTreeBuilder()
err = loadFileRows(ctx, db, func(r scanRec) {
records++
tree.add(r)
prog = newProgress(progressParams{
label: treesPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords, stalledOn: "db",
})
records, super, allDirs, err := readTree(ctx, db, prog)
prog.finish()
if err != nil {
return fmt.Errorf("database %s: %w", dbPath, err)
}
// Grouping works in memory, so it waits on nothing but the
// processor.
prog = newProgress(progressParams{
label: treesPhase("group"), unit: "directories",
total: int64(len(allDirs)), stalledOn: "cpu",
})
dupes := collectTreeGroups(allDirs, super, prog)
prog.finish()
prog = newProgress(progressParams{
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
stalledOn: "out",
})
dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
prog.finish()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr,
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
"%s reclaimable\n",
records, len(dupes), dupeTrees, humanBytes(reclaimable))
return nil
}
// readTree gives prog the number of records in db as its total, then
// reads every record in path order into the directory hierarchy,
// counting each on prog. It returns the number of records read, the
// super-root and every directory.
func readTree(ctx context.Context, db *sql.DB,
prog *progress,
) (int, *treeNode, []*treeNode, error) {
records := 0
tree := newTreeBuilder()
err := loadFileRows(ctx, db, prog.setTotal, func(r scanRec) {
records++
prog.increment()
tree.add(r)
})
if err != nil {
return 0, nil, nil, err
}
super, allDirs := tree.finish()
dupes := collectTreeGroups(allDirs, super)
return records, super, allDirs, nil
}
out := bufio.NewWriterSize(stdout, ioBufSize)
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
// a row for every member of each group but its first, counting each
// group on prog once its rows are written. stdout may be the terminal
// prog is drawn on, so the lines go through prog's writer. It returns
// the number of rows (dupe trees) and the bytes they could reclaim.
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
prog *progress,
) (int, int64, error) {
out := prog.writer(stdout)
_, err = fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
if err != nil {
return fmt.Errorf("write stdout: %w", err)
return 0, 0, fmt.Errorf("write stdout: %w", err)
}
dupeTrees := 0
@@ -81,25 +163,22 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
escapePath(first.path), escapePath(n.path),
first.fileCount, first.totalSize)
if err != nil {
return fmt.Errorf("write stdout: %w", err)
return 0, 0, fmt.Errorf("write stdout: %w", err)
}
dupeTrees++
reclaimable += first.totalSize
}
prog.increment()
}
err = out.Flush()
if err != nil {
return fmt.Errorf("write stdout: %w", err)
return 0, 0, fmt.Errorf("write stdout: %w", err)
}
fmt.Fprintf(os.Stderr,
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
"%s reclaimable\n",
records, len(dupes), dupeTrees, humanBytes(reclaimable))
return nil
return dupeTrees, reclaimable, nil
}
// treeBuilder reconstructs the directory hierarchy from records added
@@ -221,14 +300,18 @@ func fileEntry(name string, r scanRec) string {
"\x00" + r.head + "\x00" + r.tail + "\x00" + content
}
// collectTreeGroups groups directories by digest and returns every
// maximal group with two or more members, each group's members sorted
// by path, groups ordered by tree size descending then by first path
// ascending.
func collectTreeGroups(allDirs []*treeNode, super *treeNode) [][]*treeNode {
// collectTreeGroups groups directories by digest, counting each one on
// prog as it is grouped, and returns every maximal group with two or
// more members, each group's members sorted by path, groups ordered by
// tree size descending then by first path ascending.
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
prog *progress,
) [][]*treeNode {
groups := make(map[[sha256.Size]byte][]*treeNode)
for _, d := range allDirs {
groups[d.digest] = append(groups[d.digest], d)
prog.increment()
}
var dupes [][]*treeNode
trees_test.go
+107 -7
View File
@@ -3,8 +3,11 @@ package main
import (
"bytes"
"database/sql"
"os"
"slices"
"strings"
"testing"
"time"
)
// Signature hashes shared by the smoke-test records.
@@ -17,6 +20,11 @@ const (
f2Content = "f2c"
)
// treesOpenLine is the plain line of trees' open phase when it opens
// the database at once; it prints it as the phase starts and as it ends.
const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
"eta unknown"
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
// /d/t2 are identical, /d/t3 differs from them only by one filename.
func smokeTreeRecs() []scanRec {
@@ -38,7 +46,7 @@ func dbTree(t *testing.T, db *sql.DB) (*treeNode, []*treeNode) {
tree := newTreeBuilder()
err := loadFileRows(t.Context(), db, tree.add)
err := loadFileRows(t.Context(), db, func(int64) {}, tree.add)
if err != nil {
t.Fatal(err)
}
@@ -170,7 +178,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
t.Fatalf("directory paths = %q, want %q", got, want)
}
groups := collectTreeGroups(dirs, super)
groups := collectTreeGroups(dirs, super, nil)
gotGroups := groupPaths(groups)
wantGroups := [][]string{{"/a", "/c"}}
@@ -203,6 +211,98 @@ func TestRunTreesEscapesPaths(t *testing.T) {
}
}
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
// records with stderr not a terminal, and checks every line it prints
// there. After the line naming the database, each phase prints a line
// naming it with its place among the four as it starts and another as
// it ends. read counts the 6 records before it reads them, group counts
// the 8 directories, and write the 1 group. The summary comes last, and
// stdout holds the report alone.
func TestRunTreesProgressNamesEachPhase(t *testing.T) {
path := seedDatabase(t, smokeTreeRecs())
t.Setenv(databaseEnv, path)
stderr := captureStderr(t)
var stdout bytes.Buffer
code := run([]string{cmdTrees}, &stdout, os.Stderr)
if code != exitOK {
t.Fatalf("run(trees) = %d, want %d; stderr: %s",
code, exitOK, stderr())
}
report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
if got := stdout.String(); got != report {
t.Errorf("stdout = %q, want %q", got, report)
}
// How each line starts; the rate at the end of a phase varies.
want := []string{
"trees: database " + path + " (from " + databaseEnv + ")",
treesOpenLine,
treesOpenLine,
"[2/4] read: counting the records, elapsed 0s, eta unknown",
"[2/4] read: [6/6] 100%, ",
"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown",
"[3/4] group: [8/8] 100%, ",
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown",
"[4/4] write: [1/1] 100%, ",
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
}
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
if len(lines) != len(want) {
t.Fatalf("stderr lines = %q, want %d lines starting %q",
lines, len(want), want)
}
for i := range want {
if !strings.HasPrefix(lines[i], want[i]) {
t.Errorf("stderr line %d = %q, want it to start %q",
i+1, lines[i], want[i])
}
}
}
// TestWriteTreeGroupsOnTerminal writes the smoke-test report to the
// terminal its write phase's display is drawn on, as a plain "sfdupes
// trees" does: the terminal must show the header and the row whole,
// each on a line of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestWriteTreeGroupsOnTerminal(t *testing.T) {
stderr := captureStderr(t)
super, dirs := treeOf(t, smokeTreeRecs())
dupes := collectTreeGroups(dirs, super, nil)
// No timed draws: the display is drawn as it starts and as it ends.
p := startProgress(progressParams{
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
}, terminalWidth, time.Hour, time.Now)
_, _, err := writeTreeGroups(os.Stderr, dupes, p)
p.finish()
if err != nil {
t.Fatal(err)
}
// A terminal shows each line as the text after its last carriage
// return.
var shown []string
for line := range strings.SplitSeq(stderr(), "\n") {
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
}
want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
t.Errorf("terminal shows lines %q, want %q first", shown, want)
}
}
func TestTreeDigests(t *testing.T) {
t.Parallel()
@@ -254,7 +354,7 @@ func TestCollectTreeGroupsMaximal(t *testing.T) {
super, dirs := treeOf(t, smokeTreeRecs())
groups := collectTreeGroups(dirs, super)
groups := collectTreeGroups(dirs, super, nil)
want := [][]string{{"/d/t1", "/d/t2"}}
if got := groupPaths(groups); !slices.EqualFunc(got, want,
@@ -278,14 +378,14 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
super, dirs := treeOf(t, recs)
forward := groupPaths(collectTreeGroups(dirs, super))
forward := groupPaths(collectTreeGroups(dirs, super, nil))
reversed := slices.Clone(recs)
slices.Reverse(reversed)
superR, dirsR := treeOf(t, reversed)
backward := groupPaths(collectTreeGroups(dirsR, superR))
backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
if !slices.EqualFunc(forward, backward, slices.Equal) {
t.Fatalf("output depends on record order: %v vs %v",
forward, backward)
@@ -304,7 +404,7 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super))
got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/x1", "/p/x2"}}
if !slices.EqualFunc(got, want, slices.Equal) {
@@ -326,7 +426,7 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super))
got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/a/x", "/q/b/x"}}
if !slices.EqualFunc(got, want, slices.Equal) {