Compare commits

..
Author SHA1 Message Date
sneak b7e6ff9df1 scan: drop the stray [flags] from the usage line (closes #169)
check / check (push) Successful in 2m41s
The scan Use string already names every flag, so cobra no longer appends
its generic [flags]. A test checks the usage line in the help and in the
usage error against the one README.md shows.

Model: opus-5-5
2026-11-11 03:55:16 +00:00
16 changed files with 1018 additions and 1386 deletions

No files matched your search

+60 -122
View File
@@ -21,7 +21,7 @@ file takes too long. It has five subcommands. `scan` walks the filesystem and
records the files it finds in an SQLite database. SQLite keeps a whole database records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` work from the database and system's job scheduler. `report`, `trees` and `info` read the database alone and
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
first look at a new machine with one command. first look at a new machine with one command.
@@ -98,15 +98,13 @@ Each `-v` flag below makes a bind mount, which shows a host directory at a path
inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between
runs. Mount the tree to scan at the same path inside the container as on the runs. Mount the tree to scan at the same path inside the container as on the
host. Read-only access is enough. The database then records the host's paths, host. Read-only access is enough. The database then records the host's paths,
and so do the reports. `report` and `trees` need that mount too. Without it, and so do the reports.
every recorded path counts as gone, and each report holds only its header line.
```sh ```sh
make docker make docker
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \ docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
sfdupes scan /srv sfdupes scan /srv
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
sfdupes report > dupes.tsv
``` ```
### Daily scan from `cron` ### Daily scan from `cron`
@@ -161,10 +159,8 @@ to keep is your choice.
A row is a candidate, not proof: A row is a candidate, not proof:
- The reports show each file as it was when a scan last recorded it. They check - The reports show each file as it was when a scan last recorded it. A file may
only that each copy still exists: the file, or the directory at the top of a have changed or gone since.
tree. A file may have changed since, and a file inside a tree may be gone. Run
`scan` again to bring the reports up to date.
- A file of 100 MiB or more is compared only on samples of its content (see - A file of 100 MiB or more is compared only on samples of its content (see
"Duplicate detection"). "Duplicate detection").
- Two paths that are hard links, two names for one file, are listed as - Two paths that are hard links, two names for one file, are listed as
@@ -212,9 +208,8 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
of reads, at the price of a comparison that is not exact. of reads, at the price of a comparison that is not exact.
The hashes live in a database that is kept between runs. A rescan reads only new The hashes live in a database that is kept between runs. A rescan reads only new
files, and files whose size or modification time (mtime) differs from their files, and files whose size changed or whose modification time (mtime) is newer
record. All analysis then works from the database. The reports only check that than recorded. All analysis then works from the database alone.
each copy they name still exists.
The goal is whole duplicated trees, not only single files. A download extracted The goal is whole duplicated trees, not only single files. A download extracted
twice, or a project copied once too often, is such a tree. It can be removed as twice, or a project copied once too often, is such a tree. It can be removed as
@@ -240,10 +235,9 @@ In order:
the files of the directories it is reading. the files of the directories it is reading.
3. **Rescan cheaply, analyze from the database.** The scan keeps its database 3. **Rescan cheaply, analyze from the database.** The scan keeps its database
between runs, and an unchanged file is never read again. `report`, `trees` between runs, and an unchanged file is never read again. `report`, `trees`
and `info` work from the database and never read a scanned file. `report` and and `info` work from the database alone and never touch the scanned
`trees` check only that each copy they name still exists, and leave out a filesystem. `scan` is built to run from `cron`. The reports run at any time,
copy that is gone. `scan` is built to run from `cron`. The reports run at any against what the database holds.
time, against what the database holds.
4. **Keep the streams apart.** Everything on stdout is machine-readable data. 4. **Keep the streams apart.** Everything on stdout is machine-readable data.
Everything for a human reader, such as progress and warnings, goes to stderr. Everything for a human reader, such as progress and warnings, goes to stderr.
@@ -254,14 +248,15 @@ In order:
- `report` prints the file-level duplicate report from the database. - `report` prints the file-level duplicate report from the database.
- `trees` prints the duplicate-tree report from the database. - `trees` prints the duplicate-tree report from the database.
- `info` prints the database's details. - `info` prints the database's details.
- `dwim` scans `/`, then prints the duplicate-tree report. - `dwim` scans `/` with the default excludes, then prints the duplicate-tree
report.
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv sfdupes trees > dupetrees.tsv
sfdupes info sfdupes info
sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... sfdupes dwim
sfdupes --version sfdupes --version
sfdupes [command] --help sfdupes [command] --help
``` ```
@@ -291,34 +286,22 @@ The reference below is also the program's specification.
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
it. The path names the file exactly, whatever characters it holds. A relative it. The path names the file exactly, whatever characters it holds. A relative
path is relative to the working directory. path is relative to the working directory.
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once - `--workers N` (`scan`) sets how many directories `scan` reads at once while it
while it lists the trees. It also sets how many files `scan` reads at once lists the trees. It also sets how many files `scan` reads at once while it
while it hashes them. The default is the number of CPUs. `N` must be at hashes them. The default is the number of CPUs. `N` must be at least 1. A
least 1. A smaller value is a usage error, reported in one line on stderr with smaller value is a usage error, reported in one line on stderr with exit 2
exit 2 before anything is scanned. before anything is scanned.
- `-x`, `--one-file-system` (`scan`, `dwim`) keeps the walk of each operand out - `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
of directories on other filesystems, as the same flag does for `du` and directories on other filesystems, as the same flag does for `du` and `rsync`.
`rsync`. By default the walk also enters directories on other filesystems. By default the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default - `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
excludes. Those are the system and temporary directories the walk otherwise Those are the system and temporary directories the walk otherwise leaves out,
leaves out, such as `/var/tmp` on Linux (see "Rules for the walk"). such as `/var/tmp` on Linux (see "Rules for the walk").
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
below it. `PATH` may be a directory or a file. The flag may be given more than
once. The walk knows `PATH` by its device ID and inode number, as it knows the
database files (see "Rules for the walk"). So the walk leaves `PATH` out
whatever path leads to it. That includes a hard link to a file given as
`PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in
`--exclude ""`, is a usage error, reported in one line on stderr with exit 2
before anything is scanned. A `PATH` that does not exist, or whose real path
`scan` cannot find, is a fatal error. So is an operand at or below one (see
"Operands").
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and - `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
exits 0. It writes nothing to stderr. exits 0. It writes nothing to stderr.
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr - `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan` and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the
names the database. The help of `report`, `trees`, `info` and `dwim` says what default excludes of the system it runs on, and the filesystem types the walk
each prints. `sfdupes scan --help` and `sfdupes dwim --help` also list the
default excludes of the system they run on, and the filesystem types the walk
never records anything on. never records anything on.
### Duplicate detection ### Duplicate detection
@@ -372,10 +355,6 @@ default. An operand may be a directory or a regular file.
- An operand that does not exist is a fatal error, and so is a symlink whose - An operand that does not exist is a fatal error, and so is a symlink whose
target does not exist. `scan` then stops before it walks anything, and exits target does not exist. `scan` then stops before it walks anything, and exits
with code 1. with code 1.
- An operand at or below an `--exclude` path is a fatal error too. The message
names the operand and the `--exclude` path. `scan` compares the operand and
each directory above it with the `--exclude` paths by device ID and inode
number, not by name.
- Each operand is resolved to its real path before the walk, as `realpath` - Each operand is resolved to its real path before the walk, as `realpath`
prints it. A relative operand is joined to the working directory as `pwd -P` prints it. A relative operand is joined to the working directory as `pwd -P`
prints it. Every symlink in the path is then followed. A `..` after a symlink prints it. Every symlink in the path is then followed. A `..` after a symlink
@@ -431,16 +410,17 @@ operands:
- A file not yet in the database is hashed and inserted. - A file not yet in the database is hashed and inserted.
- A file already in the database is not read again when its size and mtime match - A file already in the database is not read again when its size and mtime match
its record. This is what makes a daily rescan cheap. its record. The mtime matches when it is not newer than the recorded one. This
is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan` - When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks up to seconds after unchanged. ZFS, for example, counts a new file's blocks up to seconds after
the write. A block is the unit in which a filesystem allocates disk space. A the write. A block is the unit in which a filesystem allocates disk space. A
file scanned within those seconds gets its real `disk_size` at the next scan. file scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size or its mtime differs from its record. - `scan` hashes a file again when its size differs or its mtime is newer. The
An older mtime counts as much as a newer one. A file put back by `cp -p`, mtime is compared to the nanosecond, as finely as the filesystem records it. A
`rsync -a` or a restore from backup often has an older mtime. The mtime is same-size rewrite counts as a change whenever the filesystem gives it a later
compared to the nanosecond, as finely as the filesystem records it. mtime, even within the same second.
- The scan deletes each record under a scanned operand whose file it could not - The scan deletes each record under a scanned operand whose file it could not
verify. That covers deleted files, and paths the scan could not stat or read verify. That covers deleted files, and paths the scan could not stat or read
this run. To stat a file is to read its details, such as its size, without this run. To stat a file is to read its details, such as its size, without
@@ -451,16 +431,15 @@ operands:
scanned on different schedules into one database. scanned on different schedules into one database.
- The scan keeps every record at or below a path that the walk leaves out for - The scan keeps every record at or below a path that the walk leaves out for
one of these reasons: one of these reasons:
- it is an `--exclude` path;
- it is a default exclude; - it is a default exclude;
- it is not on a persistent filesystem; - it is not on a persistent filesystem;
- with `-x`, it is a directory on another filesystem than its operand. - with `-x`, it is a directory on another filesystem than its operand.
Such records count as outside the scanned operands. So with `-x`, a Such records count as outside the scanned operands. So with `-x`, a
filesystem mounted under an operand can be scanned on its own schedule into filesystem mounted under an operand can be scanned on its own schedule into
the same database. An operand may itself lie at or inside such a path, other the same database. An operand may itself lie at or inside such a path, as
than an `--exclude` path (see "Operands"). Its walk verifies the records "Operands" describes. Its walk verifies the records below it, so they are
below it, so they are deleted when their files are gone. deleted when their files are gone.
#### Phases #### Phases
@@ -559,17 +538,6 @@ walk practical on large or busy storage pools.
The path is not counted as skipped. The records below it are kept, as The path is not counted as skipped. The records below it are kept, as
described under "What a scan does to the database". described under "What a scan does to the database".
- The walk never enters or records an `--exclude` path, nor anything below it.
For each one it reaches, `scan` prints a line on stderr naming it:
```
scan: skipped /srv/cache (--exclude)
```
The path is not counted as skipped. The scan keeps the records at and below
it, as described under "What a scan does to the database".
`--no-default-excludes` does not change this rule.
- Below an operand, the walk does not enter a default exclude. On Linux, the - Below an operand, the walk does not enter a default exclude. On Linux, the
default excludes hold temporary files and the kernel's view of the running default excludes hold temporary files and the kernel's view of the running
system. They depend on the system, and `sfdupes scan --help` lists those of system. They depend on the system, and `sfdupes scan --help` lists those of
@@ -644,8 +612,7 @@ from a thread of its own and never blocks the workers.
it repeats another or lies under another; it repeats another or lies under another;
- the progress display of each phase (see "Progress"). `scan` prints each - the progress display of each phase (see "Progress"). `scan` prints each
warning for a skipped path as it happens. It prints a line the same way for warning for a skipped path as it happens. It prints a line the same way for
each path left out as an `--exclude` path, as a default exclude or as not on a each path left out as a default exclude or as not on a persistent filesystem;
persistent filesystem;
- the summary line; - the summary line;
- a warning when a report still has the database open as the scan ends (see - a warning when a report still has the database open as the scan ends (see
"Concurrent reads"). "Concurrent reads").
@@ -688,29 +655,21 @@ and `scan` leaves it ignored.
`report` reads every record from the database. It takes no positional arguments. `report` reads every record from the database. It takes no positional arguments.
A positional argument is anything on the command line that is not a flag, such A positional argument is anything on the command line that is not a flag, such
as the paths `scan` takes. `report` never opens a scanned file. It checks that as the paths `scan` takes. `report` never touches the scanned filesystem. It
each path in a group still exists, and looks at no other path. The check is neither stats nor opens any path in the records. It reads the database and
`lstat`, the system call that looks up a path without following a symlink: writes to stdout and stderr, and nothing else. Its output is the same whether or
not the scanned filesystem is still mounted.
- A path is gone when `lstat` finds that it, or a directory above it, does not
exist.
- Any other error, such as a permission error, counts as present. So a user who
may not look inside a directory still sees its copies.
The check does not notice a file that changed after the scan. Run `scan` again
to bring the report up to date.
Processing: Processing:
- Records are grouped by signature, the four values - Records are grouped by signature, the four values
`(size, head, tail, content)`. `(size, head, tail, content)`.
- In a group with two or more paths, a path that is gone is left out. A group - Every group with two or more paths is a duplicate group.
that still has two or more paths is a duplicate group.
- Empty files form no group, because removing one frees nothing. - Empty files form no group, because removing one frees nothing.
- Within a group, paths are sorted in byte order. The first is `first`, and - Within a group, paths are sorted in byte order. The first is `first`, and
every other path is a `dupe`. every other path is a `dupe`.
- Groups are ordered by size, largest first, then by `first` path. The output is - Groups are ordered by size, largest first, then by `first` path. The output is
the same for the same database contents and the same paths on disk. the same for the same database contents.
`report` has two phases, each with its own progress display on stderr (see `report` has two phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `read` gets the number of records from "Progress"). `open` opens the database. `read` gets the number of records from
@@ -763,9 +722,8 @@ report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
`trees` reads the same database as `report` and takes no positional arguments. `trees` reads the same database as `report` and takes no positional arguments.
It reports whole duplicate directory trees. Two directories are duplicate trees It reports whole duplicate directory trees. Two directories are duplicate trees
when they hold the same relative paths, and each path holds a file with the same when they hold the same relative paths, and each path holds a file with the same
signature in both. It rebuilds the directory tree from the paths in the records, signature in both. Like `report`, it never touches the scanned filesystem. It
split on `/`. Like `report`, it never opens a scanned file. It checks only that rebuilds the directory tree from the paths in the records, split on `/`.
the directory at the top of each copy still exists, as `report` checks a path.
Definitions: Definitions:
@@ -792,13 +750,10 @@ Processing:
directories that all have the same digest. The parents' group, or an directories that all have the same digest. The parents' group, or an
ancestor's, already implies it. A group is always reported when it holds ancestor's, already implies it. A group is always reported when it holds
sibling directories, or when its members' parents have different digests. sibling directories, or when its members' parents have different digests.
- A directory that is gone is dropped from its group. A group left with fewer
than two directories is not reported.
- Within a group, paths are sorted in byte order. The first is `first`, and - Within a group, paths are sorted in byte order. The first is `first`, and
every other path is a `dupe`. every other path is a `dupe`.
- Groups are ordered by total tree size, largest first, then by `first` path. - Groups are ordered by total tree size, largest first, then by `first` path.
The output is the same for the same database contents and the same paths on The output is the same for the same database contents.
disk.
Two limitations are accepted: Two limitations are accepted:
@@ -812,9 +767,8 @@ Two limitations are accepted:
`trees` has four phases, each with its own progress display on stderr (see `trees` has four phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `read` gets the number of records from "Progress"). `open` opens the database. `read` gets the number of records from
the `totals` table, then reads them in path order. It builds the directory tree the `totals` table, then reads them in path order. It builds the directory tree
and computes the digests as it goes. `group` groups the directories by digest, and computes the digests as it goes. `group` groups the directories by digest
picks the groups to report and checks that their directories still exist. and picks the groups to report. `write` writes them to stdout.
`write` writes the groups to stdout.
#### Trees output format #### Trees output format
@@ -857,25 +811,14 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
### `dwim` mode ### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its `dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments. Like `scan`, duplicate trees in one command. It takes no positional arguments and no flags of
it should run as root, so that it can read every file. its own. Like `scan`, it should run as root, so that it can read every file.
`dwim` takes these flags of `scan`, with the same meaning (see "Settings"): `dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
excludes and crosses into other filesystems. `--workers` keeps its default. The
- `--workers N`;
- `-x`, `--one-file-system`;
- `--no-default-excludes`;
- `--exclude PATH`, which may be given more than once.
`dwim` runs what `sfdupes scan /` runs with the same flags. With no flags, the
walk skips the default excludes and crosses into other filesystems. The
database, its creation and the scan lock are those of `scan`. Then `dwim` runs database, its creation and the scan lock are those of `scan`. Then `dwim` runs
what `sfdupes trees` runs. The largest trees come first. what `sfdupes trees` runs. The largest trees come first.
```
sfdupes dwim -x --exclude /srv/cache
```
The report goes to stdout, exactly as `trees` prints it. On stderr comes what The report goes to stdout, exactly as `trees` prints it. On stderr comes what
`scan` prints, then what `trees` prints, then one last line: `scan` prints, then what `trees` prints, then one last line:
@@ -883,8 +826,8 @@ The report goes to stdout, exactly as `trees` prints it. On stderr comes what
dwim: run "sfdupes trees" to print this report again dwim: run "sfdupes trees" to print this report again
``` ```
The database keeps the scan, so `sfdupes trees` prints the report again later, The database keeps the scan, so `sfdupes trees` prints the same report later,
even after the terminal is gone. It then leaves out a copy deleted in between. even after the terminal is gone.
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan"). `SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
`dwim` then prints no report and exits 1. A fatal scan error also ends it before `dwim` then prints no report and exits 1. A fatal scan error also ends it before
@@ -1176,10 +1119,9 @@ in, and a phase that may wait in two places names both, as `disk/db`:
- `hash` says `disk/db`, because it reads files and writes their records to the - `hash` says `disk/db`, because it reads files and writes their records to the
database between them. database between them.
- `load`, `update` and the `read` phase of `trees` say `db`. - `load`, `update` and the `read` phase of `trees` say `db`.
- The `read` phase of `report` says `db/disk/out`. It checks that the paths of - The `read` phase of `report` says `db/out`, because it writes the report to
each group still exist, and writes the report to stdout, while it reads. stdout while it reads.
- `group` of `trees` says `cpu/disk`, for work in memory and the check that each - `group` of `trees` says `cpu`, for work in memory, and its `write` says `out`.
copy still exists. Its `write` says `out`.
- A phase still waiting for its total says what it waits for, and nothing more. - A phase still waiting for its total says what it waits for, and nothing more.
On a terminal: On a terminal:
@@ -1237,9 +1179,6 @@ showing the count it stopped at.
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim` - `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
stopped by either during its scan. A fatal error is one of these: stopped by either during its scan. A fatal error is one of these:
- an operand that does not exist, or whose real path `scan` cannot find; - an operand that does not exist, or whose real path `scan` cannot find;
- an `--exclude` path that does not exist, or whose real path `scan` cannot
find;
- an operand at or below an `--exclude` path;
- a list of mounted filesystems that `scan` cannot read; - a list of mounted filesystems that `scan` cannot read;
- another `scan` holding the lock (see "The scan lock"); - another `scan` holding the lock (see "The scan lock");
- a database, its directory or its lock file that cannot be created or - a database, its directory or its lock file that cannot be created or
@@ -1259,7 +1198,6 @@ showing the count it stopped at.
`--workers abc`; `--workers abc`;
- `scan` with no operand; - `scan` with no operand;
- `--workers` below 1; - `--workers` below 1;
- an empty `--exclude` value, as in `--exclude ""`;
- `report`, `trees`, `info` or `dwim` with a positional argument. - `report`, `trees`, `info` or `dwim` with a positional argument.
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
@@ -1423,8 +1361,8 @@ commands of their own:
not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
exception, because the race detector needs cgo (see `script/test`). exception, because the race detector needs cgo (see `script/test`).
- `report` and `trees` give the same output for the same database contents and - `report` and `trees` give the same output for the same database contents,
the same paths on disk, whatever order the records were inserted in. whatever order the records were inserted in.
### Definition of done ### Definition of done
@@ -1489,7 +1427,7 @@ All of the following, run in this directory, must pass:
The test suite automates this scenario (see The test suite automates this scenario (see
`internal/sfdupes/scan_test.go`). It also checks that `report` and `trees` `internal/sfdupes/scan_test.go`). It also checks that `report` and `trees`
leave out a copy removed after the scan. never touch the scanned filesystem.
## TODO ## TODO
+749 -2
View File
@@ -4,7 +4,7 @@
the tracker yet is filed as an issue first. the tracker yet is filed as an issue first.
- Branch from `next`. - Branch from `next`.
- Do the work, with tests, in small focused commits. - Do the work, with tests, in small focused commits.
- Delete Next Step from `TODO.md`, in the same commit as the work. - Record the step at the top of Completed Steps, in the same commit as the work.
- Push the branch and open a PR against `next`. Its title ends with - Push the branch and open a PR against `next`. Its title ends with
` (closes #N)`. ` (closes #N)`.
- An independent review gates each merge to `next`. Every finding is addressed - An independent review gates each merge to `next`. Every finding is addressed
@@ -15,7 +15,8 @@
- Pre-1.0. - Pre-1.0.
- The tracker is authoritative for the pre-1.0 backlog. The open issues under - The tracker is authoritative for the pre-1.0 backlog. The open issues under
the `1.0.0` milestone are what remains before the tag. the `1.0.0` milestone are what remains before the tag. This file records
history and process, not the queue.
# Next Step # Next Step
@@ -23,6 +24,752 @@
https://git.eeqj.de/sneak/sfdupes/milestone/17. The issues are not restated https://git.eeqj.de/sneak/sfdupes/milestone/17. The issues are not restated
here, because a copy drifts as soon as the tracker moves. here, because a copy drifts as soon as the tracker moves.
# Completed Steps
Newest first. An entry ends with its date and its issue or branch, where those
are known.
- The usage line of `scan` no longer ends in `[flags]`. It is the line
`README.md` shows under "Subcommands" (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/169)
- `report` and `trees` leave out empty files. `report` writes no group of size
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
counts toward a tree's digest (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/178)
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
path is now `sneak.berlin/go/sfdupes/cmd/sfdupes` (2026-11-10,
https://git.eeqj.de/sneak/sfdupes/issues/157)
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
prints the report again (2026-11-06,
https://git.eeqj.de/sneak/sfdupes/issues/154)
- `scan` resolves each operand to its real path, symlinks followed, before the
walk. A symlink operand is scanned at its target instead of skipped. With
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
walks `sub` on its own instead of dropping it and deleting its records
(2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144)
- `README.md` and a comment in `internal/sfdupes/scan.go` no longer say ZFS
counts a new file's blocks only seconds after the write. The count can land in
under a second. Both now say up to seconds after the write, as the comment on
`diskSizeOf` in `internal/sfdupes/scan_test.go` does (2026-11-03,
https://git.eeqj.de/sneak/sfdupes/issues/152)
- The three disk size tests in `internal/sfdupes/scan_test.go` no longer fail at
random on ZFS. `unix.Sync` can return before ZFS has counted a file's blocks,
and one ZFS commit can take a file through several sizes. The test that needs
the size to change waits for it, for at most 30 seconds. Each test reads the
size before and after its scan. The rewrite test rescans until the two reads
match, then compares exactly. The other two fail only on a `disk_size` outside
the two reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150)
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
sentence. Every completed step is kept, with its date and its issue or branch.
The policy-compliance checklist of 2026-07-22 is folded into its step
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
it. It leads with what the program is and how to use it. A reference section
covers every setting and every subcommand. Development and testing come last.
Every statement was checked against the code on `next`. Five comments in
`internal/sfdupes/cli_test.go` cite "Settings", where their rules now live.
One comment each in these files cites what the README now says:
- `internal/sfdupes/scan_test.go`;
- `internal/sfdupes/trees_test.go`;
- `internal/sfdupes/exclude_test.go`.
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
- Each record holds `disk_size`, the bytes the file takes on disk. That is its
allocated blocks, `st_blocks` × 512, from the `lstat` call the walk already
makes. `trees` prints a `disk_size` column after `size`. It is the sum over
the `dupe` copy's files. On a compressed or sparse file `disk_size` differs
from `size`. A rescan updates the `disk_size` of an unchanged file whose size
on disk changed, without reading the file (2026-10-27,
https://git.eeqj.de/sneak/sfdupes/issues/147)
- The database keeps the number of records and the sum of their sizes in the one
row of a `totals` table. Three triggers on `files` keep that row current.
`info` reads that row. So do `scan`'s `load` phase and the `read` phases of
`report` and `trees`, for their totals. No count reads every record any more.
The triggers make each write to `files` slower. `scan` keeps SQLite's
temporary data in memory. That wins back part of that time (2026-10-27,
https://git.eeqj.de/sneak/sfdupes/issues/146)
- an operand at or inside a directory named `.zfs` is never scanned, whether
given alone or under another operand. `scan` warns about it, counts it as
skipped and does not walk it (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/142)
- A file or directory removed while the scan runs is not an error, in the walk
or in the hash phase. The scan prints no line for it and does not count it as
skipped. When the first of several hard links to a file is gone, the hash
phase reads the next one instead. On a terminal every progress display starts
with the spinner, the hash phase's included. A phase with an exact total ends
its line with its percent and a 10-column bar at the terminal's right edge.
The bar no longer moves as the line's length changes. A line too wide for the
terminal now loses its rate before its bar. The time remaining is labelled
`remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133)
- The `head`, `tail` and `content` columns are BLOBs, each holding the raw 32
bytes of a SHA-256. That is half the size of the hex text they held. The
hashes stay raw from the hash phase through the database to `report` and
`trees`. Reading a record whose hash is not 32 bytes is an error (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/134)
- The hash phase gives three hashes to every file that the walk records. `head`
and `tail` are the SHA-256 of its first and last 64 KiB. `content` is the
SHA-256 of the whole file under 100 MiB, and of 1 MiB at each gigabyte from
100 MiB up. Gone are:
- the 10 MiB and 50 MiB thresholds;
- the size census, which counted the sizes of every walked file and of every
record outside the operands. A file was hashed only when the census showed
another file of its size;
- the records without hashes;
- the content phase.
A scan has four phases, and a rescan reads only new and changed files.
`info` no longer counts the records with `head` and `tail` hashes, nor those
with a content hash, since every record has all three (2026-10-25,
https://git.eeqj.de/sneak/sfdupes/issues/132)
- `REPO_POLICIES.md` and the four scripts that pass `CACHEBUST` were copied
again from `sneak/prompts` `next` at `0653579`. Two steps now come before
`ARG CACHEBUST`:
- the module download in the `lint`, `test` and `builder` stages of the
`Dockerfile`;
- the package install in `builder`.
Docker keeps them until one of these changes:
- the base image;
- the install line;
- `go.mod` or `go.sum`.
Every step that checks or compiles the code still comes after
`ARG CACHEBUST`. `golangci-lint run` also keeps the linter's own cache.
README says the same (2026-10-18,
https://git.eeqj.de/sneak/sfdupes/issues/109)
- The release rule was set. Until 1.0, each tip of `main` that passes the build
gets the next `v0.0.N` tag.
`go install sneak.berlin/go/sfdupes/cmd/sfdupes@latest` then installs it.
`go.mod` retracts `v0.0.1`, the early build without a database. When the
linker set no version, `sfdupes --version` prints the version Go recorded in
the binary. That is the tag, or the pseudo-version of a `go install` build. It
prints `dev` only when neither is known. README "Install" documented:
- the rule;
- what `@main` and `v0.0.1` give;
- how to check an installed binary.
(2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/116)
- `scan` creates the database's directory where the database path leads. The
directory is the path with its last element cut off as text, so a `..` in it
stays. A `..` after a symlinked directory therefore goes through the symlink,
as it does when the file is opened. With
`SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a symlink to
`/data/x`, it now creates `/data/new` and the database in it. It used to
create `/srv/new` and then fail (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/119)
- `scan` never records its own database. Once the database and its `.lock` file
are open, it notes the device and inode of each of these:
- the database file;
- its `-wal` and `-shm` files;
- the `.lock` file.
The walk passes over any file with the device and inode of one of them,
however it reaches the file. It prints no line for such a file and counts
nothing. A record an earlier scan left for one of them is deleted
(2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
- `scan` never records or reads anything on a filesystem that is not persistent,
such as `tmpfs` or `proc`. Such a filesystem is one the kernel makes up or
keeps in memory. `scan` matches them by device ID, however the walk or the
content phase reaches a path. As the scan starts, it lists the mounted
filesystems of those types, from `/proc/self/mountinfo` on Linux and from the
mount list on macOS. No flag turns that off. Below an operand, the walk does
not enter the default excludes either. So `sfdupes scan /` skips the kernel's
view of the running system and the temporary directories. On Linux those are:
```
/proc, /sys, /dev, /run, /tmp, /var/tmp
```
On macOS they are:
```
/dev, /private/tmp, /private/var/tmp, /private/var/vm
/System/Volumes/VM, /System/Volumes/Data
```
A Windows list exists in the code but is not used. An operand at or inside a
default exclude is still walked. The scan prints a line for each path it
leaves out, without counting it as skipped. The records there count as
outside the scanned operands, as for `-x`. `--no-default-excludes` turns off
the default excludes only, and `scan --help` lists them (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/124)
- A progress display's rate covers the items completed over the last 5 seconds.
When stderr is not a terminal, it covers the items since the line before. It
falls to 0 while nothing completes. A burst after a wait no longer reads as a
steady rate of over 1,000 files a second. The time remaining still comes from
the phase's average. Once no item has completed for 10 seconds, the line shows
`stalled` and where the phase waits in place of the rate, as
`stalled disk/db`. The places are:
- `disk`;
- `db`;
- `out`, for stdout;
- `cpu`.
A phase that may wait in two places names both. Each phase's word is set in
the code that defines the phase. A line too wide for the terminal loses its
bar first, then its rate or those words. Four parts stay whole:
- the phase;
- the progress;
- the elapsed time;
- the ETA, the time remaining.
The content phase had one display for its checks and one for its reads. Both
lost those words on an 80-column terminal, at counts like those of README's
example. That happened for most elapsed times and ETAs in minutes. Every
other phase kept them whenever the elapsed time and the ETA were both in
minutes. The content phase writes its last partial batch, the records not
yet written, under a display of its own (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/125)
- New subcommand `info` prints the database's details on stdout, one per line as
a name, a tab and the value:
- the path and where it came from;
- the sizes of the database file and of its `-wal` file, when there is one;
- when the database file was last changed;
- the schema version;
- the number of records;
- how many records have `head` and `tail` hashes;
- how many records have a content hash;
- the sizes of all records added up.
It opens the database as `report` does and shows two phases on stderr,
`[1/2] open` and `[2/2] count`. The schema version and the counts come from
one read-only transaction, a read of the database at one moment. The counts
come from one query (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/117)
- `report` shows its work on stderr in two phases, `[1/2] open` and
`[2/2] read`, with the progress display `scan` and `trees` use. The display
runs from the moment `report` starts until the summary line. `open` says it is
opening the database, also while it waits on a busy one. `read` counts the
records, then reads them against that count in signature order. SQLite takes
that order from the `files_signature` index without sorting, so the first
record comes at once. `report` forms the duplicate groups itself and holds
only those of the size it is reading. It writes them ordered by first path
before it reads the next size. stdout and the summary line are unchanged, and
SQLite no longer needs a temporary file to sort in (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/114)
- `trees` shows its work on stderr in four phases, from `[1/4] open` to
`[4/4] write`, each with the progress display `scan` uses. The display runs
from the moment `trees` starts until the summary line. `open` says it is
opening the database, also while it waits on a busy one. `read` counts the
records, then reads them against that count. `group` counts the directories it
groups, and `write` the groups it writes. A display's rate names what its
phase counts:
- files;
- records;
- directories;
- groups.
stdout and the summary line are unchanged. When stdout and stderr are the
same file, as one terminal is, no line of stdout shares a line with the
display. When they are not, the display keeps being drawn while stdout waits
on a slow reader (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/113)
- Each `scan` phase's progress display names the phase and its place among the
five, as `[2/5] walk`. It runs from the moment that the phase starts. It
shows:
- the progress;
- the elapsed time;
- the ETA, the time remaining.
`load` counts the records first. The walk's total is estimated from the
records under the operands. The content phase takes its total from SQLite's
grouping alone. Until the first record is returned, it says it is searching
the database. On a terminal a phase with an exact total draws its line
followed by the bar of the progress-bar library. Any other phase draws a
spinner followed by the line. Every display is drawn again several times a
second. No line reaches the terminal's last column, however narrow the
terminal is, even after it is made narrower during a phase. A line too wide
leaves out its rate, then its bar. The rest stays whole. The line is cut
short only when the rest alone does not fit, as below about 70 columns late
in a large scan. When stderr is not a terminal, the line is printed every 5
seconds whether or not an item completed (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/112)
- `scan`, `report` and `trees` first print a line on stderr naming the database
they use, as an absolute path. The line says whether that path is the default
or comes from `SFDUPES_DATABASE`. `scan` says when the database is new. README
"Database" says when `scan` writes each file (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/111)
- The canonical files and model scripts were copied again from `sneak/prompts`
`next` at `e3f79c4`:
- `golangci-lint` v2.14.0;
- lint and test are phases of the `Dockerfile` that write no image, and
`make test` runs the suite under the race detector. Gone are
`Dockerfile.lint`, `script/verify-lint-image-pin` and `make test-race`;
- every `docker build` in `script/` passes a new `CACHEBUST` value. Every
stage of the `Dockerfile` declares it first, so every `RUN` step runs on
every build. The steps that compile keep Go's build cache, and the tests
run with `-count=1`;
- `prettier` runs on the host, from the `node` and `yarn` that
`script/bootstrap` installs. The `prettier` and `markdown` stages are
gone, and the build stage installs `git` and `make` itself;
- a new push cancels the workflow's older run on the same branch, and a run
stops after 20 minutes;
- a coding agent's settings file is deleted.
Since https://git.eeqj.de/sneak/sfdupes/issues/109 two steps come before
`ARG CACHEBUST`:
- the module download in `lint`, `test` and the build stage;
- the install of `git` and `make` in the build stage.
Docker keeps them between builds. Every step that checks or compiles the
code still runs on every build (2026-10-15,
https://git.eeqj.de/sneak/sfdupes/issues/95)
- With `-x`, an operand under another operand is no longer dropped when a
directory between the two, the inner one included, is on a different
filesystem than the outer one. The inner operand is then walked on its own.
For a regular-file operand the check starts at its directory. The records of
files gone from below it are deleted (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/106)
- With `-x`, `scan` keeps the records below a directory on another filesystem
under its operand, instead of deleting them. It counts their sizes in the size
census (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
- The hash and content phases open each file without blocking and check that it
is still a regular file. A file that is now a FIFO or device node is warned
about and skipped instead of hanging the scan (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/100)
- The scan lock file is named from the database's path with symlinks resolved,
after `scan` creates the database file if it is missing. A scan through a
symlink to the database therefore fails while another scan holds the lock
(2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/101)
- On a terminal, a warning for the last file of a phase no longer erases the
phase's finished bar. The bar is drawn again below the warning (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/102)
- `scan` records the mtime to the nanosecond, as whole seconds in `mtime` plus
`mtime_nsec`, and compares it at that resolution. A same-size rewrite within
the same second is hashed again (2026-10-07,
https://git.eeqj.de/sneak/sfdupes/issues/12)
- The narration was cut from `TODO.md` Completed Steps and from the comments in
`script/`, `Dockerfile` and `Dockerfile.lint` (gone since
https://git.eeqj.de/sneak/sfdupes/issues/95). The Workflow section now
branches from and merges to `next` (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/49)
- `make test-race` ran the test suite under the race detector in a cgo-enabled
container, outside `make check`. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 `make test` itself runs the suite
under the race detector, in the `Dockerfile`'s Debian-based `test` phase.
`make test-race` is gone (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/18)
- A bare `docker build .` failed, naming `script/cibuild` and `script/docker`,
rather than serve the gates, the lint and test steps, from cache. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 a bare build succeeds, as
`REPO_POLICIES.md` requires, and may serve the gates from cache. The builds in
`script/` pass a new `CACHEBUST` value, so theirs always run (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/39)
- `make fmt` and `make fmt-check` run `prettier` over all Markdown, and the
Gitea workflow checks it. All Markdown was reformatted (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/19)
- `script/lint` writes no image, so a run no longer leaves an untagged one
behind (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/48)
- Tests cover:
- a missing database;
- `scan` keeping stdout empty;
- its skip warning;
- the `report` and `trees` summary lines;
- every subcommand going through `runE`.
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/16)
- `.golangci.yml` was replaced with the current canonical copy, which uses
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/26)
- A test fails when either `hashWorker` cancellation check in
`internal/sfdupes/scan.go` is removed (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/83)
- A database path holding `?`, `#` or `%` opens exactly the file it names
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
- `scan` rejects `--workers` below 1 as a usage error instead of running
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
- A test fails when either walk cancellation check in `internal/sfdupes/scan.go`
is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
- A test checks that `scan` refuses a database with another schema version
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
- Four inaccurate comments in `internal/sfdupes/cancel_test.go` were corrected,
and `walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles`
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/33)
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/17)
- `scan` creates the schema in one transaction. A version-0 database with a
`files` table is refused with a clear schema-version error (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/11)
- README documents install, Docker, a daily `cron` scan and how to read and
check the reports (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/54)
- The `Dockerfile` build stage kept the Go module cache out of `builder`'s home
and copied the sources with `--chown`, so no `chown -R` walked them. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 there is no `builder` user and
nothing changes owner. The build stage only compiles, as root, and the tests
run as `nobody` in the `test` phase (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/43)
- `--version` prints `sfdupes VERSION` to stdout. README documents it and
`--help` (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/15)
- `scan` stops cleanly on `SIGINT` or `SIGTERM`: it commits what it has hashed,
deletes nothing more, and exits 1 (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/5)
- `report` and `trees` stream the records instead of holding them all in memory.
The schema gains the `files_signature` index (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/14)
- Progress prints at once on a non-terminal and uses a real terminal test.
Warnings print through a spinner instead of racing its redraw (2026-10-03,
https://git.eeqj.de/sneak/sfdupes/issues/13)
- These operands are warned about and skipped, and the records beneath them are
kept:
- symlinks;
- sockets;
- FIFOs;
- device nodes;
- `.zfs` directories.
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/9)
- `scan` holds a lock on a lock file beside the database for its whole run, so a
second `scan` fails at once with exit 1 (2026-10-03,
https://git.eeqj.de/sneak/sfdupes/issues/53)
- Tests cover stdout write failures in `report` and `trees`. README states that
`| head` ends `sfdupes` by `SIGPIPE` and that `>&-` writes to `/dev/null`
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/30)
- `report` and `trees` open the database read-only, and `scan` leaves it out of
WAL mode, so reading needs only read access (2026-10-03,
https://git.eeqj.de/sneak/sfdupes/issues/8)
- Four characters are escaped in `report`, `trees` and warning paths:
- tab;
- newline;
- carriage return;
- backslash.
The root directory's path is `/` (2026-10-03,
https://git.eeqj.de/sneak/sfdupes/issues/7)
- A plain `docker build .` stamps the `git` tag or short commit instead of
`dev`. `.dockerignore` sends `.git` without `.git/config`. The build stage
stamps the `VERSION` build argument, else `git describe --tags --always`. It
fails if the context carries `.git` and the version is still empty, `dev` or
`unknown`. The Gitea workflow checks out the full history (`fetch-depth: 0`),
so it stamps the same value as `make build` (2026-10-02, branch `next`,
https://git.eeqj.de/sneak/sfdupes/issues/67)
- The 1 KiB end-window sampling was replaced with the hashes chosen by file
size, and README "Duplicate detection" documented every size threshold. A file
under 10 MiB was hashed in full, and its `head`, `tail` and `content` all held
that hash. A larger file got only its 64 KiB `head` and `tail` in the hash
phase. The content phase ran after the update phase. It read a file for
`content` only when its size, `head` and `tail` matched another record's, from
this scan or an earlier one. `content` was the whole file below 50 MiB, and
gigabyte-spaced 1 MiB samples at or above. It never read a file gone or
changed since its record was written. `report` and `trees` left out any record
without a `content` hash. The `content` column is part of the version 1
schema. Superseded by https://git.eeqj.de/sneak/sfdupes/issues/132
(2026-09-22, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/61)
- The dead `files.dat` references were removed from `Makefile`, `.gitignore` and
`.dockerignore` (2026-09-21, branch `next`,
https://git.eeqj.de/sneak/sfdupes/issues/22)
- The lint-image pin comments and `FROM` form in `Dockerfile` and
`Dockerfile.lint` were fixed. Both pins became the policy
`# image:vX.Y.Z, YYYY-MM-DD` comment over a bare `FROM image@sha256:...`,
without the false `(Debian-based)` note or the tag. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 the `Dockerfile`'s `lint` phase
holds the only `golangci-lint` pin, in that form. `Dockerfile.lint` is gone,
and so is `script/verify-lint-image-pin`, which compared the two pins
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/25)
- All linting ran in Docker, via `Dockerfile.lint` and `script/lint`. Per the
owner ruling the linter is never installed on a host, and
`golangci-lint config verify` runs before `golangci-lint run`.
`script/bootstrap` stopped installing or pinning the linter, and
`script/verify-linter-pin` was retired. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 both commands run in the
`Dockerfile`'s `lint` phase. `script/lint` builds that phase alone, and the
build stage depends on it through `COPY --from=lint /src/go.sum /dev/null`.
`Dockerfile.lint` and `script/verify-lint-image-pin` are gone. Nothing inside
an image build may run `docker`, so the phase calls `golangci-lint` directly
(2026-08-10, branch `next`, https://git.eeqj.de/sneak/sfdupes/issues/46)
- The Docker build stage's prerequisites were installed by running
`script/bootstrap` instead of `apk add --no-cache make` inline.
`script/verify-linter-pin` failed the build unless the linter copied from the
lint stage was the version that `script/bootstrap` pinned. Builds then took up
to 5m14s from an empty cache, mostly in a `chown -R` of the module cache,
filed as https://git.eeqj.de/sneak/sfdupes/issues/43. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 the build stage installs `git` and
`make` with `apk add --no-cache` and only compiles. The `lint` and `test`
phases are the gates (2026-08-09, branch `dockerfile-bootstrap`,
https://git.eeqj.de/sneak/sfdupes/issues/42)
- The Docker layer cache is defeated for the gate steps, so `script/cibuild` and
`script/docker` cannot report a pass they did not earn. The `Dockerfile`
copies the tree before its gates. On an unchanged tree Docker served them from
cache, and the build exited 0 without running them. The fix was a
`CHECK_EPOCH` build argument. Since
https://git.eeqj.de/sneak/sfdupes/issues/95 every `docker build` in `script/`
passes a new `CACHEBUST` value instead. Every stage of the `Dockerfile`
declares it before every step that checks or compiles the code. Run as root,
the tests fail `TestScanHardlinkRunFailsTogether`, because root reads through
the `chmod(0)` the test relies on. So the `test` phase runs them as `nobody`
(2026-08-09, branch `cibuild-cache-bust`,
https://git.eeqj.de/sneak/sfdupes/issues/32)
- `script/bootstrap` checked the installed `golangci-lint` version instead of
only its presence. The version lived only in `GOLANGCI_LINT_VERSION`, with the
`go install` module reference derived from it. Any installed version that was
not the pin was reinstalled, whether:
- older;
- newer;
- absent;
- unparseable.
`go install` writes into `GOBIN`, or `GOPATH/bin`, while `make lint` ran the
first `golangci-lint` on `PATH`. So bootstrap re-read the effective version
after installing. On a mismatch it printed both paths and both versions and
exited non-zero. It did not reorder `PATH` or delete anyone's binary. The
`--version` call kept its stderr and was bounded by `timeout(1)` where that
exists. `git`, `make` and `go` kept presence-only checks. The version check
was verified by bootstrapping this host from v2.10.1 to v2.12.2, and again
to a no-op. Stub runs of the script under `dash` covered:
- a thirteen-input version-parse matrix;
- a shadowed install, which must exit non-zero;
- an install destination not on `PATH`;
- `GOBIN` set;
- a wedged binary, which must hit the timeout.
`make check` and `make lint` were clean at v2.12.2, so v2.10.1 had not been
hiding findings on `main`. Superseded by
https://git.eeqj.de/sneak/sfdupes/issues/46, since the linter is no longer
installed on a host (2026-08-09, branch `bootstrap-version-check`,
https://git.eeqj.de/sneak/sfdupes/issues/24)
- The hash worker pool unwinds on the error path. The pool is an owned,
context-aware `hashPool`. Every blocking send in the feeder and the workers
selects on `ctx.Done()`, and `jobs` is closed on every path out. `hashPhase`
defers `pool.stop()`, which cancels and then drains `results` until the last
goroutine has exited. Draining is what frees a worker parked on a send. `ctx`
is passed from `cmd.Context()` as the first parameter, through:
- `runScan`;
- `syncScan`;
- both worker pools;
- the whole database layer.
The walk pool gets the same treatment, plus a `ctx.Err()` guard after the
walk. A cancelled walk yields a partial size census, and every file that it
never reached looks vanished to the update phase. That phase's own `BeginTx`
also fails on the cancelled context before deleting anything. The guard is
still the barrier once an interrupted scan may commit what it has. Tests
drive `run(scan)` against a database whose insert trigger aborts. They
assert that the scan fails instead of hanging, and that the goroutine count
returns to its pre-scan baseline. Other tests cancel a scan part-way through
the walk, deterministically, by counting the scan's own checks of
`ctx.Done()`. They assert that it stops at the guard with every record
intact. It then holds a partial size census and a still-populated record
index. Direct tests cover the remaining cancellation branches of both pools.
They test:
- `sendEvent`;
- the walk workers;
- `dispatchDirs`;
- `feedHashJobs`;
- `hashWorker`;
- `hashPhase`.
(2026-08-09, branch `hash-pool-cleanup`,
https://git.eeqj.de/sneak/sfdupes/issues/6)
- The database is closed on every fatal exit path. `fatalf` and its `os.Exit(1)`
are gone. So the deferred `db.Close()` runs when a subcommand fails, and with
it the SQLite WAL checkpoint. These return errors instead:
- `runScan`;
- `runReport`;
- `runTrees`;
- `loadRecords`;
- `resolveRoots`.
The single exit point is `run` in `internal/sfdupes/cli.go`. It maps a
`fatalError`, anything a subcommand returned, to exit 1, and cobra's own
argument and flag errors to exit 2. So a runtime failure is never reported
as a usage error, nor printed with the usage text.
`internal/sfdupes/cli_test.go` drives the CLI in-process and asserts the
exit codes from README "Error handling" and the stdout/stderr split. It also
checks that a fatal error after the database is open leaves no `-wal` or
`-shm` file behind, for `scan`, `report` or `trees` (2026-08-09, branch
`db-close-on-fatal`, https://git.eeqj.de/sneak/sfdupes/issues/4)
- `golangci-lint` was updated to v2.12.2 with the canonical config, merged as
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
`script/bootstrap` went from v2.12.1 to v2.12.2, and `.golangci.yml` was
replaced with the canonical file. Four thresholds now live under
`linters.settings`, as the v2 schema requires, so they are applied:
- `lll`;
- `funlen`;
- `cyclop`;
- `dupl`.
No new lint findings surfaced (2026-08-09, branch `golangci-v2.12.2`,
https://git.eeqj.de/sneak/sfdupes/issues/3)
- `Makefile` targets were converted to scripts-to-rule-them-all `script/`
entrypoints, like the other managed repos. All 12 entrypoints exist:
- `bootstrap`, `setup` and `projectname`;
- `test`, `lint`, `fmt`, `fmt-check` and `check`;
- `docker`, `cibuild`, `precommit` and `install-precommit`.
Every `Makefile` target does nothing but call its script (2026-07-26, commit
`3abeacf`, https://git.eeqj.de/sneak/sfdupes/issues/1)
- The binary is the default `make` target. Plain `make` builds `sfdupes`, where
it used to run `check` plus `build`. `make build` remains as an alias
(2026-07-24, branch `make-default-target`)
- The phases became scan-wide, with concurrent operands and batched updates. All
operands seed the shared walk pool, and every pass runs once over the whole
scan. So the totals and ETAs cover the whole scan. The per-operand walk, hash
and update cycles and their stderr announcements are gone. The update pass
writes in batches, each in its own transaction. The filesystem is
authoritative and the database follows it, so one scan's changes need not land
all at once (2026-07-24, branch `scan-wide-phases`)
- The stat pass was split back out of the walk. Phases are strictly sequential
again, per operand, with parallelism only inside each phase:
- walk;
- stat;
- hash;
- update.
The walk enumerates paths with per-directory workers, and the stat pass
lstats them with per-file workers. That restores the exact total and ETA of
the stat bar (2026-07-24, branch `parallel-phases`)
- Each operand is announced on stderr before its passes. With per-operand walk,
hash and update cycles, a multi-operand run such as `scan /srv/*` showed pass
totals that looked like the whole run's (2026-07-24, branch
`scan-operand-progress`)
- The walk became parallel. The walk pass was a single goroutine and took hours
at about 20 million files on a busy pool. One ZFS server took 4 hours for 22
million files. It is now a walk by a pool of per-directory workers that
records size and mtime as it goes. That folds away the separate stat pass and
halves the metadata reads. Each `PATH` operand commits in its own transaction,
so an interrupted scan keeps completed operands (2026-07-24, branch
`parallel-walk`)
- `scan` gained a persistent database. It maintains an SQLite database that
survives between runs, with the absolute path as each record's key. The driver
is `modernc.org/sqlite`, pure Go, so cgo stays disabled. A rescan hashes only
new or changed files, by mtime and size. It deletes the records of files
vanished from under the scanned operands and leaves records outside them
untouched. So `scan` can run daily from `cron`. `report` and `trees` read the
database, with no positional arguments, instead of a scan stream. The database
is at `/var/lib/sfdupes/db.sqlite`, and `SFDUPES_DATABASE` overrides that. WAL
journaling plus a single-transaction update keep a report run during a scan
safe (2026-07-24, branch `persistent-database`)
- The `origin` remote, `git@git.eeqj.de:sneak/sfdupes.git`, was added. `v0.0.1`
was tagged, and `main` and the tags were pushed (2026-07-23)
- The `scan` command line was reworked. Required `PATH...` operands, parsed by
cobra, replace the `/srv` `-root` default. The new `-x`, `--one-file-system`
flag follows the GNU convention and stops at filesystem boundaries, which the
walk crosses by default (2026-07-23, branch `scan-required-paths`)
- The repo was brought into compliance with `REPO_POLICIES.md` (the version of
2026-07-06), the existing repo checklist and the Go styleguide. The audit of
2026-07-22 found these gaps, and the branch closed them:
- `.gitignore`, with the standard patterns for secrets and for OS, editor
and Go files. The compiled `sfdupes` binary and `files.dat` had been
sitting untracked in the tree;
- `.editorconfig`;
- `LICENSE` and the README License section. MIT was assumed from house
convention, for the owner to confirm;
- `REPO_POLICIES.md` in the repo root;
- the canonical `.golangci.yml`, with its 150 findings fixed, so that
`make lint` is clean;
- the `Makefile` targets the policy requires:
- `test` and `lint`;
- `fmt` and `fmt-check`;
- `docker` and `hooks`.
`check` depended on `build`, which wrote the binary. `make check` must
not modify files, so `check` no longer depends on `build`;
- a test suite, which the policy required with a 30-second timeout and the
conditional `-v` rerun. At 64% coverage, it covers:
- parsing and grouping;
- digests and suppression;
- hashing and the scan pipeline;
- a multistage `Dockerfile` and `.dockerignore`. The `Dockerfile` had:
- hash-pinned images;
- a lint stage that failed fast;
- a build stage that ran `make check`;
- `.gitea/workflows/check.yml`, running `docker build .` on push with the
checkout action pinned by commit SHA;
- the README sections the policy requires. The first is a Description whose
first line names the project, its purpose and category, its license and
its author. Then come Getting Started and Rationale, and at the end TODO,
License and Author;
- the stale README non-goal "no git repository setup and no CI" removed;
- the pre-commit hook, installed with `make hooks` once that target existed.
The code was already `gofmt`-clean, so no standalone formatting commit was
needed (2026-07-23, branch `repo-policy-compliance`)
- `git init` was run, with a README-only first commit. The code baseline was
committed on `main` (2026-07-22)
- The `scan`, `report` and `trees` subcommands were implemented (pre-git
history)
# Future Steps # Future Steps
- Possible later features, out of scope per README, are a byte-for-byte check of - Possible later features, out of scope per README, are a byte-for-byte check of
+7 -6
View File
@@ -358,9 +358,8 @@ func TestRunScanInterrupted(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
dir := buildSmokeTree(t) dir := buildSmokeTree(t)
opts := scanOptions{workers: walkCancelWorkers} err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false)
err := runScan(t.Context(), []string{dir}, opts)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -383,7 +382,8 @@ func TestRunScanInterrupted(t *testing.T) {
go func() { go func() {
defer close(done) defer close(done)
err = runScan(cancelledContext(t), []string{dir}, opts) err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
false, false)
}() }()
awaitReturn(t, done, "runScan") awaitReturn(t, done, "runScan")
@@ -410,7 +410,8 @@ func TestRunScanInterrupted(t *testing.T) {
got, before) got, before)
} }
err = runScan(t.Context(), []string{dir}, opts) err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -436,7 +437,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
dir := buildWalkCancelTree(t) dir := buildWalkCancelTree(t)
err := runScan(newWalkClock(hashCancelAtDone), []string{dir}, err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
scanOptions{workers: walkCancelWorkers}) walkCancelWorkers, false, false)
if !errors.Is(err, errInterrupted) { if !errors.Is(err, errInterrupted) {
t.Fatalf("runScan interrupted mid-hash = %v, want %v", t.Fatalf("runScan interrupted mid-hash = %v, want %v",
err, errInterrupted) err, errInterrupted)
+37 -168
View File
@@ -10,7 +10,6 @@ import (
"os" "os"
"runtime" "runtime"
"runtime/debug" "runtime/debug"
"slices"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@@ -32,82 +31,15 @@ const (
cmdDwim = "dwim" cmdDwim = "dwim"
) )
// databaseHelp is the paragraph of the root and scan help that names
// the database (README "Location").
const databaseHelp = "scan and dwim store what they find in a database " +
"file. report, trees\nand info read it. The file is $" + databaseEnv +
" when that is set and\nnot empty, and otherwise " +
defaultDatabasePath + ", where creating it\nneeds root.\n"
// escapesHelp is the paragraph of the report and trees help on how
// paths are escaped (README "Report output format").
const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
\\ a backslash
\t a tab
\n a newline
\r a carriage return
`
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
// summary in the help of their subcommands.
const (
reportHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate file. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that file
size the file's size in bytes
Rows are ordered by size, largest first, then by path.
` + escapesHelp
treesHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate directory tree. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that tree
files the number of regular files in one copy
size their total size in bytes
disk_size the bytes the dupe copy takes on disk
Rows are ordered by size, largest first, then by path.
` + escapesHelp
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
value, separated by a tab:
database the path, and where it came from
database file size in bytes, then rounded, as 2147483648 (2.0 GiB)
-wal file size in bytes, then rounded; only if a -wal file exists
modified when the database file last changed, in UTC
schema version the version of the database's layout
records the number of records
size of recorded files their total size, in bytes, then rounded
`
dwimHelp = `dwim runs what "sfdupes scan /" runs with the same flags, then what
"sfdupes trees" runs. Run it as root, so that the scan can read every
file. The scan is stored in the database, as scan stores it, so
"sfdupes trees" prints the report again later.
`
)
// errNoSubcommand is returned by the root command when it is invoked // errNoSubcommand is returned by the root command when it is invoked
// without a subcommand. That is a usage error, and the usage text // without a subcommand. That is a usage error, and the usage text
// cobra prints for it is the whole message. // cobra prints for it is the whole message.
var errNoSubcommand = errors.New("no subcommand") var errNoSubcommand = errors.New("no subcommand")
// errWorkersBelowOne is the usage error for a --workers value below 1. // errWorkersBelowOne is the usage error for a scan --workers value
// below 1.
var errWorkersBelowOne = errors.New("--workers must be at least 1") var errWorkersBelowOne = errors.New("--workers must be at least 1")
// errEmptyExcludeValue is the usage error for an empty --exclude value,
// as from --exclude "$DIR" with DIR unset. Taken as a path, it
// would name the working directory.
var errEmptyExcludeValue = errors.New("--exclude must not be empty")
// Version is the build version, injected at link time via -ldflags // Version is the build version, injected at link time via -ldflags
// (see the Makefile); empty when the linker set none, as for a binary // (see the Makefile); empty when the linker set none, as for a binary
// built by go install. buildVersion turns it into what --version prints. // built by go install. buildVersion turns it into what --version prints.
@@ -255,116 +187,65 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}), }),
} }
root.Long = root.Short + ".\n\n" + databaseHelp dwimCmd := &cobra.Command{
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp Use: cmdDwim,
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp Short: "Scan / with the default excludes, then print the duplicate-tree report",
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", stdout)
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
newDwimCommand(stdout))
return root return root
} }
// scanOptions holds the values of the flags that scan and dwim share: // newScanCommand builds the scan subcommand, with its --workers, -x and
// --workers, -x, --no-default-excludes and --exclude. // --no-default-excludes flags. Its help lists the default excludes of
type scanOptions struct { // the system it runs on, and the filesystem types it never records
workers int // anything on.
oneFS bool
noDefaultExcludes bool
excludePaths []string
}
// scanFlagsUsage names the flags that scan and dwim share, for their
// usage lines.
const scanFlagsUsage = "[--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]..."
// addScanFlags adds the flags that scan and dwim share to cmd, which
// stores their values in opts.
func addScanFlags(cmd *cobra.Command, opts *scanOptions) {
cmd.Flags().IntVar(&opts.workers, "workers", runtime.NumCPU(),
"concurrent workers for the walk and hash phases")
cmd.Flags().BoolVarP(&opts.oneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
cmd.Flags().BoolVar(&opts.noDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
// A string array, not a slice, so a comma in a path is not a
// separator.
cmd.Flags().StringArrayVar(&opts.excludePaths, "exclude", nil,
"never enter or record `PATH` or anything below it; repeatable")
}
// check rejects the flag values that are usage errors, before the
// command runs. It is the PreRunE of scan and dwim.
func (o *scanOptions) check(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, o.workers)
if err != nil {
return err
}
return rejectEmptyExcludeValue(cmd, o.excludePaths)
}
// newScanCommand builds the scan subcommand. Its help names the
// database, and lists the default excludes of the system it runs on and
// the filesystem types it never records anything on.
func newScanCommand() *cobra.Command { func newScanCommand() *cobra.Command {
var opts scanOptions var (
scanWorkers int
scanOneFS bool
scanNoDefaultExcludes bool
)
short := "Walk trees and synchronize the scan database" short := "Walk trees and synchronize the scan database"
scanCmd := &cobra.Command{ scanCmd := &cobra.Command{
Use: cmdScan + " " + scanFlagsUsage + " PATH...", Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
// The Use line already names every flag, so cobra must not // The Use line already names every flag, so cobra must not
// append its generic "[flags]" to it. // append its generic "[flags]" to it.
DisableFlagsInUseLine: true, DisableFlagsInUseLine: true,
Short: short, Short: short,
Long: short + ".\n\n" + databaseHelp + "\n" + Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
defaultExcludesHelp(runtime.GOOS), Args: cobra.MinimumNArgs(1),
Args: cobra.MinimumNArgs(1), PreRunE: func(cmd *cobra.Command, _ []string) error {
PreRunE: opts.check, return checkScanWorkers(cmd, scanWorkers)
},
RunE: runE(func(ctx context.Context, args []string) error { RunE: runE(func(ctx context.Context, args []string) error {
ctx, stop := interruptContext(ctx) ctx, stop := interruptContext(ctx)
defer stop() defer stop()
return runScan(ctx, args, opts) return runScan(ctx, args, scanWorkers, scanOneFS,
scanNoDefaultExcludes)
}), }),
} }
addScanFlags(scanCmd, &opts) scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
"concurrent workers for the walk and hash phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
return scanCmd return scanCmd
} }
// newDwimCommand builds the dwim subcommand, which takes the scan flags // checkScanWorkers rejects a scan --workers value below 1. That is a
// with the meaning they have for scan. Its help says what dwim runs, // usage error reported in one line: cobra prints the returned message
// then lists the default excludes and the filesystem types the scan // without the usage text, and run exits 2.
// never records anything on.
func newDwimCommand(stdout io.Writer) *cobra.Command {
var opts scanOptions
short := "Scan /, then print the duplicate-tree report"
dwimCmd := &cobra.Command{
Use: cmdDwim + " " + scanFlagsUsage,
DisableFlagsInUseLine: true,
Short: short,
Long: short + ".\n\n" + dwimHelp + "\n" +
defaultExcludesHelp(runtime.GOOS),
Args: cobra.NoArgs,
PreRunE: opts.check,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", opts, stdout)
}),
}
addScanFlags(dwimCmd, &opts)
return dwimCmd
}
// checkScanWorkers rejects a --workers value below 1. That is a usage
// error reported in one line: cobra prints the returned message without
// the usage text, and run exits 2.
func checkScanWorkers(cmd *cobra.Command, workers int) error { func checkScanWorkers(cmd *cobra.Command, workers int) error {
if workers >= 1 { if workers >= 1 {
return nil return nil
@@ -375,18 +256,6 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers) return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
} }
// rejectEmptyExcludeValue rejects an empty --exclude value. That
// is a usage error reported in one line, as for --workers below 1.
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
if !slices.Contains(excludePaths, "") {
return nil
}
cmd.SilenceUsage = true
return errEmptyExcludeValue
}
// runE adapts a subcommand implementation, or the version print, to // runE adapts a subcommand implementation, or the version print, to
// cobra's RunE. Cobra prints the error and the command's usage text for // cobra's RunE. Cobra prints the error and the command's usage text for
// every error RunE returns, but a subcommand that ran and failed has no // every error RunE returns, but a subcommand that ran and failed has no
+4 -89
View File
@@ -22,12 +22,6 @@ import (
// error prints and a runtime failure must not. // error prints and a runtime failure must not.
const usageMarker = "Usage:" const usageMarker = "Usage:"
// helpFlag asks for the help text.
const helpFlag = "--help"
// workersFlag is the scan and dwim flag that sets the worker count.
const workersFlag = "--workers"
// TestMain stops the tests at once, saying why, when their temporary // TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing // directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail // there (README "Rules for the walk"), so most scan tests would fail
@@ -576,8 +570,7 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}{ }{
cmdScan: { cmdScan: {
run: func() error { run: func() error {
return runScan(t.Context(), []string{missing}, return runScan(t.Context(), []string{missing}, 1, false, false)
scanOptions{workers: 1})
}, },
want: scanWant, want: scanWant,
}, },
@@ -668,7 +661,7 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
args := []string{cmdScan, workersFlag, workers, t.TempDir()} args := []string{cmdScan, "--workers", workers, t.TempDir()}
code := run(args, &stdout, &stderr) code := run(args, &stdout, &stderr)
if code != exitUsage { if code != exitUsage {
@@ -699,7 +692,7 @@ func TestRunHelp(t *testing.T) {
// README §Settings: help goes to stderr, exits 0, and leaves stdout // README §Settings: help goes to stderr, exits 0, and leaves stdout
// empty. // empty.
cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}} cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
for _, args := range cases { for _, args := range cases {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -719,16 +712,6 @@ func TestRunHelp(t *testing.T) {
args, got) args, got)
} }
} }
// The scan help lists --exclude with its argument.
var stdout, stderr bytes.Buffer
run([]string{cmdScan, "-h"}, &stdout, &stderr)
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
t.Errorf("scan -h stderr = %q, want %s PATH", stderr.String(),
excludeFlag)
}
} }
func TestRunScanUsageLine(t *testing.T) { func TestRunScanUsageLine(t *testing.T) {
@@ -736,8 +719,7 @@ func TestRunScanUsageLine(t *testing.T) {
// README §Subcommands shows this usage line for scan, and the help // README §Subcommands shows this usage line for scan, and the help
// and the usage error both print it. // and the usage error both print it.
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] " + const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...\n"
"[--exclude PATH]... PATH...\n"
cases := map[string][]string{ cases := map[string][]string{
"help": {cmdScan, "-h"}, "help": {cmdScan, "-h"},
@@ -760,73 +742,6 @@ func TestRunScanUsageLine(t *testing.T) {
} }
} }
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
t.Parallel()
// README §Settings: the help names the database, and says what
// report, trees, info and dwim print.
database := []string{
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
}
escapes := []string{`\\`, `\t`, `\n`, `\r`}
// The subcommand whose help is printed; none for the root.
cases := []struct {
subcommand []string
want []string
}{
{nil, database},
{[]string{cmdScan}, database},
{
[]string{cmdReport},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n size ",
"largest first",
}, escapes...),
},
{
[]string{cmdTrees},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n files ",
"\n size ", "\n disk_size ", "largest first",
}, escapes...),
},
{
[]string{cmdInfo},
[]string{
"\n database ", "\n database file size ",
"\n -wal file size ", "\n modified ",
"\n schema version ", "\n records ",
"\n size of recorded files ",
},
},
{
[]string{cmdDwim},
[]string{
"sfdupes scan /", "as root", "sfdupes trees", "database",
},
},
}
for _, tc := range cases {
var stdout, stderr bytes.Buffer
args := slices.Concat(tc.subcommand, []string{helpFlag})
code := run(args, &stdout, &stderr)
if code != exitOK {
t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
}
for _, want := range tc.want {
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) help lacks %q:\n%s",
args, want, stderr.String())
}
}
}
}
func TestRunVersion(t *testing.T) { func TestRunVersion(t *testing.T) {
t.Parallel() t.Parallel()
+9 -10
View File
@@ -5,20 +5,19 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"runtime"
) )
// runDwim implements the dwim subcommand: it runs what "sfdupes scan // runDwim implements the dwim subcommand: it runs what "sfdupes scan
// root" runs with the scan flags in opts, then what "sfdupes trees" // root" runs with no flags, then what "sfdupes trees" runs, and ends
// runs, and ends with a line on stderr saying how to print the report // with a line on stderr saying how to print the report again. The dwim
// again. The dwim command passes "/" as root. It catches SIGINT and // command passes "/" as root. It catches SIGINT and SIGTERM as scan
// SIGTERM as scan does, but only during the scan. An error from the // does, but only during the scan. An error from the scan, an interrupt
// scan, an interrupt included, ends it before the report. During the // included, ends it before the report. During the report, a signal
// report, a signal ends it at once, as it ends trees. // ends it at once, as it ends trees.
func runDwim( func runDwim(ctx context.Context, root string, stdout io.Writer) error {
ctx context.Context, root string, opts scanOptions, stdout io.Writer,
) error {
scanCtx, stop := interruptContext(ctx) scanCtx, stop := interruptContext(ctx)
err := runScan(scanCtx, []string{root}, opts) err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
stop() stop()
+2 -129
View File
@@ -2,10 +2,7 @@ package sfdupes //nolint:testpackage // calls the unexported runDwim
import ( import (
"bytes" "bytes"
"errors"
"os"
"path/filepath" "path/filepath"
"runtime"
"strings" "strings"
"testing" "testing"
) )
@@ -31,7 +28,7 @@ func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
err := runDwim(t.Context(), root, scanOptions{workers: 1}, &stdout) err := runDwim(t.Context(), root, &stdout)
if err != nil { if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr()) t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
} }
@@ -70,7 +67,7 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, &stdout) err := runDwim(t.Context(), missing, &stdout)
if err == nil { if err == nil {
t.Fatal("runDwim on a missing root: error = nil, want it fatal") t.Fatal("runDwim on a missing root: error = nil, want it fatal")
} }
@@ -84,127 +81,3 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
t.Errorf("stderr = %q, want nothing from trees or dwim", got) t.Errorf("stderr = %q, want nothing from trees or dwim", got)
} }
} }
func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
// dwim passes its --exclude paths to the scan: an excluded copy is
// not recorded, so its pair is not reported.
t.Setenv(databaseEnv, testDBPath(t))
root := realTempDir(t)
kept := []string{
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
}
writeFile(t, root, "b1/f.bin", pattern(2, 5000))
excluded := filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000)))
var stdout bytes.Buffer
stderr := captureStderr(t)
opts := scanOptions{workers: 1, excludePaths: []string{excluded}}
err := runDwim(t.Context(), root, opts, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
if len(lines) != 2 {
t.Fatalf("stdout = %q, want a header and one group",
stdout.String())
}
want := kept[0] + "\t" + kept[1] + "\t1\t300\t"
if !strings.HasPrefix(lines[1], want) {
t.Errorf("group = %q, want it to start %q", lines[1], want)
}
skipped := "scan: skipped " + excluded + " (--exclude)\n"
if got := stderr(); !strings.Contains(got, skipped) {
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
}
}
func TestRunDwimTakesScanFlags(t *testing.T) {
// The dwim command parses and checks the scan flags as scan does.
// Each case stops before the walk of / begins, and before the
// database is created. "dwim --exclude /" excludes its own operand.
cases := map[string]struct {
args []string
code int
want string
}{
"--exclude /": {
[]string{cmdDwim, excludeFlag, "/"},
exitFatal,
"sfdupes: scan /: operand excluded by --exclude /\n",
},
"--exclude empty": {
[]string{cmdDwim, excludeFlag, ""},
exitUsage,
"Error: --exclude must not be empty\n",
},
"--workers 0": {
[]string{cmdDwim, workersFlag, "0", "-x", "--no-default-excludes"},
exitUsage,
"Error: --workers must be at least 1, got 0\n",
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
stdout := captureStdout(t)
stderr := captureStderr(t)
code := run(tc.args, os.Stdout, os.Stderr)
if code != tc.code {
t.Errorf("run(%q) = %d, want %d", tc.args, code, tc.code)
}
if !strings.HasSuffix(stderr(), tc.want) || stdout() != "" {
t.Errorf("stdout, stderr = %q, %q, want \"\" and stderr "+
"ending %q", stdout(), stderr(), tc.want)
}
_, err := os.Lstat(path)
if !errors.Is(err, os.ErrNotExist) {
t.Errorf("lstat %s: %v, want no database", path, err)
}
})
}
}
func TestRunDwimHelpListsScanFlags(t *testing.T) {
t.Parallel()
// README "Subcommands" shows this usage line for dwim.
const want = "\n sfdupes dwim [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]...\n"
var stdout, stderr bytes.Buffer
run([]string{cmdDwim, "-h"}, &stdout, &stderr)
if !strings.Contains(stderr.String(), want) {
t.Errorf("dwim -h stderr = %q, want the usage line %q",
stderr.String(), want)
}
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
t.Errorf("dwim -h stderr = %q, want %s PATH", stderr.String(),
excludeFlag)
}
// README "Settings": the help lists the default excludes and the
// filesystem types the walk never records anything on.
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
if !strings.Contains(stderr.String(), help) {
t.Errorf("dwim -h stderr = %q, want it to hold %q", stderr.String(),
help)
}
}
+12 -104
View File
@@ -1,9 +1,8 @@
package sfdupes package sfdupes
// This file holds three rules for the walk (README "Rules for the // This file holds two rules for the walk (README "Rules for the walk"):
// walk"): the paths given with --exclude, which the scan never enters;
// the default excludes, the paths a scan's walk does not enter below an // the default excludes, the paths a scan's walk does not enter below an
// operand unless scan is given --no-default-excludes; and the // operand unless scan is given --no-default-excludes, and the
// filesystems that are not persistent, on which the scan never records // filesystems that are not persistent, on which the scan never records
// or reads anything. // or reads anything.
@@ -13,7 +12,6 @@ import (
"fmt" "fmt"
"io" "io"
"io/fs" "io/fs"
"os"
"path/filepath" "path/filepath"
"slices" "slices"
"strings" "strings"
@@ -45,25 +43,17 @@ const typesPerHelpLine = 6
// readMountInfo reads. // readMountInfo reads.
var errMountInfoLine = errors.New("unexpected line") var errMountInfoLine = errors.New("unexpected line")
// errExcludedOperand reports an operand at or below a path given with
// --exclude.
var errExcludedOperand = errors.New("operand excluded by --exclude")
// walkRules holds the rules for the walk that a scan sets up as it // walkRules holds the rules for the walk that a scan sets up as it
// starts. excludePaths are the paths given with --exclude, each with // starts. excludes are the default excludes, or none with
// the device ID and inode number of the file it names (see // --no-default-excludes. nonPersistent maps the device ID of every
// excludePathInodes), which the walk neither enters nor records, // mounted filesystem that is not persistent to its type; the scan
// whatever path leads to them. defaultExcludes are the default // records and reads nothing on one of those, whatever the flags.
// excludes, or none with --no-default-excludes. nonPersistent maps the // databaseFiles are the scan's own database files, which the walk
// device ID of every mounted filesystem that is not persistent to its // passes over (see databaseFiles in db.go).
// type; the scan records and reads nothing on one of those, whatever
// the flags. databaseFiles are the scan's own database files, which the
// walk passes over (see databaseFiles in db.go).
type walkRules struct { type walkRules struct {
excludePaths []fileRec excludes []string
defaultExcludes []string nonPersistent map[uint64]string
nonPersistent map[uint64]string databaseFiles []fileRec
databaseFiles []fileRec
} }
// defaultExcludes returns the default excludes of the system goos // defaultExcludes returns the default excludes of the system goos
@@ -107,88 +97,6 @@ func isDefaultExclude(path string, excludes []string) bool {
return false return false
} }
// excludePathInodes returns each of paths, the --exclude values, as its
// real path (see realPaths) with the device ID and inode number of the
// file there. The walk compares what it reaches with those, as it does
// with the database files, so it also leaves out an --exclude path
// spelled another way, such as in another letter case on macOS.
func excludePathInodes(paths []string) ([]fileRec, error) {
resolved, err := realPaths(paths)
if err != nil {
return nil, err
}
files := make([]fileRec, 0, len(resolved))
for _, path := range resolved {
fi, err := os.Lstat(path)
if err != nil {
return nil, err
}
dev, ino := inodeOfInfo(fi)
files = append(files, fileRec{path: path, dev: dev, ino: ino})
}
return files, nil
}
// matchingExcludePath returns the path of the one of excludePaths with
// the device ID and inode number of fi, an lstat result, or "" when
// there is none.
func matchingExcludePath(fi fs.FileInfo, excludePaths []fileRec) string {
dev, ino := inodeOfInfo(fi)
f := fileRec{dev: dev, ino: ino}
for _, exclude := range excludePaths {
if sameInode(exclude, f) {
return exclude.path
}
}
return ""
}
// checkOperandsNotExcluded returns an error when one of roots, the
// operands, or a directory above it is one of excludePaths, the
// --exclude paths, by device ID and inode number. The error names the
// operand and the --exclude path. The roots are absolute, so the
// directories above each one end at /. A path here that lstat cannot
// read is an error too.
func checkOperandsNotExcluded(roots []string, excludePaths []fileRec) error {
for _, root := range roots {
for path := root; ; path = filepath.Dir(path) {
fi, err := os.Lstat(path)
if err != nil {
return fmt.Errorf("scan %s: %w", root, err)
}
exclude := matchingExcludePath(fi, excludePaths)
if exclude != "" {
return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand,
exclude)
}
if path == "/" {
break
}
}
}
return nil
}
// excludePathLine returns the line the scan prints for path when fi,
// its lstat, makes it one of rules.excludePaths, and "" when it does
// not.
func (rules walkRules) excludePathLine(path string, fi fs.FileInfo) string {
if matchingExcludePath(fi, rules.excludePaths) == "" {
return ""
}
return fmt.Sprintf("scan: skipped %s (--exclude)", path)
}
// nonPersistentFilesystemTypes returns the filesystem types that are // nonPersistentFilesystemTypes returns the filesystem types that are
// not persistent: filesystems that the kernel makes up or that are kept // not persistent: filesystems that the kernel makes up or that are kept
// in memory. devfs is the macOS filesystem of /dev; the others are // in memory. devfs is the macOS filesystem of /dev; the others are
@@ -254,7 +162,7 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
return devices, lines.Err() return devices, lines.Err()
} }
// defaultExcludesHelp describes, for the scan and dwim help, the // defaultExcludesHelp describes, for the scan command's help, the
// default excludes of the system goos names and, where the scan lists // default excludes of the system goos names and, where the scan lists
// the mounted filesystems, the types it never records anything on. // the mounted filesystems, the types it never records anything on.
func defaultExcludesHelp(goos string) string { func defaultExcludesHelp(goos string) string {
+3 -217
View File
@@ -14,9 +14,6 @@ import (
"golang.org/x/sys/unix" "golang.org/x/sys/unix"
) )
// excludeFlag is the scan flag that names a path the walk never enters.
const excludeFlag = "--exclude"
func TestDefaultExcludes(t *testing.T) { func TestDefaultExcludes(t *testing.T) {
t.Parallel() t.Parallel()
@@ -191,7 +188,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
unseen := writeFile(t, excluded, "new", pattern(2, 10)) unseen := writeFile(t, excluded, "new", pattern(2, 10))
st, err := syncScan(t.Context(), db, []string{dir}, 1, false, st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
walkRules{defaultExcludes: []string{excluded}}) walkRules{excludes: []string{excluded}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -237,7 +234,7 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
db := openTestDB(t) db := openTestDB(t)
st, err := syncScan(t.Context(), db, roots, 1, false, st, err := syncScan(t.Context(), db, roots, 1, false,
walkRules{defaultExcludes: []string{excluded}}) walkRules{excludes: []string{excluded}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -265,7 +262,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
f := writeFile(t, excluded, "f", pattern(1, 10)) f := writeFile(t, excluded, "f", pattern(1, 10))
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false, st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
walkRules{defaultExcludes: []string{excluded}}) walkRules{excludes: []string{excluded}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -359,217 +356,6 @@ func TestRunScanNoDefaultExcludes(t *testing.T) {
assertRecorded(t, path, f, true) assertRecorded(t, path, f, true)
} }
// TestRunScanExcludeNested scans a tree through the command with an
// --exclude path two levels below the operand. The walk records nothing
// at or below it and says so. The record an earlier scan left there is
// kept although its file is gone.
func TestRunScanExcludeNested(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
excluded := filepath.Join(dir, "a", "b")
kept := writeFile(t, dir, "a/f", pattern(1, 10))
gone := writeFile(t, excluded, "gone", pattern(2, 10))
scanOK(t, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
unseen := writeFile(t, excluded, "c/new", pattern(3, 10))
stderr := scanOK(t, excludeFlag, excluded, dir)
line := "scan: skipped " + excluded + " (--exclude)\n"
if !strings.Contains(stderr, line) {
t.Errorf("stderr = %q, want %q", stderr, line)
}
assertRecorded(t, path, kept, true)
assertRecorded(t, path, gone, true)
assertRecorded(t, path, unseen, false)
}
// TestRunScanExcludeRepeated gives --exclude twice, once for a
// directory and once for a file, the file through a symlink to it. The
// walk records neither, and records the file beside them.
func TestRunScanExcludeRepeated(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
inDir := writeFile(t, dir, "skip/f", pattern(1, 10))
file := writeFile(t, dir, "skip-file", pattern(2, 10))
kept := writeFile(t, dir, "keep", pattern(3, 10))
link := filepath.Join(t.TempDir(), "link")
symlink(t, file, link)
scanOK(t, excludeFlag, filepath.Dir(inDir), excludeFlag, link, dir)
assertRecorded(t, path, inDir, false)
assertRecorded(t, path, file, false)
assertRecorded(t, path, kept, true)
}
// TestRunScanExcludeReachedByAnotherPath gives --exclude a directory
// through a symlink and a file through "..", so the walk reaches
// neither by the path given. The walk also reaches the file through a
// hard link. It records none of these, prints a line naming each, and
// records the file beside them.
func TestRunScanExcludeReachedByAnotherPath(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
inDir := writeFile(t, dir, "a/cache/f", pattern(1, 10))
file := writeFile(t, dir, "a/big", pattern(2, 10))
kept := writeFile(t, dir, "keep", pattern(3, 10))
hardLink := filepath.Join(dir, "big-link")
err := os.Link(file, hardLink)
if err != nil {
t.Fatal(err)
}
link := filepath.Join(t.TempDir(), "link")
symlink(t, filepath.Join(dir, "a"), link)
// Joined as text, since filepath.Join would drop the "..".
dotDot := dir + "/a/cache/../big"
stderr := scanOK(t, excludeFlag, filepath.Join(link, "cache"),
excludeFlag, dotDot, dir)
for _, skipped := range []string{filepath.Dir(inDir), file, hardLink} {
line := "scan: skipped " + skipped + " (--exclude)\n"
if !strings.Contains(stderr, line) {
t.Errorf("stderr = %q, want %q", stderr, line)
}
}
assertRecorded(t, path, inDir, false)
assertRecorded(t, path, file, false)
assertRecorded(t, path, hardLink, false)
assertRecorded(t, path, kept, true)
}
// TestRunScanExcludedOperand checks that an operand at or below an
// --exclude path is a fatal error that names both, and so is an
// --exclude path that does not exist. An operand that is a hard link to
// an --exclude file is at it. Neither scan writes a database.
func TestRunScanExcludedOperand(t *testing.T) {
dir := realTempDir(t)
sub := filepath.Join(dir, "sub")
f := writeFile(t, sub, "f", pattern(1, 10))
hardLink := filepath.Join(dir, "f-link")
err := os.Link(f, hardLink)
if err != nil {
t.Fatal(err)
}
cases := map[string]struct {
args []string
want string
}{
"at it": {
[]string{excludeFlag, sub, sub},
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
},
"below it": {
[]string{excludeFlag, dir, sub},
"scan " + sub + ": operand excluded by --exclude " + dir + "\n",
},
"after an operand it does not exclude": {
[]string{dir, excludeFlag, sub, sub},
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
},
"a hard link to it": {
[]string{excludeFlag, f, hardLink},
"scan " + hardLink + ": operand excluded by --exclude " + f + "\n",
},
"exclude missing": {
[]string{excludeFlag, filepath.Join(dir, "nope"), dir},
"--exclude: resolve " + filepath.Join(dir, "nope") + ": ",
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
stdout := captureStdout(t)
stderr := captureStderr(t)
code := run(append([]string{cmdScan}, tc.args...), os.Stdout,
os.Stderr)
if code != exitFatal {
t.Errorf("run(scan %q) = %d, want %d", tc.args, code,
exitFatal)
}
assertFatalOutput(t, stderr(), stdout())
if !strings.Contains(stderr(), "sfdupes: "+tc.want) {
t.Errorf("stderr = %q, want %q", stderr(), tc.want)
}
_, err := os.Lstat(path)
if !errors.Is(err, os.ErrNotExist) {
t.Errorf("lstat %s: %v, want no database", path, err)
}
})
}
}
// TestRunScanRejectsEmptyExclude checks that an empty --exclude value,
// given as its own argument, after "=" or after another --exclude, is
// a usage error reported in one line before anything is scanned
// (README "Settings").
func TestRunScanRejectsEmptyExclude(t *testing.T) {
dir := t.TempDir()
cases := map[string][]string{
"separate": {excludeFlag, "", dir},
"joined by =": {excludeFlag + "=", dir},
"after a path": {excludeFlag, t.TempDir(), excludeFlag, "", dir},
}
for name, args := range cases {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
var stdout, stderr bytes.Buffer
code := run(append([]string{cmdScan}, args...), &stdout,
&stderr)
if code != exitUsage {
t.Errorf("run(scan %q) = %d, want %d", args, code,
exitUsage)
}
want := "Error: --exclude must not be empty\n"
if stdout.String() != "" || stderr.String() != want {
t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
stdout.String(), stderr.String(), want)
}
_, err := os.Lstat(path)
if !errors.Is(err, os.ErrNotExist) {
t.Errorf("lstat %s: %v, want no database", path, err)
}
})
}
}
// TestRunScanSkipsNonPersistentOperand scans, through the command, a // TestRunScanSkipsNonPersistentOperand scans, through the command, a
// directory on a filesystem that is not persistent, with and without // directory on a filesystem that is not persistent, with and without
// --no-default-excludes: either way it is skipped, with a line saying // --no-default-excludes: either way it is skipped, with a line saying
+2 -2
View File
@@ -642,9 +642,9 @@ func TestTerminalStalledLines(t *testing.T) {
{label: scanPhase("hash"), stalledOn: hashStalledOn}, {label: scanPhase("hash"), stalledOn: hashStalledOn},
{label: scanPhase("update"), stalledOn: "db"}, {label: scanPhase("update"), stalledOn: "db"},
{label: treesPhase("read"), stalledOn: "db"}, {label: treesPhase("read"), stalledOn: "db"},
{label: treesPhase("group"), stalledOn: "cpu/disk"}, {label: treesPhase("group"), stalledOn: "cpu"},
{label: treesPhase("write"), stalledOn: "out"}, {label: treesPhase("write"), stalledOn: "out"},
{label: reportPhase("read"), stalledOn: "db/disk/out"}, {label: reportPhase("read"), stalledOn: "db/out"},
} }
for _, params := range phases { for _, params := range phases {
+16 -43
View File
@@ -4,14 +4,11 @@ import (
"context" "context"
"crypto/sha256" "crypto/sha256"
"database/sql" "database/sql"
"errors"
"fmt" "fmt"
"io" "io"
"io/fs"
"os" "os"
"slices" "slices"
"strings" "strings"
"syscall"
"time" "time"
) )
@@ -44,10 +41,10 @@ func reportPhase(name string) string {
} }
// runReport implements the report subcommand: it prints the file-level // runReport implements the report subcommand: it prints the file-level
// duplicates report as TSV on stdout. Of the scanned filesystem it // duplicates report as TSV on stdout. It never touches the scanned
// only checks, with pathExists, that each path in a duplicate group is // filesystem; its only I/O is the database, stdout, and stderr. It
// still there. It names the database on stderr before it opens it. Any // names the database on stderr before it opens it. Any database
// database problem, including a missing database, is fatal. // problem, including a missing database, is fatal.
// //
// The work runs in two phases, each under a progress display of its // The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and read (read the number of // own on stderr: open (open the database) and read (read the number of
@@ -80,14 +77,13 @@ func runReport(ctx context.Context, stdout io.Writer) error {
defer func() { _ = db.Close() }() defer func() { _ = db.Close() }()
// The read phase checks the paths of each duplicate group on disk // The read phase writes the report to stdout while it reads the
// and writes the report to stdout while it reads the records, so it // records, so it may wait on either.
// may wait on any of the three.
prog = newProgress(progressParams{ prog = newProgress(progressParams{
label: reportPhase("read"), unit: "records", total: -1, label: reportPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords, stalledOn: "db/disk/out", waitingFor: countingRecords, stalledOn: "db/out",
}) })
report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog) report, err := readReport(ctx, db, dbPath, stdout, prog)
prog.finish() prog.finish()
@@ -108,15 +104,14 @@ func runReport(ctx context.Context, stdout io.Writer) error {
// dbPath, as its total, then reads every record in signature order, // dbPath, as its total, then reads every record in signature order,
// counting each on prog, and writes the report to stdout: the header // counting each on prog, and writes the report to stdout: the header
// line, then the rows of each size's duplicate groups once that size is // line, then the rows of each size's duplicate groups once that size is
// read. A path for which exists returns false is left out of its group. // read. stdout may be the terminal prog is drawn on, so the lines go
// stdout may be the terminal prog is drawn on, so the lines go through // through prog's writer. It returns what it wrote, with the totals for
// prog's writer. It returns what it wrote, with the totals for the // the summary line.
// summary line.
func readReport(ctx context.Context, db *sql.DB, dbPath string, func readReport(ctx context.Context, db *sql.DB, dbPath string,
stdout io.Writer, exists func(path string) bool, prog *progress, stdout io.Writer, prog *progress,
) (*reportWriter, error) { ) (*reportWriter, error) {
out := prog.writer(stdout) out := prog.writer(stdout)
report := &reportWriter{out: out, exists: exists} report := &reportWriter{out: out}
_, err := fmt.Fprintln(out, "first\tdupe\tsize") _, err := fmt.Fprintln(out, "first\tdupe\tsize")
if err != nil { if err != nil {
@@ -163,10 +158,6 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
type reportWriter struct { type reportWriter struct {
out io.Writer out io.Writer
// exists reports whether a path is still on disk. A path that is
// gone is left out of its group.
exists func(path string) bool
sig scanRec // the signature being read; its path is not used sig scanRec // the signature being read; its path is not used
paths []string // the paths read so far with sig paths []string // the paths read so far with sig
groups [][]string // the duplicate groups of sig.size, each sorted groups [][]string // the duplicate groups of sig.size, each sorted
@@ -203,18 +194,10 @@ func (w *reportWriter) add(r scanRec) error {
return nil return nil
} }
// endGroup ends the paths read with w.sig: two or more that are still // endGroup ends the paths read with w.sig: two or more are a duplicate
// on disk are a duplicate group, held with its paths sorted until // group, held with its paths sorted until writeGroups writes it. Empty
// writeGroups writes it. Empty files form no group, since removing one // files form no group, since removing one frees nothing.
// frees nothing. Only the paths of a possible group are checked, never
// a path with no duplicate or an empty file.
func (w *reportWriter) endGroup() { func (w *reportWriter) endGroup() {
if len(w.paths) >= minGroupSize && w.sig.size > 0 {
w.paths = slices.DeleteFunc(w.paths, func(p string) bool {
return !w.exists(p)
})
}
if len(w.paths) < minGroupSize || w.sig.size == 0 { if len(w.paths) < minGroupSize || w.sig.size == 0 {
// The paths are dropped, and their slice reused. // The paths are dropped, and their slice reused.
w.paths = w.paths[:0] w.paths = w.paths[:0]
@@ -263,16 +246,6 @@ func (w *reportWriter) finish() error {
return w.writeGroups() return w.writeGroups()
} }
// pathExists reports whether path is still on disk, for report and
// trees. It is false only when lstat finds that the path, or a
// directory above it, is gone. Any other error counts as present, so
// a user who may not look inside a directory still sees its copies.
func pathExists(path string) bool {
_, err := os.Lstat(path)
return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
}
// escapePath returns a path as it is written in a report column (README // escapePath returns a path as it is written in a report column (README
// "Report output format"): a backslash, tab, newline or carriage return // "Report output format"): a backslash, tab, newline or carriage return
// becomes \\, \t, \n or \r, and every other byte is kept as it is. // becomes \\, \t, \n or \r, and every other byte is kept as it is.
+9 -150
View File
@@ -72,37 +72,15 @@ type dupeGroup struct {
paths []string paths []string
} }
// everyPathExists stands in for pathExists where a test's records name
// made-up paths that are not on disk.
func everyPathExists(string) bool { return true }
// recsOnDisk returns a new directory and recs with each path moved
// under it, after creating an empty file at each moved path: report and
// trees then find every path still on disk.
func recsOnDisk(t *testing.T, recs []scanRec) (string, []scanRec) {
t.Helper()
dir := realTempDir(t)
moved := make([]scanRec, len(recs))
for i, r := range recs {
r.path = writeFile(t, dir, r.path, nil)
moved[i] = r
}
return dir, moved
}
// dupeGroups returns the duplicate groups report reads from db, in // dupeGroups returns the duplicate groups report reads from db, in
// report order, from the rows it writes: a group's rows come together, // report order, from the rows it writes: a group's rows come together,
// each naming the group's first path. Every path counts as still on // each naming the group's first path.
// disk.
func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup { func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
t.Helper() t.Helper()
var out bytes.Buffer var out bytes.Buffer
_, err := readReport(t.Context(), db, "", &out, everyPathExists, nil) _, err := readReport(t.Context(), db, "", &out, nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -149,8 +127,7 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
} }
func TestRunReportEscapesPaths(t *testing.T) { func TestRunReportEscapesPaths(t *testing.T) {
dir, recs := recsOnDisk(t, awkwardPairRecs()) t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
t.Setenv(databaseEnv, seedDatabase(t, recs))
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -161,7 +138,7 @@ func TestRunReportEscapesPaths(t *testing.T) {
} }
want := "first\tdupe\tsize\n" + want := "first\tdupe\tsize\n" +
dir + `/d/\tone\ntwo\rthree\\four/f` + "\t" + dir + "/d/A/f\t5\n" `/d/\tone\ntwo\rthree\\four/f` + "\t/d/A/f\t5\n"
if got := stdout.String(); got != want { if got := stdout.String(); got != want {
t.Errorf("stdout = %q, want %q", got, want) t.Errorf("stdout = %q, want %q", got, want)
} }
@@ -195,8 +172,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
// Counts the records read, and draws nothing. // Counts the records read, and draws nothing.
prog := &progress{now: time.Now} prog := &progress{now: time.Now}
_, err = readReport(t.Context(), db, "", failingWriter{}, _, err = readReport(t.Context(), db, "", failingWriter{}, prog)
everyPathExists, prog)
if !errors.Is(err, errWriteFailed) || if !errors.Is(err, errWriteFailed) ||
!strings.HasPrefix(err.Error(), "write stdout: ") { !strings.HasPrefix(err.Error(), "write stdout: ") {
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed) t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
@@ -227,8 +203,7 @@ const smokeReport = "first\tdupe\tsize\n" +
// it ends; read counts the 6 records before it reads them. The summary // it ends; read counts the 6 records before it reads them. The summary
// comes last, and stdout holds the report alone. // comes last, and stdout holds the report alone.
func TestRunReportProgressNamesEachPhase(t *testing.T) { func TestRunReportProgressNamesEachPhase(t *testing.T) {
dir, recs := recsOnDisk(t, smokeTreeRecs()) path := seedDatabase(t, smokeTreeRecs())
path := seedDatabase(t, recs)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
stderr := captureStderr(t) stderr := captureStderr(t)
@@ -241,9 +216,8 @@ func TestRunReportProgressNamesEachPhase(t *testing.T) {
code, exitOK, stderr()) code, exitOK, stderr())
} }
report := strings.ReplaceAll(smokeReport, "/d/", dir+"/d/") if got := stdout.String(); got != smokeReport {
if got := stdout.String(); got != report { t.Errorf("stdout = %q, want %q", got, smokeReport)
t.Errorf("stdout = %q, want %q", got, report)
} }
// How each line starts; the rate at the end of a phase varies. // How each line starts; the rate at the end of a phase varies.
@@ -292,7 +266,7 @@ func TestReadReportOnTerminal(t *testing.T) {
waitingFor: countingRecords, waitingFor: countingRecords,
}, terminalWidth, time.Hour, time.Now) }, terminalWidth, time.Hour, time.Now)
_, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p) _, err = readReport(t.Context(), db, "", os.Stderr, p)
p.finish() p.finish()
@@ -327,8 +301,6 @@ func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
scanRec{size: 50, head: a, tail: a, content: a, path: "/x/1"}, scanRec{size: 50, head: a, tail: a, content: a, path: "/x/1"},
) )
_, recs = recsOnDisk(t, recs)
reversed := slices.Clone(recs) reversed := slices.Clone(recs)
slices.Reverse(reversed) slices.Reverse(reversed)
@@ -483,119 +455,6 @@ func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
} }
} }
// TestReadReportChecksOnlyPossibleDuplicates checks that report asks
// whether a path is on disk only when its signature has two or more
// records, and never for an empty file.
func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
t.Parallel()
a, u, e := testHash("a"), testHash("u"), sha256.Sum256(nil)
recs := []scanRec{
{size: 9, head: a, tail: a, content: a, path: "/copy/1"},
{size: 9, head: a, tail: a, content: a, path: "/copy/2"},
{size: 9, head: a, tail: a, content: a, path: "/copy/3"},
{size: 9, head: u, tail: u, content: u, path: "/lonely"},
{size: 0, head: e, tail: e, content: e, path: "/empty1"},
{size: 0, head: e, tail: e, content: e, path: "/empty2"},
}
db := openTestDB(t)
err := applyChanges(t.Context(), db, recs, nil, nil)
if err != nil {
t.Fatal(err)
}
var checked []string
exists := func(path string) bool {
checked = append(checked, path)
return true
}
_, err = readReport(t.Context(), db, "", io.Discard, exists, nil)
if err != nil {
t.Fatal(err)
}
slices.Sort(checked)
want := []string{"/copy/1", "/copy/2", "/copy/3"}
if !slices.Equal(checked, want) {
t.Errorf("paths checked on disk = %q, want %q", checked, want)
}
}
// TestReadReportDropsGroupLeftWithOnePath checks that a group of two
// paths, one of them gone, is no duplicate group. If kept, it would
// write no row, but the summary line would count it.
func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
t.Parallel()
a := testHash("a")
recs := []scanRec{
{size: 9, head: a, tail: a, content: a, path: "/a/kept"},
{size: 9, head: a, tail: a, content: a, path: "/a/gone"},
}
db := openTestDB(t)
err := applyChanges(t.Context(), db, recs, nil, nil)
if err != nil {
t.Fatal(err)
}
exists := func(path string) bool { return path != "/a/gone" }
report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
if err != nil {
t.Fatal(err)
}
if report.dupeGroups != 0 {
t.Errorf("duplicate groups = %d, want 0", report.dupeGroups)
}
}
// TestReadReportKeepsGroupLeftWithTwoPaths checks that a group of three
// paths, one of them gone, is a duplicate group of the other two.
func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
t.Parallel()
a := testHash("a")
recs := []scanRec{
{size: 9, head: a, tail: a, content: a, path: "/b/kept1"},
{size: 9, head: a, tail: a, content: a, path: "/b/gone"},
{size: 9, head: a, tail: a, content: a, path: "/b/kept2"},
}
db := openTestDB(t)
err := applyChanges(t.Context(), db, recs, nil, nil)
if err != nil {
t.Fatal(err)
}
exists := func(path string) bool { return path != "/b/gone" }
var out bytes.Buffer
report, err := readReport(t.Context(), db, "", &out, exists, nil)
if err != nil {
t.Fatal(err)
}
if report.dupeGroups != 1 {
t.Errorf("duplicate groups = %d, want 1", report.dupeGroups)
}
want := "first\tdupe\tsize\n/b/kept1\t/b/kept2\t9\n"
if got := out.String(); got != want {
t.Errorf("stdout = %q, want %q", got, want)
}
}
func TestDupeGroupsContentSeparates(t *testing.T) { func TestDupeGroupsContentSeparates(t *testing.T) {
t.Parallel() t.Parallel()
+66 -85
View File
@@ -98,47 +98,44 @@ func scanPhase(name string) string {
// runs, and the lock is released after it. When ctx is cancelled, as by // runs, and the lock is released after it. When ctx is cancelled, as by
// the SIGINT or SIGTERM that interruptContext catches, the scan keeps // the SIGINT or SIGTERM that interruptContext catches, the scan keeps
// what it has hashed (see syncScan), prints how many files its walk // what it has hashed (see syncScan), prints how many files its walk
// reached, and returns errInterrupted. opts.workers must be at least 1; // reached, and returns errInterrupted. workers must be at least 1; the
// the scan and dwim commands reject anything less. Before the lock is // scan command rejects anything less. Before the lock is taken, the
// taken, the scan lists the filesystems that are not persistent, on // scan lists the filesystems that are not persistent, on which the walk
// which the walk records nothing; unless opts.noDefaultExcludes is set, // records nothing; unless noDefaultExcludes is set, the walk does not
// the walk does not enter the default excludes either. The walk neither // enter the default excludes either. Once the database is open, the
// enters nor records the paths in opts.excludePaths, the --exclude // scan notes its files, which the walk passes over.
// values; an operand at or below one is a fatal error. Once the func runScan(ctx context.Context, roots []string, workers int,
// database is open, the scan notes its files, which the walk passes oneFS, noDefaultExcludes bool,
// over. ) error {
func runScan(ctx context.Context, roots []string, opts scanOptions) error {
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
return err return err
} }
printDatabasePath(dbPath, source) // A database file missing now is created by lockScanDatabase below.
what := "database"
roots, err = realPaths(roots) _, err = os.Stat(dbPath)
if errors.Is(err, fs.ErrNotExist) {
what = "new database"
}
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
source)
roots, err = resolveRoots(roots)
if err != nil { if err != nil {
return err return err
} }
var rules walkRules nonPersistent, err := nonPersistentFilesystems()
rules.excludePaths, err = excludePathInodes(opts.excludePaths)
if err != nil {
return fmt.Errorf("--exclude: %w", err)
}
err = checkOperandsNotExcluded(roots, rules.excludePaths)
if err != nil {
return err
}
rules.nonPersistent, err = nonPersistentFilesystems()
if err != nil { if err != nil {
return fmt.Errorf("list the mounted filesystems: %w", err) return fmt.Errorf("list the mounted filesystems: %w", err)
} }
if !opts.noDefaultExcludes { rules := walkRules{nonPersistent: nonPersistent}
rules.defaultExcludes = defaultExcludes(runtime.GOOS) if !noDefaultExcludes {
rules.excludes = defaultExcludes(runtime.GOOS)
} }
lock, err := lockScanDatabase(dbPath) lock, err := lockScanDatabase(dbPath)
@@ -166,7 +163,7 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
return err return err
} }
st, err := syncScan(ctx, db, roots, opts.workers, opts.oneFS, rules) st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
if errors.Is(err, context.Canceled) { if errors.Is(err, context.Canceled) {
return interrupted(st.walked) return interrupted(st.walked)
} }
@@ -184,21 +181,6 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
return nil return nil
} }
// printDatabasePath prints on stderr the line that names the database
// at dbPath, with source, saying whether it is new. A database file
// missing now is new: lockScanDatabase creates it.
func printDatabasePath(dbPath, source string) {
what := "database"
_, err := os.Stat(dbPath)
if errors.Is(err, fs.ErrNotExist) {
what = "new database"
}
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
source)
}
// interruptContext returns a copy of ctx that the first SIGINT or // interruptContext returns a copy of ctx that the first SIGINT or
// SIGTERM cancels; the scan command runs the scan under it. stop // SIGTERM cancels; the scan command runs the scan under it. stop
// releases the signals. // releases the signals.
@@ -227,25 +209,24 @@ func interrupted(walked int) error {
return errInterrupted return errInterrupted
} }
// realPaths returns the real path of each of paths, the PATH operands // resolveRoots converts each PATH operand to its real path: absolute,
// or the --exclude values: absolute, with every symlink in it followed. // with every symlink in it followed. Database records are keyed by
// Database records are keyed by that path, so scan results depend // that path, so scan results depend neither on the working directory
// neither on the working directory nor on the symlinks an operand was // nor on the symlinks an operand was given through. An operand that
// given through. A path that cannot be resolved, because it or a // cannot be resolved, because it or a symlink's target does not exist
// symlink's target does not exist or cannot be read, is a fatal error // or cannot be read, is a fatal error before any scanning.
// before any scanning. func resolveRoots(roots []string) ([]string, error) {
func realPaths(paths []string) ([]string, error) { resolved := make([]string, 0, len(roots))
resolved := make([]string, 0, len(paths))
for _, path := range paths { for _, root := range roots {
a, err := absolutePath(path) a, err := absolutePath(root)
if err != nil { if err != nil {
return nil, fmt.Errorf("resolve %s: %w", path, err) return nil, fmt.Errorf("resolve %s: %w", root, err)
} }
r, err := filepath.EvalSymlinks(a) r, err := filepath.EvalSymlinks(a)
if err != nil { if err != nil {
return nil, fmt.Errorf("resolve %s: %w", path, err) return nil, fmt.Errorf("resolve %s: %w", root, err)
} }
resolved = append(resolved, r) resolved = append(resolved, r)
@@ -314,7 +295,7 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
} }
for ; dir != above; dir = filepath.Dir(dir) { for ; dir != above; dir = filepath.Dir(dir) {
if isDefaultExclude(dir, rules.defaultExcludes) { if isDefaultExclude(dir, rules.excludes) {
return false return false
} }
@@ -437,7 +418,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
// walkableRoots returns the operands the walk can start from: regular // walkableRoots returns the operands the walk can start from: regular
// files, and directories not named .zfs, none of them inside a // files, and directories not named .zfs, none of them inside a
// directory named .zfs. The operands are real paths from realPaths, // directory named .zfs. The operands are real paths from resolveRoots,
// so these checks apply to what a symlink operand points at. Every // so these checks apply to what a symlink operand points at. Every
// other operand is warned about, counted as skipped, and dropped. A // other operand is warned about, counted as skipped, and dropped. A
// dropped operand is no longer a root, so the records stored beneath it // dropped operand is no longer a root, so the records stored beneath it
@@ -512,11 +493,10 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// as verified. When its disk_size differs from the record's, it is // as verified. When its disk_size differs from the record's, it is
// queued for the update phase, which writes the new disk_size. A record // queued for the update phase, which writes the new disk_size. A record
// at or below a path the walk left out, because -x stopped at it, it is // at or below a path the walk left out, because -x stopped at it, it is
// an --exclude path or a default exclude, or it is not on a persistent // a default exclude, or it is not on a persistent filesystem, counts as
// filesystem, counts as outside the roots: it is not deleted. That does // outside the roots: it is not deleted. That does not hold below a root
// not hold below a root at or inside that path, since that root's own // at or inside that path, since that root's own walk verifies the
// walk verifies the records below it; nor is a line printed for such a // records below it; nor is a line printed for such a root.
// root.
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) { func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
// The records the last scan left under the roots estimate how many // The records the last scan left under the roots estimate how many
// files the walk will find. A first scan has none, and no estimate. // files the walk will find. A first scan has none, and no estimate.
@@ -565,7 +545,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
prog.increment() prog.increment()
old, ok := s.existing[ev.rec.path] old, ok := s.existing[ev.rec.path]
if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) { if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) {
s.toHash = append(s.toHash, ev.rec) s.toHash = append(s.toHash, ev.rec)
continue continue
@@ -800,6 +780,16 @@ func (s *scanState) updatePhase(ctx context.Context) error {
return applyChanges(ctx, s.db, nil, deletes, prog) return applyChanges(ctx, s.db, nil, deletes, prog)
} }
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under // underAnyRoot reports whether path is any of the roots or lies under
// one of them. // one of them.
func underAnyRoot(path string, roots []string) bool { func underAnyRoot(path string, roots []string) bool {
@@ -854,9 +844,9 @@ type dirJob struct {
// file's statted record, a warning when fail is set, or a path the walk // file's statted record, a warning when fail is set, or a path the walk
// left out: when otherFSDir is set, a directory -x did not enter // left out: when otherFSDir is set, a directory -x did not enter
// because it is on another filesystem than its operand, and when // because it is on another filesystem than its operand, and when
// leftOut is set, an --exclude path, a default exclude or a directory or // leftOut is set, a default exclude or a directory or file on a
// file on a filesystem that is not persistent, with warn holding the // filesystem that is not persistent, with warn holding the line that
// line that says so. // says so.
type walkEvent struct { type walkEvent struct {
rec fileRec rec fileRec
warn string warn string
@@ -907,7 +897,7 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
// operandWarning returns the one-line warning for an operand the walk // operandWarning returns the one-line warning for an operand the walk
// does not start from, naming the path and what it is, or "" for one it // does not start from, naming the path and what it is, or "" for one it
// does: a regular file, or a directory not named .zfs, that is not // does: a regular file, or a directory not named .zfs, that is not
// inside a directory named .zfs. realPaths has already followed the // inside a directory named .zfs. resolveRoots has already followed the
// operand's symlinks, so a symlink here is an operand replaced by one // operand's symlinks, so a symlink here is an operand replaced by one
// after it was resolved, and is not followed. // after it was resolved, and is not followed.
func operandWarning(root string, fi fs.FileInfo) string { func operandWarning(root string, fi fs.FileInfo) string {
@@ -1138,9 +1128,8 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
// no longer exists, and one that has stopped being a regular file is // no longer exists, and one that has stopped being a regular file is
// skipped. A file on a filesystem in rules that is not persistent, as // skipped. A file on a filesystem in rules that is not persistent, as
// through a bind mount of one file, is not recorded: it gets an event // through a bind mount of one file, is not recorded: it gets an event
// naming it instead, and so does a file given with --exclude, reached // naming it instead. One of the scan's own database files in rules gets
// by any path, a hard link to it included. One of the scan's own // no event at all.
// database files in rules gets no event at all.
func emitFile(ctx context.Context, p string, e fs.DirEntry, func emitFile(ctx context.Context, p string, e fs.DirEntry,
rules walkRules, events chan<- walkEvent, rules walkRules, events chan<- walkEvent,
) { ) {
@@ -1164,10 +1153,6 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
} }
line := rules.nonPersistentLine(p, info) line := rules.nonPersistentLine(p, info)
if line == "" {
line = rules.excludePathLine(p, info)
}
if line != "" { if line != "" {
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line}) sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
@@ -1192,11 +1177,11 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
// subdirJob applies the descent rules to directory p: never enter // subdirJob applies the descent rules to directory p: never enter
// .zfs (ZFS snapshot pseudo-dirs would list every file once per // .zfs (ZFS snapshot pseudo-dirs would list every file once per
// snapshot), never enter a directory on a filesystem in rules that is // snapshot), never enter a directory on a filesystem in rules that is
// not persistent, nor an --exclude path or a default exclude in rules, // not persistent, nor a default exclude in rules, and with -x never
// and with -x never enter a directory on a different filesystem than // enter a directory on a different filesystem than its operand. For the
// its operand. For all but .zfs it sends an event that names the // last three it sends an event that names the directory, so the scan
// directory, so the scan keeps the records below it, except those below // keeps the records below it, except those below an operand at or
// an operand at or inside it. // inside it.
func subdirJob(ctx context.Context, p string, e fs.DirEntry, func subdirJob(ctx context.Context, p string, e fs.DirEntry,
parent dirJob, oneFS bool, rules walkRules, parent dirJob, oneFS bool, rules walkRules,
events chan<- walkEvent, events chan<- walkEvent,
@@ -1221,17 +1206,13 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
} }
line := rules.nonPersistentLine(p, info) line := rules.nonPersistentLine(p, info)
if line == "" {
line = rules.excludePathLine(p, info)
}
if line != "" { if line != "" {
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line}) sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
return dirJob{}, false return dirJob{}, false
} }
if isDefaultExclude(p, rules.defaultExcludes) { if isDefaultExclude(p, rules.excludes) {
sendEvent(ctx, events, walkEvent{ sendEvent(ctx, events, walkEvent{
leftOut: p, leftOut: p,
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p), warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
+13 -117
View File
@@ -1215,7 +1215,7 @@ func assertSmokeTreeGroups(t *testing.T, dir string, db *sql.DB) {
super, dirs := dbTree(t, db) super, dirs := dbTree(t, db)
tg := collectTreeGroups(dirs, super, everyPathExists, nil) tg := collectTreeGroups(dirs, super, nil)
if len(tg) != 1 { if len(tg) != 1 {
t.Fatalf("len(tree groups) = %d, want 1", len(tg)) t.Fatalf("len(tree groups) = %d, want 1", len(tg))
} }
@@ -1384,47 +1384,6 @@ func TestSyncScanMtimeBump(t *testing.T) {
} }
} }
// TestSyncScanMtimeMovedBack replaces a file with other content of the
// same size and an older mtime, as cp -p or a restore from backup does.
// The next scan must hash the file again.
func TestSyncScanMtimeMovedBack(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err := os.Chtimes(a, recorded, recorded)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
replaced := pattern(3, 500)
writeFile(t, dir, "a.bin", replaced)
older := recorded.Add(-time.Hour)
err = os.Chtimes(a, older, older)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
assertWholeFileHashed(t, db, a, replaced)
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
t.Fatalf("mtime = %v, want %v", r.mtime, older)
}
}
// assertWholeFileHashed fails unless the record for path holds the // assertWholeFileHashed fails unless the record for path holds the
// whole-file hash of data as its head, tail, and content. // whole-file hash of data as its head, tail, and content.
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string, func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
@@ -2196,7 +2155,7 @@ func TestScanHashWriteFailureUnwindsPool(t *testing.T) {
var stderr bytes.Buffer var stderr bytes.Buffer
code := run([]string{ code := run([]string{
cmdScan, workersFlag, strconv.Itoa(hashLeakWorkers), dir, cmdScan, "--workers", strconv.Itoa(hashLeakWorkers), dir,
}, io.Discard, &stderr) }, io.Discard, &stderr)
if code != exitFatal { if code != exitFatal {
t.Fatalf("run(scan) = %d, want %d; stderr: %s", t.Fatalf("run(scan) = %d, want %d; stderr: %s",
@@ -2289,7 +2248,7 @@ func TestPruneRoots(t *testing.T) {
// The walk of /p does not enter /p/q when it is a default exclude, so // The walk of /p does not enter /p/q when it is a default exclude, so
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped. // /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
got = pruneRoots(nested, false, walkRules{defaultExcludes: []string{pqRoot}}, got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
device) device)
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want) t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
@@ -2307,86 +2266,23 @@ func TestPruneRoots(t *testing.T) {
} }
} }
// TestRunReportsLeaveOutRemovedCopies scans the smoke tree, then func TestReportsNeverTouchFilesystem(t *testing.T) {
// removes copies the reports name. report leaves out each removed file,
// and a group left with one path. trees checks only the root of each
// duplicate tree: with a file under t2 removed it still names t2, and
// with t2 removed it names nothing.
func TestRunReportsLeaveOutRemovedCopies(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
dir := buildSmokeTree(t)
p := func(name string) string { return filepath.Join(dir, name) }
scanOK(t, dir)
for _, name := range []string{"b/copy2.bin", "tiny2", "t2/sub/f2"} {
err := os.Remove(p(name))
if err != nil {
t.Fatal(err)
}
}
want := "first\tdupe\tsize\n" +
p("t1/f1") + "\t" + p("t2/f1") + "\t3000\n" +
p("t1/f1") + "\t" + p("t3/f1") + "\t3000\n" +
p("a/one.bin") + "\t" + p("b/copy.bin") + "\t2000\n" +
p("t1/sub/f2") + "\t" + p("t3/sub/f2renamed") + "\t100\n"
if got := runStdout(t, cmdReport); got != want {
t.Errorf("report stdout = %q, want %q", got, want)
}
// disk_size, the last column, depends on the filesystem.
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
want = header + p("t1") + "\t" + p("t2") + "\t2\t3100\t"
if got := runStdout(t, cmdTrees); !strings.HasPrefix(got, want) ||
strings.Count(got, "\n") != 2 {
t.Errorf("trees stdout = %q, want one row starting %q", got, want)
}
err := os.RemoveAll(p("t2"))
if err != nil {
t.Fatal(err)
}
if got := runStdout(t, cmdTrees); got != header {
t.Errorf("trees stdout after removing t2 = %q, want %q", got, header)
}
}
func TestPathExists(t *testing.T) {
t.Parallel() t.Parallel()
dir := t.TempDir() dir := buildSmokeTree(t)
file := writeFile(t, dir, "file", nil) db := openTestDB(t)
locked := writeFile(t, dir, "locked/file", nil)
// Without search permission on its directory, a file cannot be syncTree(t, db, dir)
// looked at by anyone but root.
err := os.Chmod(filepath.Dir(locked), 0) // Remove the scanned tree entirely; the analysis must be
// unaffected because it reads the database alone.
err := os.RemoveAll(dir)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
// Runs before t.TempDir's own cleanup, which must delete the files. assertSmokeDupeGroups(t, dir, db)
t.Cleanup(func() { assertSmokeTreeGroups(t, dir, db)
//nolint:gosec // removing the directory needs its search bit back
_ = os.Chmod(filepath.Dir(locked), 0o700)
})
cases := map[string]bool{
dir: true,
file: true,
locked: true,
filepath.Join(dir, "gone"): false,
filepath.Join(file, "under-a-file"): false,
}
for path, want := range cases {
if got := pathExists(path); got != want {
t.Errorf("pathExists(%q) = %v, want %v", path, got, want)
}
}
} }
func TestUnderRoot(t *testing.T) { func TestUnderRoot(t *testing.T) {
+13 -26
View File
@@ -38,11 +38,10 @@ func treesPhase(name string) string {
// runTrees implements the trees subcommand: it reads every record from // runTrees implements the trees subcommand: it reads every record from
// the database in path order, reconstructs the directory hierarchy from // the database in path order, reconstructs the directory hierarchy from
// the record paths, computes a Merkle-style digest per directory, and // the record paths, computes a Merkle-style digest per directory, and
// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned // prints maximal duplicate-tree groups as TSV on stdout. It never
// filesystem it only checks, with pathExists, that the root of each // touches the scanned filesystem; its only I/O is the database, stdout,
// duplicate tree is still there, never the files below it. It names the // and stderr. It names the database on stderr before it opens it. Any
// database on stderr before it opens it. Any database problem, // database problem, including a missing database, is fatal.
// including a missing database, is fatal.
// //
// The work runs in four phases, each under a progress display of its // The work runs in four phases, each under a progress display of its
// own on stderr: open (open the database), read (read the number of // own on stderr: open (open the database), read (read the number of
@@ -87,13 +86,13 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
return fmt.Errorf("database %s: %w", dbPath, err) return fmt.Errorf("database %s: %w", dbPath, err)
} }
// Grouping works in memory, then checks the root of each duplicate // Grouping works in memory, so it waits on nothing but the
// tree on disk, so it waits on the processor or the disk. // processor.
prog = newProgress(progressParams{ prog = newProgress(progressParams{
label: treesPhase("group"), unit: "directories", label: treesPhase("group"), unit: "directories",
total: int64(len(allDirs)), stalledOn: "cpu/disk", total: int64(len(allDirs)), stalledOn: "cpu",
}) })
dupes := collectTreeGroups(allDirs, super, pathExists, prog) dupes := collectTreeGroups(allDirs, super, prog)
prog.finish() prog.finish()
@@ -301,14 +300,12 @@ func fileEntry(name string, r scanRec) string {
// collectTreeGroups groups directories by digest, counting each one on // collectTreeGroups groups directories by digest, counting each one on
// prog as it is grouped, and returns every maximal group with two or // prog as it is grouped, and returns every maximal group with two or
// more members for which exists returns true, each group's members // more members, each group's members sorted by path, groups ordered by
// sorted by path, groups ordered by tree size descending then by first // tree size descending then by first path ascending. A group whose
// path ascending. A group whose trees hold 0 bytes is left out, since // trees hold 0 bytes is left out, since removing one frees nothing. The
// removing one frees nothing. The groups it implies hold 0 bytes too, // groups it implies hold 0 bytes too, so leaving it out hides nothing.
// so leaving it out hides nothing. exists is called only on the members
// of maximal groups.
func collectTreeGroups(allDirs []*treeNode, super *treeNode, func collectTreeGroups(allDirs []*treeNode, super *treeNode,
exists func(path string) bool, prog *progress, prog *progress,
) [][]*treeNode { ) [][]*treeNode {
groups := make(map[[sha256.Size]byte][]*treeNode) groups := make(map[[sha256.Size]byte][]*treeNode)
for _, d := range allDirs { for _, d := range allDirs {
@@ -326,16 +323,6 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
continue continue
} }
// A member that is gone leaves the group. A suppressed group is
// still implied: its members still on disk have parents still
// on disk, and those parents' group is checked the same way.
g = slices.DeleteFunc(g, func(n *treeNode) bool {
return !exists(n.path)
})
if len(g) < minGroupSize {
continue
}
slices.SortFunc(g, func(a, b *treeNode) int { slices.SortFunc(g, func(a, b *treeNode) int {
return strings.Compare(a.path, b.path) return strings.Compare(a.path, b.path)
}) })
+16 -116
View File
@@ -6,7 +6,6 @@ import (
"database/sql" "database/sql"
"os" "os"
"slices" "slices"
"strconv"
"strings" "strings"
"testing" "testing"
"time" "time"
@@ -180,7 +179,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
t.Fatalf("directory paths = %q, want %q", got, want) t.Fatalf("directory paths = %q, want %q", got, want)
} }
groups := collectTreeGroups(dirs, super, everyPathExists, nil) groups := collectTreeGroups(dirs, super, nil)
gotGroups := groupPaths(groups) gotGroups := groupPaths(groups)
wantGroups := [][]string{{"/a", "/c"}} wantGroups := [][]string{{"/a", "/c"}}
@@ -196,8 +195,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
} }
func TestRunTreesEscapesPaths(t *testing.T) { func TestRunTreesEscapesPaths(t *testing.T) {
dir, recs := recsOnDisk(t, awkwardPairRecs()) t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
t.Setenv(databaseEnv, seedDatabase(t, recs))
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -208,7 +206,7 @@ func TestRunTreesEscapesPaths(t *testing.T) {
} }
want := "first\tdupe\tfiles\tsize\tdisk_size\n" + want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
dir + `/d/\tone\ntwo\rthree\\four` + "\t" + dir + "/d/A\t1\t5\t4096\n" `/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\t4096\n"
if got := stdout.String(); got != want { if got := stdout.String(); got != want {
t.Errorf("stdout = %q, want %q", got, want) t.Errorf("stdout = %q, want %q", got, want)
} }
@@ -226,11 +224,9 @@ func TestRunTreesDiskSize(t *testing.T) {
{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"}, {size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
} }
dir, recs := recsOnDisk(t, recs)
t.Setenv(databaseEnv, seedDatabase(t, recs)) t.Setenv(databaseEnv, seedDatabase(t, recs))
want := "first\tdupe\tfiles\tsize\tdisk_size\n" + want := "first\tdupe\tfiles\tsize\tdisk_size\n/a\t/b\t2\t30\t1536\n"
dir + "/a\t" + dir + "/b\t2\t30\t1536\n"
if got := runStdout(t, cmdTrees); got != want { if got := runStdout(t, cmdTrees); got != want {
t.Errorf("stdout = %q, want %q", got, want) t.Errorf("stdout = %q, want %q", got, want)
} }
@@ -241,11 +237,10 @@ func TestRunTreesDiskSize(t *testing.T) {
// there. After the line naming the database, each phase prints a line // there. After the line naming the database, each phase prints a line
// naming it with its place among the four as it starts and another as // naming it with its place among the four as it starts and another as
// it ends. read counts the 6 records before it reads them, group counts // it ends. read counts the 6 records before it reads them, group counts
// the directories, and write the 1 group. The summary comes last, and // the 8 directories, and write the 1 group. The summary comes last, and
// stdout holds the report alone. // stdout holds the report alone.
func TestRunTreesProgressNamesEachPhase(t *testing.T) { func TestRunTreesProgressNamesEachPhase(t *testing.T) {
dir, recs := recsOnDisk(t, smokeTreeRecs()) path := seedDatabase(t, smokeTreeRecs())
path := seedDatabase(t, recs)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
stderr := captureStderr(t) stderr := captureStderr(t)
@@ -259,15 +254,11 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
} }
report := "first\tdupe\tfiles\tsize\tdisk_size\n" + report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
dir + "/d/t1\t" + dir + "/d/t2\t2\t3100\t8192\n" "/d/t1\t/d/t2\t2\t3100\t8192\n"
if got := stdout.String(); got != report { if got := stdout.String(); got != report {
t.Errorf("stdout = %q, want %q", got, report) t.Errorf("stdout = %q, want %q", got, report)
} }
// The smoke-test records name 8 directories, / included. Moved under
// dir, they name one more for each slash in dir.
dirs := strconv.Itoa(8 + strings.Count(dir, "/"))
// How each line starts; the rate at the end of a phase varies. // How each line starts; the rate at the end of a phase varies.
want := []string{ want := []string{
"trees: database " + path + " (from " + databaseEnv + ")", "trees: database " + path + " (from " + databaseEnv + ")",
@@ -275,9 +266,8 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
treesOpenLine, treesOpenLine,
"[2/4] read: counting the records, elapsed 0s, remain unknown", "[2/4] read: counting the records, elapsed 0s, remain unknown",
"[2/4] read: [6/6] 100%, ", "[2/4] read: [6/6] 100%, ",
"[3/4] group: [0/" + dirs + "] 0%, 0 directories/s, elapsed 0s, " + "[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, remain unknown",
"remain unknown", "[3/4] group: [8/8] 100%, ",
"[3/4] group: [" + dirs + "/" + dirs + "] 100%, ",
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown", "[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown",
"[4/4] write: [1/1] 100%, ", "[4/4] write: [1/1] 100%, ",
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ", "trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
@@ -306,7 +296,7 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
func TestWriteTreeGroupsOnTerminal(t *testing.T) { func TestWriteTreeGroupsOnTerminal(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
super, dirs := treeOf(t, smokeTreeRecs()) super, dirs := treeOf(t, smokeTreeRecs())
dupes := collectTreeGroups(dirs, super, everyPathExists, nil) dupes := collectTreeGroups(dirs, super, nil)
// No timed draws: the display is drawn as it starts and as it ends. // No timed draws: the display is drawn as it starts and as it ends.
p := startProgress(progressParams{ p := startProgress(progressParams{
@@ -387,7 +377,7 @@ func TestCollectTreeGroupsMaximal(t *testing.T) {
super, dirs := treeOf(t, smokeTreeRecs()) super, dirs := treeOf(t, smokeTreeRecs())
groups := collectTreeGroups(dirs, super, everyPathExists, nil) groups := collectTreeGroups(dirs, super, nil)
want := [][]string{{"/d/t1", "/d/t2"}} want := [][]string{{"/d/t1", "/d/t2"}}
if got := groupPaths(groups); !slices.EqualFunc(got, want, if got := groupPaths(groups); !slices.EqualFunc(got, want,
@@ -411,16 +401,14 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
super, dirs := treeOf(t, recs) super, dirs := treeOf(t, recs)
forward := groupPaths(collectTreeGroups(dirs, super, everyPathExists, forward := groupPaths(collectTreeGroups(dirs, super, nil))
nil))
reversed := slices.Clone(recs) reversed := slices.Clone(recs)
slices.Reverse(reversed) slices.Reverse(reversed)
superR, dirsR := treeOf(t, reversed) superR, dirsR := treeOf(t, reversed)
backward := groupPaths(collectTreeGroups(dirsR, superR, everyPathExists, backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
nil))
if !slices.EqualFunc(forward, backward, slices.Equal) { if !slices.EqualFunc(forward, backward, slices.Equal) {
t.Fatalf("output depends on record order: %v vs %v", t.Fatalf("output depends on record order: %v vs %v",
forward, backward) forward, backward)
@@ -440,7 +428,7 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
super, dirs := treeOf(t, recs) super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil)) got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/x1", "/p/x2"}} want := [][]string{{"/p/x1", "/p/x2"}}
if !slices.EqualFunc(got, want, slices.Equal) { if !slices.EqualFunc(got, want, slices.Equal) {
@@ -469,7 +457,7 @@ func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
super, dirs := treeOf(t, recs) super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil)) got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/x1", "/p/x2"}} want := [][]string{{"/p/x1", "/p/x2"}}
if !slices.EqualFunc(got, want, slices.Equal) { if !slices.EqualFunc(got, want, slices.Equal) {
@@ -492,98 +480,10 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
super, dirs := treeOf(t, recs) super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil)) got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/a/x", "/q/b/x"}} want := [][]string{{"/p/a/x", "/q/b/x"}}
if !slices.EqualFunc(got, want, slices.Equal) { if !slices.EqualFunc(got, want, slices.Equal) {
t.Fatalf("groups = %v, want %v", got, want) t.Fatalf("groups = %v, want %v", got, want)
} }
} }
// TestCollectTreeGroupsChecksOnlyReportedCopies checks that trees asks
// whether a directory is on disk only for the members of the groups it
// returns. It never asks about /a/proj/sub, which lies inside a copy
// and whose group the group of /a/proj implies.
func TestCollectTreeGroupsChecksOnlyReportedCopies(t *testing.T) {
t.Parallel()
f, g, u := testHash("f"), testHash("g"), testHash("u")
recs := []scanRec{
{size: 10, head: f, tail: f, content: f, path: "/a/proj/f"},
{size: 20, head: g, tail: g, content: g, path: "/a/proj/sub/g"},
{size: 10, head: f, tail: f, content: f, path: "/b/proj/f"},
{size: 20, head: g, tail: g, content: g, path: "/b/proj/sub/g"},
// /a lacks this file, so /a and /b are no copies of each other.
{size: 30, head: u, tail: u, content: u, path: "/b/lonely"},
}
super, dirs := treeOf(t, recs)
var checked []string
exists := func(path string) bool {
checked = append(checked, path)
return true
}
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
want := [][]string{{"/a/proj", "/b/proj"}}
if !slices.EqualFunc(got, want, slices.Equal) {
t.Fatalf("groups = %v, want %v", got, want)
}
slices.Sort(checked)
if !slices.Equal(checked, want[0]) {
t.Errorf("directories checked on disk = %q, want %q",
checked, want[0])
}
}
// TestCollectTreeGroupsDropsGroupLeftWithOneCopy checks that a group of
// two copies, one of them gone, is not returned. If kept, it would
// write no row, but the summary line would count it.
func TestCollectTreeGroupsDropsGroupLeftWithOneCopy(t *testing.T) {
t.Parallel()
f := testHash("f")
recs := []scanRec{
{size: 10, head: f, tail: f, content: f, path: "/c/kept/f"},
{size: 10, head: f, tail: f, content: f, path: "/c/gone/f"},
}
super, dirs := treeOf(t, recs)
exists := func(path string) bool { return path != "/c/gone" }
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
if len(got) != 0 {
t.Errorf("groups = %v, want none", got)
}
}
// TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies checks that a group
// of three copies, one of them gone, is returned with the other two.
func TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies(t *testing.T) {
t.Parallel()
f := testHash("f")
recs := []scanRec{
{size: 10, head: f, tail: f, content: f, path: "/d/kept1/f"},
{size: 10, head: f, tail: f, content: f, path: "/d/gone/f"},
{size: 10, head: f, tail: f, content: f, path: "/d/kept2/f"},
}
super, dirs := treeOf(t, recs)
exists := func(path string) bool { return path != "/d/gone" }
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
want := [][]string{{"/d/kept1", "/d/kept2"}}
if !slices.EqualFunc(got, want, slices.Equal) {
t.Errorf("groups = %v, want %v", got, want)
}
}