Merge pull request 'next into main: the command moves to cmd/sfdupes' (#159) from next into main

Reviewed-on: #159

README.md

@@ -21,7 +21,7 @@ file takes too long. It has five subcommands. `scan` walks the filesystem and
 records the files it finds in an SQLite database. SQLite keeps a whole database
 in one file. The database is kept between runs. A rescan reads only new and
 changed files. That makes `scan` cheap enough to run every day from `cron`, the
-system's job scheduler. `report`, `trees` and `info` work from the database and
+system's job scheduler. `report`, `trees` and `info` read the database alone and
 can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
 first look at a new machine with one command.
 
@@ -98,15 +98,13 @@ Each `-v` flag below makes a bind mount, which shows a host directory at a path
 inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between
 runs. Mount the tree to scan at the same path inside the container as on the
 host. Read-only access is enough. The database then records the host's paths,
-and so do the reports. `report` and `trees` need that mount too. Without it,
+and so do the reports.
-every recorded path counts as gone, and each report holds only its header line.
 
 ```sh
 make docker
 docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
     sfdupes scan /srv
-docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
+docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
-    sfdupes report > dupes.tsv
 ```
 
 ### Daily scan from `cron`
@@ -161,10 +159,8 @@ to keep is your choice.
 
 A row is a candidate, not proof:
 
-- The reports show each file as it was when a scan last recorded it. They check
+- The reports show each file as it was when a scan last recorded it. A file may
-  only that each copy still exists: the file, or the directory at the top of a
+  have changed or gone since.
-  tree. A file may have changed since, and a file inside a tree may be gone. Run
-  `scan` again to bring the reports up to date.
 - A file of 100 MiB or more is compared only on samples of its content (see
   "Duplicate detection").
 - Two paths that are hard links, two names for one file, are listed as
@@ -212,9 +208,8 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
 of reads, at the price of a comparison that is not exact.
 
 The hashes live in a database that is kept between runs. A rescan reads only new
-files, and files whose size or modification time (mtime) differs from their
+files, and files whose size changed or whose modification time (mtime) is newer
-record. All analysis then works from the database. The reports only check that
+than recorded. All analysis then works from the database alone.
-each copy they name still exists.
 
 The goal is whole duplicated trees, not only single files. A download extracted
 twice, or a project copied once too often, is such a tree. It can be removed as
@@ -240,10 +235,9 @@ In order:
    the files of the directories it is reading.
 3. **Rescan cheaply, analyze from the database.** The scan keeps its database
    between runs, and an unchanged file is never read again. `report`, `trees`
-   and `info` work from the database and never read a scanned file. `report` and
+   and `info` work from the database alone and never touch the scanned
-   `trees` check only that each copy they name still exists, and leave out a
+   filesystem. `scan` is built to run from `cron`. The reports run at any time,
-   copy that is gone. `scan` is built to run from `cron`. The reports run at any
+   against what the database holds.
-   time, against what the database holds.
 4. **Keep the streams apart.** Everything on stdout is machine-readable data.
    Everything for a human reader, such as progress and warnings, goes to stderr.
 
@@ -258,7 +252,7 @@ In order:
   report.
 
 ```
-sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
+sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
 sfdupes report > dupes.tsv
 sfdupes trees > dupetrees.tsv
 sfdupes info
@@ -303,24 +297,12 @@ The reference below is also the program's specification.
 - `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
   Those are the system and temporary directories the walk otherwise leaves out,
   such as `/var/tmp` on Linux (see "Rules for the walk").
-- `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
-  it. `PATH` may be a directory or a file. The flag may be given more than once.
-  The walk knows `PATH` by its device ID and inode number, as it knows the
-  database files (see "Rules for the walk"). So the walk leaves `PATH` out
-  whatever path leads to it. That includes a hard link to a file given as
-  `PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in
-  `--exclude ""`, is a usage error, reported in one line on stderr with exit 2
-  before anything is scanned. A `PATH` that does not exist, or whose real path
-  `scan` cannot find, is a fatal error. So is an operand at or below one (see
-  "Operands").
 - `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
   exits 0. It writes nothing to stderr.
 - `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
-  and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
+  and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the
-  names the database. The help of `report`, `trees`, `info` and `dwim` says what
+  default excludes of the system it runs on, and the filesystem types the walk
-  each prints. `sfdupes scan --help` also lists the default excludes of the
+  never records anything on.
-  system it runs on, and the filesystem types the walk never records anything
-  on.
 
 ### Duplicate detection
 
@@ -373,10 +355,6 @@ default. An operand may be a directory or a regular file.
 - An operand that does not exist is a fatal error, and so is a symlink whose
   target does not exist. `scan` then stops before it walks anything, and exits
   with code 1.
-- An operand at or below an `--exclude` path is a fatal error too. The message
-  names the operand and the `--exclude` path. `scan` compares the operand and
-  each directory above it with the `--exclude` paths by device ID and inode
-  number, not by name.
 - Each operand is resolved to its real path before the walk, as `realpath`
   prints it. A relative operand is joined to the working directory as `pwd -P`
   prints it. Every symlink in the path is then followed. A `..` after a symlink
@@ -432,16 +410,17 @@ operands:
 
 - A file not yet in the database is hashed and inserted.
 - A file already in the database is not read again when its size and mtime match
-  its record. This is what makes a daily rescan cheap.
+  its record. The mtime matches when it is not newer than the recorded one. This
+  is what makes a daily rescan cheap.
 - When such a file's size on disk differs from its recorded `disk_size`, `scan`
   updates `disk_size` alone. It still does not read the file, and counts it as
   unchanged. ZFS, for example, counts a new file's blocks up to seconds after
   the write. A block is the unit in which a filesystem allocates disk space. A
   file scanned within those seconds gets its real `disk_size` at the next scan.
-- `scan` hashes a file again when its size or its mtime differs from its record.
+- `scan` hashes a file again when its size differs or its mtime is newer. The
-  An older mtime counts as much as a newer one. A file put back by `cp -p`,
+  mtime is compared to the nanosecond, as finely as the filesystem records it. A
-  `rsync -a` or a restore from backup often has an older mtime. The mtime is
+  same-size rewrite counts as a change whenever the filesystem gives it a later
-  compared to the nanosecond, as finely as the filesystem records it.
+  mtime, even within the same second.
 - The scan deletes each record under a scanned operand whose file it could not
   verify. That covers deleted files, and paths the scan could not stat or read
   this run. To stat a file is to read its details, such as its size, without
@@ -452,16 +431,15 @@ operands:
   scanned on different schedules into one database.
 - The scan keeps every record at or below a path that the walk leaves out for
   one of these reasons:
-    - it is an `--exclude` path;
     - it is a default exclude;
     - it is not on a persistent filesystem;
     - with `-x`, it is a directory on another filesystem than its operand.
 
     Such records count as outside the scanned operands. So with `-x`, a
     filesystem mounted under an operand can be scanned on its own schedule into
-    the same database. An operand may itself lie at or inside such a path, other
+    the same database. An operand may itself lie at or inside such a path, as
-    than an `--exclude` path (see "Operands"). Its walk verifies the records
+    "Operands" describes. Its walk verifies the records below it, so they are
-    below it, so they are deleted when their files are gone.
+    deleted when their files are gone.
 
 #### Phases
 
@@ -560,17 +538,6 @@ walk practical on large or busy storage pools.
     The path is not counted as skipped. The records below it are kept, as
     described under "What a scan does to the database".
 
-- The walk never enters or records an `--exclude` path, nor anything below it.
-  For each one it reaches, `scan` prints a line on stderr naming it:
-
-    ```
-    scan: skipped /srv/cache (--exclude)
-    ```
-
-    The path is not counted as skipped. The scan keeps the records at and below
-    it, as described under "What a scan does to the database".
-    `--no-default-excludes` does not change this rule.
-
 - Below an operand, the walk does not enter a default exclude. On Linux, the
   default excludes hold temporary files and the kernel's view of the running
   system. They depend on the system, and `sfdupes scan --help` lists those of
@@ -645,8 +612,7 @@ from a thread of its own and never blocks the workers.
   it repeats another or lies under another;
 - the progress display of each phase (see "Progress"). `scan` prints each
   warning for a skipped path as it happens. It prints a line the same way for
-  each path left out as an `--exclude` path, as a default exclude or as not on a
+  each path left out as a default exclude or as not on a persistent filesystem;
-  persistent filesystem;
 - the summary line;
 - a warning when a report still has the database open as the scan ends (see
   "Concurrent reads").
@@ -689,29 +655,20 @@ and `scan` leaves it ignored.
 
 `report` reads every record from the database. It takes no positional arguments.
 A positional argument is anything on the command line that is not a flag, such
-as the paths `scan` takes. `report` never opens a scanned file. It checks that
+as the paths `scan` takes. `report` never touches the scanned filesystem. It
-each path in a group still exists, and looks at no other path. The check is
+neither stats nor opens any path in the records. It reads the database and
-`lstat`, the system call that looks up a path without following a symlink:
+writes to stdout and stderr, and nothing else. Its output is the same whether or
-
+not the scanned filesystem is still mounted.
-- A path is gone when `lstat` finds that it, or a directory above it, does not
-  exist.
-- Any other error, such as a permission error, counts as present. So a user who
-  may not look inside a directory still sees its copies.
-
-The check does not notice a file that changed after the scan. Run `scan` again
-to bring the report up to date.
 
 Processing:
 
 - Records are grouped by signature, the four values
   `(size, head, tail, content)`.
-- In a group with two or more paths, a path that is gone is left out. A group
+- Every group with two or more paths is a duplicate group.
-  that still has two or more paths is a duplicate group.
-- Empty files form no group, because removing one frees nothing.
 - Within a group, paths are sorted in byte order. The first is `first`, and
   every other path is a `dupe`.
 - Groups are ordered by size, largest first, then by `first` path. The output is
-  the same for the same database contents and the same paths on disk.
+  the same for the same database contents.
 
 `report` has two phases, each with its own progress display on stderr (see
 "Progress"). `open` opens the database. `read` gets the number of records from
@@ -756,7 +713,7 @@ The summary line on stderr gives:
 - the reclaimable bytes, the sum of `size` over all rows.
 
 ```
-report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
+report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable
 ```
 
 ### `trees` mode
@@ -764,9 +721,8 @@ report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
 `trees` reads the same database as `report` and takes no positional arguments.
 It reports whole duplicate directory trees. Two directories are duplicate trees
 when they hold the same relative paths, and each path holds a file with the same
-signature in both. It rebuilds the directory tree from the paths in the records,
+signature in both. Like `report`, it never touches the scanned filesystem. It
-split on `/`. Like `report`, it never opens a scanned file. It checks only that
+rebuilds the directory tree from the paths in the records, split on `/`.
-the directory at the top of each copy still exists, as `report` checks a path.
 
 Definitions:
 
@@ -785,21 +741,15 @@ Processing:
 - `trees` builds the directory tree and computes every directory's digest. It
   then groups the directories by digest. Every group with two or more
   directories is a duplicate-tree group.
-- `trees` leaves out a group whose trees hold 0 bytes, because removing one
-  frees nothing. An empty file still counts toward the digest of a tree that
-  holds other files.
 - `trees` leaves out a group that a group of its members' parents already
   implies. A group is left out when its members' parents are distinct
   directories that all have the same digest. The parents' group, or an
   ancestor's, already implies it. A group is always reported when it holds
   sibling directories, or when its members' parents have different digests.
-- A directory that is gone is dropped from its group. A group left with fewer
-  than two directories is not reported.
 - Within a group, paths are sorted in byte order. The first is `first`, and
   every other path is a `dupe`.
 - Groups are ordered by total tree size, largest first, then by `first` path.
-  The output is the same for the same database contents and the same paths on
+  The output is the same for the same database contents.
-  disk.
 
 Two limitations are accepted:
 
@@ -813,9 +763,8 @@ Two limitations are accepted:
 `trees` has four phases, each with its own progress display on stderr (see
 "Progress"). `open` opens the database. `read` gets the number of records from
 the `totals` table, then reads them in path order. It builds the directory tree
-and computes the digests as it goes. `group` groups the directories by digest,
+and computes the digests as it goes. `group` groups the directories by digest
-picks the groups to report and checks that their directories still exist.
+and picks the groups to report. `write` writes them to stdout.
-`write` writes the groups to stdout.
 
 #### Trees output format
 
@@ -873,8 +822,8 @@ The report goes to stdout, exactly as `trees` prints it. On stderr comes what
 dwim: run "sfdupes trees" to print this report again
 ```
 
-The database keeps the scan, so `sfdupes trees` prints the report again later,
+The database keeps the scan, so `sfdupes trees` prints the same report later,
-even after the terminal is gone. It then leaves out a copy deleted in between.
+even after the terminal is gone.
 
 `SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
 `dwim` then prints no report and exits 1. A fatal scan error also ends it before
@@ -1166,10 +1115,9 @@ in, and a phase that may wait in two places names both, as `disk/db`:
 - `hash` says `disk/db`, because it reads files and writes their records to the
   database between them.
 - `load`, `update` and the `read` phase of `trees` say `db`.
-- The `read` phase of `report` says `db/disk/out`. It checks that the paths of
+- The `read` phase of `report` says `db/out`, because it writes the report to
-  each group still exist, and writes the report to stdout, while it reads.
+  stdout while it reads.
-- `group` of `trees` says `cpu/disk`, for work in memory and the check that each
+- `group` of `trees` says `cpu`, for work in memory, and its `write` says `out`.
-  copy still exists. Its `write` says `out`.
 - A phase still waiting for its total says what it waits for, and nothing more.
 
 On a terminal:
@@ -1227,9 +1175,6 @@ showing the count it stopped at.
 - `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
   stopped by either during its scan. A fatal error is one of these:
     - an operand that does not exist, or whose real path `scan` cannot find;
-    - an `--exclude` path that does not exist, or whose real path `scan` cannot
-      find;
-    - an operand at or below an `--exclude` path;
     - a list of mounted filesystems that `scan` cannot read;
     - another `scan` holding the lock (see "The scan lock");
     - a database, its directory or its lock file that cannot be created or
@@ -1249,7 +1194,6 @@ showing the count it stopped at.
       `--workers abc`;
     - `scan` with no operand;
     - `--workers` below 1;
-    - an empty `--exclude` value, as in `--exclude ""`;
     - `report`, `trees`, `info` or `dwim` with a positional argument.
 
 A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
@@ -1413,8 +1357,8 @@ commands of their own:
   not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
   cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
   exception, because the race detector needs cgo (see `script/test`).
-- `report` and `trees` give the same output for the same database contents and
+- `report` and `trees` give the same output for the same database contents,
-  the same paths on disk, whatever order the records were inserted in.
+  whatever order the records were inserted in.
 
 ### Definition of done
 
@@ -1438,7 +1382,7 @@ All of the following, run in this directory, must pass:
     head -c 2000 /dev/urandom > "$d/a/unique.bin"     # same size, different content
     printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2"  # 1-byte duplicates
     printf 'y' > "$d/tiny3"                           # 1-byte non-duplicate
-    : > "$d/empty1"; : > "$d/empty2"                  # empty files
+    : > "$d/empty1"; : > "$d/empty2"                  # empty duplicates
     # duplicate trees: t1 and t2 are identical; t3 differs by one filename
     mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
     head -c 3000 /dev/urandom > "$d/t1/f1"
@@ -1463,8 +1407,8 @@ All of the following, run in this directory, must pass:
       is the path that sorts first.
     - `t1/f1`, `t2/f1` and `t3/f1` form one group.
     - `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
-    - `tiny1` and `tiny2` form a pair.
+    - `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`.
-    - `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
+    - `unique.bin` and `tiny3` appear nowhere.
     - The groups come in order of size, largest first.
 
     `trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
@@ -1479,7 +1423,7 @@ All of the following, run in this directory, must pass:
 
     The test suite automates this scenario (see
     `internal/sfdupes/scan_test.go`). It also checks that `report` and `trees`
-    leave out a copy removed after the scan.
+    never touch the scanned filesystem.
 
 ## TODO
 

TODO.md

@@ -29,28 +29,6 @@
 Newest first. An entry ends with its date and its issue or branch, where those
 are known.
 
-- The help of `sfdupes` and of `scan` names the database, its default path and
-  `SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
-  order of the rows and the escapes. The help of `info` lists its lines. The
-  help of `dwim` says that it scans `/`, should run as root and keeps the scan
-  in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
-- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
-  enters nor records an excluded path, and keeps the records below it. An
-  operand at or below one is a fatal error (2026-11-11,
-  https://git.eeqj.de/sneak/sfdupes/issues/163)
-
-- `report` and `trees` leave out a copy that was deleted after the scan. Each
-  checks only the copy itself with `lstat`: the file, or the directory at the
-  top of a tree. A group left with one copy is not reported. A missing `scan`
-  operand is still a fatal error (2026-11-11,
-  https://git.eeqj.de/sneak/sfdupes/issues/173)
-- The usage line of `scan` no longer ends in `[flags]`. It is the line
-  `README.md` shows under "Subcommands" (2026-11-11,
-  https://git.eeqj.de/sneak/sfdupes/issues/169)
-- `report` and `trees` leave out empty files. `report` writes no group of size
-  0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
-  counts toward a tree's digest (2026-11-11,
-  https://git.eeqj.de/sneak/sfdupes/issues/178)
 - The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
   command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
   became `internal/sfdupes/cli.go`. Every other file kept its name. The install

internal/sfdupes/cancel_test.go

@@ -359,7 +359,7 @@ func TestRunScanInterrupted(t *testing.T) {
 	dir := buildSmokeTree(t)
 
 	err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
-		false, nil)
+		false)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -383,7 +383,7 @@ func TestRunScanInterrupted(t *testing.T) {
 		defer close(done)
 
 		err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
-			false, false, nil)
+			false, false)
 	}()
 
 	awaitReturn(t, done, "runScan")
@@ -411,7 +411,7 @@ func TestRunScanInterrupted(t *testing.T) {
 	}
 
 	err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
-		false, nil)
+		false)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -437,7 +437,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
 	dir := buildWalkCancelTree(t)
 
 	err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
-		walkCancelWorkers, false, false, nil)
+		walkCancelWorkers, false, false)
 	if !errors.Is(err, errInterrupted) {
 		t.Fatalf("runScan interrupted mid-hash = %v, want %v",
 			err, errInterrupted)

internal/sfdupes/cli.go

@@ -10,7 +10,6 @@ import (
 	"os"
 	"runtime"
 	"runtime/debug"
-	"slices"
 
 	"github.com/spf13/cobra"
 )
@@ -32,69 +31,6 @@ const (
 	cmdDwim   = "dwim"
 )
 
-// databaseHelp is the paragraph of the root and scan help that names
-// the database (README "Location").
-const databaseHelp = "scan and dwim store what they find in a database " +
-	"file. report, trees\nand info read it. The file is $" + databaseEnv +
-	" when that is set and\nnot empty, and otherwise " +
-	defaultDatabasePath + ", where creating it\nneeds root.\n"
-
-// escapesHelp is the paragraph of the report and trees help on how
-// paths are escaped (README "Report output format").
-const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
-
-  \\  a backslash
-  \t  a tab
-  \n  a newline
-  \r  a carriage return
-`
-
-// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
-// summary in the help of their subcommands.
-const (
-	reportHelp = `The report is tab-separated text on stdout: a header line, then one
-row per duplicate file. The columns are:
-
-  first   the path that sorts first, byte by byte, in its group of copies
-  dupe    another copy of that file
-  size    the file's size in bytes
-
-Rows are ordered by size, largest first, then by path.
-
-` + escapesHelp
-
-	treesHelp = `The report is tab-separated text on stdout: a header line, then one
-row per duplicate directory tree. The columns are:
-
-  first      the path that sorts first, byte by byte, in its group of copies
-  dupe       another copy of that tree
-  files      the number of regular files in one copy
-  size       their total size in bytes
-  disk_size  the bytes the dupe copy takes on disk
-
-Rows are ordered by size, largest first, then by path.
-
-` + escapesHelp
-
-	infoHelp = `The details go to stdout, one per line. Each line holds a name and its
-value, separated by a tab:
-
-  database                the path, and where it came from
-  database file size      in bytes, then rounded, as 2147483648 (2.0 GiB)
-  -wal file size          in bytes, then rounded; only if a -wal file exists
-  modified                when the database file last changed, in UTC
-  schema version          the version of the database's layout
-  records                 the number of records
-  size of recorded files  their total size, in bytes, then rounded
-`
-
-	dwimHelp = `dwim runs what "sfdupes scan /" runs, then what "sfdupes trees" runs.
-Run it as root, so that the scan can read every file. The scan is stored
-in the database, as scan stores it, so "sfdupes trees" prints the report
-again later.
-`
-)
-
 // errNoSubcommand is returned by the root command when it is invoked
 // without a subcommand. That is a usage error, and the usage text
 // cobra prints for it is the whole message.
@@ -104,11 +40,6 @@ var errNoSubcommand = errors.New("no subcommand")
 // below 1.
 var errWorkersBelowOne = errors.New("--workers must be at least 1")
 
-// errEmptyExcludeValue is the usage error for an empty scan --exclude
-// value, as from --exclude "$DIR" with DIR unset. Taken as a path, it
-// would name the working directory.
-var errEmptyExcludeValue = errors.New("--exclude must not be empty")
-
 // Version is the build version, injected at link time via -ldflags
 // (see the Makefile); empty when the linker set none, as for a binary
 // built by go install. buildVersion turns it into what --version prints.
@@ -265,55 +196,38 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 		}),
 	}
 
-	root.Long = root.Short + ".\n\n" + databaseHelp
-	reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
-	treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
-	infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
-	dwimCmd.Long = dwimCmd.Short + ".\n\n" + dwimHelp
-
 	root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
 
 	return root
 }
 
-// newScanCommand builds the scan subcommand, with its --workers, -x,
+// newScanCommand builds the scan subcommand, with its --workers, -x and
-// --no-default-excludes and --exclude flags. Its help names the
+// --no-default-excludes flags. Its help lists the default excludes of
-// database, and lists the default excludes of the system it runs on and
+// the system it runs on, and the filesystem types it never records
-// the filesystem types it never records anything on.
+// anything on.
 func newScanCommand() *cobra.Command {
 	var (
 		scanWorkers           int
 		scanOneFS             bool
 		scanNoDefaultExcludes bool
-		scanExclude           []string
 	)
 
 	short := "Walk trees and synchronize the scan database"
 
 	scanCmd := &cobra.Command{
-		Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " +
+		Use:   cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
-			"[--exclude PATH]... PATH...",
+		Short: short,
-		// The Use line already names every flag, so cobra must not
+		Long:  short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
-		// append its generic "[flags]" to it.
+		Args:  cobra.MinimumNArgs(1),
-		DisableFlagsInUseLine: true,
-		Short:                 short,
-		Long: short + ".\n\n" + databaseHelp + "\n" +
-			defaultExcludesHelp(runtime.GOOS),
-		Args: cobra.MinimumNArgs(1),
 		PreRunE: func(cmd *cobra.Command, _ []string) error {
-			err := checkScanWorkers(cmd, scanWorkers)
+			return checkScanWorkers(cmd, scanWorkers)
-			if err != nil {
-				return err
-			}
-
-			return rejectEmptyExcludeValue(cmd, scanExclude)
 		},
 		RunE: runE(func(ctx context.Context, args []string) error {
 			ctx, stop := interruptContext(ctx)
 			defer stop()
 
 			return runScan(ctx, args, scanWorkers, scanOneFS,
-				scanNoDefaultExcludes, scanExclude)
+				scanNoDefaultExcludes)
 		}),
 	}
 	scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
@@ -322,10 +236,6 @@ func newScanCommand() *cobra.Command {
 		"do not cross filesystem boundaries")
 	scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
 		false, "enter the default excludes, the paths listed above")
-	// A string array, not a slice, so a comma in a path is not a
-	// separator.
-	scanCmd.Flags().StringArrayVar(&scanExclude, "exclude", nil,
-		"never enter or record `PATH` or anything below it; repeatable")
 
 	return scanCmd
 }
@@ -343,18 +253,6 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
 	return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
 }
 
-// rejectEmptyExcludeValue rejects an empty scan --exclude value. That
-// is a usage error reported in one line, as for --workers below 1.
-func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
-	if !slices.Contains(excludePaths, "") {
-		return nil
-	}
-
-	cmd.SilenceUsage = true
-
-	return errEmptyExcludeValue
-}
-
 // runE adapts a subcommand implementation, or the version print, to
 // cobra's RunE. Cobra prints the error and the command's usage text for
 // every error RunE returns, but a subcommand that ran and failed has no

internal/sfdupes/cli_test.go

@@ -22,9 +22,6 @@ import (
 // error prints and a runtime failure must not.
 const usageMarker = "Usage:"
 
-// helpFlag asks for the help text.
-const helpFlag = "--help"
-
 // TestMain stops the tests at once, saying why, when their temporary
 // directory is not on a persistent filesystem: scan records nothing
 // there (README "Rules for the walk"), so most scan tests would fail
@@ -573,8 +570,7 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
 	}{
 		cmdScan: {
 			run: func() error {
-				return runScan(t.Context(), []string{missing}, 1, false, false,
+				return runScan(t.Context(), []string{missing}, 1, false, false)
-					nil)
 			},
 			want: scanWant,
 		},
@@ -696,7 +692,7 @@ func TestRunHelp(t *testing.T) {
 
 	// README §Settings: help goes to stderr, exits 0, and leaves stdout
 	// empty.
-	cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
+	cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
 
 	for _, args := range cases {
 		var stdout, stderr bytes.Buffer
@@ -716,112 +712,6 @@ func TestRunHelp(t *testing.T) {
 				args, got)
 		}
 	}
-
-	// The scan help lists --exclude with its argument.
-	var stdout, stderr bytes.Buffer
-
-	run([]string{cmdScan, "-h"}, &stdout, &stderr)
-
-	if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
-		t.Errorf("scan -h stderr = %q, want %s PATH", stderr.String(),
-			excludeFlag)
-	}
-}
-
-func TestRunScanUsageLine(t *testing.T) {
-	t.Parallel()
-
-	// README §Subcommands shows this usage line for scan, and the help
-	// and the usage error both print it.
-	const want = "\n  sfdupes scan [--workers N] [-x] [--no-default-excludes] " +
-		"[--exclude PATH]... PATH...\n"
-
-	cases := map[string][]string{
-		"help":          {cmdScan, "-h"},
-		"missing paths": {cmdScan},
-	}
-
-	for name, args := range cases {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			var stdout, stderr bytes.Buffer
-
-			run(args, &stdout, &stderr)
-
-			if !strings.Contains(stderr.String(), want) {
-				t.Errorf("run(%v) stderr = %q, want the usage line %q",
-					args, stderr.String(), want)
-			}
-		})
-	}
-}
-
-func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
-	t.Parallel()
-
-	// README §Settings: the help names the database, and says what
-	// report, trees, info and dwim print.
-	database := []string{
-		"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
-	}
-	escapes := []string{`\\`, `\t`, `\n`, `\r`}
-
-	// The subcommand whose help is printed; none for the root.
-	cases := []struct {
-		subcommand []string
-		want       []string
-	}{
-		{nil, database},
-		{[]string{cmdScan}, database},
-		{
-			[]string{cmdReport},
-			append([]string{
-				"tab-separated", "\n  first ", "\n  dupe ", "\n  size ",
-				"largest first",
-			}, escapes...),
-		},
-		{
-			[]string{cmdTrees},
-			append([]string{
-				"tab-separated", "\n  first ", "\n  dupe ", "\n  files ",
-				"\n  size ", "\n  disk_size ", "largest first",
-			}, escapes...),
-		},
-		{
-			[]string{cmdInfo},
-			[]string{
-				"\n  database ", "\n  database file size ",
-				"\n  -wal file size ", "\n  modified ",
-				"\n  schema version ", "\n  records ",
-				"\n  size of recorded files ",
-			},
-		},
-		{
-			[]string{cmdDwim},
-			[]string{
-				"sfdupes scan /", "as root", "sfdupes trees", "database",
-			},
-		},
-	}
-
-	for _, tc := range cases {
-		var stdout, stderr bytes.Buffer
-
-		args := slices.Concat(tc.subcommand, []string{helpFlag})
-
-		code := run(args, &stdout, &stderr)
-		if code != exitOK {
-			t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
-		}
-
-		for _, want := range tc.want {
-			if !strings.Contains(stderr.String(), want) {
-				t.Errorf("run(%v) help lacks %q:\n%s",
-					args, want, stderr.String())
-			}
-		}
-	}
 }
 
 func TestRunVersion(t *testing.T) {

internal/sfdupes/dwim.go

@@ -17,8 +17,7 @@ import (
 // ends it at once, as it ends trees.
 func runDwim(ctx context.Context, root string, stdout io.Writer) error {
 	scanCtx, stop := interruptContext(ctx)
-	err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false,
+	err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
-		nil)
 
 	stop()
 

internal/sfdupes/exclude.go

@@ -1,9 +1,8 @@
 package sfdupes
 
-// This file holds three rules for the walk (README "Rules for the
+// This file holds two rules for the walk (README "Rules for the walk"):
-// walk"): the paths given with --exclude, which the scan never enters;
 // the default excludes, the paths a scan's walk does not enter below an
-// operand unless scan is given --no-default-excludes; and the
+// operand unless scan is given --no-default-excludes, and the
 // filesystems that are not persistent, on which the scan never records
 // or reads anything.
 
@@ -13,7 +12,6 @@ import (
 	"fmt"
 	"io"
 	"io/fs"
-	"os"
 	"path/filepath"
 	"slices"
 	"strings"
@@ -45,25 +43,17 @@ const typesPerHelpLine = 6
 // readMountInfo reads.
 var errMountInfoLine = errors.New("unexpected line")
 
-// errExcludedOperand reports an operand at or below a path given with
-// --exclude.
-var errExcludedOperand = errors.New("operand excluded by --exclude")
-
 // walkRules holds the rules for the walk that a scan sets up as it
-// starts. excludePaths are the paths given with --exclude, each with
+// starts. excludes are the default excludes, or none with
-// the device ID and inode number of the file it names (see
+// --no-default-excludes. nonPersistent maps the device ID of every
-// excludePathInodes), which the walk neither enters nor records,
+// mounted filesystem that is not persistent to its type; the scan
-// whatever path leads to them. defaultExcludes are the default
+// records and reads nothing on one of those, whatever the flags.
-// excludes, or none with --no-default-excludes. nonPersistent maps the
+// databaseFiles are the scan's own database files, which the walk
-// device ID of every mounted filesystem that is not persistent to its
+// passes over (see databaseFiles in db.go).
-// type; the scan records and reads nothing on one of those, whatever
-// the flags. databaseFiles are the scan's own database files, which the
-// walk passes over (see databaseFiles in db.go).
 type walkRules struct {
-	excludePaths    []fileRec
+	excludes      []string
-	defaultExcludes []string
+	nonPersistent map[uint64]string
-	nonPersistent   map[uint64]string
+	databaseFiles []fileRec
-	databaseFiles   []fileRec
 }
 
 // defaultExcludes returns the default excludes of the system goos
@@ -107,88 +97,6 @@ func isDefaultExclude(path string, excludes []string) bool {
 	return false
 }
 
-// excludePathInodes returns each of paths, the --exclude values, as its
-// real path (see realPaths) with the device ID and inode number of the
-// file there. The walk compares what it reaches with those, as it does
-// with the database files, so it also leaves out an --exclude path
-// spelled another way, such as in another letter case on macOS.
-func excludePathInodes(paths []string) ([]fileRec, error) {
-	resolved, err := realPaths(paths)
-	if err != nil {
-		return nil, err
-	}
-
-	files := make([]fileRec, 0, len(resolved))
-
-	for _, path := range resolved {
-		fi, err := os.Lstat(path)
-		if err != nil {
-			return nil, err
-		}
-
-		dev, ino := inodeOfInfo(fi)
-		files = append(files, fileRec{path: path, dev: dev, ino: ino})
-	}
-
-	return files, nil
-}
-
-// matchingExcludePath returns the path of the one of excludePaths with
-// the device ID and inode number of fi, an lstat result, or "" when
-// there is none.
-func matchingExcludePath(fi fs.FileInfo, excludePaths []fileRec) string {
-	dev, ino := inodeOfInfo(fi)
-	f := fileRec{dev: dev, ino: ino}
-
-	for _, exclude := range excludePaths {
-		if sameInode(exclude, f) {
-			return exclude.path
-		}
-	}
-
-	return ""
-}
-
-// checkOperandsNotExcluded returns an error when one of roots, the
-// operands, or a directory above it is one of excludePaths, the
-// --exclude paths, by device ID and inode number. The error names the
-// operand and the --exclude path. The roots are absolute, so the
-// directories above each one end at /. A path here that lstat cannot
-// read is an error too.
-func checkOperandsNotExcluded(roots []string, excludePaths []fileRec) error {
-	for _, root := range roots {
-		for path := root; ; path = filepath.Dir(path) {
-			fi, err := os.Lstat(path)
-			if err != nil {
-				return fmt.Errorf("scan %s: %w", root, err)
-			}
-
-			exclude := matchingExcludePath(fi, excludePaths)
-			if exclude != "" {
-				return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand,
-					exclude)
-			}
-
-			if path == "/" {
-				break
-			}
-		}
-	}
-
-	return nil
-}
-
-// excludePathLine returns the line the scan prints for path when fi,
-// its lstat, makes it one of rules.excludePaths, and "" when it does
-// not.
-func (rules walkRules) excludePathLine(path string, fi fs.FileInfo) string {
-	if matchingExcludePath(fi, rules.excludePaths) == "" {
-		return ""
-	}
-
-	return fmt.Sprintf("scan: skipped %s (--exclude)", path)
-}
-
 // nonPersistentFilesystemTypes returns the filesystem types that are
 // not persistent: filesystems that the kernel makes up or that are kept
 // in memory. devfs is the macOS filesystem of /dev; the others are

internal/sfdupes/exclude_test.go

@@ -14,9 +14,6 @@ import (
 	"golang.org/x/sys/unix"
 )
 
-// excludeFlag is the scan flag that names a path the walk never enters.
-const excludeFlag = "--exclude"
-
 func TestDefaultExcludes(t *testing.T) {
 	t.Parallel()
 
@@ -191,7 +188,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
 	unseen := writeFile(t, excluded, "new", pattern(2, 10))
 
 	st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
-		walkRules{defaultExcludes: []string{excluded}})
+		walkRules{excludes: []string{excluded}})
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -237,7 +234,7 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
 			db := openTestDB(t)
 
 			st, err := syncScan(t.Context(), db, roots, 1, false,
-				walkRules{defaultExcludes: []string{excluded}})
+				walkRules{excludes: []string{excluded}})
 			if err != nil {
 				t.Fatal(err)
 			}
@@ -265,7 +262,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
 	f := writeFile(t, excluded, "f", pattern(1, 10))
 
 	st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
-		walkRules{defaultExcludes: []string{excluded}})
+		walkRules{excludes: []string{excluded}})
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -359,217 +356,6 @@ func TestRunScanNoDefaultExcludes(t *testing.T) {
 	assertRecorded(t, path, f, true)
 }
 
-// TestRunScanExcludeNested scans a tree through the command with an
-// --exclude path two levels below the operand. The walk records nothing
-// at or below it and says so. The record an earlier scan left there is
-// kept although its file is gone.
-func TestRunScanExcludeNested(t *testing.T) {
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	dir := realTempDir(t)
-	excluded := filepath.Join(dir, "a", "b")
-	kept := writeFile(t, dir, "a/f", pattern(1, 10))
-	gone := writeFile(t, excluded, "gone", pattern(2, 10))
-
-	scanOK(t, dir)
-
-	err := os.Remove(gone)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	unseen := writeFile(t, excluded, "c/new", pattern(3, 10))
-
-	stderr := scanOK(t, excludeFlag, excluded, dir)
-
-	line := "scan: skipped " + excluded + " (--exclude)\n"
-	if !strings.Contains(stderr, line) {
-		t.Errorf("stderr = %q, want %q", stderr, line)
-	}
-
-	assertRecorded(t, path, kept, true)
-	assertRecorded(t, path, gone, true)
-	assertRecorded(t, path, unseen, false)
-}
-
-// TestRunScanExcludeRepeated gives --exclude twice, once for a
-// directory and once for a file, the file through a symlink to it. The
-// walk records neither, and records the file beside them.
-func TestRunScanExcludeRepeated(t *testing.T) {
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	dir := realTempDir(t)
-	inDir := writeFile(t, dir, "skip/f", pattern(1, 10))
-	file := writeFile(t, dir, "skip-file", pattern(2, 10))
-	kept := writeFile(t, dir, "keep", pattern(3, 10))
-
-	link := filepath.Join(t.TempDir(), "link")
-	symlink(t, file, link)
-
-	scanOK(t, excludeFlag, filepath.Dir(inDir), excludeFlag, link, dir)
-
-	assertRecorded(t, path, inDir, false)
-	assertRecorded(t, path, file, false)
-	assertRecorded(t, path, kept, true)
-}
-
-// TestRunScanExcludeReachedByAnotherPath gives --exclude a directory
-// through a symlink and a file through "..", so the walk reaches
-// neither by the path given. The walk also reaches the file through a
-// hard link. It records none of these, prints a line naming each, and
-// records the file beside them.
-func TestRunScanExcludeReachedByAnotherPath(t *testing.T) {
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	dir := realTempDir(t)
-	inDir := writeFile(t, dir, "a/cache/f", pattern(1, 10))
-	file := writeFile(t, dir, "a/big", pattern(2, 10))
-	kept := writeFile(t, dir, "keep", pattern(3, 10))
-
-	hardLink := filepath.Join(dir, "big-link")
-
-	err := os.Link(file, hardLink)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	link := filepath.Join(t.TempDir(), "link")
-	symlink(t, filepath.Join(dir, "a"), link)
-
-	// Joined as text, since filepath.Join would drop the "..".
-	dotDot := dir + "/a/cache/../big"
-
-	stderr := scanOK(t, excludeFlag, filepath.Join(link, "cache"),
-		excludeFlag, dotDot, dir)
-
-	for _, skipped := range []string{filepath.Dir(inDir), file, hardLink} {
-		line := "scan: skipped " + skipped + " (--exclude)\n"
-		if !strings.Contains(stderr, line) {
-			t.Errorf("stderr = %q, want %q", stderr, line)
-		}
-	}
-
-	assertRecorded(t, path, inDir, false)
-	assertRecorded(t, path, file, false)
-	assertRecorded(t, path, hardLink, false)
-	assertRecorded(t, path, kept, true)
-}
-
-// TestRunScanExcludedOperand checks that an operand at or below an
-// --exclude path is a fatal error that names both, and so is an
-// --exclude path that does not exist. An operand that is a hard link to
-// an --exclude file is at it. Neither scan writes a database.
-func TestRunScanExcludedOperand(t *testing.T) {
-	dir := realTempDir(t)
-	sub := filepath.Join(dir, "sub")
-	f := writeFile(t, sub, "f", pattern(1, 10))
-
-	hardLink := filepath.Join(dir, "f-link")
-
-	err := os.Link(f, hardLink)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	cases := map[string]struct {
-		args []string
-		want string
-	}{
-		"at it": {
-			[]string{excludeFlag, sub, sub},
-			"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
-		},
-		"below it": {
-			[]string{excludeFlag, dir, sub},
-			"scan " + sub + ": operand excluded by --exclude " + dir + "\n",
-		},
-		"after an operand it does not exclude": {
-			[]string{dir, excludeFlag, sub, sub},
-			"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
-		},
-		"a hard link to it": {
-			[]string{excludeFlag, f, hardLink},
-			"scan " + hardLink + ": operand excluded by --exclude " + f + "\n",
-		},
-		"exclude missing": {
-			[]string{excludeFlag, filepath.Join(dir, "nope"), dir},
-			"--exclude: resolve " + filepath.Join(dir, "nope") + ": ",
-		},
-	}
-
-	for name, tc := range cases {
-		t.Run(name, func(t *testing.T) {
-			path := testDBPath(t)
-			t.Setenv(databaseEnv, path)
-
-			stdout := captureStdout(t)
-			stderr := captureStderr(t)
-
-			code := run(append([]string{cmdScan}, tc.args...), os.Stdout,
-				os.Stderr)
-			if code != exitFatal {
-				t.Errorf("run(scan %q) = %d, want %d", tc.args, code,
-					exitFatal)
-			}
-
-			assertFatalOutput(t, stderr(), stdout())
-
-			if !strings.Contains(stderr(), "sfdupes: "+tc.want) {
-				t.Errorf("stderr = %q, want %q", stderr(), tc.want)
-			}
-
-			_, err := os.Lstat(path)
-			if !errors.Is(err, os.ErrNotExist) {
-				t.Errorf("lstat %s: %v, want no database", path, err)
-			}
-		})
-	}
-}
-
-// TestRunScanRejectsEmptyExclude checks that an empty --exclude value,
-// given as its own argument, after "=" or after another --exclude, is
-// a usage error reported in one line before anything is scanned
-// (README "Settings").
-func TestRunScanRejectsEmptyExclude(t *testing.T) {
-	dir := t.TempDir()
-
-	cases := map[string][]string{
-		"separate":     {excludeFlag, "", dir},
-		"joined by =":  {excludeFlag + "=", dir},
-		"after a path": {excludeFlag, t.TempDir(), excludeFlag, "", dir},
-	}
-
-	for name, args := range cases {
-		t.Run(name, func(t *testing.T) {
-			path := testDBPath(t)
-			t.Setenv(databaseEnv, path)
-
-			var stdout, stderr bytes.Buffer
-
-			code := run(append([]string{cmdScan}, args...), &stdout,
-				&stderr)
-			if code != exitUsage {
-				t.Errorf("run(scan %q) = %d, want %d", args, code,
-					exitUsage)
-			}
-
-			want := "Error: --exclude must not be empty\n"
-			if stdout.String() != "" || stderr.String() != want {
-				t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
-					stdout.String(), stderr.String(), want)
-			}
-
-			_, err := os.Lstat(path)
-			if !errors.Is(err, os.ErrNotExist) {
-				t.Errorf("lstat %s: %v, want no database", path, err)
-			}
-		})
-	}
-}
-
 // TestRunScanSkipsNonPersistentOperand scans, through the command, a
 // directory on a filesystem that is not persistent, with and without
 // --no-default-excludes: either way it is skipped, with a line saying

internal/sfdupes/progress_test.go

@@ -642,9 +642,9 @@ func TestTerminalStalledLines(t *testing.T) {
 		{label: scanPhase("hash"), stalledOn: hashStalledOn},
 		{label: scanPhase("update"), stalledOn: "db"},
 		{label: treesPhase("read"), stalledOn: "db"},
-		{label: treesPhase("group"), stalledOn: "cpu/disk"},
+		{label: treesPhase("group"), stalledOn: "cpu"},
 		{label: treesPhase("write"), stalledOn: "out"},
-		{label: reportPhase("read"), stalledOn: "db/disk/out"},
+		{label: reportPhase("read"), stalledOn: "db/out"},
 	}
 
 	for _, params := range phases {

internal/sfdupes/report.go

@@ -4,14 +4,11 @@ import (
 	"context"
 	"crypto/sha256"
 	"database/sql"
-	"errors"
 	"fmt"
 	"io"
-	"io/fs"
 	"os"
 	"slices"
 	"strings"
-	"syscall"
 	"time"
 )
 
@@ -44,10 +41,10 @@ func reportPhase(name string) string {
 }
 
 // runReport implements the report subcommand: it prints the file-level
-// duplicates report as TSV on stdout. Of the scanned filesystem it
+// duplicates report as TSV on stdout. It never touches the scanned
-// only checks, with pathExists, that each path in a duplicate group is
+// filesystem; its only I/O is the database, stdout, and stderr. It
-// still there. It names the database on stderr before it opens it. Any
+// names the database on stderr before it opens it. Any database
-// database problem, including a missing database, is fatal.
+// problem, including a missing database, is fatal.
 //
 // The work runs in two phases, each under a progress display of its
 // own on stderr: open (open the database) and read (read the number of
@@ -80,14 +77,13 @@ func runReport(ctx context.Context, stdout io.Writer) error {
 
 	defer func() { _ = db.Close() }()
 
-	// The read phase checks the paths of each duplicate group on disk
+	// The read phase writes the report to stdout while it reads the
-	// and writes the report to stdout while it reads the records, so it
+	// records, so it may wait on either.
-	// may wait on any of the three.
 	prog = newProgress(progressParams{
 		label: reportPhase("read"), unit: "records", total: -1,
-		waitingFor: countingRecords, stalledOn: "db/disk/out",
+		waitingFor: countingRecords, stalledOn: "db/out",
 	})
-	report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog)
+	report, err := readReport(ctx, db, dbPath, stdout, prog)
 
 	prog.finish()
 
@@ -108,15 +104,14 @@ func runReport(ctx context.Context, stdout io.Writer) error {
 // dbPath, as its total, then reads every record in signature order,
 // counting each on prog, and writes the report to stdout: the header
 // line, then the rows of each size's duplicate groups once that size is
-// read. A path for which exists returns false is left out of its group.
+// read. stdout may be the terminal prog is drawn on, so the lines go
-// stdout may be the terminal prog is drawn on, so the lines go through
+// through prog's writer. It returns what it wrote, with the totals for
-// prog's writer. It returns what it wrote, with the totals for the
+// the summary line.
-// summary line.
 func readReport(ctx context.Context, db *sql.DB, dbPath string,
-	stdout io.Writer, exists func(path string) bool, prog *progress,
+	stdout io.Writer, prog *progress,
 ) (*reportWriter, error) {
 	out := prog.writer(stdout)
-	report := &reportWriter{out: out, exists: exists}
+	report := &reportWriter{out: out}
 
 	_, err := fmt.Fprintln(out, "first\tdupe\tsize")
 	if err != nil {
@@ -163,10 +158,6 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
 type reportWriter struct {
 	out io.Writer
 
-	// exists reports whether a path is still on disk. A path that is
-	// gone is left out of its group.
-	exists func(path string) bool
-
 	sig    scanRec    // the signature being read; its path is not used
 	paths  []string   // the paths read so far with sig
 	groups [][]string // the duplicate groups of sig.size, each sorted
@@ -203,20 +194,11 @@ func (w *reportWriter) add(r scanRec) error {
 	return nil
 }
 
-// endGroup ends the paths read with w.sig: two or more that are still
+// endGroup ends the paths read with w.sig: two or more are a duplicate
-// on disk are a duplicate group, held with its paths sorted until
+// group, held with its paths sorted until writeGroups writes it.
-// writeGroups writes it. Empty files form no group, since removing one
-// frees nothing. Only the paths of a possible group are checked, never
-// a path with no duplicate or an empty file.
 func (w *reportWriter) endGroup() {
-	if len(w.paths) >= minGroupSize && w.sig.size > 0 {
+	if len(w.paths) < minGroupSize {
-		w.paths = slices.DeleteFunc(w.paths, func(p string) bool {
+		// A path with no duplicate is dropped, and its slice reused.
-			return !w.exists(p)
-		})
-	}
-
-	if len(w.paths) < minGroupSize || w.sig.size == 0 {
-		// The paths are dropped, and their slice reused.
 		w.paths = w.paths[:0]
 
 		return
@@ -263,16 +245,6 @@ func (w *reportWriter) finish() error {
 	return w.writeGroups()
 }
 
-// pathExists reports whether path is still on disk, for report and
-// trees. It is false only when lstat finds that the path, or a
-// directory above it, is gone. Any other error counts as present, so
-// a user who may not look inside a directory still sees its copies.
-func pathExists(path string) bool {
-	_, err := os.Lstat(path)
-
-	return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
-}
-
 // escapePath returns a path as it is written in a report column (README
 // "Report output format"): a backslash, tab, newline or carriage return
 // becomes \\, \t, \n or \r, and every other byte is kept as it is.

internal/sfdupes/report_test.go

@@ -72,37 +72,15 @@ type dupeGroup struct {
 	paths []string
 }
 
-// everyPathExists stands in for pathExists where a test's records name
-// made-up paths that are not on disk.
-func everyPathExists(string) bool { return true }
-
-// recsOnDisk returns a new directory and recs with each path moved
-// under it, after creating an empty file at each moved path: report and
-// trees then find every path still on disk.
-func recsOnDisk(t *testing.T, recs []scanRec) (string, []scanRec) {
-	t.Helper()
-
-	dir := realTempDir(t)
-	moved := make([]scanRec, len(recs))
-
-	for i, r := range recs {
-		r.path = writeFile(t, dir, r.path, nil)
-		moved[i] = r
-	}
-
-	return dir, moved
-}
-
 // dupeGroups returns the duplicate groups report reads from db, in
 // report order, from the rows it writes: a group's rows come together,
-// each naming the group's first path. Every path counts as still on
+// each naming the group's first path.
-// disk.
 func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
 	t.Helper()
 
 	var out bytes.Buffer
 
-	_, err := readReport(t.Context(), db, "", &out, everyPathExists, nil)
+	_, err := readReport(t.Context(), db, "", &out, nil)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -149,8 +127,7 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
 }
 
 func TestRunReportEscapesPaths(t *testing.T) {
-	dir, recs := recsOnDisk(t, awkwardPairRecs())
+	t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
-	t.Setenv(databaseEnv, seedDatabase(t, recs))
 
 	var stdout, stderr bytes.Buffer
 
@@ -161,7 +138,7 @@ func TestRunReportEscapesPaths(t *testing.T) {
 	}
 
 	want := "first\tdupe\tsize\n" +
-		dir + `/d/\tone\ntwo\rthree\\four/f` + "\t" + dir + "/d/A/f\t5\n"
+		`/d/\tone\ntwo\rthree\\four/f` + "\t/d/A/f\t5\n"
 	if got := stdout.String(); got != want {
 		t.Errorf("stdout = %q, want %q", got, want)
 	}
@@ -195,8 +172,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
 	// Counts the records read, and draws nothing.
 	prog := &progress{now: time.Now}
 
-	_, err = readReport(t.Context(), db, "", failingWriter{},
+	_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
-		everyPathExists, prog)
 	if !errors.Is(err, errWriteFailed) ||
 		!strings.HasPrefix(err.Error(), "write stdout: ") {
 		t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
@@ -227,8 +203,7 @@ const smokeReport = "first\tdupe\tsize\n" +
 // it ends; read counts the 6 records before it reads them. The summary
 // comes last, and stdout holds the report alone.
 func TestRunReportProgressNamesEachPhase(t *testing.T) {
-	dir, recs := recsOnDisk(t, smokeTreeRecs())
+	path := seedDatabase(t, smokeTreeRecs())
-	path := seedDatabase(t, recs)
 	t.Setenv(databaseEnv, path)
 
 	stderr := captureStderr(t)
@@ -241,9 +216,8 @@ func TestRunReportProgressNamesEachPhase(t *testing.T) {
 			code, exitOK, stderr())
 	}
 
-	report := strings.ReplaceAll(smokeReport, "/d/", dir+"/d/")
+	if got := stdout.String(); got != smokeReport {
-	if got := stdout.String(); got != report {
+		t.Errorf("stdout = %q, want %q", got, smokeReport)
-		t.Errorf("stdout = %q, want %q", got, report)
 	}
 
 	// How each line starts; the rate at the end of a phase varies.
@@ -292,7 +266,7 @@ func TestReadReportOnTerminal(t *testing.T) {
 		waitingFor: countingRecords,
 	}, terminalWidth, time.Hour, time.Now)
 
-	_, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p)
+	_, err = readReport(t.Context(), db, "", os.Stderr, p)
 
 	p.finish()
 
@@ -327,8 +301,6 @@ func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
 		scanRec{size: 50, head: a, tail: a, content: a, path: "/x/1"},
 	)
 
-	_, recs = recsOnDisk(t, recs)
-
 	reversed := slices.Clone(recs)
 	slices.Reverse(reversed)
 
@@ -462,140 +434,6 @@ func TestDupeGroups(t *testing.T) {
 	}
 }
 
-// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
-// duplicate group, while 1-byte files still do.
-func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
-	t.Parallel()
-
-	e, x := sha256.Sum256(nil), testHash("x")
-	recs := []scanRec{
-		{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
-		{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
-		{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
-		{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
-		{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
-	}
-
-	groups := dupeGroupsOf(t, recs)
-	if len(groups) != 1 || groups[0].size != 1 ||
-		!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
-		t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
-	}
-}
-
-// TestReadReportChecksOnlyPossibleDuplicates checks that report asks
-// whether a path is on disk only when its signature has two or more
-// records, and never for an empty file.
-func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
-	t.Parallel()
-
-	a, u, e := testHash("a"), testHash("u"), sha256.Sum256(nil)
-	recs := []scanRec{
-		{size: 9, head: a, tail: a, content: a, path: "/copy/1"},
-		{size: 9, head: a, tail: a, content: a, path: "/copy/2"},
-		{size: 9, head: a, tail: a, content: a, path: "/copy/3"},
-		{size: 9, head: u, tail: u, content: u, path: "/lonely"},
-		{size: 0, head: e, tail: e, content: e, path: "/empty1"},
-		{size: 0, head: e, tail: e, content: e, path: "/empty2"},
-	}
-
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	var checked []string
-
-	exists := func(path string) bool {
-		checked = append(checked, path)
-
-		return true
-	}
-
-	_, err = readReport(t.Context(), db, "", io.Discard, exists, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	slices.Sort(checked)
-
-	want := []string{"/copy/1", "/copy/2", "/copy/3"}
-	if !slices.Equal(checked, want) {
-		t.Errorf("paths checked on disk = %q, want %q", checked, want)
-	}
-}
-
-// TestReadReportDropsGroupLeftWithOnePath checks that a group of two
-// paths, one of them gone, is no duplicate group. If kept, it would
-// write no row, but the summary line would count it.
-func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
-	t.Parallel()
-
-	a := testHash("a")
-	recs := []scanRec{
-		{size: 9, head: a, tail: a, content: a, path: "/a/kept"},
-		{size: 9, head: a, tail: a, content: a, path: "/a/gone"},
-	}
-
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	exists := func(path string) bool { return path != "/a/gone" }
-
-	report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if report.dupeGroups != 0 {
-		t.Errorf("duplicate groups = %d, want 0", report.dupeGroups)
-	}
-}
-
-// TestReadReportKeepsGroupLeftWithTwoPaths checks that a group of three
-// paths, one of them gone, is a duplicate group of the other two.
-func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
-	t.Parallel()
-
-	a := testHash("a")
-	recs := []scanRec{
-		{size: 9, head: a, tail: a, content: a, path: "/b/kept1"},
-		{size: 9, head: a, tail: a, content: a, path: "/b/gone"},
-		{size: 9, head: a, tail: a, content: a, path: "/b/kept2"},
-	}
-
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	exists := func(path string) bool { return path != "/b/gone" }
-
-	var out bytes.Buffer
-
-	report, err := readReport(t.Context(), db, "", &out, exists, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if report.dupeGroups != 1 {
-		t.Errorf("duplicate groups = %d, want 1", report.dupeGroups)
-	}
-
-	want := "first\tdupe\tsize\n/b/kept1\t/b/kept2\t9\n"
-	if got := out.String(); got != want {
-		t.Errorf("stdout = %q, want %q", got, want)
-	}
-}
-
 func TestDupeGroupsContentSeparates(t *testing.T) {
 	t.Parallel()
 

internal/sfdupes/scan.go

@@ -102,44 +102,40 @@ func scanPhase(name string) string {
 // scan command rejects anything less. Before the lock is taken, the
 // scan lists the filesystems that are not persistent, on which the walk
 // records nothing; unless noDefaultExcludes is set, the walk does not
-// enter the default excludes either. The walk neither enters nor
+// enter the default excludes either. Once the database is open, the
-// records the paths in excludePaths, the --exclude values; an operand
+// scan notes its files, which the walk passes over.
-// at or below one is a fatal error. Once the database is open, the scan
-// notes its files, which the walk passes over.
 func runScan(ctx context.Context, roots []string, workers int,
-	oneFS, noDefaultExcludes bool, excludePaths []string,
+	oneFS, noDefaultExcludes bool,
 ) error {
 	dbPath, source, err := databasePath()
 	if err != nil {
 		return err
 	}
 
-	printDatabasePath(dbPath, source)
+	// A database file missing now is created by lockScanDatabase below.
+	what := "database"
 
-	roots, err = realPaths(roots)
+	_, err = os.Stat(dbPath)
+	if errors.Is(err, fs.ErrNotExist) {
+		what = "new database"
+	}
+
+	fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
+		source)
+
+	roots, err = resolveRoots(roots)
 	if err != nil {
 		return err
 	}
 
-	var rules walkRules
+	nonPersistent, err := nonPersistentFilesystems()
-
-	rules.excludePaths, err = excludePathInodes(excludePaths)
-	if err != nil {
-		return fmt.Errorf("--exclude: %w", err)
-	}
-
-	err = checkOperandsNotExcluded(roots, rules.excludePaths)
-	if err != nil {
-		return err
-	}
-
-	rules.nonPersistent, err = nonPersistentFilesystems()
 	if err != nil {
 		return fmt.Errorf("list the mounted filesystems: %w", err)
 	}
 
+	rules := walkRules{nonPersistent: nonPersistent}
 	if !noDefaultExcludes {
-		rules.defaultExcludes = defaultExcludes(runtime.GOOS)
+		rules.excludes = defaultExcludes(runtime.GOOS)
 	}
 
 	lock, err := lockScanDatabase(dbPath)
@@ -185,21 +181,6 @@ func runScan(ctx context.Context, roots []string, workers int,
 	return nil
 }
 
-// printDatabasePath prints on stderr the line that names the database
-// at dbPath, with source, saying whether it is new. A database file
-// missing now is new: lockScanDatabase creates it.
-func printDatabasePath(dbPath, source string) {
-	what := "database"
-
-	_, err := os.Stat(dbPath)
-	if errors.Is(err, fs.ErrNotExist) {
-		what = "new database"
-	}
-
-	fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
-		source)
-}
-
 // interruptContext returns a copy of ctx that the first SIGINT or
 // SIGTERM cancels; the scan command runs the scan under it. stop
 // releases the signals.
@@ -228,25 +209,24 @@ func interrupted(walked int) error {
 	return errInterrupted
 }
 
-// realPaths returns the real path of each of paths, the PATH operands
+// resolveRoots converts each PATH operand to its real path: absolute,
-// or the --exclude values: absolute, with every symlink in it followed.
+// with every symlink in it followed. Database records are keyed by
-// Database records are keyed by that path, so scan results depend
+// that path, so scan results depend neither on the working directory
-// neither on the working directory nor on the symlinks an operand was
+// nor on the symlinks an operand was given through. An operand that
-// given through. A path that cannot be resolved, because it or a
+// cannot be resolved, because it or a symlink's target does not exist
-// symlink's target does not exist or cannot be read, is a fatal error
+// or cannot be read, is a fatal error before any scanning.
-// before any scanning.
+func resolveRoots(roots []string) ([]string, error) {
-func realPaths(paths []string) ([]string, error) {
+	resolved := make([]string, 0, len(roots))
-	resolved := make([]string, 0, len(paths))
 
-	for _, path := range paths {
+	for _, root := range roots {
-		a, err := absolutePath(path)
+		a, err := absolutePath(root)
 		if err != nil {
-			return nil, fmt.Errorf("resolve %s: %w", path, err)
+			return nil, fmt.Errorf("resolve %s: %w", root, err)
 		}
 
 		r, err := filepath.EvalSymlinks(a)
 		if err != nil {
-			return nil, fmt.Errorf("resolve %s: %w", path, err)
+			return nil, fmt.Errorf("resolve %s: %w", root, err)
 		}
 
 		resolved = append(resolved, r)
@@ -315,7 +295,7 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
 	}
 
 	for ; dir != above; dir = filepath.Dir(dir) {
-		if isDefaultExclude(dir, rules.defaultExcludes) {
+		if isDefaultExclude(dir, rules.excludes) {
 			return false
 		}
 
@@ -438,7 +418,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 
 // walkableRoots returns the operands the walk can start from: regular
 // files, and directories not named .zfs, none of them inside a
-// directory named .zfs. The operands are real paths from realPaths,
+// directory named .zfs. The operands are real paths from resolveRoots,
 // so these checks apply to what a symlink operand points at. Every
 // other operand is warned about, counted as skipped, and dropped. A
 // dropped operand is no longer a root, so the records stored beneath it
@@ -513,11 +493,10 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 // as verified. When its disk_size differs from the record's, it is
 // queued for the update phase, which writes the new disk_size. A record
 // at or below a path the walk left out, because -x stopped at it, it is
-// an --exclude path or a default exclude, or it is not on a persistent
+// a default exclude, or it is not on a persistent filesystem, counts as
-// filesystem, counts as outside the roots: it is not deleted. That does
+// outside the roots: it is not deleted. That does not hold below a root
-// not hold below a root at or inside that path, since that root's own
+// at or inside that path, since that root's own walk verifies the
-// walk verifies the records below it; nor is a line printed for such a
+// records below it; nor is a line printed for such a root.
-// root.
 func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
 	// The records the last scan left under the roots estimate how many
 	// files the walk will find. A first scan has none, and no estimate.
@@ -566,7 +545,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
 		prog.increment()
 
 		old, ok := s.existing[ev.rec.path]
-		if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) {
+		if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) {
 			s.toHash = append(s.toHash, ev.rec)
 
 			continue
@@ -801,6 +780,16 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 	return applyChanges(ctx, s.db, nil, deletes, prog)
 }
 
+// mtimeAfter reports whether mtime a is later than mtime b.
+// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
+func mtimeAfter(a, b time.Time) bool {
+	if a.Unix() != b.Unix() {
+		return a.Unix() > b.Unix()
+	}
+
+	return a.Nanosecond() > b.Nanosecond()
+}
+
 // underAnyRoot reports whether path is any of the roots or lies under
 // one of them.
 func underAnyRoot(path string, roots []string) bool {
@@ -855,9 +844,9 @@ type dirJob struct {
 // file's statted record, a warning when fail is set, or a path the walk
 // left out: when otherFSDir is set, a directory -x did not enter
 // because it is on another filesystem than its operand, and when
-// leftOut is set, an --exclude path, a default exclude or a directory or
+// leftOut is set, a default exclude or a directory or file on a
-// file on a filesystem that is not persistent, with warn holding the
+// filesystem that is not persistent, with warn holding the line that
-// line that says so.
+// says so.
 type walkEvent struct {
 	rec        fileRec
 	warn       string
@@ -908,7 +897,7 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
 // operandWarning returns the one-line warning for an operand the walk
 // does not start from, naming the path and what it is, or "" for one it
 // does: a regular file, or a directory not named .zfs, that is not
-// inside a directory named .zfs. realPaths has already followed the
+// inside a directory named .zfs. resolveRoots has already followed the
 // operand's symlinks, so a symlink here is an operand replaced by one
 // after it was resolved, and is not followed.
 func operandWarning(root string, fi fs.FileInfo) string {
@@ -1139,9 +1128,8 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 // no longer exists, and one that has stopped being a regular file is
 // skipped. A file on a filesystem in rules that is not persistent, as
 // through a bind mount of one file, is not recorded: it gets an event
-// naming it instead, and so does a file given with --exclude, reached
+// naming it instead. One of the scan's own database files in rules gets
-// by any path, a hard link to it included. One of the scan's own
+// no event at all.
-// database files in rules gets no event at all.
 func emitFile(ctx context.Context, p string, e fs.DirEntry,
 	rules walkRules, events chan<- walkEvent,
 ) {
@@ -1165,10 +1153,6 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
 	}
 
 	line := rules.nonPersistentLine(p, info)
-	if line == "" {
-		line = rules.excludePathLine(p, info)
-	}
-
 	if line != "" {
 		sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
 
@@ -1193,11 +1177,11 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
 // subdirJob applies the descent rules to directory p: never enter
 // .zfs (ZFS snapshot pseudo-dirs would list every file once per
 // snapshot), never enter a directory on a filesystem in rules that is
-// not persistent, nor an --exclude path or a default exclude in rules,
+// not persistent, nor a default exclude in rules, and with -x never
-// and with -x never enter a directory on a different filesystem than
+// enter a directory on a different filesystem than its operand. For the
-// its operand. For all but .zfs it sends an event that names the
+// last three it sends an event that names the directory, so the scan
-// directory, so the scan keeps the records below it, except those below
+// keeps the records below it, except those below an operand at or
-// an operand at or inside it.
+// inside it.
 func subdirJob(ctx context.Context, p string, e fs.DirEntry,
 	parent dirJob, oneFS bool, rules walkRules,
 	events chan<- walkEvent,
@@ -1222,17 +1206,13 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
 	}
 
 	line := rules.nonPersistentLine(p, info)
-	if line == "" {
-		line = rules.excludePathLine(p, info)
-	}
-
 	if line != "" {
 		sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
 
 		return dirJob{}, false
 	}
 
-	if isDefaultExclude(p, rules.defaultExcludes) {
+	if isDefaultExclude(p, rules.excludes) {
 		sendEvent(ctx, events, walkEvent{
 			leftOut: p,
 			warn:    fmt.Sprintf("scan: skipped %s (default exclude)", p),

internal/sfdupes/scan_test.go

@@ -1185,12 +1185,11 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
 	t.Helper()
 
 	groups := dupeGroups(t, db)
-	if len(groups) != 4 {
+	if len(groups) != 5 {
-		t.Fatalf("len(groups) = %d, want 4", len(groups))
+		t.Fatalf("len(groups) = %d, want 5", len(groups))
 	}
 
-	// The empty files empty1 and empty2 form no group.
+	wantSizes := []int64{3000, 2000, 100, 1, 0}
-	wantSizes := []int64{3000, 2000, 100, 1}
 	for i, g := range groups {
 		if g.size != wantSizes[i] {
 			t.Errorf("groups[%d].size = %d, want %d",
@@ -1215,7 +1214,7 @@ func assertSmokeTreeGroups(t *testing.T, dir string, db *sql.DB) {
 
 	super, dirs := dbTree(t, db)
 
-	tg := collectTreeGroups(dirs, super, everyPathExists, nil)
+	tg := collectTreeGroups(dirs, super, nil)
 	if len(tg) != 1 {
 		t.Fatalf("len(tree groups) = %d, want 1", len(tg))
 	}
@@ -1384,47 +1383,6 @@ func TestSyncScanMtimeBump(t *testing.T) {
 	}
 }
 
-// TestSyncScanMtimeMovedBack replaces a file with other content of the
-// same size and an older mtime, as cp -p or a restore from backup does.
-// The next scan must hash the file again.
-func TestSyncScanMtimeMovedBack(t *testing.T) {
-	t.Parallel()
-
-	dir := t.TempDir()
-	db := openTestDB(t)
-	a := writeFile(t, dir, "a.bin", pattern(1, 500))
-
-	recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
-
-	err := os.Chtimes(a, recorded, recorded)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	syncTree(t, db, dir)
-
-	replaced := pattern(3, 500)
-	writeFile(t, dir, "a.bin", replaced)
-
-	older := recorded.Add(-time.Hour)
-
-	err = os.Chtimes(a, older, older)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	st := syncTree(t, db, dir)
-	if st != (scanStats{walked: 1, updated: 1}) {
-		t.Fatalf("rescan stats = %+v, want 1 updated", st)
-	}
-
-	assertWholeFileHashed(t, db, a, replaced)
-
-	if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
-		t.Fatalf("mtime = %v, want %v", r.mtime, older)
-	}
-}
-
 // assertWholeFileHashed fails unless the record for path holds the
 // whole-file hash of data as its head, tail, and content.
 func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
@@ -2289,7 +2247,7 @@ func TestPruneRoots(t *testing.T) {
 
 	// The walk of /p does not enter /p/q when it is a default exclude, so
 	// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
-	got = pruneRoots(nested, false, walkRules{defaultExcludes: []string{pqRoot}},
+	got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
 		device)
 	if !slices.Equal(got, want) {
 		t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
@@ -2307,86 +2265,23 @@ func TestPruneRoots(t *testing.T) {
 	}
 }
 
-// TestRunReportsLeaveOutRemovedCopies scans the smoke tree, then
+func TestReportsNeverTouchFilesystem(t *testing.T) {
-// removes copies the reports name. report leaves out each removed file,
-// and a group left with one path. trees checks only the root of each
-// duplicate tree: with a file under t2 removed it still names t2, and
-// with t2 removed it names nothing.
-func TestRunReportsLeaveOutRemovedCopies(t *testing.T) {
-	t.Setenv(databaseEnv, testDBPath(t))
-
-	dir := buildSmokeTree(t)
-	p := func(name string) string { return filepath.Join(dir, name) }
-
-	scanOK(t, dir)
-
-	for _, name := range []string{"b/copy2.bin", "tiny2", "t2/sub/f2"} {
-		err := os.Remove(p(name))
-		if err != nil {
-			t.Fatal(err)
-		}
-	}
-
-	want := "first\tdupe\tsize\n" +
-		p("t1/f1") + "\t" + p("t2/f1") + "\t3000\n" +
-		p("t1/f1") + "\t" + p("t3/f1") + "\t3000\n" +
-		p("a/one.bin") + "\t" + p("b/copy.bin") + "\t2000\n" +
-		p("t1/sub/f2") + "\t" + p("t3/sub/f2renamed") + "\t100\n"
-	if got := runStdout(t, cmdReport); got != want {
-		t.Errorf("report stdout = %q, want %q", got, want)
-	}
-
-	// disk_size, the last column, depends on the filesystem.
-	header := "first\tdupe\tfiles\tsize\tdisk_size\n"
-
-	want = header + p("t1") + "\t" + p("t2") + "\t2\t3100\t"
-	if got := runStdout(t, cmdTrees); !strings.HasPrefix(got, want) ||
-		strings.Count(got, "\n") != 2 {
-		t.Errorf("trees stdout = %q, want one row starting %q", got, want)
-	}
-
-	err := os.RemoveAll(p("t2"))
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if got := runStdout(t, cmdTrees); got != header {
-		t.Errorf("trees stdout after removing t2 = %q, want %q", got, header)
-	}
-}
-
-func TestPathExists(t *testing.T) {
 	t.Parallel()
 
-	dir := t.TempDir()
+	dir := buildSmokeTree(t)
-	file := writeFile(t, dir, "file", nil)
+	db := openTestDB(t)
-	locked := writeFile(t, dir, "locked/file", nil)
 
-	// Without search permission on its directory, a file cannot be
+	syncTree(t, db, dir)
-	// looked at by anyone but root.
+
-	err := os.Chmod(filepath.Dir(locked), 0)
+	// Remove the scanned tree entirely; the analysis must be
+	// unaffected because it reads the database alone.
+	err := os.RemoveAll(dir)
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	// Runs before t.TempDir's own cleanup, which must delete the files.
+	assertSmokeDupeGroups(t, dir, db)
-	t.Cleanup(func() {
+	assertSmokeTreeGroups(t, dir, db)
-		//nolint:gosec // removing the directory needs its search bit back
-		_ = os.Chmod(filepath.Dir(locked), 0o700)
-	})
-
-	cases := map[string]bool{
-		dir:                                 true,
-		file:                                true,
-		locked:                              true,
-		filepath.Join(dir, "gone"):          false,
-		filepath.Join(file, "under-a-file"): false,
-	}
-	for path, want := range cases {
-		if got := pathExists(path); got != want {
-			t.Errorf("pathExists(%q) = %v, want %v", path, got, want)
-		}
-	}
 }
 
 func TestUnderRoot(t *testing.T) {
@@ -2472,8 +2367,8 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
 			report := runStdout(t, cmdReport)
 			if strings.Contains(report, "db.sqlite") ||
 				strings.Contains(report, hardLink) ||
-				!strings.Contains(report, filepath.Join(tree, "b")) {
+				!strings.Contains(report, filepath.Join(tree, "empty2")) {
-				t.Errorf("report = %q, want a and b and none of "+
+				t.Errorf("report = %q, want the empty files and none of "+
 					"the database's files", report)
 			}
 		})

internal/sfdupes/trees.go

@@ -38,11 +38,10 @@ func treesPhase(name string) string {
 // runTrees implements the trees subcommand: it reads every record from
 // the database in path order, reconstructs the directory hierarchy from
 // the record paths, computes a Merkle-style digest per directory, and
-// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned
+// prints maximal duplicate-tree groups as TSV on stdout. It never
-// filesystem it only checks, with pathExists, that the root of each
+// touches the scanned filesystem; its only I/O is the database, stdout,
-// duplicate tree is still there, never the files below it. It names the
+// and stderr. It names the database on stderr before it opens it. Any
-// database on stderr before it opens it. Any database problem,
+// database problem, including a missing database, is fatal.
-// including a missing database, is fatal.
 //
 // The work runs in four phases, each under a progress display of its
 // own on stderr: open (open the database), read (read the number of
@@ -87,13 +86,13 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
 		return fmt.Errorf("database %s: %w", dbPath, err)
 	}
 
-	// Grouping works in memory, then checks the root of each duplicate
+	// Grouping works in memory, so it waits on nothing but the
-	// tree on disk, so it waits on the processor or the disk.
+	// processor.
 	prog = newProgress(progressParams{
 		label: treesPhase("group"), unit: "directories",
-		total: int64(len(allDirs)), stalledOn: "cpu/disk",
+		total: int64(len(allDirs)), stalledOn: "cpu",
 	})
-	dupes := collectTreeGroups(allDirs, super, pathExists, prog)
+	dupes := collectTreeGroups(allDirs, super, prog)
 
 	prog.finish()
 
@@ -301,14 +300,10 @@ func fileEntry(name string, r scanRec) string {
 
 // collectTreeGroups groups directories by digest, counting each one on
 // prog as it is grouped, and returns every maximal group with two or
-// more members for which exists returns true, each group's members
+// more members, each group's members sorted by path, groups ordered by
-// sorted by path, groups ordered by tree size descending then by first
+// tree size descending then by first path ascending.
-// path ascending. A group whose trees hold 0 bytes is left out, since
-// removing one frees nothing. The groups it implies hold 0 bytes too,
-// so leaving it out hides nothing. exists is called only on the members
-// of maximal groups.
 func collectTreeGroups(allDirs []*treeNode, super *treeNode,
-	exists func(path string) bool, prog *progress,
+	prog *progress,
 ) [][]*treeNode {
 	groups := make(map[[sha256.Size]byte][]*treeNode)
 	for _, d := range allDirs {
@@ -320,19 +315,7 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
 	var dupes [][]*treeNode
 
 	for _, g := range groups {
-		// The members share a digest, so they share a total size.
+		if len(g) < minGroupSize || suppressed(g, super) {
-		if len(g) < minGroupSize || g[0].totalSize == 0 ||
-			suppressed(g, super) {
-			continue
-		}
-
-		// A member that is gone leaves the group. A suppressed group is
-		// still implied: its members still on disk have parents still
-		// on disk, and those parents' group is checked the same way.
-		g = slices.DeleteFunc(g, func(n *treeNode) bool {
-			return !exists(n.path)
-		})
-		if len(g) < minGroupSize {
 			continue
 		}
 

internal/sfdupes/trees_test.go

@@ -2,11 +2,9 @@ package sfdupes //nolint:testpackage // checks the unexported treeNode results
 
 import (
 	"bytes"
-	"crypto/sha256"
 	"database/sql"
 	"os"
 	"slices"
-	"strconv"
 	"strings"
 	"testing"
 	"time"
@@ -180,7 +178,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
 		t.Fatalf("directory paths = %q, want %q", got, want)
 	}
 
-	groups := collectTreeGroups(dirs, super, everyPathExists, nil)
+	groups := collectTreeGroups(dirs, super, nil)
 
 	gotGroups := groupPaths(groups)
 	wantGroups := [][]string{{"/a", "/c"}}
@@ -196,8 +194,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
 }
 
 func TestRunTreesEscapesPaths(t *testing.T) {
-	dir, recs := recsOnDisk(t, awkwardPairRecs())
+	t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
-	t.Setenv(databaseEnv, seedDatabase(t, recs))
 
 	var stdout, stderr bytes.Buffer
 
@@ -208,7 +205,7 @@ func TestRunTreesEscapesPaths(t *testing.T) {
 	}
 
 	want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
-		dir + `/d/\tone\ntwo\rthree\\four` + "\t" + dir + "/d/A\t1\t5\t4096\n"
+		`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\t4096\n"
 	if got := stdout.String(); got != want {
 		t.Errorf("stdout = %q, want %q", got, want)
 	}
@@ -226,11 +223,9 @@ func TestRunTreesDiskSize(t *testing.T) {
 		{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
 	}
 
-	dir, recs := recsOnDisk(t, recs)
 	t.Setenv(databaseEnv, seedDatabase(t, recs))
 
-	want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
+	want := "first\tdupe\tfiles\tsize\tdisk_size\n/a\t/b\t2\t30\t1536\n"
-		dir + "/a\t" + dir + "/b\t2\t30\t1536\n"
 	if got := runStdout(t, cmdTrees); got != want {
 		t.Errorf("stdout = %q, want %q", got, want)
 	}
@@ -241,11 +236,10 @@ func TestRunTreesDiskSize(t *testing.T) {
 // there. After the line naming the database, each phase prints a line
 // naming it with its place among the four as it starts and another as
 // it ends. read counts the 6 records before it reads them, group counts
-// the directories, and write the 1 group. The summary comes last, and
+// the 8 directories, and write the 1 group. The summary comes last, and
 // stdout holds the report alone.
 func TestRunTreesProgressNamesEachPhase(t *testing.T) {
-	dir, recs := recsOnDisk(t, smokeTreeRecs())
+	path := seedDatabase(t, smokeTreeRecs())
-	path := seedDatabase(t, recs)
 	t.Setenv(databaseEnv, path)
 
 	stderr := captureStderr(t)
@@ -259,15 +253,11 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
 	}
 
 	report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
-		dir + "/d/t1\t" + dir + "/d/t2\t2\t3100\t8192\n"
+		"/d/t1\t/d/t2\t2\t3100\t8192\n"
 	if got := stdout.String(); got != report {
 		t.Errorf("stdout = %q, want %q", got, report)
 	}
 
-	// The smoke-test records name 8 directories, / included. Moved under
-	// dir, they name one more for each slash in dir.
-	dirs := strconv.Itoa(8 + strings.Count(dir, "/"))
-
 	// How each line starts; the rate at the end of a phase varies.
 	want := []string{
 		"trees: database " + path + " (from " + databaseEnv + ")",
@@ -275,9 +265,8 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
 		treesOpenLine,
 		"[2/4] read: counting the records, elapsed 0s, remain unknown",
 		"[2/4] read: [6/6] 100%, ",
-		"[3/4] group: [0/" + dirs + "] 0%, 0 directories/s, elapsed 0s, " +
+		"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, remain unknown",
-			"remain unknown",
+		"[3/4] group: [8/8] 100%, ",
-		"[3/4] group: [" + dirs + "/" + dirs + "] 100%, ",
 		"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown",
 		"[4/4] write: [1/1] 100%, ",
 		"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
@@ -306,7 +295,7 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
 func TestWriteTreeGroupsOnTerminal(t *testing.T) {
 	stderr := captureStderr(t)
 	super, dirs := treeOf(t, smokeTreeRecs())
-	dupes := collectTreeGroups(dirs, super, everyPathExists, nil)
+	dupes := collectTreeGroups(dirs, super, nil)
 
 	// No timed draws: the display is drawn as it starts and as it ends.
 	p := startProgress(progressParams{
@@ -387,7 +376,7 @@ func TestCollectTreeGroupsMaximal(t *testing.T) {
 
 	super, dirs := treeOf(t, smokeTreeRecs())
 
-	groups := collectTreeGroups(dirs, super, everyPathExists, nil)
+	groups := collectTreeGroups(dirs, super, nil)
 
 	want := [][]string{{"/d/t1", "/d/t2"}}
 	if got := groupPaths(groups); !slices.EqualFunc(got, want,
@@ -411,16 +400,14 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
 
 	super, dirs := treeOf(t, recs)
 
-	forward := groupPaths(collectTreeGroups(dirs, super, everyPathExists,
+	forward := groupPaths(collectTreeGroups(dirs, super, nil))
-		nil))
 
 	reversed := slices.Clone(recs)
 	slices.Reverse(reversed)
 
 	superR, dirsR := treeOf(t, reversed)
 
-	backward := groupPaths(collectTreeGroups(dirsR, superR, everyPathExists,
+	backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
-		nil))
 	if !slices.EqualFunc(forward, backward, slices.Equal) {
 		t.Fatalf("output depends on record order: %v vs %v",
 			forward, backward)
@@ -440,36 +427,7 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
 
 	super, dirs := treeOf(t, recs)
 
-	got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
+	got := groupPaths(collectTreeGroups(dirs, super, nil))
-
-	want := [][]string{{"/p/x1", "/p/x2"}}
-	if !slices.EqualFunc(got, want, slices.Equal) {
-		t.Fatalf("groups = %v, want %v", got, want)
-	}
-}
-
-// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
-// holding only empty files form no group, and that an empty file still
-// counts toward the digest of a tree that holds other files.
-func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
-	t.Parallel()
-
-	h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
-		sha256.Sum256(nil)
-	recs := []scanRec{
-		{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
-		{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
-		{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
-		{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
-		{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
-		{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
-		// Without the empty file, /p/x3 is no copy of /p/x1.
-		{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
-	}
-
-	super, dirs := treeOf(t, recs)
-
-	got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
 
 	want := [][]string{{"/p/x1", "/p/x2"}}
 	if !slices.EqualFunc(got, want, slices.Equal) {
@@ -492,98 +450,10 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
 
 	super, dirs := treeOf(t, recs)
 
-	got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
+	got := groupPaths(collectTreeGroups(dirs, super, nil))
 
 	want := [][]string{{"/p/a/x", "/q/b/x"}}
 	if !slices.EqualFunc(got, want, slices.Equal) {
 		t.Fatalf("groups = %v, want %v", got, want)
 	}
 }
-
-// TestCollectTreeGroupsChecksOnlyReportedCopies checks that trees asks
-// whether a directory is on disk only for the members of the groups it
-// returns. It never asks about /a/proj/sub, which lies inside a copy
-// and whose group the group of /a/proj implies.
-func TestCollectTreeGroupsChecksOnlyReportedCopies(t *testing.T) {
-	t.Parallel()
-
-	f, g, u := testHash("f"), testHash("g"), testHash("u")
-	recs := []scanRec{
-		{size: 10, head: f, tail: f, content: f, path: "/a/proj/f"},
-		{size: 20, head: g, tail: g, content: g, path: "/a/proj/sub/g"},
-		{size: 10, head: f, tail: f, content: f, path: "/b/proj/f"},
-		{size: 20, head: g, tail: g, content: g, path: "/b/proj/sub/g"},
-		// /a lacks this file, so /a and /b are no copies of each other.
-		{size: 30, head: u, tail: u, content: u, path: "/b/lonely"},
-	}
-
-	super, dirs := treeOf(t, recs)
-
-	var checked []string
-
-	exists := func(path string) bool {
-		checked = append(checked, path)
-
-		return true
-	}
-
-	got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
-
-	want := [][]string{{"/a/proj", "/b/proj"}}
-	if !slices.EqualFunc(got, want, slices.Equal) {
-		t.Fatalf("groups = %v, want %v", got, want)
-	}
-
-	slices.Sort(checked)
-
-	if !slices.Equal(checked, want[0]) {
-		t.Errorf("directories checked on disk = %q, want %q",
-			checked, want[0])
-	}
-}
-
-// TestCollectTreeGroupsDropsGroupLeftWithOneCopy checks that a group of
-// two copies, one of them gone, is not returned. If kept, it would
-// write no row, but the summary line would count it.
-func TestCollectTreeGroupsDropsGroupLeftWithOneCopy(t *testing.T) {
-	t.Parallel()
-
-	f := testHash("f")
-	recs := []scanRec{
-		{size: 10, head: f, tail: f, content: f, path: "/c/kept/f"},
-		{size: 10, head: f, tail: f, content: f, path: "/c/gone/f"},
-	}
-
-	super, dirs := treeOf(t, recs)
-
-	exists := func(path string) bool { return path != "/c/gone" }
-
-	got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
-	if len(got) != 0 {
-		t.Errorf("groups = %v, want none", got)
-	}
-}
-
-// TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies checks that a group
-// of three copies, one of them gone, is returned with the other two.
-func TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies(t *testing.T) {
-	t.Parallel()
-
-	f := testHash("f")
-	recs := []scanRec{
-		{size: 10, head: f, tail: f, content: f, path: "/d/kept1/f"},
-		{size: 10, head: f, tail: f, content: f, path: "/d/gone/f"},
-		{size: 10, head: f, tail: f, content: f, path: "/d/kept2/f"},
-	}
-
-	super, dirs := treeOf(t, recs)
-
-	exists := func(path string) bool { return path != "/d/gone" }
-
-	got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
-
-	want := [][]string{{"/d/kept1", "/d/kept2"}}
-	if !slices.EqualFunc(got, want, slices.Equal) {
-		t.Errorf("groups = %v, want %v", got, want)
-	}
-}