Name the database each command uses on stderr (closes #111)

scan, report and trees now print one line on stderr before anything
else, naming the database and whether its path is the default or comes
from SFDUPES_DATABASE; scan also says when the file is new. report and
trees print it before opening the database, so a missing-database error
follows it. A relative path gets the working directory put in front of
it without cleaning: cleaning drops a ".." with the directory before
it, which names another file when that directory is a symlink. README
"Database" lists when scan writes each file.

Unverified: the line on a terminal has no test of its own.

Model: opus-5-5

README.md

@@ -38,9 +38,8 @@ database persists between runs; a rescan only hashes files that are new or
 changed, or that may have gained a duplicate since the last scan, and removes
 records for files that no longer exist. `report` reads the database and prints
 the file-level duplicates report. `trees` reads the same database and prints the
-duplicate-tree report. `info` prints the database's details: its path, the sizes
+duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
-of its files and the number of its records. A missing/invalid subcommand — or a
+with no `PATH` operand — prints a usage message and exits 2.
-`scan` invocation with no `PATH` operand — prints a usage message and exits 2.
 
 The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
 by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
@@ -49,40 +48,15 @@ results are as fresh as the last completed scan.
 
 ### Install
 
-With Go installed, this builds and installs the newest release:
+With Go installed, this builds and installs the current `main` branch:
-
-```sh
-go install sneak.berlin/go/sfdupes@latest
-```
-
-Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time
-`main` moves and its tip passes `script/cibuild`, that tip is tagged with the
-next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build.
-`@main` installs the tip before it is tagged:
 
 ```sh
 go install sneak.berlin/go/sfdupes@main
 ```
 
-`v0.0.1` is an early build without a database. It does not work as this README
+The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. A
-describes, and `go.mod` retracts it.
+binary installed this way reports its version as `dev`; one built from a clone
-
+or into the Docker image carries the git tag or commit it was built from.
-The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To
-see which version an installed binary is:
-
-```sh
-sfdupes --version
-go version -m "$(command -v sfdupes)"
-```
-
-`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`,
-`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`,
-or for `@main` a pseudo-version ending in the first 12 characters of the
-commit's hash. A binary built from a clone or into the Docker image carries the
-git tag or commit it was built from instead, and `VERSION` is `dev` only when
-the binary carries no version at all. `go version -m` prints what Go recorded in
-the binary; its `mod` line names the module's version. Binaries from `v0.0.2`
-and earlier print `dev`, so check those with `go version -m`.
 
 From a clone, `make build` writes the binary to `./sfdupes`:
 
@@ -123,13 +97,10 @@ As a file `/etc/cron.d/sfdupes`:
   `SFDUPES_DATABASE=/path/to/db.sqlite` before the command on the same line.
 - `scan` writes nothing to stdout. Its stderr, appended here to
   `/var/log/sfdupes.log`, holds a first line naming the database (see
-  "Database"), plain progress lines naming each phase and its place among the
+  "Database"), a plain progress line as each phase starts and then at most every
-  five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
+  5 seconds, a warning for each path it skips, and the summary line (see
-  runs, and one as it ends), a warning for each path it skips, a line for each
+  "Progress" and "`scan` mode"). The log grows with every scan; rotate it like
-  default exclude its walk does not enter and for each path it leaves out
+  any other.
-  because it is not on a persistent filesystem (see "Rules for the walk"), and
-  the summary line (see "Progress" and "`scan` mode"). The log grows with every
-  scan; rotate it like any other.
 - Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
   scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
   reason among the last lines of the log; `tail` prints them, and cron mails
@@ -137,9 +108,9 @@ As a file `/etc/cron.d/sfdupes`:
 - A scan still running when the next one starts carries on. The new one fails at
   once, and the lines cron mails include
   `sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
-- `report`, `trees` and `info` need only read access to the database (see
+- `report` and `trees` need only read access to the database (see "Database").
-  "Database"). Under the usual umask of `022` the first scan creates it readable
+  Under the usual umask of `022` the first scan creates it readable by every
-  by every user, so an unprivileged user can run them against root's database.
+  user, so an unprivileged user can run them against root's database.
 
 ### Reading the reports
 
@@ -227,13 +198,12 @@ Goals, in order:
     under resilver). Holding one small record (path, size, mtime) per file in
     memory during a scan is acceptable; holding every file's hashes is not (they
     stay in the database). The reporting commands do not hold every file's
-    hashes either: `report` reads the records in signature order and holds the
+    hashes either: `report` lets SQLite group and order the records and writes
-    duplicate paths of one size at a time, so its memory grows with the
+    each row as it reads it, so its memory does not grow with the database, and
-    duplicates of the size that has the most of them, and `trees` reads the
+    `trees` reads the records in path order and keeps each directory's path,
-    records in path order and keeps each directory's path, digest and totals,
+    digest and totals, plus the hashes of only the files in the directories
-    plus the hashes of only the files in the directories holding the record
+    holding the record being read, so its memory grows with the number of
-    being read, so its memory grows with the number of directories and with the
+    directories and with the size of the largest directory.
-    size of the largest directory.
 3.  **Scan incrementally, analyze offline.** The expensive filesystem scan
     maintains a persistent database; an unchanged file is never read again on a
     rescan, except to compute its content hash once a file of 10 MiB or more
@@ -263,7 +233,7 @@ Goals, in order:
 
 ### Subcommands
 
-Four subcommands, all implemented:
+Three subcommands, all implemented:
 
 1.  `scan` — walk the filesystem and synchronize the database: one signature
     record per regular file.
@@ -271,30 +241,26 @@ Four subcommands, all implemented:
 3.  `trees` — tree-level duplicate report: reconstruct the directory hierarchy
     from the database records, compute a Merkle-style digest per directory, and
     report maximal groups of identical trees.
-4.  `info` — the database's details: its path, the sizes of its files, its
-    schema version and the counts of its records.
 
 ```
-sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
+sfdupes scan [--workers N] [-x] PATH...
 sfdupes report > dupes.tsv
 sfdupes trees > dupetrees.tsv
-sfdupes info
 sfdupes --version
 sfdupes [command] --help
 ```
 
 `--workers N` sets the size of each `scan` worker pool (default: the number of
-CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
+CPUs), and `-x` (`--one-file-system`) keeps the walk of each operand on that
-operand's filesystem, and `--no-default-excludes` lets the walk enter the
+operand's filesystem; both are described under "`scan` mode".
-default excludes, a list of paths; all three are described under "`scan` mode".
+`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
-`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see
+exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
-"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`,
+subcommand, prints the help text to stderr and exits 0, writing nothing to
-alone or after a subcommand, prints the help text to stderr and exits 0, writing
+stdout.
-nothing to stdout.
 
 ### Database
 
-All four subcommands operate on a single SQLite database file:
+All three subcommands operate on a single SQLite database file:
 
 - Location: the value of the `SFDUPES_DATABASE` environment variable when set
   and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
@@ -313,17 +279,13 @@ All four subcommands operate on a single SQLite database file:
     trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
     ```
 
-    The working directory put in front is the one `pwd -P` prints, with its
+    `scan` says `new database` when no file is at that path as it starts.
-    symlinks resolved. After a `cd` through a symlink to a directory, a path
+    `report` and `trees` print the line before they open the database, so the
-    starting `../` therefore names a file beside the directory the symlink leads
+    error for a missing database comes after it.
-    to, as it does for any other program. `scan` says `new database` when no
-    file is at that path as it starts. `report`, `trees` and `info` print the
-    line before they open the database, so the error for a missing database
-    comes after it.
 
-- `scan` creates the database (and its parent directory) on first use. `report`,
+- `scan` creates the database (and its parent directory) on first use. `report`
-  `trees` and `info` require an existing database; a missing database file is a
+  and `trees` require an existing database; a missing database file is a fatal
-  fatal error (exit 1) telling the user to run `scan` first.
+  error (exit 1) telling the user to run `scan` first.
 - Only one `scan` runs against a database at a time. For its whole run, `scan`
   holds an exclusive `flock(2)` lock on a lock file beside the database, named
   by appending `.lock` to the database's path with symlinks resolved
@@ -339,8 +301,8 @@ All four subcommands operate on a single SQLite database file:
   on. The lock file is created on first use, open to its owner only, and left in
   place: a leftover file blocks nothing, because the lock ends with the process
   holding it however it ends, a fatal error or an interrupt included, and
-  deleting the file while a scan runs would let a second scan start. `report`,
+  deleting the file while a scan runs would let a second scan start. `report`
-  `trees` and `info` never take the lock, so they run during a scan.
+  and `trees` never take the lock, so they run during a scan.
 - While `scan` runs, the database is in WAL journal mode with a busy timeout, so
   running a report while a cron `scan` is in progress is safe. The filesystem is
   authoritative; the database is an eventually-consistent reflection of it.
@@ -356,9 +318,9 @@ All four subcommands operate on a single SQLite database file:
   needs the database to itself: a `scan` that starts while a report is still
   reading waits for it up to the 10-second busy timeout, then fails; a `scan`
   that ends while a report has the database open warns and leaves the database
-  in WAL mode until the next scan. `report` writes its rows while it reads the
+  in WAL mode until the next scan. `report` writes each row as it reads it, so
-  records, so it is still reading while its output is paused (a pager, a stalled
+  it is still reading while its output is paused (a pager, a stalled pipe), and
-  pipe), and a `scan` started then fails after the busy timeout.
+  a `scan` started then fails after the busy timeout.
 - A `scan` writes to the database and the files beside it at these points (its
   phases are described under "`scan` mode"):
     - As it starts, before the walk: the directory, an empty database file and
@@ -373,10 +335,10 @@ All four subcommands operate on a single SQLite database file:
     - In the content phase: content hashes, again committed 10,000 at a time.
     - At the end, as it closes the database: the `-wal` and `-shm` files go
       away, unless a report still has the database open (see above).
-- `report`, `trees` and `info` open the database read-only and need only read
+- `report` and `trees` open the database read-only and need only read access to
-  access to the database file, and no write access to its directory. While the
+  the database file, and no write access to its directory. While the database is
-  database is in WAL mode they also read the `-wal` and `-shm` files beside it,
+  in WAL mode they also read the `-wal` and `-shm` files beside it, which SQLite
-  which SQLite creates with the database file's permissions.
+  creates with the database file's permissions.
 - Schema (`PRAGMA user_version` is the schema version, currently 1; a database
   with any other version is a fatal error. `scan` creates the schema and sets
   the version in one transaction, so a first scan stopped while doing so leaves
@@ -412,8 +374,8 @@ All four subcommands operate on a single SQLite database file:
     a file of 10 MiB or more, `content` stays empty until the content phase of a
     scan (see "`scan` mode" below) has read the file. A record with an empty
     `content` is never part of a duplicate group, though it still defines the
-    file for tree reconstruction. The `files_signature` index lets `report` read
+    file for tree reconstruction. The `files_signature` index lets SQLite group
-    the records in signature order without sorting the table.
+    the records by signature for `report` without sorting the whole table.
 
 ### Duplicate detection
 
@@ -467,12 +429,11 @@ results do not depend on the working directory. All operands belong to a single
 scan and are enumerated concurrently: every operand seeds the shared walk worker
 pool. Overlapping operands are harmless — an operand that duplicates another or
 lies under another is dropped before walking, so every file is reached exactly
-once and produces one database record. An operand under another one is kept,
+once and produces one database record. With `-x`, though, an operand under
-though, and walked on its own when any directory from it (from its directory,
+another one is kept and walked on its own when any directory from it (from its
-for a regular file) up to but not including the operand covering it is a default
+directory, for a regular file) up to but not including the operand covering it
-exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
+is on a different filesystem than that operand, because that operand's walk does
-`-x`, is on a different filesystem than that operand, because that operand's
+not enter such a directory and so does not reach it.
-walk does not enter such a directory and so does not reach it.
 
 An operand that is a symlink (never followed, not even as an operand), socket,
 FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -482,10 +443,7 @@ beneath it is still scanned. The records stored beneath it are not deleted: they
 are treated like any other record outside the scanned operands, including the
 content-phase exception below. If it lies under another operand, they are under
 that operand instead, and are treated like any other record there. This is not
-an error: a scan whose every operand is dropped walks nothing and exits 0. An
+an error: a scan whose every operand is dropped walks nothing and exits 0.
-operand that is not on a persistent filesystem (see "Rules for the walk") is
-dropped the same way, except that the line `scan` prints for it is not a
-warning, and it is not counted as skipped.
 
 `scan` synchronizes the database with the filesystem state under the scanned
 operands:
@@ -521,26 +479,19 @@ operands:
   can be scanned on different schedules into the same database. The one
   exception is the content phase below: a stored file of 10 MiB or more without
   a `content` hash is read for one, wherever it lies, once its size, `head`, and
-  `tail` match another record's. If that file is gone, has changed since its
+  `tail` match another record's. If that file is gone or has changed since its
-  record was written, or is not on a persistent filesystem, it is not read and
+  record was written, the record is left as it is.
-  the record is left as it is.
+- With `-x`, a record below a directory the walk did not enter because that
-- A record at or below a path the walk left out, because that path is a default
+  directory is on another filesystem than its operand counts as outside the
-  exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
+  scanned operands: it is not deleted, and its size joins the size census. A
-  another filesystem than its operand, counts as outside the scanned operands:
+  filesystem mounted under an operand can therefore be scanned on its own
-  it is not deleted, and its size joins the size census. A filesystem mounted
+  schedule into the same database. Records below an operand at or inside such a
-  under an operand can therefore be scanned on its own schedule into the same
+  directory are not kept this way, since that operand's own walk verifies them.
-  database. Records below an operand at or inside such a path are not kept this
-  way, since that operand's own walk verifies them.
 
-`scan` runs **five sequential phases over the whole scan**. Parallelism lives
+`scan` runs **four sequential phases over the whole scan**. Parallelism lives
-inside the walk, hash and content phases; batched database writes begin during
+inside each phase; batched database writes begin during the hash phase:
-the hash phase:
 
-1.  **load** — read the existing records. Each record under the operands is kept
+1.  **walk + stat** — enumerate the trees under all `PATH` operands concurrently
-    in memory, by path, with its size, its mtime and whether it has hashes, for
-    the walk to compare each file with; the size of every other record joins the
-    size census.
-2.  **walk + stat** — enumerate the trees under all `PATH` operands concurrently
     with the walk worker pool: every operand seeds the shared queue, and each
     worker reads one directory at a time, handing discovered subdirectories back
     to the queue and running `lstat` on each regular file as it is discovered
@@ -550,7 +501,7 @@ the hash phase:
     or busy pools. The walk builds the size census and resolves unchanged
     already-hashed files on the fly; every other file is carried to the hash
     phase as a (path, size, mtime) record.
-3.  **hash** — with the census complete, each carried file's size decides its
+2.  **hash** — with the census complete, each carried file's size decides its
     fate. Size-unique files are never read: new or changed ones are recorded
     without hashes in the update phase, unchanged unhashed ones simply keep
     their records. Every file with a shared size is hashed by the worker pool as
@@ -566,30 +517,29 @@ the hash phase:
     **while hashing runs**, so a scan interrupted after hours keeps everything
     hashed so far and the next scan resumes cheaply, skipping records already
     written.
-4.  **update** — commit the final partial batch, the hash-less records for
+3.  **update** — commit the final partial batch, the hash-less records for
     size-unique new and changed files, and the deletions for records the scan
     did not verify (vanished files, plus paths that failed to stat or hash).
-5.  **content** — find every record of 10 MiB or more without a `content` hash
+4.  **content** — find every record of 10 MiB or more without a `content` hash
     whose size, `head`, and `tail` equal another record's, anywhere in the
     database: records from this scan and records stored by earlier scans, inside
     or outside the scanned operands. SQLite finds them, so only the records to
     be read are kept in memory, never every file's hashes. Every record sharing
     their size, `head`, and `tail`, including one that already has a `content`
     hash, has its file checked with `lstat` first. A file that is gone, is no
-    longer a regular file, has changed (a different size, or an mtime newer than
+    longer a regular file, or has changed (a different size, or an mtime newer
-    recorded), or is not on a persistent filesystem (see "Rules for the walk")
+    than recorded) keeps its record as it is and does not count as a match for
-    keeps its record as it is and does not count as a match for the others. Any
+    the others. Any other `lstat` error is warned about and counted as skipped,
-    other `lstat` error is warned about and counted as skipped, with the same
+    with the same result. If such a record has no `content` hash, it stays out
-    result. If such a record has no `content` hash, it stays out of duplicate
+    of duplicate groups; if it has one, it is still reported until a scan
-    groups; if it has one, it is still reported until a scan covering its own
+    covering its own tree updates or removes it. The files that pass and have no
-    tree updates or removes it. The files that pass and have no `content` hash
+    `content` hash are read only if at least two of those records pass, so a
-    are read only if at least two of those records pass, so a file whose only
+    file whose only matches are stale costs no read; a file that already has a
-    matches are stale costs no read; a file that already has a `content` hash is
+    `content` hash is never read again. They are read by a worker pool as in the
-    never read again. They are read by a worker pool as in the hash phase, in
+    hash phase, in inode order and once per inode, and their content hashes are
-    inode order and once per inode, and their content hashes are committed in
+    committed in batches. A failed read is warned about and counted as skipped;
-    batches. A failed read is warned about and counted as skipped; its record
+    its record keeps an empty `content`, so it is not a duplicate, and a later
-    keeps an empty `content`, so it is not a duplicate, and a later scan tries
+    scan tries again.
-    again.
 
 Rules for the walk:
 
@@ -600,15 +550,6 @@ Rules for the walk:
 - Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
   them would list every file once per snapshot), not even when it is an operand;
   such an operand is dropped the same way.
-- Never record the scan's own database. Once the database and its `.lock` file
-  are open, `scan` notes the device and inode of the database file, its `-wal`
-  and `-shm` files and its `.lock` file, and the walk passes over a file with
-  one of them, however it reaches it: as an operand, below one, or through a
-  symlinked directory. It prints no line for such a file and does not count it
-  as skipped. A record that an earlier scan left for one of them is deleted,
-  like the record of any file the scan no longer verifies. There is no
-  `-journal` file during the walk, because the database is in WAL mode then (see
-  "Database").
 - Filesystem boundaries are crossed by default. With `-x` (long form
   `--one-file-system`, following the GNU `du`/`rsync` convention), never descend
   into a directory on a different filesystem than its `PATH` operand; each
@@ -616,74 +557,6 @@ Rules for the walk:
   unless they lie below an operand at or inside that directory, count as outside
   the scanned operands, as described in "`scan` mode" above: they are not
   deleted, and their sizes join the size census.
-- Never record anything that is not on a persistent filesystem. A filesystem is
-  not persistent when the kernel makes it up or keeps it in memory: one of type
-  `proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
-  `debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
-  `mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
-  `rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
-  it names the database and before it takes the lock, `scan` lists the mounted
-  filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
-  of mounted filesystems on macOS (on any other system it lists none), and notes
-  the device ID of each one of those types. On Linux that is the device ID the
-  list gives, so a filesystem that a later mount hides still gives its own. On
-  macOS it is read with `lstat` on the mount point, and where one filesystem is
-  mounted over another on the same mount point, the one mounted last counts. If
-  `scan` cannot list them, it fails (exit 1). The walk then never enters a
-  directory, and never records a file, on one of those devices, however it
-  reaches it: as an operand, below one, or through a bind mount, which keeps the
-  device ID of what it binds. The content phase never reads a file on one of
-  them either. No flag turns this off. For each operand, directory or file it
-  leaves out this way, `scan` prints a line on stderr naming it and the type of
-  its filesystem, as it prints a warning, so it never lands inside a progress
-  display:
-
-    ```
-    scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
-    ```
-
-    That line is not a warning, and the path is not counted as skipped. Records
-    that an earlier scan left at or below it, unless they lie below an operand
-    at or inside it, count as outside the scanned operands, as described in
-    "`scan` mode" above: they are not deleted, and their sizes join the size
-    census.
-
-- Below an operand, never descend into a default exclude: a path that holds the
-  kernel's view of the running system, or temporary files, so that
-  `sfdupes scan /` walks only the files worth comparing. The default excludes
-  depend on the system, and `sfdupes scan --help` lists those of the system it
-  runs on:
-    - On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
-    - On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
-      `/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
-      everything in `/System/Volumes/Data` through the firmlinks, so walking it
-      as well would record each of those files twice.
-    - On Windows, relative to the root of the drive an operand lies on: the
-      page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
-      `hiberfil.sys`, and the directories `System Volume Information`,
-      `$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
-      matches one path component, as it does in any default exclude. sfdupes
-      does not build for Windows yet, so this list is not used.
-    - On any other system there are none.
-
-    `--no-default-excludes` turns them all off; what is not on a persistent
-    filesystem stays left out. An operand at or inside a default exclude is
-    still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
-    persistent filesystem. For each default exclude its walk does not enter,
-    `scan` prints a line on stderr naming it, as it prints a warning:
-
-    ```
-    scan: skipped /var/tmp (default exclude)
-    ```
-
-    That line is not a warning, and the directory is not counted as skipped. A
-    default exclude that is not on a persistent filesystem, as `/proc` is, gets
-    the line for its filesystem instead, and one that is itself an operand gets
-    no line, since it is walked as that operand. The records below a default
-    exclude the walk does not enter, unless they lie below an operand at or
-    inside it, count as outside the scanned operands, as described in "`scan`
-    mode" above: they are not deleted, and their sizes join the size census.
-
 - On any per-path error (permission denied, file vanished between passes,
   unreadable): print a one-line warning to stderr, skip the path, and continue.
   Per-file errors never abort the run; the final summary reports how many were
@@ -701,9 +574,8 @@ files. `--workers` must be at least 1: a smaller value is a usage error,
 reported in one line on stderr with exit 2 before anything is scanned. All three
 phases are seek-bound on spinning disks, so raising `--workers` well past the
 core count can help on pools with many spindles. The main goroutine owns
-partitioning, database writes, and the progress counts, which each progress
+partitioning, database writes, and progress rendering; progress display must
-display draws from a goroutine of its own; progress display must never block the
+never block the workers.
-workers.
 
 `scan` writes nothing to stdout. The summary line on stderr reports the files
 seen this run broken down by disposition, plus skips:
@@ -721,8 +593,11 @@ total.)
 
 **`report` must never touch the filesystem being analyzed.** It does not stat,
 open, or otherwise access any path that appears in the records; its only I/O is
-reading the database and writing stdout/stderr. `report` must produce identical
+reading the database, writing stdout/stderr, and the temporary file SQLite sorts
-output whether or not the scanned filesystem is still mounted.
+in when the duplicate rows do not fit in memory. SQLite puts that file in
+`$SQLITE_TMPDIR` or `$TMPDIR` when set, otherwise in `/var/tmp` (or `/tmp`), and
+deletes it as soon as it has opened it. `report` must produce identical output
+whether or not the scanned filesystem is still mounted.
 
 Processing:
 
@@ -736,16 +611,6 @@ Processing:
   by `first` path ascending. Output must be fully deterministic for a given
   database state.
 
-`report` runs in two phases, each with its own progress display on stderr (see
-"Progress"): `open` opens the database; `read` counts the records, then reads
-them in signature order, largest size first. That is the order of the
-`files_signature` index read backwards, so SQLite sorts nothing and returns the
-first record at once. The records sharing a signature come together, and so do
-the records of one size, so `report` holds only the duplicate groups of the size
-it is reading, and writes them, ordered by `first` path, once it reaches a
-smaller size. Its memory holds at most the duplicate paths of one size at a
-time, never every record.
-
 #### Report output format
 
 TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
@@ -819,12 +684,6 @@ Processing:
 - Order groups by total tree size descending, tie-broken by `first` path
   ascending. Output must be fully deterministic for a given input.
 
-`trees` runs in four phases, each with its own progress display on stderr (see
-"Progress"): `open` opens the database; `read` counts the records, then reads
-them in path order, building the hierarchy and computing the digests as it goes;
-`group` groups the directories by digest and picks the maximal groups; `write`
-writes the groups to stdout.
-
 #### Trees output format
 
 TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
@@ -843,192 +702,57 @@ Summary to stderr: records read, number of duplicate-tree groups, number of dupe
 trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
 units.
 
-### `info` mode
-
-`info` takes no positional arguments and prints the database's details on
-stdout, one per line: a name, a tab, and the value. It opens the database the
-way `report` does, read-only and without the scan lock, so it runs during a scan
-and needs only read access (see "Database"). Like `report`, it never touches the
-scanned filesystem.
-
-```
-database	/var/lib/sfdupes/db.sqlite (default)
-database file size	2147483648 (2.0 GiB)
--wal file size	4124152 (3.9 MiB)
-modified	2026-10-16T12:02:50Z
-schema version	1
-records	12345678
-records with head and tail hashes	1234567
-records with content hash	123456
-size of recorded files	150000000000000 (136.4 TiB)
-```
-
-- `database` is the path and where it came from, as on the line every subcommand
-  prints first on stderr (see "Database").
-- `database file size` and `-wal file size` are in bytes, then in human units.
-  The `-wal` line is there only when a `-wal` file is beside the database, as
-  while a scan runs.
-- `modified` is the time the database file was last changed, in UTC. While the
-  database is in WAL mode, the newest changes may be in the `-wal` file alone.
-- `schema version` is the database's `PRAGMA user_version`. A database with
-  another version than this build's is refused, as by `report`.
-- `records` is the number of records. `records with head and tail hashes` and
-  `records with content hash` count the records that have those hashes (see
-  "Database"), and `size of recorded files` adds up the sizes of all records, in
-  bytes, then in human units.
-
-`info` runs in two phases, each with its own progress display on stderr (see
-"Progress"): `open` opens the database; `count` reads the schema version and
-counts the records with one query, a single pass over the table, both in one
-read-only transaction, so the numbers agree with each other while a scan is
-committing. `info` writes its lines once the `count` display has ended, so on a
-terminal they come after the displays.
-
 ### Progress
 
-Use the progress-bar library for all progress of `scan`, `report`, `trees` and
+Use the progress-bar library for all scan progress; rendering in the style of
-`info`; rendering in the style of `pv` is the model. All progress goes to
+`pv` is the model. All progress goes to stderr.
-stderr.
 
-`scan`, `report`, `trees` and `info` must always show that they are working and
+Each phase gets its own display, rendered the moment the phase starts — a scan
-not hung. Each phase gets its own display from the moment it starts, naming the
+must never look hung. Loading the existing-record index (`load`) and the walk
-phase and its place among the command's phases and showing its progress, its
+have no known totals while running: show a live count, rate, and elapsed time
-elapsed time and its estimated time remaining (ETA).
+(spinner-style, no percentage or ETA). The content phase's display (`content`)
+starts the same way, counting the records checked while SQLite finds the files
+to read and `lstat` checks them, then shows a bar once reading starts. The hash
+and update phases, and the content phase's reads, have exact totals — only files
+that actually need hashing appear in the hash and content totals, so their ETAs
+are meaningful. Required elements for the bars with known totals:
 
-A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
-`[4/5] update`, `[5/5] content`):
-
-- `load` counts the records first, then reads them against that count. While the
-  count runs, the display says it is counting the records.
-- `walk` has no exact total. Its estimate is the number of records the last scan
-  left under the operands, known once `load` is done and shown with a `~`. Once
-  the walk passes the estimate, or when there are no such records, as on a first
-  scan, the ETA is unknown.
-- `hash` and `update` have exact totals. Only files that actually need hashing
-  appear in the hash total, so its ETA is meaningful.
-- `content` first counts the records whose files it checks with `lstat`, which
-  SQLite does by adding up the number of records with each matching size,
-  `head`, and `tail`, without fetching the records. It then says it is searching
-  the database until SQLite returns the first record; the checks, and their rate
-  and ETA, start there. The files to read then get a display of their own, with
-  their count as its total, and so does the writing of the last of their records
-  to the database, with the count of those records as its total.
-
-`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
-`[4/4] write`, see "`trees` mode"):
-
-- `open` has no total. While it opens the database, waiting on a busy one
-  included, the display says it is opening the database.
-- `read` counts the records first, then reads them against that count. While the
-  count runs, the display says it is counting the records.
-- `group` has the directories `read` found as its total, and `write` the
-  duplicate-tree groups.
-
-`report` has two phases (`[1/2] open`, `[2/2] read`, see "`report` mode"):
-
-- `open` has no total. While it opens the database, waiting on a busy one
-  included, the display says it is opening the database.
-- `read` counts the records first, then reads them against that count, writing
-  the rows to stdout as it goes. While the count runs, the display says it is
-  counting the records.
-
-`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
-
-- `open` has no total. While it opens the database, waiting on a busy one
-  included, the display says it is opening the database.
-- `count` has no total either, because one query counts everything: the display
-  says it is counting the records until the query returns, and its ETA is
-  unknown.
-
-Each display is one line holding:
-
-- the phase and its place among the phases
-- a `[m/n] x%` display (items processed / total items, percent); before a phase
-  has a total, the items processed, or what the phase is waiting for
-- current rate (items/s), naming the items the phase counts: files, records,
-  directories or groups. It covers the items completed over the last 5 seconds,
-  or, when stderr is not a terminal, since the line before, so it falls to 0
-  while nothing completes, and a burst of items after a wait shows as a burst
-  rather than as a steady rate. Once no item has completed for 10 seconds, the
-  word `stalled` and where the phase waits take the rate's place (see below).
 - elapsed time
-- estimated time remaining, or `unknown`. It comes from the phase's average rate
+- estimated time remaining
-  since it started counting, so that it does not swing with each burst.
+- a `[m/n] x%` display (items processed / total items, percent)
+- current rate (items/s)
 
 Example shape (exact layout is flexible, content is not):
 
 ```
-[3/5] hash: [12345/98765] 12% |████      | 92 files/s elapsed 2:32 eta 17:54
+hash: [12345/98765] 12% |████      | 92 files/s elapsed 2:32 eta 17:54
-[2/5] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
-[3/5] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
 ```
 
-When no item has completed for 10 seconds, as while one large file is read, a
-disk is slow or the database is busy, the line says `stalled` and where the
-phase waits in place of the rate, which would read 0: `disk` for files, `db` for
-the database, `out` for stdout, and two of them, as `disk/db`, for a phase that
-may wait on either. `walk` says `disk`. `hash` and the content phase's reads say
-`disk/db`, as they read files and write their records to the database between
-them, and so do the content phase's checks, which `lstat` each record's file as
-the database search returns it. `load`, `update`, the content phase's last write
-and the `read` phase of `trees` say `db`. `trees`' `write` says `out`, and its
-`group`, which works in memory, says `cpu`. The `read` phase of `report` writes
-the report while it reads the records, so it says `db/out`. A phase still
-waiting for its total already says what it waits for, and says nothing more.
-
 Additional requirements:
 
-- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
-  the `open` phase of `report` and `trees`, and both phases of `info`, once it
-  has counted what it will do) is drawn as its line followed by the progress-bar
-  library's bar, 10 columns wide, which shows the percent in place of the line's
-  own. Any other phase, and a phase whose bar has no room, is drawn as the
-  library's spinner followed by its line. Each display is drawn again several
-  times a second, so its elapsed time keeps moving while a phase waits for its
-  next item.
-- On a terminal no line reaches the terminal's last column, however narrow the
-  terminal is. A line too wide for the terminal first loses its bar, which gives
-  way to the spinner, then its rate, the words of a stalled phase, or what the
-  phase is waiting for. The phase and its place, the progress, the elapsed time
-  and the ETA are kept whole. On an 80-column terminal, at counts like those of
-  the stalled line in the example above, a stalled line keeps its words whenever
-  the elapsed time and the ETA are both in minutes, except in the content
-  phase's checks and reads. Their lines are the widest, and at most times in
-  minutes they lose the words. Where the words and the ETA do not both fit, as
-  at millions of files and times in hours, the words give way. Only when those
-  four alone are too wide, as on a terminal narrower than about 70 columns hours
-  into a scan of millions of files, is the line cut short at the terminal's
-  edge, so the end of the ETA is lost first. A terminal made narrower while a
-  phase runs gets the same: the next draw erases the line within the new width
-  and fits the line to it, and the display goes on redrawing in place.
 - When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
-  redraws: print the line the moment each phase starts, every 5 seconds while it
+  redraws: print a plain one-line progress update the moment each phase starts,
-  runs (whether or not an item has completed, and never more often), and once
+  then no more often than every 5 seconds.
-  more as the phase ends.
+- Progress updates are driven from the main goroutine and must be non-blocking
-- Progress must be non-blocking with respect to the worker pool: the main
+  with respect to the worker pool. On a terminal the spinner-style displays also
-  goroutine counts the items, and each display draws them from a goroutine of
+  redraw on their own several times a second, so their count and elapsed time
-  its own.
+  stay current while a phase waits for its next item.
 - A warning printed during a phase always lands on a line of its own, never
-  inside the progress display. So does every line written to stdout when stdout
+  inside the progress display.
-  and stderr are the same file (the same device and inode): one terminal, as in
+- A bar whose phase stops short of its total, as an interrupted one does, is
-  a plain `sfdupes report` or `sfdupes trees`, or one pipe or file that both are
+  left as last drawn rather than filled up.
-  sent to. When they are not, stdout is written without waiting on the display,
+- `report` and `trees` modes need no progress display, only their stderr
-  and the display keeps being drawn while a slow reader of stdout holds up a
+  summaries.
-  write.
-- A phase that stops short of its total, as an interrupted one does, is left
-  showing the count it stopped at rather than filled up.
 
 ### Error handling and exit codes
 
 - `0`: success, even if individual files were skipped with warnings.
 - `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is
   already running against the same database, the database cannot be
-  created/opened/read/written, a missing database for `report`/`trees`/`info`,
+  created/opened/read/written, a missing database for `report`/`trees`, stdout
-  stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see
+  write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see below).
-  below).
 - `2`: usage error (including `scan` with no `PATH` operand, `scan` with
-  `--workers` below 1, and `report`/`trees`/`info` with any positional
+  `--workers` below 1, and `report`/`trees` with any positional argument).
-  argument).
 
 A stdout write failure, such as a full disk, is reported in one line on stderr
 and exits 1. Two cases never reach sfdupes as a failed write:
@@ -1195,6 +919,10 @@ All of the following, run in this directory, must pass:
     ./sfdupes report
     ```
 
+    (The database lives in a temp directory of its own: inside `$d`, the scan
+    would record it, and its empty lock file would join the `empty1`/`empty2`
+    group.)
+
     Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
     group (two dupe rows, `first` is the lexicographically smallest path);
     `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/

TODO.md

@@ -29,113 +29,6 @@
 
 # Completed Steps
 
-- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so
-  `go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has
-  passed the build; README "Install" says so, says what `@main` and `v0.0.1`
-  give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the
-  early build without a database. When the linker set no version,
-  `sfdupes --version` prints the main module's version that Go recorded in the
-  binary, the tag or the pseudo-version of a `go install` build, and `dev` only
-  when neither is known (2026-10-17,
-  https://git.eeqj.de/sneak/sfdupes/issues/116)
-
-- `scan` creates the database's directory where the database path leads: the
-  path with its last element cut off as text, never cleaned, so a `..` after a
-  symlinked directory is followed through the symlink, as when the file is
-  opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
-  symlink to `/data/x`, it creates `/data/new` and the database in it, where it
-  used to create `/srv/new` and then fail (2026-10-17,
-  https://git.eeqj.de/sneak/sfdupes/issues/119)
-
-- `scan` never records its own database: once the database and its `.lock` file
-  are open, it notes the device and inode of the database file, its `-wal` and
-  `-shm` files and its `.lock` file, and the walk passes over a file with one of
-  them wherever the operands reach it, through a symlinked directory included,
-  printing no line and counting nothing. A record an earlier scan left for one
-  of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
-
-- `scan` never records or reads anything that is not on a persistent filesystem
-  (`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
-  matched by the device ID of each such filesystem, listed as the scan starts
-  from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
-  however the walk or the content phase reaches it; no flag turns that off.
-  Below an operand, its walk does not enter the default excludes either, so
-  `sfdupes scan /` skips the kernel's view of the running system and the
-  temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
-  `/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
-  `/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
-  Windows list, not used yet. An operand at or inside a default exclude is still
-  walked. The scan prints a line for each path it leaves out, without counting
-  it as skipped, and the records there count as outside the scanned operands, as
-  for `-x`. `--no-default-excludes` turns off the default excludes only, and
-  `scan --help` lists them (2026-10-17,
-  https://git.eeqj.de/sneak/sfdupes/issues/124)
-
-- the rate a progress display shows covers the items completed over the last 5
-  seconds, or since the line before when stderr is not a terminal, so it falls
-  to 0 while nothing completes, and a burst after a wait no longer reads as a
-  steady rate of over 1,000 files a second; the ETA still comes from the phase's
-  average. Once no item has completed for 10 seconds, the word `stalled` and
-  where the phase waits take the rate's place, as `stalled disk/db`: `disk`,
-  `db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
-  either, each phase giving its word where it is defined. A line too wide for
-  the terminal loses its bar first, then its rate or those words, so the phase,
-  progress, elapsed time and ETA stay whole. On an 80-column terminal, at counts
-  like those of README's example, the content phase's checks and reads lose
-  those words at most times in minutes, and every other phase keeps them
-  whenever the elapsed time and the ETA are both in minutes. The content phase's
-  last partial batch is now written under a display of its own (2026-10-16,
-  https://git.eeqj.de/sneak/sfdupes/issues/125)
-
-- new subcommand `info` prints the database's details on stdout, one per line as
-  a name, a tab and the value: its path and where it came from, the sizes of the
-  database file and of its `-wal` file when there is one, when the database file
-  was last changed, the schema version, the number of records, how many have
-  head and tail hashes and how many a content hash, and the sizes of all records
-  added up. It opens the database as `report` does, and shows two phases on
-  stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come
-  from one read-only transaction, the counts from one query (2026-10-16,
-  https://git.eeqj.de/sneak/sfdupes/issues/117)
-
-- `report` shows its work on stderr in two phases, `[1/2] open` and
-  `[2/2] read`, with the progress display `scan` and `trees` use, from the
-  moment it starts until the summary line: `open` says it is opening the
-  database, also while it waits on a busy one; `read` counts the records, then
-  reads them against that count in signature order, which SQLite takes from the
-  `files_signature` index without sorting, so the first record comes at once.
-  `report` forms the duplicate groups itself, holding only those of the size it
-  is reading, and writes them ordered by first path before it reads the next
-  size. stdout and the summary line are unchanged, and SQLite no longer needs a
-  temporary file to sort in (2026-10-16,
-  https://git.eeqj.de/sneak/sfdupes/issues/114)
-
-- `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`,
-  `[3/4] group` and `[4/4] write`, each with the progress display `scan` uses,
-  from the moment it starts until the summary line: `open` says it is opening
-  the database, also while it waits on a busy one; `read` counts the records,
-  then reads them against that count; `group` counts the directories it groups,
-  and `write` the groups it writes. A display's rate names what its phase
-  counts: files, records, directories or groups. stdout and the summary line are
-  unchanged. When stdout and stderr are the same file, as one terminal is, none
-  of stdout's lines shares a line with the display; when they are not, the
-  display keeps being drawn while stdout waits on a slow reader (2026-10-16,
-  https://git.eeqj.de/sneak/sfdupes/issues/113)
-
-- each `scan` phase's progress display names the phase and its place among the
-  five, as `[2/5] walk`, from the moment it starts, with its progress, elapsed
-  time and ETA: `load` counts the records first, the walk's total is estimated
-  from the records under the operands, and the content phase counts the records
-  it checks from SQLite's grouping alone, then says it is searching the database
-  until the first one is returned. On a terminal a phase with an exact total
-  draws its line followed by the library's bar, and any other a spinner followed
-  by the line; every display is drawn again several times a second. No line
-  reaches the terminal's last column, however narrow, also after the terminal is
-  made narrower during a phase: a line too wide leaves out its rate, then its
-  bar, keeping the phase, progress, elapsed time and ETA whole, and is cut short
-  only when those alone do not fit, as below about 70 columns late in a large
-  scan. In plain mode the line is printed every 5 seconds whether or not an item
-  completed (2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/112)
-
 - `scan`, `report` and `trees` first print a line on stderr naming the database
   they use, as an absolute path, and whether that path is the default or comes
   from `SFDUPES_DATABASE`; `scan` says when the database is new. README

cancel_test.go

@@ -196,7 +196,7 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
 	base := baselineGoroutines(t)
 
 	st, err := syncScan(newWalkClock(walkCancelAtDone), db,
-		[]string{dir}, walkCancelWorkers, false, walkRules{})
+		[]string{dir}, walkCancelWorkers, false)
 
 	assertWalkGuardAborted(t, st, err)
 	assertRecordsIntact(t, db, before)
@@ -268,8 +268,7 @@ func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
 	ctx, cancel := context.WithCancel(t.Context())
 	cancel()
 
-	st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false,
+	st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false)
-		walkRules{})
 	if !errors.Is(err, context.Canceled) {
 		t.Fatalf("syncScan on a cancelled context = %v, want %v",
 			err, context.Canceled)
@@ -302,7 +301,7 @@ func TestSyncScanCancelledMidHashKeepsHashedRecords(t *testing.T) {
 	db := openTestDB(t)
 
 	st, err := syncScan(newWalkClock(hashCancelAtDone), db,
-		[]string{dir}, walkCancelWorkers, false, walkRules{})
+		[]string{dir}, walkCancelWorkers, false)
 	if !errors.Is(err, context.Canceled) {
 		t.Fatalf("syncScan cancelled mid-hash = %v, want %v",
 			err, context.Canceled)
@@ -357,8 +356,7 @@ func TestRunScanInterrupted(t *testing.T) {
 	stderr := captureStderr(t)
 	dir := buildSmokeTree(t)
 
-	err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
+	err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
-		false)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -382,7 +380,7 @@ func TestRunScanInterrupted(t *testing.T) {
 		defer close(done)
 
 		err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
-			false, false)
+			false)
 	}()
 
 	awaitReturn(t, done, "runScan")
@@ -409,8 +407,7 @@ func TestRunScanInterrupted(t *testing.T) {
 			got, before)
 	}
 
-	err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
+	err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
-		false)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -436,7 +433,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
 	dir := buildWalkCancelTree(t)
 
 	err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
-		walkCancelWorkers, false, false)
+		walkCancelWorkers, false)
 	if !errors.Is(err, errInterrupted) {
 		t.Fatalf("runScan interrupted mid-hash = %v, want %v",
 			err, errInterrupted)
@@ -620,8 +617,7 @@ func TestWalkOneDirStopsWhenCancelled(t *testing.T) {
 	// Unbuffered and unread: on a cancelled scan every send gives up.
 	events := make(chan walkEvent)
 
-	subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false,
+	subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false, events)
-		walkRules{}, events)
 	if len(subs) != 0 {
 		t.Errorf("cancelled walkOneDir returned %+v to descend into, "+
 			"want none", subs)
@@ -641,8 +637,7 @@ func TestWalkWorkersDropQueuedDirs(t *testing.T) {
 
 	missing := filepath.Join(t.TempDir(), "missing")
 
-	jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false,
+	jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false)
-		walkRules{})
 
 	for range 64 {
 		jobs <- dirJob{path: missing}
@@ -673,7 +668,7 @@ func TestWalkWorkerAbandonsSubdirHandoff(t *testing.T) {
 	ctx, cancel := context.WithCancel(t.Context())
 	defer cancel()
 
-	jobs, subdirs, events := startWalkWorkers(ctx, 1, false, walkRules{})
+	jobs, subdirs, events := startWalkWorkers(ctx, 1, false)
 
 	// Fill the hand-back channel to its capacity — one slot per worker
 	// — so the worker's own hand-back is certain to block.

db.go

@@ -61,8 +61,8 @@ CREATE TABLE files (
 ) WITHOUT ROWID
 `
 
-// createIndexSQL indexes the records by signature, so report can read
+// createIndexSQL indexes the records by signature, so report can have
-// them in signature order without sorting the table.
+// SQLite group them without sorting the whole table.
 const createIndexSQL = `
 CREATE INDEX files_signature ON files (size, head, tail, content)
 `
@@ -79,8 +79,7 @@ ON CONFLICT (path) DO UPDATE SET
     content = excluded.content
 `
 
-// errNoDatabase reports a missing database file for report, trees and
+// errNoDatabase reports a missing database file for report/trees.
-// info.
 var errNoDatabase = errors.New(
 	"no database (run \"sfdupes scan\" first, or set " + databaseEnv + ")")
 
@@ -94,9 +93,9 @@ var errScanRunning = errors.New("another scan is running")
 
 // databasePath resolves the database location: SFDUPES_DATABASE when
 // set and non-empty, the compiled-in default otherwise. It returns the
-// path, with the working directory, its symlinks resolved, put in front
+// path, with the working directory put in front of it when it is
-// of it when it is relative, and where the path came from, "default" or
+// relative, and where the path came from, "default" or "from
-// "from SFDUPES_DATABASE", for the line each subcommand prints first on
+// SFDUPES_DATABASE", for the line each subcommand prints first on
 // stderr.
 func databasePath() (string, string, error) {
 	path, source := defaultDatabasePath, "default"
@@ -108,10 +107,7 @@ func databasePath() (string, string, error) {
 		return path, source, nil
 	}
 
-	// The working directory as the kernel reports it, with no symlink
+	wd, err := os.Getwd()
-	// in it, as README "Database" says. os.Getwd may return PWD
-	// instead, which can name it through a symlink.
-	wd, err := unix.Getwd()
 	if err != nil {
 		return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
 	}
@@ -123,24 +119,6 @@ func databasePath() (string, string, error) {
 	return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
 }
 
-// createDatabaseDirectory creates the directory the database at path
-// goes in, and any directory above it, if missing. path is absolute, as
-// databasePath returns it. The directory is path with its last element
-// cut off as text, by filepath.Split. filepath.Dir would clean it,
-// dropping each ".." together with the directory name before it; when
-// that name is a symlink, the cleaned path names another directory than
-// the one the system reaches when it opens path.
-func createDatabaseDirectory(path string) error {
-	dir, _ := filepath.Split(path)
-
-	err := os.MkdirAll(dir, dbDirPerm)
-	if err != nil {
-		return fmt.Errorf("create database directory: %w", err)
-	}
-
-	return nil
-}
-
 // scanParams are the connection parameters for scan: read-write, with
 // WAL journaling and a busy timeout, so a report can run while a cron
 // scan is in progress. closeScanDatabase leaves WAL mode again.
@@ -148,9 +126,9 @@ const scanParams = "_pragma=busy_timeout(10000)" +
 	"&_pragma=journal_mode(WAL)" +
 	"&_pragma=synchronous(NORMAL)"
 
-// reportParams are the connection parameters for report, trees and
+// reportParams are the connection parameters for report and trees:
-// info: read-only, with the same busy timeout. They set no journal
+// read-only, with the same busy timeout. They set no journal mode,
-// mode, because setting one is a write.
+// because setting one is a write.
 const reportParams = "mode=ro" +
 	"&_pragma=busy_timeout(10000)" +
 	"&_pragma=query_only(1)"
@@ -195,9 +173,9 @@ func openDB(path, params string) (*sql.DB, error) {
 // that deleted it would let the next scan lock a new file while
 // another still holds the old one.
 func lockScanDatabase(path string) (*os.File, error) {
-	err := createDatabaseDirectory(path)
+	err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
 	if err != nil {
-		return nil, err
+		return nil, fmt.Errorf("create database directory: %w", err)
 	}
 
 	// Like SQLite, this follows a symlink at path and creates its
@@ -241,9 +219,9 @@ func lockScanDatabase(path string) (*os.File, error) {
 // openScanDatabase opens the database for the scan subcommand, creating
 // the file, its parent directory, and the schema as needed.
 func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
-	err := createDatabaseDirectory(path)
+	err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
 	if err != nil {
-		return nil, err
+		return nil, fmt.Errorf("create database directory: %w", err)
 	}
 
 	db, err := openDB(path, scanParams)
@@ -261,42 +239,6 @@ func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
 	return db, nil
 }
 
-// databaseFiles returns, by device and inode, the files of the database
-// at path that the scan keeps open: the database file, its -wal and
-// -shm files and its .lock file, each named from the database's path
-// with symlinks resolved, as SQLite and lockScanDatabase name them. The
-// scan calls it once the database and the lock file are open, so that
-// its walk passes over them however it reaches them, through a
-// symlinked directory included. A file that does not exist then is left
-// out, since no walk can reach it. There is no -journal file while the
-// database is in WAL mode.
-func databaseFiles(path string) ([]fileRec, error) {
-	resolved, err := filepath.EvalSymlinks(path)
-	if err != nil {
-		return nil, fmt.Errorf("resolve database path: %w", err)
-	}
-
-	var files []fileRec
-
-	for _, p := range []string{
-		resolved, resolved + "-wal", resolved + "-shm", resolved + ".lock",
-	} {
-		fi, err := os.Lstat(p)
-		if errors.Is(err, fs.ErrNotExist) {
-			continue
-		}
-
-		if err != nil {
-			return nil, fmt.Errorf("database: %w", err)
-		}
-
-		dev, ino := inodeOfInfo(fi)
-		files = append(files, fileRec{path: p, dev: dev, ino: ino})
-	}
-
-	return files, nil
-}
-
 // closeScanDatabase switches the database at path from WAL back to
 // rollback-journal mode and closes it. Out of WAL mode the database
 // file alone holds the whole database, so a reader needs no -wal or
@@ -315,9 +257,9 @@ func closeScanDatabase(ctx context.Context, db *sql.DB, path string) {
 	_ = db.Close()
 }
 
-// openReportDatabase opens an existing database for the report, trees
+// openReportDatabase opens an existing database for the report and
-// and info subcommands. A missing database file is an error directing
+// trees subcommands. A missing database file is an error directing the
-// the user to run scan first; the schema version must match exactly.
+// user to run scan first; the schema version must match exactly.
 func openReportDatabase(ctx context.Context,
 	path string,
 ) (*sql.DB, error) {
@@ -456,32 +398,10 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
 	return v, nil
 }
 
-// loadFileRows gives total the number of records, then streams every
+// loadFileRows streams every record to fn in path order: byte order,
-// record to fn in path order: byte order, which is the order of the
+// which is the order of the primary key, so SQLite does not sort.
-// primary key, so SQLite does not sort. The count and the records are
+func loadFileRows(ctx context.Context, db *sql.DB, fn func(r scanRec)) error {
-// read in one transaction, so they agree while a scan is committing.
+	rows, err := db.QueryContext(ctx,
-func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
-	fn func(r scanRec),
-) error {
-	// Everything goes through tx: db has one connection, so a query on
-	// db would wait for tx forever.
-	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
-	if err != nil {
-		return fmt.Errorf("read records: %w", err)
-	}
-
-	defer func() { _ = tx.Rollback() }()
-
-	var n int64
-
-	err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
-	if err != nil {
-		return fmt.Errorf("count records: %w", err)
-	}
-
-	total(n)
-
-	rows, err := tx.QueryContext(ctx,
 		"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
 			"FROM files ORDER BY path")
 	if err != nil {
@@ -516,140 +436,77 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
 	return nil
 }
 
-// signatureRowsSQL selects every record's signature and path in
+// dupeRowsSQL selects every record in a duplicate group, with the
-// signature order, largest size first. That is the order of the
+// group's first path. A group is the records with a content hash that
-// files_signature index read backwards, so SQLite reads the index alone
+// share a size, head, tail, and content, when there are two or more of
-// and returns the first row at once, with nothing to sort.
+// them. The rows come in report order: groups by size descending, then
-const signatureRowsSQL = `
+// by first path, and each group's paths ascending.
-SELECT size, head, tail, content, path
+const dupeRowsSQL = `
-FROM files
+SELECT g.first, f.path, f.size
-ORDER BY size DESC, head DESC, tail DESC, content DESC
+FROM files AS f
+JOIN (
+    SELECT size, head, tail, content, MIN(path) AS first
+    FROM files
+    WHERE content <> ''
+    GROUP BY size, head, tail, content
+    HAVING COUNT(*) > 1
+) AS g USING (size, head, tail, content)
+ORDER BY f.size DESC, g.first, f.path
 `
 
-// loadSignatureRows gives total the number of records, then streams
+// loadDupeRows streams the rows of dupeRowsSQL to fn and returns the
-// every record's signature and path to fn, without its mtime, in the
+// number of records in the database. The count and the rows are read
-// order of signatureRowsSQL. The count and the records are read in one
+// in one transaction, so they agree while a scan is committing. An
-// transaction, so they agree while a scan is committing. An error from
+// error from fn stops the reading and is returned as it is.
-// fn stops the reading and is returned as it is.
+func loadDupeRows(ctx context.Context, db *sql.DB,
-func loadSignatureRows(ctx context.Context, db *sql.DB,
+	fn func(first, path string, size int64) error,
-	total func(n int64), fn func(r scanRec) error,
+) (int, error) {
-) error {
 	// Everything goes through tx: the report connection is the only
 	// one, so a query on db would wait for tx forever.
 	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
 	if err != nil {
-		return fmt.Errorf("read records: %w", err)
+		return 0, fmt.Errorf("read records: %w", err)
 	}
 
 	defer func() { _ = tx.Rollback() }()
 
-	var n int64
+	var records int
 
-	err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
+	err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&records)
 	if err != nil {
-		return fmt.Errorf("count records: %w", err)
+		return 0, fmt.Errorf("read records: %w", err)
 	}
 
-	total(n)
+	rows, err := tx.QueryContext(ctx, dupeRowsSQL)
-
-	rows, err := tx.QueryContext(ctx, signatureRowsSQL)
 	if err != nil {
-		return fmt.Errorf("read records: %w", err)
+		return 0, fmt.Errorf("read records: %w", err)
 	}
 
 	defer func() { _ = rows.Close() }()
 
 	for rows.Next() {
 		var (
-			path []byte
+			first, path []byte
-			r    scanRec
+			size        int64
 		)
 
-		err = rows.Scan(&r.size, &r.head, &r.tail, &r.content, &path)
+		err = rows.Scan(&first, &path, &size)
 		if err != nil {
-			return fmt.Errorf("read record: %w", err)
+			return 0, fmt.Errorf("read record: %w", err)
 		}
 
-		r.path = string(path)
+		err = fn(string(first), string(path), size)
-
-		err = fn(r)
 		if err != nil {
-			return err
+			return 0, err
 		}
 	}
 
 	err = rows.Err()
 	if err != nil {
-		return fmt.Errorf("read records: %w", err)
+		return 0, fmt.Errorf("read records: %w", err)
 	}
 
-	return nil
+	return records, nil
-}
-
-// countRecords returns the number of records in the database.
-func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
-	var n int64
-
-	err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
-	if err != nil {
-		return 0, fmt.Errorf("count records: %w", err)
-	}
-
-	return n, nil
-}
-
-// databaseCountsSQL counts the records, those with head and tail
-// hashes and those with a content hash, and adds up their sizes, in one
-// pass over the table. A SUM over no rows is NULL, so on an empty table
-// each SUM falls back to 0.
-const databaseCountsSQL = `
-SELECT COUNT(*),
-       COALESCE(SUM(head <> '' AND tail <> ''), 0),
-       COALESCE(SUM(content <> ''), 0),
-       COALESCE(SUM(size), 0)
-FROM files
-`
-
-// databaseCounts is what info reads from the database: its schema
-// version and the counts of databaseCountsSQL.
-type databaseCounts struct {
-	schemaVersion int
-	records       int64 // every record
-	headTail      int64 // records with head and tail hashes
-	content       int64 // records with a content hash
-	size          int64 // the sizes of all records added up
-}
-
-// readDatabaseCounts reads the schema version, then the counts of
-// databaseCountsSQL, in one read-only transaction, so both describe the
-// database at one moment while a scan is committing.
-func readDatabaseCounts(ctx context.Context,
-	db *sql.DB,
-) (databaseCounts, error) {
-	var c databaseCounts
-
-	// Everything goes through tx: the report connection is the only
-	// one, so a query on db would wait for tx forever.
-	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
-	if err != nil {
-		return c, fmt.Errorf("count records: %w", err)
-	}
-
-	defer func() { _ = tx.Rollback() }()
-
-	err = tx.QueryRowContext(ctx, "PRAGMA user_version").
-		Scan(&c.schemaVersion)
-	if err != nil {
-		return c, fmt.Errorf("read schema version: %w", err)
-	}
-
-	err = tx.QueryRowContext(ctx, databaseCountsSQL).
-		Scan(&c.records, &c.headTail, &c.content, &c.size)
-	if err != nil {
-		return c, fmt.Errorf("count records: %w", err)
-	}
-
-	return c, nil
 }
 
 // loadFileMeta streams every record's path, size, mtime, and whether
@@ -690,46 +547,26 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
 	return nil
 }
 
-// contentGroupsSQL selects each group (the records sharing a size,
+// contentCandidatesSQL selects every record of at least headTailMin
-// head, and tail) of records of at least headTailMin bytes that has more
+// bytes whose size, head, and tail equal another record's, in each
-// than one record and at least one record without a content hash, with
+// group (the records sharing a size, head, and tail) where at least one
-// the number of its records.
+// record has no content hash, with whether each record has one. SQLite
-const contentGroupsSQL = `
+// does the grouping, so no other record's hashes are loaded into
-SELECT size, head, tail, COUNT(*) AS records
+// memory; the rows come ordered by size, head, and tail, so each
-FROM files
+// group's rows arrive together.
-WHERE size >= ? AND head <> ''
-GROUP BY size, head, tail
-HAVING COUNT(*) > 1 AND SUM(content = '') > 0
-`
-
-// contentCandidatesSQL selects every record of the groups
-// contentGroupsSQL selects, with whether each record has a content
-// hash. SQLite does the grouping, so no other record's hashes are
-// loaded into memory; the rows come ordered by size, head, and tail, so
-// each group's rows arrive together.
 const contentCandidatesSQL = `
 SELECT f.path, f.size, f.mtime, f.mtime_nsec, f.head, f.tail, f.content <> ''
 FROM files AS f
-JOIN (` + contentGroupsSQL + `) AS g USING (size, head, tail)
+JOIN (
+    SELECT size, head, tail
+    FROM files
+    WHERE size >= ? AND head <> ''
+    GROUP BY size, head, tail
+    HAVING COUNT(*) > 1 AND SUM(content = '') > 0
+) AS g USING (size, head, tail)
 ORDER BY size, head, tail
 `
 
-// countContentCandidates returns the number of rows of
-// contentCandidatesSQL, adding up the records of the groups
-// contentGroupsSQL selects, so it neither joins nor sorts.
-func countContentCandidates(ctx context.Context, db *sql.DB) (int64, error) {
-	var n int64
-
-	err := db.QueryRowContext(ctx,
-		"SELECT COALESCE(SUM(records), 0) FROM ("+contentGroupsSQL+")",
-		headTailMin).Scan(&n)
-	if err != nil {
-		return 0, fmt.Errorf("count content candidates: %w", err)
-	}
-
-	return n, nil
-}
-
 // loadContentCandidates streams the rows of contentCandidatesSQL to fn:
 // each record, without its content hash, and whether it has one.
 func loadContentCandidates(ctx context.Context, db *sql.DB,

db_test.go

@@ -5,7 +5,6 @@ import (
 	"database/sql"
 	"errors"
 	"fmt"
-	"io/fs"
 	"os"
 	"path/filepath"
 	"slices"
@@ -45,14 +44,9 @@ func openTestDB(t *testing.T) *sql.DB {
 
 func TestDatabasePath(t *testing.T) {
 	// README §Database: SFDUPES_DATABASE when set and non-empty, the
-	// default otherwise, with the working directory, its symlinks
+	// default otherwise, with the working directory put in front of a
-	// resolved, put in front of a relative path. Nothing is cleaned, so
+	// relative path. Nothing is cleaned, so a ".." stays.
-	// a ".." stays.
+	dir := t.TempDir()
-	dir, err := filepath.EvalSymlinks(t.TempDir())
-	if err != nil {
-		t.Fatal(err)
-	}
-
 	t.Chdir(dir)
 
 	fromEnv := "from " + databaseEnv
@@ -119,116 +113,6 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
 	}
 }
 
-// TestScanCreatesDatabaseDirectoryAfterSymlink has a database path go
-// through a symlink to a directory, then "..", to a missing directory.
-// With srv/link a symlink to data/x, srv/link/../new/db.sqlite leads to
-// data/new/db.sqlite. lockScanDatabase and openScanDatabase, the two
-// places scan creates the database's directory, must each create the
-// database there, and nothing at srv/new, where the path leads once
-// cleaned as text.
-func TestScanCreatesDatabaseDirectoryAfterSymlink(t *testing.T) {
-	t.Parallel()
-
-	creates := map[string]func(ctx context.Context, path string) error{
-		"lockScanDatabase": func(_ context.Context, path string) error {
-			lock, err := lockScanDatabase(path)
-			if err == nil {
-				_ = lock.Close()
-			}
-
-			return err
-		},
-		"openScanDatabase": func(ctx context.Context, path string) error {
-			db, err := openScanDatabase(ctx, path)
-			if err == nil {
-				_ = db.Close()
-			}
-
-			return err
-		},
-	}
-
-	for name, create := range creates {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			dir, err := filepath.EvalSymlinks(t.TempDir())
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			err = os.MkdirAll(filepath.Join(dir, "data", "x"), 0o750)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			err = os.Mkdir(filepath.Join(dir, "srv"), 0o750)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			err = os.Symlink(filepath.Join(dir, "data", "x"),
-				filepath.Join(dir, "srv", "link"))
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			err = create(t.Context(), dir+"/srv/link/../new/db.sqlite")
-			if err != nil {
-				t.Fatalf("%s: %v", name, err)
-			}
-
-			_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
-			if err != nil {
-				t.Errorf("database the path leads to: %v", err)
-			}
-
-			_, err = os.Lstat(filepath.Join(dir, "srv", "new"))
-			if !errors.Is(err, fs.ErrNotExist) {
-				t.Errorf("srv/new: stat error %v, want it missing", err)
-			}
-		})
-	}
-}
-
-// TestDatabaseFiles opens a database through a symlink to its file:
-// databaseFiles finds the database file and its -wal and -shm files
-// beside the symlink's target, where SQLite keeps them, and leaves out
-// the .lock file, which only a scan creates.
-func TestDatabaseFiles(t *testing.T) {
-	t.Parallel()
-
-	path := testDBPath(t)
-	link := filepath.Join(t.TempDir(), "link.sqlite")
-
-	err := os.Symlink(path, link)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	db, err := openScanDatabase(t.Context(), link)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = db.Close() }()
-
-	files, err := databaseFiles(link)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	got := make([]string, 0, len(files))
-	for _, f := range files {
-		got = append(got, f.path)
-	}
-
-	want := []string{path, path + "-wal", path + "-shm"}
-	if !slices.Equal(got, want) {
-		t.Errorf("databaseFiles(%s) = %q, want %q", link, got, want)
-	}
-}
-
 func TestOpenDatabaseUnversionedForeign(t *testing.T) {
 	t.Parallel()
 
@@ -395,128 +279,12 @@ func TestCloseScanDatabaseWhileReportOpen(t *testing.T) {
 
 	defer func() { _ = reportDB.Close() }()
 
-	err = loadFileRows(t.Context(), reportDB, func(int64) {},
+	err = loadFileRows(t.Context(), reportDB, func(scanRec) {})
-		func(scanRec) {})
 	if err != nil {
 		t.Fatalf("loadFileRows: %v", err)
 	}
 }
 
-// TestLoadRowsCountAgreeDuringScan has a scan commit records after
-// loadFileRows, as trees calls it, or loadSignatureRows, as report
-// calls it, has counted the records and before it reads them, as when
-// trees or report runs during a scan: each must read exactly as many
-// records as it counted.
-func TestLoadRowsCountAgreeDuringScan(t *testing.T) {
-	t.Parallel()
-
-	loads := map[string]func(context.Context, *sql.DB, func(int64),
-		func(scanRec)) error{
-		"loadFileRows": loadFileRows,
-		"loadSignatureRows": func(ctx context.Context, db *sql.DB,
-			total func(int64), fn func(scanRec),
-		) error {
-			return loadSignatureRows(ctx, db, total, func(r scanRec) error {
-				fn(r)
-
-				return nil
-			})
-		},
-	}
-
-	for name, load := range loads {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			path := testDBPath(t)
-
-			scanDB, err := openScanDatabase(t.Context(), path)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			defer func() { _ = scanDB.Close() }()
-
-			recs := smokeTreeRecs()
-
-			err = applyChanges(t.Context(), scanDB, recs[:3], nil, nil)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			reportDB, err := openReportDatabase(t.Context(), path)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			defer func() { _ = reportDB.Close() }()
-
-			var counted, read int64
-
-			err = load(t.Context(), reportDB, func(n int64) {
-				counted = n
-
-				err := applyChanges(t.Context(), scanDB, recs[3:], nil, nil)
-				if err != nil {
-					t.Error(err)
-				}
-			}, func(scanRec) { read++ })
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			if counted != 3 || read != counted {
-				t.Errorf("counted %d records and read %d, want 3 and 3",
-					counted, read)
-			}
-		})
-	}
-}
-
-// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
-// from the files_signature index alone, with nothing to sort: the
-// first record comes at once, so report's read phase counts each
-// record as it comes rather than waiting on SQLite for all of them.
-func TestSignatureRowsNeedNoSort(t *testing.T) {
-	t.Parallel()
-
-	db := openTestDB(t)
-
-	rows, err := db.QueryContext(t.Context(),
-		"EXPLAIN QUERY PLAN "+signatureRowsSQL)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = rows.Close() }()
-
-	var plan []string
-
-	for rows.Next() {
-		var (
-			id, parent, unused int
-			detail             string
-		)
-
-		err = rows.Scan(&id, &parent, &unused, &detail)
-		if err != nil {
-			t.Fatal(err)
-		}
-
-		plan = append(plan, detail)
-	}
-
-	err = rows.Err()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	want := []string{"SCAN files USING COVERING INDEX files_signature"}
-	if !slices.Equal(plan, want) {
-		t.Errorf("query plan = %q, want %q", plan, want)
-	}
-}
-
 func TestApplyChangesRoundTrip(t *testing.T) {
 	t.Parallel()
 
@@ -536,10 +304,8 @@ func TestApplyChangesRoundTrip(t *testing.T) {
 		},
 	}
 
-	prog := newProgress(progressParams{label: "update", total: 4})
+	err := applyChanges(t.Context(), db, recs, nil,
-	defer prog.finish()
+		newProgress("update", 2))
-
-	err := applyChanges(t.Context(), db, recs, nil, prog)
 	if err != nil {
 		t.Fatalf("applyChanges: %v", err)
 	}
@@ -558,7 +324,7 @@ func TestApplyChangesRoundTrip(t *testing.T) {
 	}
 
 	err = applyChanges(t.Context(), db, []scanRec{upd},
-		[]string{"/a/tab\tnew\nline"}, prog)
+		[]string{"/a/tab\tnew\nline"}, newProgress("update", 2))
 	if err != nil {
 		t.Fatalf("applyChanges: %v", err)
 	}
@@ -586,10 +352,8 @@ func TestApplyChangesBatching(t *testing.T) {
 		})
 	}
 
-	prog := newProgress(progressParams{label: "update", total: int64(2 * n)})
+	err := applyChanges(t.Context(), db, recs, nil,
-	defer prog.finish()
+		newProgress("update", int64(n)))
-
-	err := applyChanges(t.Context(), db, recs, nil, prog)
 	if err != nil {
 		t.Fatalf("applyChanges: %v", err)
 	}
@@ -603,7 +367,8 @@ func TestApplyChangesBatching(t *testing.T) {
 		deletes = append(deletes, r.path)
 	}
 
-	err = applyChanges(t.Context(), db, nil, deletes, prog)
+	err = applyChanges(t.Context(), db, nil, deletes,
+		newProgress("update", int64(n)))
 	if err != nil {
 		t.Fatalf("applyChanges deletes: %v", err)
 	}
@@ -612,62 +377,3 @@ func TestApplyChangesBatching(t *testing.T) {
 		t.Fatalf("records = %d, want 0", len(got))
 	}
 }
-
-// TestCountContentCandidates checks that countContentCandidates counts
-// the records the content phase's search returns: those of each group
-// of two or more records of headTailMin or more with a head, where at
-// least one has no content hash.
-func TestCountContentCandidates(t *testing.T) {
-	t.Parallel()
-
-	db := openTestDB(t)
-
-	// Each group's records: size, head, and content, one per record.
-	groups := []struct {
-		size     int64
-		head     string
-		contents []string
-	}{
-		{headTailMin, "h", []string{"", "c", "c"}}, // returned
-		{headTailMin + 1, "h", []string{"", ""}},   // returned
-		{headTailMin + 2, "h", []string{"c", "c"}}, // all hashed
-		{headTailMin + 3, "h", []string{""}},       // one record
-		{headTailMin - 1, "h", []string{"", ""}},   // too small
-		{headTailMin + 4, "", []string{"", ""}},    // no head
-	}
-
-	var recs []scanRec
-
-	for _, g := range groups {
-		for _, content := range g.contents {
-			recs = append(recs, scanRec{
-				size: g.size, mtime: time.Unix(1, 0), head: g.head,
-				tail: "t", content: content,
-				path: fmt.Sprintf("/r/%d", len(recs)),
-			})
-		}
-	}
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatalf("applyChanges: %v", err)
-	}
-
-	got, err := countContentCandidates(t.Context(), db)
-	if err != nil {
-		t.Fatalf("countContentCandidates: %v", err)
-	}
-
-	var returned int64
-
-	err = loadContentCandidates(t.Context(), db,
-		func(scanRec, bool) { returned++ })
-	if err != nil {
-		t.Fatalf("loadContentCandidates: %v", err)
-	}
-
-	if got != 5 || returned != 5 {
-		t.Errorf("count = %d, records returned = %d, want 5 and 5",
-			got, returned)
-	}
-}

exclude.go

@@ -1,195 +0,0 @@
-package main
-
-// This file holds two rules for the walk (README "Rules for the walk"):
-// the default excludes, the paths a scan's walk does not enter below an
-// operand unless scan is given --no-default-excludes, and the
-// filesystems that are not persistent, on which the scan never records
-// or reads anything.
-
-import (
-	"bufio"
-	"errors"
-	"fmt"
-	"io"
-	"io/fs"
-	"path/filepath"
-	"slices"
-	"strings"
-
-	"golang.org/x/sys/unix"
-)
-
-// The systems that have default excludes, as runtime.GOOS names them.
-const (
-	goosLinux   = "linux"
-	goosDarwin  = "darwin"
-	goosWindows = "windows"
-)
-
-// The fields of a line of mountInfoPath that readMountInfo reads,
-// counted from 0 (see proc(5)): the device ID, as major:minor, is the
-// third field, and after the sixth come optional fields, then a field
-// holding "-", then the filesystem type.
-const (
-	deviceField         = 2
-	optionalFieldsStart = 6
-)
-
-// typesPerHelpLine is how many filesystem types the scan command's help
-// lists on each line.
-const typesPerHelpLine = 6
-
-// errMountInfoLine reports a line of mountInfoPath without the fields
-// readMountInfo reads.
-var errMountInfoLine = errors.New("unexpected line")
-
-// walkRules holds the rules for the walk that a scan sets up as it
-// starts. excludes are the default excludes, or none with
-// --no-default-excludes. nonPersistent maps the device ID of every
-// mounted filesystem that is not persistent to its type; the scan
-// records and reads nothing on one of those, whatever the flags.
-// databaseFiles are the scan's own database files, which the walk
-// passes over (see databaseFiles in db.go).
-type walkRules struct {
-	excludes      []string
-	nonPersistent map[uint64]string
-	databaseFiles []fileRec
-}
-
-// defaultExcludes returns the default excludes of the system goos
-// names, as runtime.GOOS does: a fixed list on Linux, macOS and
-// Windows, and none on any other system.
-func defaultExcludes(goos string) []string {
-	switch goos {
-	case goosLinux:
-		return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
-	case goosDarwin:
-		// A walk of / reaches everything in /System/Volumes/Data through
-		// the firmlinks, so walking it as well would record each of
-		// those files twice.
-		return []string{
-			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
-			"/System/Volumes/VM", "/System/Volumes/Data",
-		}
-	case goosWindows:
-		// Relative to the root of the drive an operand lies on. The
-		// first three are the page, swap and hibernation files.
-		return []string{
-			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
-			"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
-			`Users\*\AppData\Local\Temp`,
-		}
-	default:
-		return nil
-	}
-}
-
-// isDefaultExclude reports whether path is one of excludes, in which a
-// * matches one path component.
-func isDefaultExclude(path string, excludes []string) bool {
-	for _, pattern := range excludes {
-		// The patterns are fixed, so none is malformed.
-		if ok, _ := filepath.Match(pattern, path); ok {
-			return true
-		}
-	}
-
-	return false
-}
-
-// nonPersistentFilesystemTypes returns the filesystem types that are
-// not persistent: filesystems that the kernel makes up or that are kept
-// in memory. devfs is the macOS filesystem of /dev; the others are
-// Linux names.
-func nonPersistentFilesystemTypes() []string {
-	return []string{
-		"proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "ramfs",
-		"cgroup", "cgroup2", "debugfs", "tracefs", "securityfs", "pstore",
-		"bpf", "configfs", "fusectl", "mqueue", "hugetlbfs", "efivarfs",
-		"binfmt_misc", "autofs", "nsfs", "rpc_pipefs", "selinuxfs", "devfs",
-	}
-}
-
-// nonPersistentLine returns the line the scan prints for path when fi,
-// its lstat, puts it on a filesystem in rules.nonPersistent, and ""
-// when it is on any other.
-func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
-	dev, ok := deviceOfInfo(fi)
-	if !ok {
-		return ""
-	}
-
-	fsType, found := rules.nonPersistent[dev]
-	if !found {
-		return ""
-	}
-
-	return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
-		path, fsType)
-}
-
-// readMountInfo reads mountinfo, laid out as mountInfoPath is, and
-// returns the device ID of every mounted filesystem that is not
-// persistent, with its type. The device ID is the one each line gives,
-// not that of the file now at its mount point, so a mount that a later
-// mount hides never adds the device of another filesystem.
-func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
-	types := nonPersistentFilesystemTypes()
-	devices := make(map[uint64]string)
-	lines := bufio.NewScanner(mountinfo)
-
-	for lines.Scan() {
-		fields := strings.Fields(lines.Text())
-
-		sep := slices.Index(fields, "-")
-		if sep < optionalFieldsStart || sep+1 >= len(fields) {
-			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
-		}
-
-		var major, minor uint32
-
-		_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
-		if err != nil {
-			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
-		}
-
-		fsType := fields[sep+1]
-		if slices.Contains(types, fsType) {
-			devices[unix.Mkdev(major, minor)] = fsType
-		}
-	}
-
-	return devices, lines.Err()
-}
-
-// defaultExcludesHelp describes, for the scan command's help, the
-// default excludes of the system goos names and, where the scan lists
-// the mounted filesystems, the types it never records anything on.
-func defaultExcludesHelp(goos string) string {
-	var help strings.Builder
-
-	dirs := defaultExcludes(goos)
-	if len(dirs) == 0 {
-		help.WriteString("This system has no default excludes.\n")
-	} else {
-		help.WriteString("Below an operand, the walk does not enter these " +
-			"default excludes, unless\n--no-default-excludes is given:\n\n")
-
-		for _, dir := range dirs {
-			fmt.Fprintf(&help, "  %s\n", dir)
-		}
-	}
-
-	if goos == goosLinux || goos == goosDarwin {
-		help.WriteString("\nWith or without --no-default-excludes, the scan " +
-			"records nothing on a\nfilesystem that is not persistent, " +
-			"one of these types:\n\n")
-
-		types := nonPersistentFilesystemTypes()
-		for line := range slices.Chunk(types, typesPerHelpLine) {
-			fmt.Fprintf(&help, "  %s\n", strings.Join(line, ", "))
-		}
-	}
-
-	return help.String()
-}

exclude_darwin.go

@@ -1,52 +0,0 @@
-//go:build darwin
-
-package main
-
-import (
-	"slices"
-
-	"golang.org/x/sys/unix"
-)
-
-// nonPersistentFilesystems returns the device ID of every mounted
-// filesystem that is not persistent, with its type, from the system's
-// list of mounted filesystems. The device ID is read with lstat on the
-// mount point. Where one filesystem is mounted over another on the same
-// mount point, the later one in the list counts. A mount point that
-// lstat cannot reach is left out.
-func nonPersistentFilesystems() (map[uint64]string, error) {
-	// The first call counts the filesystems, the second lists them.
-	n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
-	if err != nil {
-		return nil, err
-	}
-
-	list := make([]unix.Statfs_t, n)
-
-	n, err = unix.Getfsstat(list, unix.MNT_NOWAIT)
-	if err != nil {
-		return nil, err
-	}
-
-	mounts := make(map[string]string, n)
-
-	for _, st := range list[:n] {
-		mounts[unix.ByteSliceToString(st.Mntonname[:])] =
-			unix.ByteSliceToString(st.Fstypename[:])
-	}
-
-	types := nonPersistentFilesystemTypes()
-	devices := make(map[uint64]string)
-
-	for mountPoint, fsType := range mounts {
-		if !slices.Contains(types, fsType) {
-			continue
-		}
-
-		if dev, ok := deviceOfPath(mountPoint); ok {
-			devices[dev] = fsType
-		}
-	}
-
-	return devices, nil
-}

exclude_other.go

@@ -1,36 +0,0 @@
-//go:build !darwin
-
-package main
-
-import (
-	"fmt"
-	"os"
-	"runtime"
-)
-
-// mountInfoPath is the file in which Linux lists the mounts the reading
-// process sees.
-const mountInfoPath = "/proc/self/mountinfo"
-
-// nonPersistentFilesystems returns the device ID of every mounted
-// filesystem that is not persistent, with its type: on Linux read from
-// mountInfoPath (see readMountInfo), and on any other system none.
-func nonPersistentFilesystems() (map[uint64]string, error) {
-	if runtime.GOOS != goosLinux {
-		return map[uint64]string{}, nil
-	}
-
-	f, err := os.Open(mountInfoPath)
-	if err != nil {
-		return nil, err
-	}
-
-	defer func() { _ = f.Close() }()
-
-	devices, err := readMountInfo(f)
-	if err != nil {
-		return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
-	}
-
-	return devices, nil
-}

exclude_test.go

@@ -1,583 +0,0 @@
-package main
-
-import (
-	"bytes"
-	"errors"
-	"maps"
-	"os"
-	"path/filepath"
-	"runtime"
-	"slices"
-	"strings"
-	"testing"
-
-	"golang.org/x/sys/unix"
-)
-
-func TestDefaultExcludes(t *testing.T) {
-	t.Parallel()
-
-	// README "Rules for the walk": each system's list.
-	cases := map[string][]string{
-		goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
-		goosDarwin: {
-			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
-			"/System/Volumes/VM", "/System/Volumes/Data",
-		},
-		goosWindows: {
-			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
-			"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
-			`Users\*\AppData\Local\Temp`,
-		},
-		"freebsd": nil,
-	}
-
-	for goos, want := range cases {
-		if got := defaultExcludes(goos); !slices.Equal(got, want) {
-			t.Errorf("defaultExcludes(%q) = %q, want %q", goos, got, want)
-		}
-	}
-}
-
-func TestIsDefaultExclude(t *testing.T) {
-	t.Parallel()
-
-	// A * matches one path component, no more and no fewer.
-	excludes := []string{"/scratch", "/home/*/cache"}
-
-	cases := map[string]bool{
-		"/scratch":        true,
-		"/scratch/x":      false,
-		"/home/u/cache":   true,
-		"/home/u/v/cache": false,
-		"/home/cache":     false,
-	}
-
-	for path, want := range cases {
-		if got := isDefaultExclude(path, excludes); got != want {
-			t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
-		}
-	}
-}
-
-func TestReadMountInfo(t *testing.T) {
-	t.Parallel()
-
-	// The second-to-last line has no optional fields. The tmpfs on
-	// /data/a/b is hidden by the root filesystem's /srv, mounted over
-	// /data/a after it: the tmpfs still gives its own device, and the
-	// root filesystem's device is not listed.
-	mountinfo := strings.Join([]string{
-		"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
-		"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
-		"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
-		"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw",
-		"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw",
-		"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
-	}, "\n")
-
-	got, err := readMountInfo(strings.NewReader(mountinfo))
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	want := map[uint64]string{
-		unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs",
-		unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs",
-	}
-	if !maps.Equal(got, want) {
-		t.Errorf("readMountInfo = %v, want %v", got, want)
-	}
-
-	for _, line := range []string{
-		"22 1 8:1 / / rw,relatime\n",
-		"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n",
-	} {
-		_, err = readMountInfo(strings.NewReader(line))
-		if !errors.Is(err, errMountInfoLine) {
-			t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
-				errMountInfoLine)
-		}
-	}
-}
-
-func TestNonPersistentFilesystems(t *testing.T) {
-	t.Parallel()
-
-	if runtime.GOOS != goosLinux {
-		t.Skip("the test knows a mount only on Linux: /proc")
-	}
-
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	dev, ok := deviceOfPath("/proc")
-	if !ok {
-		t.Fatal("no device for /proc")
-	}
-
-	if got := nonPersistent[dev]; got != "proc" {
-		t.Errorf("filesystem of /proc = %q, want %q", got, "proc")
-	}
-}
-
-func TestDefaultExcludesHelp(t *testing.T) {
-	t.Parallel()
-
-	for _, goos := range []string{goosLinux, goosDarwin, goosWindows} {
-		help := defaultExcludesHelp(goos)
-
-		for _, dir := range defaultExcludes(goos) {
-			if !strings.Contains(help, "\n  "+dir+"\n") {
-				t.Errorf("help for %s does not list %s:\n%s", goos, dir, help)
-			}
-		}
-
-		// Only Linux and macOS list their mounted filesystems.
-		for _, fsType := range nonPersistentFilesystemTypes() {
-			listed := strings.Contains(help, " "+fsType+",") ||
-				strings.Contains(help, " "+fsType+"\n")
-			if listed != (goos != goosWindows) {
-				t.Errorf("help for %s lists %s: %v", goos, fsType, listed)
-			}
-		}
-	}
-
-	want := "This system has no default excludes.\n"
-	if got := defaultExcludesHelp("freebsd"); got != want {
-		t.Errorf("help for freebsd = %q, want %q", got, want)
-	}
-
-	// scan's help shows those of the system it runs on.
-	var stdout, stderr bytes.Buffer
-
-	run([]string{cmdScan, "-h"}, &stdout, &stderr)
-
-	help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
-	if !strings.Contains(stderr.String(), help) {
-		t.Errorf("scan --help = %q, want it to hold %q", stderr.String(), help)
-	}
-}
-
-// TestScanDoesNotEnterDefaultExclude checks that the walk does not
-// enter a default exclude below an operand. It says so on a line of its
-// own and does not count it as skipped. The records below it count as
-// outside the operands: the record of a file gone from it is kept, and
-// its size joins the size census, while a file added to it is not
-// recorded.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
-	stderr := captureStderr(t)
-	dir := t.TempDir()
-	db := openTestDB(t)
-	excluded := filepath.Join(dir, "tmp")
-
-	// A scan with no default excludes records the file in excluded.
-	gone := writeFile(t, excluded, "gone", pattern(1, 100))
-	syncTree(t, db, dir)
-
-	err := os.Remove(gone)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	unseen := writeFile(t, excluded, "new", pattern(2, 10))
-
-	// The same size as gone, so it is hashed only if the census holds the
-	// size of gone's record.
-	added := writeFile(t, dir, "added", pattern(3, 100))
-
-	st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
-		walkRules{excludes: []string{excluded}})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.skipped != 0 {
-		t.Errorf("skipped = %d, want 0", st.skipped)
-	}
-
-	recs := dbRecords(t, db)
-	recordByPath(t, recs, gone)
-
-	if recordByPath(t, recs, added).head == "" {
-		t.Errorf("%s not hashed: the census lacks the size of %s",
-			added, gone)
-	}
-
-	if slices.Contains(recordPaths(recs), unseen) {
-		t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
-	}
-
-	line := "\nscan: skipped " + excluded + " (default exclude)\n"
-	if !strings.Contains(stderr(), line) {
-		t.Errorf("stderr = %q, want %q", stderr(), line)
-	}
-}
-
-// TestScanWalksOperandInDefaultExclude checks that an operand at or
-// inside a default exclude is walked, also when it lies under another
-// operand, whose walk does not enter the default exclude.
-func TestScanWalksOperandInDefaultExclude(t *testing.T) {
-	t.Parallel()
-
-	dir := t.TempDir()
-	excluded := filepath.Join(dir, "tmp")
-	f := writeFile(t, excluded, "sub/f", pattern(1, 10))
-	sub := filepath.Dir(f)
-
-	cases := map[string][]string{
-		"at it":                      {excluded},
-		"inside it":                  {sub},
-		"inside it under an operand": {dir, sub},
-	}
-
-	for name, roots := range cases {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			db := openTestDB(t)
-
-			st, err := syncScan(t.Context(), db, roots, 1, false,
-				walkRules{excludes: []string{excluded}})
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			if st.added != 1 {
-				t.Errorf("added = %d, want %s recorded once", st.added, f)
-			}
-
-			recordByPath(t, dbRecords(t, db), f)
-		})
-	}
-}
-
-// TestScanNamesNoDefaultExcludeThatIsAnOperand checks that a default
-// exclude given as an operand under another operand, which the walk of
-// that other operand does not enter, gets no line saying it was
-// skipped: it is walked as an operand of its own.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
-	stderr := captureStderr(t)
-	dir := t.TempDir()
-	db := openTestDB(t)
-	excluded := filepath.Join(dir, "tmp")
-	f := writeFile(t, excluded, "f", pattern(1, 10))
-
-	st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
-		walkRules{excludes: []string{excluded}})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.added != 1 {
-		t.Errorf("added = %d, want %s recorded once", st.added, f)
-	}
-
-	recordByPath(t, dbRecords(t, db), f)
-
-	if strings.Contains(stderr(), "scan: skipped") {
-		t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
-	}
-}
-
-// TestRunScanTempDirOperand scans a tree in the tests' temporary
-// directory, which on Linux lies inside the default exclude /tmp unless
-// TMPDIR names another place, with the system's own default excludes:
-// the scan walks the whole tree and reports no default exclude.
-func TestRunScanTempDirOperand(t *testing.T) {
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	dir := t.TempDir()
-	if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
-		t.Skip(dir + " is not inside a default exclude")
-	}
-
-	f := writeFile(t, dir, "sub/f", pattern(1, 10))
-
-	stderr := scanOK(t, dir)
-	if strings.Contains(stderr, "scan: skipped") {
-		t.Errorf("stderr = %q, want no line saying a path was skipped",
-			stderr)
-	}
-
-	assertRecorded(t, path, f, true)
-}
-
-// TestRunScanNoDefaultExcludes scans /var through the command. Its
-// /var/tmp, on a persistent filesystem, is a default exclude on Linux:
-// without --no-default-excludes the walk does not enter it and says so,
-// and with that flag the walk records a file the test puts there.
-func TestRunScanNoDefaultExcludes(t *testing.T) {
-	if runtime.GOOS != goosLinux {
-		t.Skip("only Linux has /var/tmp among its default excludes")
-	}
-
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	rules := walkRules{nonPersistent: nonPersistent}
-
-	for _, dir := range []string{"/var", "/var/tmp"} {
-		fi, lstatErr := os.Lstat(dir)
-		if lstatErr != nil || !fi.IsDir() ||
-			rules.nonPersistentLine(dir, fi) != "" {
-			t.Skip(dir + " is not a directory on a persistent filesystem here")
-		}
-	}
-
-	//nolint:usetesting // the directory must be inside /var/tmp
-	tmp, err := os.MkdirTemp("/var/tmp", "sfdupes-test-")
-	if err != nil {
-		t.Skip("cannot write to /var/tmp here: " + err.Error())
-	}
-
-	t.Cleanup(func() { _ = os.RemoveAll(tmp) })
-
-	f := writeFile(t, tmp, "f", pattern(1, 10))
-
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	line := "scan: skipped /var/tmp (default exclude)\n"
-
-	if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
-		t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
-	}
-
-	assertRecorded(t, path, f, false)
-
-	stderr := scanOK(t, "--no-default-excludes", "/var")
-	if strings.Contains(stderr, line) {
-		t.Errorf("scan --no-default-excludes /var: stderr = %q, want no %q",
-			stderr, line)
-	}
-
-	assertRecorded(t, path, f, true)
-}
-
-// TestRunScanSkipsNonPersistentOperand scans, through the command, a
-// directory on a filesystem that is not persistent, with and without
-// --no-default-excludes: either way it is skipped, with a line saying
-// so, and nothing in it is recorded.
-func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
-	shm, fsType, _ := shmTempDir(t)
-	writeFile(t, shm, "f", pattern(1, 10))
-
-	t.Setenv(databaseEnv, testDBPath(t))
-
-	line := "scan: skipped " + shm + " (" + fsType +
-		", not a persistent filesystem)\n"
-	summary := "scan: 0 files seen"
-
-	for _, flags := range [][]string{nil, {"--no-default-excludes"}} {
-		stderr := scanOK(t, append(flags, shm)...)
-		if !strings.Contains(stderr, line) ||
-			!strings.Contains(stderr, summary) {
-			t.Errorf("scan %q: stderr = %q, want %q and %q", flags, stderr,
-				line, summary)
-		}
-	}
-}
-
-// TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem
-// that is not persistent beside one on the tests' temporary directory.
-// The records an earlier scan left in it count as outside the operands:
-// the record of a file gone from it is kept, and its size joins the size
-// census, while a file added to it is not recorded, and its size does
-// not join the census.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
-	shm, fsType, rules := shmTempDir(t)
-	stderr := captureStderr(t)
-	dir := t.TempDir()
-	db := openTestDB(t)
-
-	// A scan that, without the list of filesystems, records gone.
-	gone := writeFile(t, shm, "gone", pattern(1, 100))
-	syncTree(t, db, shm)
-
-	err := os.Remove(gone)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	unseen := writeFile(t, shm, "new", pattern(2, 10))
-
-	// added has the size of gone, and lonely that of unseen: each is
-	// hashed only if the census holds that size.
-	added := writeFile(t, dir, "added", pattern(3, 100))
-	lonely := writeFile(t, dir, "lonely", pattern(4, 10))
-
-	st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.skipped != 0 {
-		t.Errorf("skipped = %d, want 0", st.skipped)
-	}
-
-	recs := dbRecords(t, db)
-	recordByPath(t, recs, gone)
-
-	if recordByPath(t, recs, added).head == "" {
-		t.Errorf("%s not hashed: the census lacks the size of %s",
-			added, gone)
-	}
-
-	if recordByPath(t, recs, lonely).head != "" {
-		t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen)
-	}
-
-	if slices.Contains(recordPaths(recs), unseen) {
-		t.Errorf("%s recorded", unseen)
-	}
-
-	line := "scan: skipped " + shm + " (" + fsType +
-		", not a persistent filesystem)\n"
-	if !strings.Contains(stderr(), line) {
-		t.Errorf("stderr = %q, want %q", stderr(), line)
-	}
-}
-
-// TestScanContentSkipsNonPersistent checks that the content phase does
-// not read a file on a filesystem that is not persistent, though an
-// earlier scan left a record for it: it is treated as a changed file,
-// so its record is left as it is and its identical copy in another
-// tree, having no other match, is not read either.
-func TestScanContentSkipsNonPersistent(t *testing.T) {
-	t.Parallel()
-
-	shm, _, rules := shmTempDir(t)
-	db := openTestDB(t)
-
-	// A scan that, without the list of filesystems, records a with a
-	// head and tail but no content hash.
-	a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
-
-	syncTree(t, db, shm)
-
-	before := recordByPath(t, dbRecords(t, db), a)
-	if before.head == "" || before.content != "" {
-		t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
-	}
-
-	b := sparseFile(t, t.TempDir(), "b", headTailMin)
-
-	st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
-		rules)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.skipped != 0 {
-		t.Errorf("skipped = %d, want 0", st.skipped)
-	}
-
-	recs := dbRecords(t, db)
-	if r := recordByPath(t, recs, a); r != before {
-		t.Errorf("record = %+v, want it left as %+v", r, before)
-	}
-
-	if r := recordByPath(t, recs, b); r.content != "" {
-		t.Errorf("%s: content = %q, want none: its only match is on %s",
-			b, r.content, shm)
-	}
-}
-
-// TestWalkLeavesOutNonPersistent starts the walk itself in a directory
-// on a filesystem that is not persistent, which a scan never does: it
-// drops such an operand first. Below a directory on a persistent
-// filesystem, one on another takes a mount, which a test cannot make.
-// The walk enters no directory and records no file there, and names
-// each.
-func TestWalkLeavesOutNonPersistent(t *testing.T) {
-	t.Parallel()
-
-	shm, fsType, rules := shmTempDir(t)
-	f := writeFile(t, shm, "f", pattern(1, 10))
-	sub := filepath.Dir(writeFile(t, shm, "sub/g", pattern(1, 10)))
-
-	var leftOut []string
-
-	for ev := range startWalk(t.Context(), []string{shm}, false, rules, 1) {
-		if ev.rec.path != "" {
-			t.Errorf("walk recorded %s", ev.rec.path)
-		}
-
-		if ev.leftOut == "" {
-			continue
-		}
-
-		leftOut = append(leftOut, ev.leftOut)
-
-		want := "scan: skipped " + ev.leftOut + " (" + fsType +
-			", not a persistent filesystem)"
-		if ev.warn != want {
-			t.Errorf("line = %q, want %q", ev.warn, want)
-		}
-	}
-
-	slices.Sort(leftOut)
-
-	if want := []string{f, sub}; !slices.Equal(leftOut, want) {
-		t.Errorf("walk left out %q, want %q", leftOut, want)
-	}
-}
-
-// shmTempDir returns a new directory in /dev/shm, removed when the test
-// ends, the type of its filesystem, and the rules holding the real
-// filesystems that are not persistent. It skips the test where /dev/shm
-// is on a persistent filesystem, or cannot be written to.
-func shmTempDir(t *testing.T) (string, string, walkRules) {
-	t.Helper()
-
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	dev, ok := deviceOfPath("/dev/shm")
-
-	fsType, found := nonPersistent[dev]
-	if !ok || !found {
-		t.Skip("/dev/shm is not a memory-backed filesystem here")
-	}
-
-	//nolint:usetesting // the directory must be inside /dev/shm
-	dir, err := os.MkdirTemp("/dev/shm", "sfdupes-test-")
-	if err != nil {
-		t.Skip("cannot write to /dev/shm here: " + err.Error())
-	}
-
-	t.Cleanup(func() { _ = os.RemoveAll(dir) })
-
-	return dir, fsType, walkRules{nonPersistent: nonPersistent}
-}
-
-// assertRecorded checks that the database at dbPath holds a record for
-// path when want is set, and none when it is not.
-func assertRecorded(t *testing.T, dbPath, path string, want bool) {
-	t.Helper()
-
-	db, err := openDB(dbPath, reportParams)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = db.Close() }()
-
-	if got := slices.Contains(recordPaths(dbRecords(t, db)), path); got != want {
-		t.Errorf("%s recorded: %v, want %v", path, got, want)
-	}
-}

go.mod

@@ -24,5 +24,3 @@ require (
 	modernc.org/mathutil v1.7.1 // indirect
 	modernc.org/memory v1.11.0 // indirect
 )
-
-retract v0.0.1 // early build without a database; does not work as the README describes

info.go

@@ -1,128 +0,0 @@
-package main
-
-import (
-	"context"
-	"errors"
-	"fmt"
-	"io"
-	"io/fs"
-	"os"
-	"path/filepath"
-	"strings"
-	"time"
-)
-
-// infoPhase names one phase of info with its place among all of them,
-// as "[2/2] count", for the phase's progress display. The list here is
-// the one place the phases are counted.
-func infoPhase(name string) string {
-	return phaseLabel([]string{openPhase, "count"}, name)
-}
-
-// runInfo implements the info subcommand: it prints the database's
-// details on stdout (see writeInfo). It opens the database as report
-// does, read-only and without the scan lock, and never touches the
-// scanned filesystem. It names the database on stderr before it opens
-// it. Any database problem, including a missing database, is fatal.
-//
-// The work runs in two phases, each under a progress display of its
-// own on stderr: open (open the database) and count (count the records,
-// with one query). Each display ends before the next phase starts, and
-// before an error is returned to be printed. The details are written
-// once the last display has ended, so no display is drawn while they
-// are written.
-func runInfo(ctx context.Context, stdout io.Writer) error {
-	dbPath, source, err := databasePath()
-	if err != nil {
-		return err
-	}
-
-	fmt.Fprintf(os.Stderr, "info: database %s (%s)\n",
-		escapePath(dbPath), source)
-
-	// Opening can wait on a busy database for as long as the busy
-	// timeout.
-	prog := newProgress(progressParams{
-		label: infoPhase(openPhase), total: -1,
-		waitingFor: openingDatabase,
-	})
-	db, err := openReportDatabase(ctx, dbPath)
-
-	prog.finish()
-
-	if err != nil {
-		return err
-	}
-
-	defer func() { _ = db.Close() }()
-
-	// One query counts everything, so the phase never has a total.
-	prog = newProgress(progressParams{
-		label: infoPhase("count"), total: -1,
-		waitingFor: countingRecords,
-	})
-	counts, err := readDatabaseCounts(ctx, db)
-
-	prog.finish()
-
-	if err != nil {
-		return fmt.Errorf("database %s: %w", dbPath, err)
-	}
-
-	return writeInfo(stdout, dbPath, source, counts)
-}
-
-// writeInfo writes the details of the database at dbPath to stdout,
-// one per line, as a name, a tab and the value (README "`info` mode"):
-// the path and source as databasePath returns them, the sizes of the
-// database file and of its -wal file when there is one, when the
-// database file was last changed, and the schema version and record
-// counts that readDatabaseCounts read.
-func writeInfo(stdout io.Writer, dbPath, source string,
-	counts databaseCounts,
-) error {
-	// SQLite keeps the -wal file beside the database file, named from
-	// its path with symlinks resolved.
-	file, err := filepath.EvalSymlinks(dbPath)
-	if err != nil {
-		return fmt.Errorf("database: %w", err)
-	}
-
-	dbFile, err := os.Stat(file)
-	if err != nil {
-		return fmt.Errorf("database: %w", err)
-	}
-
-	walFile, err := os.Stat(file + "-wal")
-	if err != nil && !errors.Is(err, fs.ErrNotExist) {
-		return fmt.Errorf("database: %w", err)
-	}
-
-	var b strings.Builder
-
-	fmt.Fprintf(&b, "database\t%s (%s)\n", escapePath(dbPath), source)
-	fmt.Fprintf(&b, "database file size\t%d (%s)\n",
-		dbFile.Size(), humanBytes(dbFile.Size()))
-
-	if walFile != nil {
-		fmt.Fprintf(&b, "-wal file size\t%d (%s)\n",
-			walFile.Size(), humanBytes(walFile.Size()))
-	}
-
-	fmt.Fprintf(&b, "modified\t%s\n",
-		dbFile.ModTime().UTC().Format(time.RFC3339))
-	fmt.Fprintf(&b, "schema version\t%d\n", counts.schemaVersion)
-	fmt.Fprintf(&b, "records\t%d\n", counts.records)
-	fmt.Fprintf(&b, "records with head and tail hashes\t%d\n",
-		counts.headTail)
-	fmt.Fprintf(&b, "records with content hash\t%d\n", counts.content)
-	fmt.Fprintf(&b, "size of recorded files\t%d (%s)\n",
-		counts.size, humanBytes(counts.size))
-
-	_, err = io.WriteString(stdout, b.String())
-	if err != nil {
-		return fmt.Errorf("write stdout: %w", err)
-	}
-
-	return nil
-}

info_test.go

@@ -1,168 +0,0 @@
-package main
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"strings"
-	"testing"
-	"time"
-)
-
-// infoOpenLine is the plain line of info's open phase when it opens the
-// database at once; it prints it as the phase starts and as it ends.
-const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
-	"eta unknown"
-
-// infoCountLine is the plain line of info's count phase when the count
-// takes under a second; it prints it as the phase starts and as it
-// ends.
-const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " +
-	"eta unknown"
-
-// infoRecs are four records: a duplicate pair with every hash, a file
-// of 20 MiB with head and tail hashes but no content hash yet, and a
-// file never hashed.
-func infoRecs() []scanRec {
-	return []scanRec{
-		{size: 5, head: "h", tail: "t", content: "c", path: "/d/a"},
-		{size: 5, head: "h", tail: "t", content: "c", path: "/d/b"},
-		{size: 20 << 20, head: "bh", tail: "bt", path: "/d/big"},
-		{size: 7, path: "/d/unique"},
-	}
-}
-
-// TestRunInfo runs info on an empty and on a filled database, with
-// stdout and stderr the same file that is not a terminal, and checks
-// everything written there: the line naming the database, each phase's
-// line as it starts and as it ends, and then the details, each on a
-// line of its own. No -wal file is beside a database scan has closed.
-func TestRunInfo(t *testing.T) {
-	cases := map[string]struct {
-		recs []scanRec
-		want string // the details after the database file's time
-	}{
-		"empty": {
-			want: "schema version\t1\n" +
-				"records\t0\n" +
-				"records with head and tail hashes\t0\n" +
-				"records with content hash\t0\n" +
-				"size of recorded files\t0 (0 B)\n",
-		},
-		"filled": {
-			recs: infoRecs(),
-			want: "schema version\t1\n" +
-				"records\t4\n" +
-				"records with head and tail hashes\t3\n" +
-				"records with content hash\t2\n" +
-				"size of recorded files\t20971537 (20.0 MiB)\n",
-		},
-	}
-
-	for name, tc := range cases {
-		t.Run(name, func(t *testing.T) {
-			path := seedDatabase(t, tc.recs)
-			t.Setenv(databaseEnv, path)
-
-			file, err := os.Stat(path)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			stderr := captureStderr(t)
-
-			code := run([]string{cmdInfo}, os.Stderr, os.Stderr)
-			if code != exitOK {
-				t.Fatalf("run(info) = %d, want %d; stderr: %s",
-					code, exitOK, stderr())
-			}
-
-			named := path + " (from " + databaseEnv + ")\n"
-			want := "info: database " + named +
-				infoOpenLine + "\n" + infoOpenLine + "\n" +
-				infoCountLine + "\n" + infoCountLine + "\n" +
-				"database\t" + named +
-				fmt.Sprintf("database file size\t%d (%s)\n",
-					file.Size(), humanBytes(file.Size())) +
-				"modified\t" + file.ModTime().UTC().Format(time.RFC3339) +
-				"\n" + tc.want
-
-			if got := stderr(); got != want {
-				t.Errorf("stdout and stderr = %q, want %q", got, want)
-			}
-		})
-	}
-}
-
-// TestRunInfoDuringScan runs info while a scan holds the lock and has
-// the database open in WAL mode, with records committed to the -wal
-// file only. info must count those records and show the -wal file's
-// size, also when the database path is a symlink: SQLite keeps the
-// -wal file beside the symlink's target.
-func TestRunInfoDuringScan(t *testing.T) {
-	path := testDBPath(t)
-	link := filepath.Join(t.TempDir(), "link.sqlite")
-
-	err := os.Symlink(path, link)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	db, err := openScanDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	t.Cleanup(func() { _ = db.Close() })
-
-	err = applyChanges(t.Context(), db, infoRecs(), nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	holdScanLock(t, path)
-
-	wal, err := os.Stat(path + "-wal")
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	for _, named := range []string{path, link} {
-		t.Run(filepath.Base(named), func(t *testing.T) {
-			t.Setenv(databaseEnv, named)
-
-			stdout := runStdout(t, cmdInfo)
-
-			for _, want := range []string{
-				"database\t" + named + " (from " + databaseEnv + ")\n",
-				fmt.Sprintf("\n-wal file size\t%d (%s)\n",
-					wal.Size(), humanBytes(wal.Size())),
-				"\nrecords\t4\n",
-			} {
-				if !strings.Contains(stdout, want) {
-					t.Errorf("stdout = %q, want it to hold %q", stdout, want)
-				}
-			}
-		})
-	}
-}
-
-func TestRunInfoNeedsOnlyReadAccess(t *testing.T) {
-	// README "`info` mode": like report, info needs only read access to
-	// the database file and none to its directory.
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	scanFixture(t)
-	makeReadOnly(t, path)
-
-	stdout := runStdout(t, cmdInfo)
-
-	if want := "\nrecords\t2\n"; !strings.Contains(stdout, want) {
-		t.Errorf("stdout = %q, want it to hold %q", stdout, want)
-	}
-
-	if strings.Contains(stdout, "-wal") {
-		t.Errorf("stdout = %q, want no -wal file between scans", stdout)
-	}
-}

main.go

@@ -8,14 +8,13 @@
 // under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
 // or larger. scan maintains a persistent SQLite database of file
 // signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
-// that the other subcommands read.
+// that the reporting subcommands read.
 //
 // Usage:
 //
-//	sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
+//	sfdupes scan [--workers N] [-x] PATH...
 //	sfdupes report > dupes.tsv
 //	sfdupes trees > dupetrees.tsv
-//	sfdupes info
 //	sfdupes --version
 //
 // See README.md for the complete specification.
@@ -28,7 +27,6 @@ import (
 	"io"
 	"os"
 	"runtime"
-	"runtime/debug"
 
 	"github.com/spf13/cobra"
 )
@@ -46,7 +44,6 @@ const (
 	cmdScan   = "scan"
 	cmdReport = "report"
 	cmdTrees  = "trees"
-	cmdInfo   = "info"
 )
 
 // errNoSubcommand is returned by the root command when it is invoked
@@ -59,32 +56,10 @@ var errNoSubcommand = errors.New("no subcommand")
 var errWorkersBelowOne = errors.New("--workers must be at least 1")
 
 // Version is the build version, injected at link time via -ldflags
-// (see the Makefile); empty when the linker set none, as for a binary
+// (see the Makefile); "dev" for a plain go build.
-// built by go install. buildVersion turns it into what --version prints.
 //
 //nolint:gochecknoglobals // written only by the linker
-var Version string
+var Version = "dev"
-
-// buildVersion returns the version that sfdupes --version prints
-// (README "Install"): the version the linker set, when it set one;
-// otherwise the main module's version that Go recorded in the binary,
-// a tag such as v0.0.2 or the pseudo-version of an @main build; and
-// "dev" when neither is known. info is what debug.ReadBuildInfo
-// returns, nil when the binary carries none. Go records "(devel)" for
-// a build that has no version, such as one inside a copy of the source
-// without git.
-func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
-	if linkerVersion != "" {
-		return linkerVersion
-	}
-
-	if info == nil || info.Main.Version == "" ||
-		info.Main.Version == "(devel)" {
-		return "dev"
-	}
-
-	return info.Main.Version
-}
 
 func main() {
 	// Once the reader of a stdout pipe has gone, as in "sfdupes report |
@@ -98,8 +73,8 @@ func main() {
 // exit code. It is the program's single exit point: the subcommands
 // return their errors instead of exiting, so every deferred cleanup —
 // above all closing the database, which checkpoints the SQLite WAL —
-// runs before the process ends. The report, trees and info subcommands
+// runs before the process ends. The report and trees subcommands write
-// write their data to stdout.
+// their data to stdout.
 func run(args []string, stdout, stderr io.Writer) int {
 	// A nil slice makes cobra fall back to os.Args, which would let a
 	// test binary's own flags reach the command tree.
@@ -139,10 +114,7 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 	var showVersion bool
 
 	printVersion := runE(func(context.Context, []string) error {
-		info, _ := debug.ReadBuildInfo()
+		_, err := fmt.Fprintf(stdout, "sfdupes %s\n", Version)
-
-		_, err := fmt.Fprintf(stdout, "sfdupes %s\n",
-			buildVersion(Version, info))
 		if err != nil {
 			return fmt.Errorf("write stdout: %w", err)
 		}
@@ -176,6 +148,30 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 	root.Flags().BoolVarP(&showVersion, "version", "v", false,
 		"print the version to stdout")
 
+	var (
+		scanWorkers int
+		scanOneFS   bool
+	)
+
+	scanCmd := &cobra.Command{
+		Use:   cmdScan + " [--workers N] [-x] PATH...",
+		Short: "Walk trees and synchronize the scan database",
+		Args:  cobra.MinimumNArgs(1),
+		PreRunE: func(cmd *cobra.Command, _ []string) error {
+			return checkScanWorkers(cmd, scanWorkers)
+		},
+		RunE: runE(func(ctx context.Context, args []string) error {
+			ctx, stop := interruptContext(ctx)
+			defer stop()
+
+			return runScan(ctx, args, scanWorkers, scanOneFS)
+		}),
+	}
+	scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
+		"concurrent workers for the walk, hash, and content phases")
+	scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
+		"do not cross filesystem boundaries")
+
 	reportCmd := &cobra.Command{
 		Use:   cmdReport,
 		Short: "Read the scan database and print the file-level duplicates report",
@@ -194,59 +190,11 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 		}),
 	}
 
-	infoCmd := &cobra.Command{
+	root.AddCommand(scanCmd, reportCmd, treesCmd)
-		Use:   cmdInfo,
-		Short: "Read the scan database and print its details",
-		Args:  cobra.NoArgs,
-		RunE: runE(func(ctx context.Context, _ []string) error {
-			return runInfo(ctx, stdout)
-		}),
-	}
-
-	root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
 
 	return root
 }
 
-// newScanCommand builds the scan subcommand, with its --workers, -x and
-// --no-default-excludes flags. Its help lists the default excludes of
-// the system it runs on, and the filesystem types it never records
-// anything on.
-func newScanCommand() *cobra.Command {
-	var (
-		scanWorkers           int
-		scanOneFS             bool
-		scanNoDefaultExcludes bool
-	)
-
-	short := "Walk trees and synchronize the scan database"
-
-	scanCmd := &cobra.Command{
-		Use:   cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
-		Short: short,
-		Long:  short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
-		Args:  cobra.MinimumNArgs(1),
-		PreRunE: func(cmd *cobra.Command, _ []string) error {
-			return checkScanWorkers(cmd, scanWorkers)
-		},
-		RunE: runE(func(ctx context.Context, args []string) error {
-			ctx, stop := interruptContext(ctx)
-			defer stop()
-
-			return runScan(ctx, args, scanWorkers, scanOneFS,
-				scanNoDefaultExcludes)
-		}),
-	}
-	scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
-		"concurrent workers for the walk, hash, and content phases")
-	scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
-		"do not cross filesystem boundaries")
-	scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
-		false, "enter the default excludes, the paths listed above")
-
-	return scanCmd
-}
-
 // checkScanWorkers rejects a scan --workers value below 1. That is a
 // usage error reported in one line: cobra prints the returned message
 // without the usage text, and run exits 2.

main_test.go

@@ -4,12 +4,10 @@ import (
 	"bytes"
 	"context"
 	"errors"
-	"fmt"
 	"io"
 	"io/fs"
 	"os"
 	"path/filepath"
-	"runtime/debug"
 	"slices"
 	"strconv"
 	"strings"
@@ -20,36 +18,6 @@ import (
 // error prints and a runtime failure must not.
 const usageMarker = "Usage:"
 
-// TestMain stops the tests at once, saying why, when their temporary
-// directory is not on a persistent filesystem: scan records nothing
-// there (README "Rules for the walk"), so most scan tests would fail
-// without saying why.
-func TestMain(m *testing.M) {
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		fmt.Fprintf(os.Stderr, "list the mounted filesystems: %v\n", err)
-		os.Exit(1)
-	}
-
-	dir := os.TempDir()
-
-	fi, err := os.Stat(dir)
-	if err != nil {
-		fmt.Fprintln(os.Stderr, err)
-		os.Exit(1)
-	}
-
-	dev, ok := deviceOfInfo(fi)
-	if fsType, found := nonPersistent[dev]; ok && found {
-		fmt.Fprintf(os.Stderr, "%s is on %s, not a persistent filesystem, "+
-			"and scan records nothing there: set TMPDIR to a directory on a "+
-			"persistent filesystem\n", dir, fsType)
-		os.Exit(1)
-	}
-
-	os.Exit(m.Run())
-}
-
 // walSuffixes are the SQLite sidecar files a WAL-mode database keeps
 // while it is open. A clean close checkpoints the WAL and removes
 // both; finding either afterwards means the database was never closed.
@@ -316,16 +284,10 @@ func assertFatalOutput(t *testing.T, stderr, stdout string) {
 }
 
 func TestRunMissingDatabaseIsFatal(t *testing.T) {
-	// README §Database: report, trees and info need an existing
+	// README §Database: report and trees need an existing database; a
-	// database; a missing one exits 1 with a message telling the user to
+	// missing one exits 1 with a message telling the user to run scan,
-	// run scan, after the line naming the database.
+	// after the line naming the database.
-	openLines := map[string]string{
+	for _, name := range []string{cmdReport, cmdTrees} {
-		cmdReport: reportOpenLine,
-		cmdTrees:  treesOpenLine,
-		cmdInfo:   infoOpenLine,
-	}
-
-	for name, openLine := range openLines {
 		t.Run(name, func(t *testing.T) {
 			path := testDBPath(t)
 			t.Setenv(databaseEnv, path)
@@ -341,10 +303,8 @@ func TestRunMissingDatabaseIsFatal(t *testing.T) {
 				t.Errorf("run(%s) = %d, want %d", name, code, exitFatal)
 			}
 
-			// Each shows its open phase, as it starts and as it ends,
-			// before the error.
 			want := name + ": database " + path + " (from " + databaseEnv +
-				")\n" + openLine + "\n" + openLine + "\n" +
+				")\n" +
 				"sfdupes: " + path + ": no database (run \"sfdupes " +
 				"scan\" first, or set " + databaseEnv + ")\n"
 			if got := stderr(); got != want {
@@ -399,14 +359,10 @@ func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) {
 	// use that file, relative or absolute, and its first line must name
 	// the path as set, with the working directory put in front of a
 	// relative one.
-	dir, err := filepath.EvalSymlinks(t.TempDir())
+	dir := t.TempDir()
-	if err != nil {
-		t.Fatal(err)
-	}
-
 	t.Chdir(dir)
 
-	err = os.MkdirAll(filepath.Join(dir, "real", "sub"), 0o750)
+	err := os.MkdirAll(filepath.Join(dir, "real", "sub"), 0o750)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -462,92 +418,14 @@ func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) {
 	}
 }
 
-func TestRunScanFromSymlinkedWorkingDirectory(t *testing.T) {
-	// README §Database: a relative path gets the working directory, its
-	// symlinks resolved, put in front of it. With home/proj a symlink
-	// to data/proj, entered as the working directory through the
-	// symlink, ../new/db.sqlite names data/new/db.sqlite. scan must
-	// create that directory and file, and nothing beside home/proj.
-	dir, err := filepath.EvalSymlinks(t.TempDir())
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	err = os.MkdirAll(filepath.Join(dir, "data", "proj"), 0o750)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	err = os.Mkdir(filepath.Join(dir, "home"), 0o750)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	err = os.Symlink(filepath.Join(dir, "data", "proj"),
-		filepath.Join(dir, "home", "proj"))
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// t.Chdir also sets PWD to the path through the symlink, as a
-	// shell does after cd.
-	t.Chdir(filepath.Join(dir, "home", "proj"))
-	t.Setenv(databaseEnv, "../new/db.sqlite")
-
-	want := "scan: new database " + dir + "/data/proj/../new/db.sqlite" +
-		" (from " + databaseEnv + ")\n"
-	if got := scanOK(t, t.TempDir()); !strings.HasPrefix(got, want) {
-		t.Errorf("scan stderr = %q, want it to start %q", got, want)
-	}
-
-	_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
-	if err != nil {
-		t.Errorf("database the path names: %v", err)
-	}
-
-	_, err = os.Lstat(filepath.Join(dir, "home", "new"))
-	if !errors.Is(err, fs.ErrNotExist) {
-		t.Errorf("home/new: stat error %v, want it missing", err)
-	}
-}
-
-func TestRunEscapesDatabasePath(t *testing.T) {
-	// README §Database: each subcommand's first line escapes the
-	// database path like the paths in a report, so a tab, newline or
-	// backslash in it keeps the line one line.
-	dir := t.TempDir()
-	t.Setenv(databaseEnv, dir+"/a\tb\nc\\d.sqlite")
-
-	escaped := dir + `/a\tb\nc\\d.sqlite`
-	from := " (from " + databaseEnv + ")\n"
-
-	want := "scan: new database " + escaped + from
-	if got := scanOK(t, t.TempDir()); !strings.HasPrefix(got, want) {
-		t.Errorf("scan stderr = %q, want it to start %q", got, want)
-	}
-
-	for _, name := range []string{cmdReport, cmdTrees} {
-		stderr := captureStderr(t)
-
-		code := run([]string{name}, io.Discard, os.Stderr)
-
-		want = name + ": database " + escaped + from
-		if got := stderr(); code != exitOK || !strings.HasPrefix(got, want) {
-			t.Errorf("run(%s) = %d with stderr %q, want %d and it to "+
-				"start %q", name, code, got, exitOK, want)
-		}
-	}
-}
-
 func TestRunNamesDefaultDatabase(t *testing.T) {
 	// README §Database: with SFDUPES_DATABASE empty, each subcommand's
 	// first line on stderr names the default database. A database may
 	// exist at the default path on the machine running the test, so
 	// nothing here may create or read one: scan gets an operand that
 	// does not exist, which stops it before it takes the lock, and
-	// report, trees and info a cancelled context, which stops them
+	// report and trees a cancelled context, which stops them before
-	// before they read anything. They show their open phase after the
+	// they read anything.
-	// line.
 	t.Setenv(databaseEnv, "")
 
 	// Whether scan calls the database new depends on that machine too.
@@ -567,7 +445,7 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
 	}{
 		cmdScan: {
 			run: func() error {
-				return runScan(t.Context(), []string{missing}, 1, false, false)
+				return runScan(t.Context(), []string{missing}, 1, false)
 			},
 			want: scanWant,
 		},
@@ -575,22 +453,13 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
 			run: func() error {
 				return runReport(cancelledContext(t), io.Discard)
 			},
-			want: "report: database " + defaultDatabasePath + " (default)\n" +
+			want: "report: database " + defaultDatabasePath + " (default)\n",
-				reportOpenLine + "\n" + reportOpenLine + "\n",
 		},
 		cmdTrees: {
 			run: func() error {
 				return runTrees(cancelledContext(t), io.Discard)
 			},
-			want: "trees: database " + defaultDatabasePath + " (default)\n" +
+			want: "trees: database " + defaultDatabasePath + " (default)\n",
-				treesOpenLine + "\n" + treesOpenLine + "\n",
-		},
-		cmdInfo: {
-			run: func() error {
-				return runInfo(cancelledContext(t), io.Discard)
-			},
-			want: "info: database " + defaultDatabasePath + " (default)\n" +
-				infoOpenLine + "\n" + infoOpenLine + "\n",
 		},
 	}
 
@@ -618,7 +487,6 @@ func TestRunUsageErrors(t *testing.T) {
 		"scan without paths": {[]string{cmdScan}, usageMarker},
 		"report with args":   {[]string{cmdReport, "x"}, usageMarker},
 		"trees with args":    {[]string{cmdTrees, "x"}, usageMarker},
-		"info with args":     {[]string{cmdInfo, "x"}, usageMarker},
 		"unknown flag":       {[]string{cmdScan, "--nope", "/"}, usageMarker},
 		"unknown subcommand": {[]string{"nope"}, "unknown command"},
 	}
@@ -715,8 +583,6 @@ func TestRunVersion(t *testing.T) {
 
 	// README §Subcommands: the version is one line on stdout, with
 	// nothing on stderr, and exits 0.
-	info, _ := debug.ReadBuildInfo()
-
 	for _, arg := range []string{"--version", "-v"} {
 		var stdout, stderr bytes.Buffer
 
@@ -725,7 +591,7 @@ func TestRunVersion(t *testing.T) {
 			t.Errorf("run(%s) = %d, want %d", arg, code, exitOK)
 		}
 
-		want := "sfdupes " + buildVersion(Version, info) + "\n"
+		want := "sfdupes " + Version + "\n"
 		if got := stdout.String(); got != want {
 			t.Errorf("run(%s) stdout = %q, want %q", arg, got, want)
 		}
@@ -736,40 +602,6 @@ func TestRunVersion(t *testing.T) {
 	}
 }
 
-func TestBuildVersion(t *testing.T) {
-	t.Parallel()
-
-	// README "Install": the linker's version wins; otherwise the main
-	// module's version that Go recorded; "dev" when neither is known.
-	tagged := &debug.BuildInfo{Main: debug.Module{Version: "v0.0.2"}}
-	pseudo := &debug.BuildInfo{Main: debug.Module{
-		Version: "v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1",
-	}}
-	devel := &debug.BuildInfo{Main: debug.Module{Version: "(devel)"}}
-
-	cases := []struct {
-		name   string
-		linker string
-		info   *debug.BuildInfo
-		want   string
-	}{
-		{"linker set", "v0.0.2-3-g4e4a94d", tagged, "v0.0.2-3-g4e4a94d"},
-		{"linker set, no build info", "4e4a94d", nil, "4e4a94d"},
-		{"build info with a tag", "", tagged, "v0.0.2"},
-		{"build info with a pseudo-version", "", pseudo,
-			"v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1"},
-		{"(devel)", "", devel, "dev"},
-		{"no build info", "", nil, "dev"},
-	}
-
-	for _, c := range cases {
-		if got := buildVersion(c.linker, c.info); got != c.want {
-			t.Errorf("%s: buildVersion(%q, ...) = %q, want %q",
-				c.name, c.linker, got, c.want)
-		}
-	}
-}
-
 func TestRunVersionWriteFailureIsFatal(t *testing.T) {
 	t.Parallel()
 
@@ -995,16 +827,11 @@ func TestRunReportSucceeds(t *testing.T) {
 		t.Errorf("stdout = %q, want %q", got, want)
 	}
 
-	// Its phases come between the two lines; see
+	want = "report: database " + path + " (from " + databaseEnv + ")\n" +
-	// TestRunReportProgressNamesEachPhase.
+		"report: 2 records read, 1 duplicate groups, 1 dupe files, " +
-	first := "report: database " + path + " (from " + databaseEnv + ")\n"
-	last := "\nreport: 2 records read, 1 duplicate groups, 1 dupe files, " +
 		"300 B reclaimable\n"
-
+	if got := stderr(); got != want {
-	if got := stderr(); !strings.HasPrefix(got, first) ||
+		t.Errorf("stderr = %q, want %q", got, want)
-		!strings.HasSuffix(got, last) {
-		t.Errorf("stderr = %q, want it to start %q and end %q",
-			got, first, last)
 	}
 
 	assertNoSidecars(t, path)
@@ -1034,16 +861,11 @@ func TestRunTreesSucceeds(t *testing.T) {
 		t.Errorf("stdout = %q, want %q", got, want)
 	}
 
-	// Its phases come between the two lines; see
+	want = "trees: database " + path + " (from " + databaseEnv + ")\n" +
-	// TestRunTreesProgressNamesEachPhase.
+		"trees: 2 records read, 1 duplicate tree groups, 1 dupe trees, " +
-	first := "trees: database " + path + " (from " + databaseEnv + ")\n"
+		"300 B reclaimable\n"
-	last := "\ntrees: 2 records read, 1 duplicate tree groups, " +
+	if got := stderr(); got != want {
-		"1 dupe trees, 300 B reclaimable\n"
+		t.Errorf("stderr = %q, want %q", got, want)
-
-	if got := stderr(); !strings.HasPrefix(got, first) ||
-		!strings.HasSuffix(got, last) {
-		t.Errorf("stderr = %q, want it to start %q and end %q",
-			got, first, last)
 	}
 
 	assertNoSidecars(t, path)
@@ -1335,7 +1157,7 @@ func TestRunReportsDuringScan(t *testing.T) {
 func TestRunStdoutClosedIsFatal(t *testing.T) {
 	// README §Error handling: a stdout write failure exits 1, reported
 	// in one line on stderr.
-	for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
+	for _, name := range []string{cmdReport, cmdTrees} {
 		t.Run(name, func(t *testing.T) {
 			t.Setenv(databaseEnv, testDBPath(t))
 
@@ -1385,7 +1207,6 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
 	cases := map[string]func(context.Context, io.Writer) error{
 		cmdReport: runReport,
 		cmdTrees:  runTrees,
-		cmdInfo:   runInfo,
 	}
 
 	for name, fn := range cases {

progress.go

@@ -2,333 +2,122 @@ package main
 
 import (
 	"fmt"
-	"io"
-	"math"
 	"os"
-	"slices"
-	"strings"
-	"sync"
 	"time"
 
 	"github.com/schollz/progressbar/v3"
 	"golang.org/x/term"
 )
 
-// plainInterval is the time between progress lines while a phase runs,
+// plainInterval is the minimum time between progress lines when stderr
-// when stderr is not a TTY.
+// is not a TTY.
 const plainInterval = 5 * time.Second
 
-// redrawInterval is the time between redraws of the progress display on
+// barThrottle caps how often the TTY progress bar redraws.
-// a TTY.
+const barThrottle = 100 * time.Millisecond
-const redrawInterval = 100 * time.Millisecond
 
-// rateWindow is the time the rate a display shows covers: the items
+// walkSpinnerType selects the progressbar library's spinner style used
-// completed over the last rateWindow, which is the time since the line
+// for the walk pass, which has no known total.
-// before when stderr is not a TTY.
+const walkSpinnerType = 14
-const rateWindow = 5 * time.Second
-
-// stallTime is how long a phase goes with no item completed before its
-// line says it has stalled, and where it waits, in place of the rate.
-const stallTime = 10 * time.Second
-
-// spinnerType selects the progressbar library's spinner style.
-const spinnerType = 14
-
-// barWidth is the width of the progressbar library's bar between its
-// two ends, as in the example line in README "Progress".
-const barWidth = 10
 
 // percentScale converts a fraction to a percentage.
 const percentScale = 100
 
-// fallbackWidth is the width taken for a terminal whose width cannot be
-// read, as the progressbar library takes it.
-const fallbackWidth = 80
-
-// spinnerMargin is the columns of a spinner's line besides its text:
-// the spinner and the three spaces the library draws around the text,
-// and the terminal's last column, which every line stays clear of so
-// that no terminal wraps it.
-const spinnerMargin = 5
-
-// barMargin is the columns of a bar's line besides its text: the
-// percent and the bar the library draws after the text, the spaces
-// around them, and the terminal's last column.
-const barMargin = barWidth + 11
-
-// countingRecords is what the display of a phase that reads the records
-// says while it counts them, before it has its total.
-const countingRecords = "counting the records"
-
-// openPhase is the name of the phase in which report, trees and info
-// open the database, and openingDatabase what its display says while
-// it does.
-const (
-	openPhase       = "open"
-	openingDatabase = "opening the database"
-)
-
 // stderrIsTTY reports whether stderr is attached to a terminal.
 func stderrIsTTY() bool {
 	return term.IsTerminal(int(os.Stderr.Fd()))
 }
 
-// terminalWidth returns the width of the terminal on stderr, or
+// progress renders one scan pass's progress on stderr. On a TTY it
-// fallbackWidth, as the progressbar library does, when it cannot read
+// delegates to the progressbar library (spinner style when the total is
-// one.
+// unknown, full bar with count/percent/rate/elapsed/ETA otherwise). When
-func terminalWidth() int {
+// stderr is not a TTY it emits no ANSI redraws: it prints a plain
-	width, _, err := term.GetSize(int(os.Stderr.Fd()))
+// one-line update as the pass starts, then no more often than every
-	if err != nil || width <= 0 {
+// plainInterval.
-		return fallbackWidth
-	}
-
-	return width
-}
-
-// phaseLabel names phase with its place among phases, as "[2/5] walk".
-func phaseLabel(phases []string, phase string) string {
-	return fmt.Sprintf("[%d/%d] %s",
-		slices.Index(phases, phase)+1, len(phases), phase)
-}
-
-// progressParams describe one phase's progress display as it starts.
-type progressParams struct {
-	label      string // the phase and its place, from phaseLabel
-	unit       string // what the phase counts, as "files"
-	total      int64  // the items the phase will complete; -1 when not known
-	estimate   bool   // total is an estimate, which the count may pass
-	waitingFor string // what the phase waits for until setTotal is called
-	stalledOn  string // where the phase waits, as "disk/db"; see stallTime
-}
-
-// drawnCount is the count of items completed as of one draw, and when
-// that draw was.
-type drawnCount struct {
-	at    time.Time
-	count int64
-}
-
-// progress renders one phase's progress on stderr. On a TTY the
-// progressbar library draws line, fitted to the terminal's width: after
-// the library's spinner, or, for a phase with an exact total, before the
-// library's bar when the terminal has room for the bar beside it. It is
-// drawn again every redrawInterval, so its elapsed time keeps moving
-// while the phase waits for its next item. When stderr is not a TTY it
-// emits no ANSI redraws: it prints line as the phase starts, every
-// plainInterval while it runs, and as it ends.
 //
-// The draws come from a goroutine of their own, so mu guards the fields
+// All methods must be called from the main goroutine only. On a TTY
-// below it. Every method must be called from the main goroutine. A nil
+// the library also redraws a spinner from its own goroutine, several
-// *progress is a valid no-display receiver: every method is a no-op, so
+// times a second, so its count and elapsed time stay current while a
-// batched database flushes during the hash phase can reuse the update
+// pass waits for its next item. A nil *progress is a valid
-// phase's helpers without rendering anything, and its writer writes
+// no-display receiver: every method is a no-op, so batched database
-// with no display to clear.
+// flushes during the streaming pass can reuse the update-pass helpers
+// without rendering anything.
 type progress struct {
-	now   func() time.Time // reads the time
-	width func() int       // reads the terminal's width; nil when not a TTY
-	stop  chan struct{}    // closed by finish to end the draws
-	done  chan struct{}    // closed once the draws have ended
-
-	mu         sync.Mutex
-	bar        *progressbar.ProgressBar // what the library draws; nil until then
-	withBar    bool                     // bar is the library's bar, not its spinner
-	widest     int                      // the widest line bar has drawn, in columns
 	label string
-	unit       string
+	total int64 // -1 when unknown (walk pass)
-	total      int64
+	bar   *progressbar.ProgressBar
-	estimate   bool
-	waitingFor string
-	stalledOn  string
 	count int64
-	start      time.Time // when the phase started, for the elapsed time
+	start time.Time
-	rateStart  time.Time // when the count started, for the ETA
+	last  time.Time
-	lastItem   time.Time // when the last item was completed, or the count started
-	// drawn holds the count at each draw over the last rateWindow, oldest
-	// first; the rate counts from the first. It keeps the draw before the
-	// latest even when that is older, as the line before is when lines
-	// are plainInterval apart and a tick comes late.
-	drawn []drawnCount
 }
 
-// newProgress starts the display of one phase on stderr.
+func newProgress(label string, total int64) *progress {
-func newProgress(params progressParams) *progress {
+	p := &progress{label: label, total: total, start: time.Now()}
 	if stderrIsTTY() {
-		return startProgress(params, terminalWidth, redrawInterval, time.Now)
+		p.bar = newBar(label, total)
+
+		return p
 	}
 
-	return startProgress(params, nil, plainInterval, time.Now)
+	// Print the zero state at once: the first item may take minutes,
-}
+	// and a pass must never look hung.
-
+	p.last = p.start
-// startProgress shows the phase params describes at once, and again
+	fmt.Fprintln(os.Stderr, p.plainLine())
-// every interval until finish: with the progressbar library on a
-// terminal, whose width each draw reads by calling width, or as a plain
-// line when width is nil. The display reads the time by calling now.
-func startProgress(params progressParams, width func() int,
-	interval time.Duration, now func() time.Time,
-) *progress {
-	start := now()
-	p := &progress{
-		now: now, width: width,
-		stop: make(chan struct{}), done: make(chan struct{}),
-		label: params.label, unit: params.unit, total: params.total,
-		estimate: params.estimate, waitingFor: params.waitingFor,
-		stalledOn: params.stalledOn, start: start, rateStart: start,
-		lastItem: start, drawn: []drawnCount{{at: start}},
-	}
-
-	// Show the zero state at once: the first item may take minutes,
-	// and a phase must never look hung.
-	p.draw()
-
-	go func() {
-		defer close(p.done)
-
-		ticker := time.NewTicker(interval)
-		defer ticker.Stop()
-
-		for {
-			select {
-			case <-p.stop:
-				return
-			case <-ticker.C:
-				p.draw()
-			}
-		}
-	}()
 
 	return p
 }
 
-// draw shows the phase's progress: on a TTY in place of the line
+// newBar builds the TTY display for newProgress.
-// before, and otherwise as a plain line. It first adds the count to
+func newBar(label string, total int64) *progressbar.ProgressBar {
-// drawn, then drops from drawn the draws older than rateWindow, except
+	opts := []progressbar.Option{
-// the one before this draw.
-func (p *progress) draw() {
-	p.mu.Lock()
-	defer p.mu.Unlock()
-
-	now := p.now()
-
-	p.drawn = append(p.drawn, drawnCount{at: now, count: p.count})
-	for len(p.drawn) > 2 && now.Sub(p.drawn[0].at) > rateWindow {
-		p.drawn = p.drawn[1:]
-	}
-
-	if p.width == nil {
-		fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true))
-
-		return
-	}
-
-	width := p.width()
-
-	// A phase with an exact total gets the library's bar when its whole
-	// line, without the percent the bar shows, fits beside the bar: the
-	// bar gives way before any part of the line. A total of 0 leaves the
-	// bar nothing to fill.
-	text, withBar := "", false
-	if p.total > 0 && !p.estimate {
-		text = p.line(now, math.MaxInt, false)
-		withBar = len(text) <= width-barMargin
-	}
-
-	margin := barMargin
-	if !withBar {
-		margin = spinnerMargin
-		text = p.line(now, width-margin, true)
-	}
-
-	// The library makes a bar or a spinner, and before each draw erases
-	// its line with as many spaces as the widest line it has drawn. A
-	// display gets a new one when it turns from one to the other, as its
-	// line grows or the terminal is resized, and when the terminal has
-	// been made too narrow for that erase, which would wrap and leave a
-	// line behind at every draw.
-	if p.bar == nil || p.withBar != withBar || p.widest >= width {
-		p.newBar(withBar, width)
-	}
-
-	if withBar {
-		_ = p.bar.Set64(p.count)
-	}
-
-	p.bar.Describe(text)
-
-	// The library drew the text and its margin, less the terminal's last
-	// column, which the margin keeps clear.
-	p.widest = max(p.widest, len(text)+margin-1)
-}
-
-// newBar has the library make what draw draws: a bar for the phase's
-// total when withBar is set, and a spinner when not. Either shows the
-// text draw gives it, and nothing of its own besides the spinner, or
-// the bar and its percent. p.mu must be held.
-func (p *progress) newBar(withBar bool, width int) {
-	if p.bar != nil {
-		// Erase what was drawn before, short of the last column of a
-		// terminal that may have been made narrower since: the library's
-		// own erase is as wide as the widest line it drew.
-		blank := strings.Repeat(" ", min(p.widest, width-1))
-		fmt.Fprint(os.Stderr, "\r"+blank+"\r")
-	}
-
-	total := int64(-1) // the library's spinner
-	if withBar {
-		total = p.total
-	}
-
-	p.bar = progressbar.NewOptions64(total,
 		progressbar.OptionSetWriter(os.Stderr),
-		progressbar.OptionSetWidth(barWidth),
+		progressbar.OptionSetDescription(label),
-		progressbar.OptionSpinnerType(spinnerType),
+		progressbar.OptionShowCount(),
-		// The spinner turns at each draw, not on a timer of the
+		progressbar.OptionShowIts(),
-		// library's own.
+		progressbar.OptionSetItsString("files"),
-		progressbar.OptionSetSpinnerChangeInterval(0),
+		progressbar.OptionSetElapsedTime(true),
-		// The text carries the elapsed time and the ETA.
+		progressbar.OptionThrottle(barThrottle),
-		progressbar.OptionSetElapsedTime(false),
+		// Render at zero immediately: a phase must be visible the
+		// moment it starts, even before its first item completes.
+		progressbar.OptionSetRenderBlankState(true),
+	}
+	if total >= 0 {
+		opts = append(opts,
+			progressbar.OptionSetPredictTime(true),
+			progressbar.OptionShowElapsedTimeOnFinish(),
+		)
+	} else {
+		opts = append(opts,
 			progressbar.OptionSetPredictTime(false),
+			progressbar.OptionSpinnerType(walkSpinnerType),
 		)
-	p.withBar, p.widest = withBar, 0
-}
-
-// setTotal gives a phase that started without a total its total, once
-// it is known. The rate, the ETA and the time with no item completed
-// count from then on: the time spent waiting for the total says nothing
-// about how fast items go.
-func (p *progress) setTotal(total int64) {
-	if p == nil {
-		return
 	}
 
-	now := p.now()
+	return progressbar.NewOptions64(total, opts...)
-
-	p.mu.Lock()
-	defer p.mu.Unlock()
-
-	p.total = total
-	p.rateStart, p.lastItem = now, now
-	p.drawn = []drawnCount{{at: now, count: p.count}}
 }
 
-// increment records one completed item, which the display shows at its
+// increment records one completed item and refreshes the display.
-// next draw.
 func (p *progress) increment() {
 	if p == nil {
 		return
 	}
 
-	now := p.now()
-
-	p.mu.Lock()
-	defer p.mu.Unlock()
-
 	p.count++
-	p.lastItem = now
+	if p.bar != nil {
+		_ = p.bar.Add(1)
+
+		return
+	}
+
+	if time.Since(p.last) >= plainInterval {
+		p.last = time.Now()
+		fmt.Fprintln(os.Stderr, p.plainLine())
+	}
 }
 
-// warnf prints a one-line warning to stderr without corrupting the
+// warnf prints a one-line warning to stderr without corrupting the bar.
-// display. The whole message is escaped like a report's path columns,
+// The whole message is escaped like a report's path columns, so a path
-// so a path holding a newline cannot split the warning.
+// holding a newline cannot split the warning.
 func (p *progress) warnf(format string, args ...any) {
 	if p == nil {
 		return
@@ -336,186 +125,58 @@ func (p *progress) warnf(format string, args ...any) {
 
 	msg := escapePath(fmt.Sprintf(format, args...))
 
-	p.mu.Lock()
+	if p.bar != nil && p.total < 0 {
-	defer p.mu.Unlock()
+		// The library also redraws a spinner from its own goroutine, so
+		// a direct write could land inside a redraw. The bar prints the
+		// warning itself, just before its next redraw.
+		_, _ = progressbar.Bprintln(p.bar, msg)
+
+		return
+	}
 
 	if p.bar != nil {
-		// The warning takes the display's line; the next draw shows
-		// the display again, on the line below.
 		_ = p.bar.Clear()
 	}
 
 	fmt.Fprintln(os.Stderr, msg)
+
+	// The library never draws a finished bar again, so the warning for
+	// a pass's last item would erase it for good: draw its last frame
+	// again below the warning.
+	if p.bar != nil && p.bar.IsFinished() {
+		fmt.Fprint(os.Stderr, p.bar.String())
+	}
 }
 
-// progressWriter writes a phase's output to out, which may be the file
+// finish terminates the pass's display. A bar whose pass stopped short
-// the phase's display is drawn on, as stdout is in a plain "sfdupes
+// of its total, as an interrupted one does, is left as last drawn; the
-// trees" on a terminal. It holds what is written to it and writes that
+// library's Finish would fill it up.
-// to out in pieces of about ioBufSize bytes. When out is the file
-// stderr is, each piece is written under the display's lock with the
-// display cleared first, as warnf does for a warning: no draw lands in
-// the middle of a piece, and the next draw shows the display again, on
-// the line below. Every write to it must end with a newline, as a row
-// written with one fmt.Fprintf does, so that each piece ends a line.
-// Flush writes what it holds once the last row is written.
-type progressWriter struct {
-	// p is the display to clear, nil unless out is the file stderr is.
-	p   *progress
-	out io.Writer
-	buf []byte
-}
-
-// writer returns a progressWriter writing to out around p's display.
-// It clears the display and takes its lock only when out is the file
-// stderr is: the same device and inode, as os.SameFile compares them,
-// which is one terminal, or one pipe or file that both are sent to.
-// Otherwise it writes to out without the lock, so the display keeps
-// being drawn while a write to out waits for a slow reader.
-func (p *progress) writer(out io.Writer) *progressWriter {
-	w := &progressWriter{out: out}
-
-	if f, ok := out.(*os.File); ok {
-		outInfo, outErr := f.Stat()
-		errInfo, errErr := os.Stderr.Stat()
-
-		if outErr == nil && errErr == nil && os.SameFile(outInfo, errInfo) {
-			w.p = p
-		}
-	}
-
-	return w
-}
-
-// Write holds b, and writes what it holds to out once that comes to
-// ioBufSize bytes.
-func (w *progressWriter) Write(b []byte) (int, error) {
-	w.buf = append(w.buf, b...)
-	if len(w.buf) < ioBufSize {
-		return len(b), nil
-	}
-
-	return len(b), w.Flush()
-}
-
-// Flush writes what w holds to out. When out is the file stderr is, the
-// display is cleared first and not drawn again until it is written.
-func (w *progressWriter) Flush() error {
-	if w.p != nil {
-		w.p.mu.Lock()
-		defer w.p.mu.Unlock()
-
-		if w.p.bar != nil {
-			_ = w.p.bar.Clear()
-		}
-	}
-
-	_, err := w.out.Write(w.buf)
-	w.buf = w.buf[:0]
-
-	return err
-}
-
-// finish ends the phase's display, drawn once more as the phase ends.
-// A phase stopped short of its total, as an interrupted one is, shows
-// the count it stopped at rather than a full bar.
 func (p *progress) finish() {
 	if p == nil {
 		return
 	}
 
-	close(p.stop)
+	if p.bar != nil {
-	<-p.done
+		if p.total >= 0 && p.count < p.total {
-
+			_ = p.bar.Exit()
-	p.draw()
-
-	if p.width != nil {
-		// The display stays on the TTY as drawn; what follows starts
-		// on a line of its own.
-		fmt.Fprintln(os.Stderr)
-	}
-}
-
-// line formats the phase's progress as of now in at most room columns,
-// as
-//
-//	[3/5] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s
-//
-// where "files" is the phase's unit, without the percent when percent
-// is false, as before the library's bar, which shows its own. A phase
-// with no total shows the items done instead, or what it waits for. The
-// rate is that since the first draw in drawn, and the ETA comes from
-// the rate since the count started. Once no item has been completed for
-// stallTime, the rate, which would read 0, gives way to the word
-// "stalled" and where the phase waits, as "stalled disk/db".
-//
-// The phase and its place, the progress, the elapsed time and the ETA
-// are kept whole. A line wider than room leaves out the rate, the words
-// of a stalled phase, or what the phase waits for; only if it is still
-// wider is it cut at room. Every part is ASCII, so its length is its
-// width in columns. p.mu must be held.
-func (p *progress) line(now time.Time, room int, percent bool) string {
-	elapsed := now.Sub(p.start).Round(time.Second)
-
-	// head and tail hold the parts kept whole; extra, between them, is
-	// the part left out.
-	var head, extra, tail string
-
-	if p.total < 0 && p.waitingFor != "" {
-		head, extra = p.label+": ", p.waitingFor+", "
-		tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed)
 		} else {
-		since := p.drawn[0]
+			_ = p.bar.Finish()
-		rate := perSecond(p.count-since.count, now.Sub(since.at))
-		average := perSecond(p.count, now.Sub(p.rateStart))
-
-		head = p.label + ": " + p.itemsDone(percent) + ", "
-		extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit)
-		tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average))
-
-		if now.Sub(p.lastItem) >= stallTime {
-			extra = "stalled " + p.stalledOn + ", "
-		}
 		}
 
-	full, kept := head+extra+tail, head+tail
+		fmt.Fprintln(os.Stderr)
 
-	switch {
+		return
-	case len(full) <= room:
-		return full
-	case len(kept) <= room:
-		return kept
 	}
 
-	return kept[:max(room, 0)]
+	fmt.Fprintln(os.Stderr, p.plainLine())
 }
 
-// perSecond returns count items over d as items a second, or 0 when no
+// plainLine formats a non-TTY progress line.
-// time has passed.
+func (p *progress) plainLine() string {
-func perSecond(count int64, d time.Duration) float64 {
+	elapsed := time.Since(p.start)
-	if d <= 0 {
-		return 0
-	}
-
-	return float64(count) / d.Seconds()
-}
-
-// itemsDone formats the items done, as "[1234/5000] 25%", with "~"
-// before a total that is an estimate and without the percent when
-// percent is false, or as "1234 files", in the phase's unit, when the
-// phase has no total. p.mu must be held.
-func (p *progress) itemsDone(percent bool) string {
 	if p.total < 0 {
-		return fmt.Sprintf("%d %s", p.count, p.unit)
+		return fmt.Sprintf("%s: %d files, elapsed %s",
-	}
+			p.label, p.count, elapsed.Round(time.Second))
-
-	approx := ""
-	if p.estimate {
-		approx = "~"
-	}
-
-	done := fmt.Sprintf("[%d/%s%d]", p.count, approx, p.total)
-	if !percent {
-		return done
 	}
 
 	pct := float64(percentScale)
@@ -523,24 +184,19 @@ func (p *progress) itemsDone(percent bool) string {
 		pct = percentScale * float64(p.count) / float64(p.total)
 	}
 
-	return fmt.Sprintf("%s %.0f%%", done, pct)
+	rate := 0.0
-}
+	if s := elapsed.Seconds(); s > 0 {
-
+		rate = float64(p.count) / s
-// eta returns the time the phase has left at rate items a second, or
-// "unknown": with no total, before the first item, and once the count
-// passes an estimate. p.mu must be held.
-func (p *progress) eta(rate float64) string {
-	switch {
-	case p.total < 0, p.estimate && p.count > p.total:
-		return "unknown"
-	case p.count >= p.total:
-		return "0s"
-	case rate <= 0:
-		return "unknown"
 	}
 
-	remaining := time.Duration(
+	eta := "?"
-		float64(p.total-p.count) / rate * float64(time.Second))
 
-	return remaining.Round(time.Second).String()
+	if rate > 0 && p.count <= p.total {
+		remaining := time.Duration(float64(p.total-p.count) / rate * float64(time.Second))
+		eta = remaining.Round(time.Second).String()
+	}
+
+	return fmt.Sprintf("%s: [%d/%d] %.0f%% %.0f files/s elapsed %s eta %s",
+		p.label, p.count, p.total, pct, rate,
+		elapsed.Round(time.Second), eta)
 }

report.go

@@ -1,12 +1,11 @@
 package main
 
 import (
+	"bufio"
 	"context"
-	"database/sql"
 	"fmt"
 	"io"
 	"os"
-	"slices"
 	"strings"
 	"time"
 )
@@ -30,25 +29,13 @@ type scanRec struct {
 	path    string
 }
 
-// reportPhase names one phase of report with its place among all of
-// them, as "[2/2] read", for the phase's progress display. The list
-// here is the one place the phases are counted.
-func reportPhase(name string) string {
-	return phaseLabel([]string{openPhase, "read"}, name)
-}
-
 // runReport implements the report subcommand: it prints the file-level
-// duplicates report as TSV on stdout. It never touches the scanned
+// duplicates report as TSV on stdout. SQLite groups and orders the
-// filesystem; its only I/O is the database, stdout, and stderr. It
+// records, and each row is written as it is read, so no group is held
-// names the database on stderr before it opens it. Any database
+// in memory. It never touches the scanned filesystem; its only I/O is
+// the database (with SQLite's temporary sort file), stdout, and stderr.
+// It names the database on stderr before it opens it. Any database
 // problem, including a missing database, is fatal.
-//
-// The work runs in two phases, each under a progress display of its
-// own on stderr: open (open the database) and read (count the records,
-// then read them in signature order, writing each size's duplicate
-// groups to stdout once that size is read; see readReport). Each
-// display ends before the next phase starts, and before an error is
-// returned to be printed.
 func runReport(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {
@@ -58,194 +45,63 @@ func runReport(ctx context.Context, stdout io.Writer) error {
 	fmt.Fprintf(os.Stderr, "report: database %s (%s)\n",
 		escapePath(dbPath), source)
 
-	// Opening can wait on a busy database for as long as the busy
-	// timeout.
-	prog := newProgress(progressParams{
-		label: reportPhase(openPhase), total: -1,
-		waitingFor: openingDatabase,
-	})
 	db, err := openReportDatabase(ctx, dbPath)
-
-	prog.finish()
-
 	if err != nil {
 		return err
 	}
 
 	defer func() { _ = db.Close() }()
 
-	// The read phase writes the report to stdout while it reads the
+	out := bufio.NewWriterSize(stdout, ioBufSize)
-	// records, so it may wait on either.
-	prog = newProgress(progressParams{
-		label: reportPhase("read"), unit: "records", total: -1,
-		waitingFor: countingRecords, stalledOn: "db/out",
-	})
-	report, err := readReport(ctx, db, dbPath, stdout, prog)
-
-	prog.finish()
 
+	_, err = fmt.Fprintln(out, "first\tdupe\tsize")
 	if err != nil {
-		return err
+		return fmt.Errorf("write stdout: %w", err)
 	}
 
-	fmt.Fprintf(os.Stderr,
+	var (
-		"report: %d records read, %d duplicate groups, %d dupe files, "+
+		groups, dupeFiles int
-			"%s reclaimable\n",
+		reclaimable       int64
-		report.records, report.dupeGroups, report.dupeFiles,
+		writeErr          error
-		humanBytes(report.reclaimable))
+	)
+
+	records, err := loadDupeRows(ctx, db,
+		func(first, path string, size int64) error {
+			// A group's first path is its first row; every other
+			// path is a dupe.
+			if path == first {
+				groups++
 
 				return nil
-}
-
-// readReport gives prog the number of records in db, the database at
-// dbPath, as its total, then reads every record in signature order,
-// counting each on prog, and writes the report to stdout: the header
-// line, then the rows of each size's duplicate groups once that size is
-// read. stdout may be the terminal prog is drawn on, so the lines go
-// through prog's writer. It returns what it wrote, with the totals for
-// the summary line.
-func readReport(ctx context.Context, db *sql.DB, dbPath string,
-	stdout io.Writer, prog *progress,
-) (*reportWriter, error) {
-	out := prog.writer(stdout)
-	report := &reportWriter{out: out}
-
-	_, err := fmt.Fprintln(out, "first\tdupe\tsize")
-	if err != nil {
-		return nil, fmt.Errorf("write stdout: %w", err)
 			}
 
-	var writeErr error
+			_, writeErr = fmt.Fprintf(out, "%s\t%s\t%d\n",
-
+				escapePath(first), escapePath(path), size)
-	err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
+			dupeFiles++
-		prog.increment()
+			reclaimable += size
-
-		writeErr = report.add(r)
 
 			return writeErr
 		})
 
 	if writeErr != nil {
-		return nil, fmt.Errorf("write stdout: %w", writeErr)
+		return fmt.Errorf("write stdout: %w", writeErr)
 	}
 
 	if err != nil {
-		return nil, fmt.Errorf("database %s: %w", dbPath, err)
+		return fmt.Errorf("database %s: %w", dbPath, err)
 	}
 
-	err = report.finish()
-	if err == nil {
 	err = out.Flush()
-	}
-
 	if err != nil {
-		return nil, fmt.Errorf("write stdout: %w", err)
+		return fmt.Errorf("write stdout: %w", err)
 	}
 
-	return report, nil
+	fmt.Fprintf(os.Stderr,
-}
+		"report: %d records read, %d duplicate groups, %d dupe files, "+
-
+			"%s reclaimable\n",
-// reportWriter writes the report's rows to out from the records, added
+		records, groups, dupeFiles, humanBytes(reclaimable))
-// in signature order, largest size first, as loadSignatureRows reads
-// them. The records sharing a signature come together, and so do the
-// records of one size, so it holds only the duplicate groups of the
-// size being read, and writes them, ordered by first path, once a
-// record of a smaller size comes. It counts what the summary line
-// reports.
-type reportWriter struct {
-	out io.Writer
-
-	sig    scanRec    // the signature being read; its path is not used
-	paths  []string   // the paths read so far with sig
-	groups [][]string // the duplicate groups of sig.size, each sorted
-
-	records, dupeGroups, dupeFiles int
-	reclaimable                    int64
-}
-
-// add takes the next record. One with the signature being read joins
-// its paths; any other ends them, and one of a smaller size first
-// writes the groups of the size before.
-func (w *reportWriter) add(r scanRec) error {
-	w.records++
-
-	// A record without a content hash has unknown content (README
-	// "Database"), so it is never a duplicate.
-	if r.content == "" {
-		return nil
-	}
-
-	if r.size == w.sig.size && r.head == w.sig.head &&
-		r.tail == w.sig.tail && r.content == w.sig.content {
-		w.paths = append(w.paths, r.path)
 
 	return nil
-	}
-
-	w.endGroup()
-
-	if r.size != w.sig.size {
-		err := w.writeGroups()
-		if err != nil {
-			return err
-		}
-	}
-
-	w.sig = r
-	w.paths = append(w.paths, r.path)
-
-	return nil
-}
-
-// endGroup ends the paths read with w.sig: two or more are a duplicate
-// group, held with its paths sorted until writeGroups writes it.
-func (w *reportWriter) endGroup() {
-	if len(w.paths) < minGroupSize {
-		// A path with no duplicate is dropped, and its slice reused.
-		w.paths = w.paths[:0]
-
-		return
-	}
-
-	slices.Sort(w.paths)
-	w.groups = append(w.groups, w.paths)
-	w.paths = nil
-}
-
-// writeGroups writes the duplicate groups held, all of size w.sig.size,
-// ordered by first path: for each group, a row for every path but the
-// first. No two groups share a first path, since no two records share
-// a path.
-func (w *reportWriter) writeGroups() error {
-	slices.SortFunc(w.groups, func(a, b []string) int {
-		return strings.Compare(a[0], b[0])
-	})
-
-	for _, g := range w.groups {
-		for _, path := range g[1:] {
-			_, err := fmt.Fprintf(w.out, "%s\t%s\t%d\n",
-				escapePath(g[0]), escapePath(path), w.sig.size)
-			if err != nil {
-				return err
-			}
-
-			w.dupeFiles++
-			w.reclaimable += w.sig.size
-		}
-
-		w.dupeGroups++
-	}
-
-	w.groups = nil
-
-	return nil
-}
-
-// finish writes the groups of the last size read.
-func (w *reportWriter) finish() error {
-	w.endGroup()
-
-	return w.writeGroups()
 }
 
 // escapePath returns a path as it is written in a report column (README

report_test.go

@@ -9,7 +9,6 @@ import (
 	"os"
 	"path/filepath"
 	"slices"
-	"strconv"
 	"strings"
 	"testing"
 	"time"
@@ -29,8 +28,7 @@ func awkwardPairRecs() []scanRec {
 	}
 }
 
-// seedDatabase writes recs into a fresh database, closes it as scan
+// seedDatabase writes recs into a fresh database and returns its path.
-// does, out of WAL mode, and returns its path.
 func seedDatabase(t *testing.T, recs []scanRec) string {
 	t.Helper()
 
@@ -46,7 +44,10 @@ func seedDatabase(t *testing.T, recs []scanRec) string {
 		t.Fatal(err)
 	}
 
-	closeScanDatabase(t.Context(), db, path)
+	err = db.Close()
+	if err != nil {
+		t.Fatal(err)
+	}
 
 	return path
 }
@@ -59,39 +60,25 @@ type dupeGroup struct {
 }
 
 // dupeGroups returns the duplicate groups report reads from db, in
-// report order, from the rows it writes: a group's rows come together,
+// report order.
-// each naming the group's first path.
 func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
 	t.Helper()
 
-	var out bytes.Buffer
-
-	_, err := readReport(t.Context(), db, "", &out, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
 	var groups []dupeGroup
 
-	rows := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n")
+	_, err := loadDupeRows(t.Context(), db,
-	for _, row := range rows[1:] { // after the header
+		func(first, path string, size int64) error {
-		fields := strings.Split(row, "\t")
+			if path == first {
-		if len(fields) != 3 {
+				groups = append(groups, dupeGroup{size: size})
-			t.Fatalf("row %q, want first, dupe and size", row)
-		}
-
-		size, err := strconv.ParseInt(fields[2], 10, 64)
-		if err != nil {
-			t.Fatal(err)
-		}
-
-		if len(groups) == 0 || groups[len(groups)-1].paths[0] != fields[0] {
-			groups = append(groups,
-				dupeGroup{size: size, paths: []string{fields[0]}})
 			}
 
 			g := &groups[len(groups)-1]
-		g.paths = append(g.paths, fields[1])
+			g.paths = append(g.paths, path)
+
+			return nil
+		})
+	if err != nil {
+		t.Fatal(err)
 	}
 
 	return groups
@@ -131,147 +118,26 @@ func TestRunReportEscapesPaths(t *testing.T) {
 }
 
 func TestReportStdoutFailsWhileReading(t *testing.T) {
-	t.Parallel()
+	// Each row holds two paths longer than dir, so the report is more
-
+	// than twice the stdout buffer and stdout fails while rows are
-	// Every two records are a duplicate pair of a size of their own, and
+	// still being read, not at the final flush.
-	// a pair's row is written once a smaller size is read. Each row
-	// holds two paths longer than dir, so the rows come to more than
-	// twice the stdout buffer, and stdout fails while records are still
-	// being read, not once every record is.
 	dir := "/" + strings.Repeat("d", 4096)
 
-	recs := make([]scanRec, 2*ioBufSize/len(dir))
+	recs := make([]scanRec, ioBufSize/len(dir))
 	for i := range recs {
 		recs[i] = scanRec{
-			size: int64(i/2) + 1, head: "h", tail: "t", content: "c",
+			size: 1, head: "h", tail: "t", content: "c",
 			path: fmt.Sprintf("%s/%d", dir, i),
 		}
 	}
 
-	db := openTestDB(t)
+	t.Setenv(databaseEnv, seedDatabase(t, recs))
 
-	err := applyChanges(t.Context(), db, recs, nil, nil)
+	err := runReport(t.Context(), failingWriter{})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// Counts the records read, and draws nothing.
-	prog := &progress{now: time.Now}
-
-	_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
 	if !errors.Is(err, errWriteFailed) ||
 		!strings.HasPrefix(err.Error(), "write stdout: ") {
 		t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
 	}
-
-	if prog.count >= int64(len(recs)) {
-		t.Errorf("stdout failed after all %d records were read, want it "+
-			"to stop the read", prog.count)
-	}
-}
-
-// reportOpenLine is the plain line of report's open phase when it opens
-// the database at once; it prints it as the phase starts and as it ends.
-const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
-	"eta unknown"
-
-// smokeReport is report's stdout for the smoke-test records.
-const smokeReport = "first\tdupe\tsize\n" +
-	"/d/t1/f1\t/d/t2/f1\t3000\n" +
-	"/d/t1/f1\t/d/t3/f1\t3000\n" +
-	"/d/t1/sub/f2\t/d/t2/sub/f2\t100\n" +
-	"/d/t1/sub/f2\t/d/t3/sub/f2renamed\t100\n"
-
-// TestRunReportProgressNamesEachPhase runs report on the smoke-test
-// records with stderr not a terminal, and checks every line it prints
-// there. After the line naming the database, each phase prints a line
-// naming it with its place among the two as it starts and another as
-// it ends; read counts the 6 records before it reads them. The summary
-// comes last, and stdout holds the report alone.
-func TestRunReportProgressNamesEachPhase(t *testing.T) {
-	path := seedDatabase(t, smokeTreeRecs())
-	t.Setenv(databaseEnv, path)
-
-	stderr := captureStderr(t)
-
-	var stdout bytes.Buffer
-
-	code := run([]string{cmdReport}, &stdout, os.Stderr)
-	if code != exitOK {
-		t.Fatalf("run(report) = %d, want %d; stderr: %s",
-			code, exitOK, stderr())
-	}
-
-	if got := stdout.String(); got != smokeReport {
-		t.Errorf("stdout = %q, want %q", got, smokeReport)
-	}
-
-	// How each line starts; the rate at the end of a phase varies.
-	want := []string{
-		"report: database " + path + " (from " + databaseEnv + ")",
-		reportOpenLine,
-		reportOpenLine,
-		"[2/2] read: counting the records, elapsed 0s, eta unknown",
-		"[2/2] read: [6/6] 100%, ",
-		"report: 6 records read, 2 duplicate groups, 4 dupe files, " +
-			"6.1 KiB reclaimable",
-	}
-
-	lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
-	if len(lines) != len(want) {
-		t.Fatalf("stderr lines = %q, want %d lines starting %q",
-			lines, len(want), want)
-	}
-
-	for i := range want {
-		if !strings.HasPrefix(lines[i], want[i]) {
-			t.Errorf("stderr line %d = %q, want it to start %q",
-				i+1, lines[i], want[i])
-		}
-	}
-}
-
-// TestReadReportOnTerminal writes the smoke-test report to the terminal
-// its read phase's display is drawn on, as a plain "sfdupes report"
-// does: the terminal must show the header and every row whole, each on
-// a line of its own.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestReadReportOnTerminal(t *testing.T) {
-	stderr := captureStderr(t)
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, smokeTreeRecs(), nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// No timed draws: the display is drawn as it starts and as it ends.
-	p := startProgress(progressParams{
-		label: reportPhase("read"), unit: "records", total: -1,
-		waitingFor: countingRecords,
-	}, terminalWidth, time.Hour, time.Now)
-
-	_, err = readReport(t.Context(), db, "", os.Stderr, p)
-
-	p.finish()
-
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// A terminal shows each line as the text after its last carriage
-	// return.
-	var shown []string
-
-	for line := range strings.SplitSeq(stderr(), "\n") {
-		shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
-	}
-
-	want := strings.Split(strings.TrimSuffix(smokeReport, "\n"), "\n")
-	if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
-		t.Errorf("terminal shows lines %q, want %q first", shown, want)
-	}
 }
 
 func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
@@ -488,37 +354,6 @@ func TestDupeGroupsTieBreak(t *testing.T) {
 	}
 }
 
-func TestDupeGroupsOrderedByFirstPath(t *testing.T) {
-	t.Parallel()
-
-	// report reads the groups of size 50 in signature order, here /a,
-	// /z, /k, which is neither the order of their first paths nor its
-	// reverse, and must write them ordered by first path. The group of
-	// size 7 comes after them all, though its first path sorts first.
-	recs := []scanRec{
-		{size: 50, head: "a", tail: "a", content: "a", path: "/k/2"},
-		{size: 50, head: "a", tail: "a", content: "a", path: "/k/1"},
-		{size: 50, head: "b", tail: "b", content: "b", path: "/z/1"},
-		{size: 50, head: "b", tail: "b", content: "b", path: "/z/2"},
-		{size: 50, head: "c", tail: "c", content: "c", path: "/a/2"},
-		{size: 50, head: "c", tail: "c", content: "c", path: "/a/1"},
-		{size: 7, head: "d", tail: "d", content: "d", path: "/0/2"},
-		{size: 7, head: "d", tail: "d", content: "d", path: "/0/1"},
-	}
-
-	groups := dupeGroupsOf(t, recs)
-
-	got := make([]string, 0, len(groups))
-	for _, g := range groups {
-		got = append(got, g.paths[0])
-	}
-
-	want := []string{"/a/1", "/k/1", "/z/1", "/0/1"}
-	if !slices.Equal(got, want) {
-		t.Errorf("groups' first paths = %q, want %q", got, want)
-	}
-}
-
 func TestDupeGroupsDeterministic(t *testing.T) {
 	t.Parallel()
 

scan.go

@@ -13,7 +13,6 @@ import (
 	"os"
 	"os/signal"
 	"path/filepath"
-	"runtime"
 	"slices"
 	"strings"
 	"sync"
@@ -85,43 +84,27 @@ type fileMeta struct {
 	hashed bool
 }
 
-// scanUnit is what every scan phase counts, as its progress display
+// runScan implements the scan subcommand: four sequential phases —
-// names it.
+// walk (which stats each file as it is discovered), hash, update,
-const scanUnit = "files"
+// content — that synchronize the persistent database with the
-
+// filesystem state under the PATH operands. Only files whose size at
-// scanPhase names one phase of a scan with its place among all of them,
+// least one other file shares are ever hashed: a size-unique file
-// as "[2/5] walk", for the phase's progress display. The list here is
+// cannot be a duplicate. A file of headTailMin or more gets its content
-// the one place the phases are counted.
+// hash only when its size, head, and tail match another file's. Flag
-func scanPhase(name string) string {
+// parsing and the at-least-one-operand check are done by cobra. Before
-	return phaseLabel(
+// anything else, the scan names the database on stderr, saying whether
-		[]string{"load", "walk", "hash", "update", "content"}, name)
+// it is new. The scan holds the lock on the database for its whole
-}
+// run, so a second scan fails before it walks the filesystem or opens
-
+// the database.
-// runScan implements the scan subcommand: five sequential phases —
-// load (which reads the existing records), walk (which stats each file
-// as it is discovered), hash, update, content — that synchronize the
-// persistent database with the filesystem state under the PATH
-// operands. Only files whose size at least one other file shares are
-// ever hashed: a size-unique file cannot be a duplicate. A file of
-// headTailMin or more gets its content hash only when its size, head,
-// and tail match another file's. Flag parsing and the
-// at-least-one-operand check are done by cobra. Before anything else,
-// the scan names the database on stderr, saying whether it is new. The
-// scan holds the lock on the database for its whole run, so a second
-// scan fails before it walks the filesystem or opens the database.
 // Errors are returned rather than exiting, so that the deferred close —
 // which takes the database out of WAL mode — always runs, and the lock
 // is released after it. When ctx is cancelled, as by the SIGINT or
 // SIGTERM that interruptContext catches, the scan keeps what it has
 // hashed (see syncScan), prints how many files its walk reached, and
 // returns errInterrupted. workers must be at least 1; the scan command
-// rejects anything less. Before the lock is taken, the scan lists the
+// rejects anything less.
-// filesystems that are not persistent, on which the walk records
-// nothing; unless noDefaultExcludes is set, the walk does not enter the
-// default excludes either. Once the database is open, the scan notes
-// its files, which the walk passes over.
 func runScan(ctx context.Context, roots []string, workers int,
-	oneFS, noDefaultExcludes bool,
+	oneFS bool,
 ) error {
 	dbPath, source, err := databasePath()
 	if err != nil {
@@ -144,16 +127,6 @@ func runScan(ctx context.Context, roots []string, workers int,
 		return err
 	}
 
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		return fmt.Errorf("list the mounted filesystems: %w", err)
-	}
-
-	rules := walkRules{nonPersistent: nonPersistent}
-	if !noDefaultExcludes {
-		rules.excludes = defaultExcludes(runtime.GOOS)
-	}
-
 	lock, err := lockScanDatabase(dbPath)
 	if err != nil {
 		return err
@@ -174,12 +147,7 @@ func runScan(ctx context.Context, roots []string, workers int,
 
 	defer closeScanDatabase(ctx, db, dbPath)
 
-	rules.databaseFiles, err = databaseFiles(dbPath)
+	st, err := syncScan(ctx, db, roots, workers, oneFS)
-	if err != nil {
-		return err
-	}
-
-	st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
 	if errors.Is(err, context.Canceled) {
 		return interrupted(st.walked)
 	}
@@ -251,12 +219,14 @@ func resolveRoots(roots []string) ([]string, error) {
 }
 
 // pruneRoots drops operands already covered by another operand:
-// duplicates and any operand lying under another one. An operand under
+// duplicates and any operand lying under another one. With -x, an
-// another one is kept, though, when the walk of the deepest kept operand
+// operand under another one is kept when the walk of the deepest kept
-// above it does not reach it (see walkReaches). Every remaining file is
+// operand above it cannot reach it: when device puts any directory from
-// then reachable through exactly one root, so walked paths are unique
+// it (from its directory, for a regular file) up to but not including
-// without keeping a scan-wide set of every path seen.
+// that operand on a different filesystem than that operand. Every
-func pruneRoots(roots []string, oneFS bool, rules walkRules,
+// remaining file is then reachable through exactly one root, so walked
+// paths are unique without keeping a scan-wide set of every path seen.
+func pruneRoots(roots []string, oneFS bool,
 	device func(path string) (uint64, bool),
 ) []string {
 	// Shorter-first ordering guarantees an ancestor is kept before any
@@ -280,28 +250,20 @@ func pruneRoots(roots []string, oneFS bool, rules walkRules,
 			continue
 		}
 
-		// A duplicate is dropped, and so is an operand the walk of above
+		// Without -x, and always for a duplicate, r is dropped.
-		// reaches.
+		if !oneFS || above == r {
-		if above != r && !walkReaches(above, r, oneFS, rules, device) {
+			continue
-			kept = append(kept, r)
-		}
 		}
 
-	return kept
+		aboveDev, aboveOK := device(above)
-}
+		if !aboveOK {
-
+			continue
-// walkReaches reports whether the walk of operand above reaches r, an
+		}
-// operand under it. That walk does not enter a default exclude in
-// rules, nor a directory that device puts on a filesystem in rules that
-// is not persistent, nor, with -x, one that device puts on a different
-// filesystem than above, when device knows above's. It reaches every
-// regular file in a directory it enters, so for a regular-file operand
-// the directories checked start at its directory; they end below above.
-func walkReaches(above, r string, oneFS bool, rules walkRules,
-	device func(path string) (uint64, bool),
-) bool {
-	aboveDev, aboveDevOK := device(above)
 
+		// The walk of above enters a directory only when it is on
+		// above's filesystem, and then reaches every regular file in it
+		// whatever device the file reports, so for a regular-file
+		// operand the directories checked start at its directory.
 		dir := r
 
 		fi, err := os.Lstat(r)
@@ -310,25 +272,16 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
 		}
 
 		for ; dir != above; dir = filepath.Dir(dir) {
-		if isDefaultExclude(dir, rules.excludes) {
-			return false
-		}
-
 			dev, ok := device(dir)
-		if !ok {
+			if ok && dev != aboveDev {
-			continue
+				kept = append(kept, r)
-		}
 
-		if _, found := rules.nonPersistent[dev]; found {
+				break
-			return false
 			}
-
-		if oneFS && aboveDevOK && dev != aboveDev {
-			return false
 		}
 	}
 
-	return true
+	return kept
 }
 
 // deviceOfPath returns the filesystem device ID of the file at path,
@@ -372,11 +325,11 @@ type scanState struct {
 // the hashed records still waiting in the batch, starts no other write
 // or deletion, and returns the cancellation.
 func syncScan(ctx context.Context, db *sql.DB, roots []string,
-	workers int, oneFS bool, rules walkRules,
+	workers int, oneFS bool,
 ) (scanStats, error) {
 	s := &scanState{db: db}
 
-	err := s.runPhases(ctx, roots, workers, oneFS, rules)
+	err := s.runPhases(ctx, roots, workers, oneFS)
 	if err == nil || ctx.Err() == nil {
 		return s.st, err
 	}
@@ -391,27 +344,24 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
 }
 
 // runPhases synchronizes the database with the filesystem under roots
-// in five sequential phases: load (read the existing records), walk
+// in four sequential phases: walk (enumerate and stat every file,
-// (enumerate and stat every file, building a complete size census),
+// building a complete size census), hash (read only the new or
-// hash (read only the new or changed — or previously unhashed — files
+// changed — or previously unhashed — files whose size at least one
-// whose size at least one other file shares, committing results in
+// other file shares, committing results in batches as they arrive),
-// batches as they arrive), update (record the size-unique files without
+// update (record the size-unique files without reading them, and
-// reading them, and delete the records the scan no longer verifies),
+// delete the records the scan no longer verifies), and content (fill
-// and content (fill in the content hash of every record of headTailMin
+// in the content hash of every record of headTailMin or more whose
-// or more whose size, head, and tail match another record's). Records
+// size, head, and tail match another record's). Records outside the
-// outside the roots are never touched, except that the content phase
+// roots are never touched, except that the content phase fills in
-// fills in their content hash. Operands the walk cannot start from, or
+// their content hash. Operands the walk cannot start from are dropped
-// that are not on a persistent filesystem, are dropped first, so the
+// first, so the records beneath them are treated as if those operands
-// records beneath them are treated as if those operands had not been
+// had not been given.
-// given. Below an operand, the walk follows rules.
 func (s *scanState) runPhases(ctx context.Context, roots []string,
-	workers int, oneFS bool, rules walkRules,
+	workers int, oneFS bool,
 ) error {
-	// Types and filesystems are checked before pruning so that an
+	// Types are checked before pruning so that an operand under a
-	// operand under a dropped one is still scanned, not dropped as lying
+	// dropped one is still scanned, not dropped as lying under it.
-	// under it.
+	roots = pruneRoots(s.walkableRoots(roots), oneFS, deviceOfPath)
-	roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
-		deviceOfPath)
 
 	err := s.loadIndex(ctx, roots)
 	if err != nil {
@@ -419,7 +369,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 	}
 
 	changed, unhashed := s.walkPhase(
-		startWalk(ctx, roots, oneFS, rules, workers), roots)
+		startWalk(ctx, roots, oneFS, workers), roots)
 
 	// A cancelled walk stops early, so its size census covers only part
 	// of the roots, and every file it never reached would look vanished
@@ -441,7 +391,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 		return err
 	}
 
-	return s.contentPhase(ctx, workers, rules)
+	return s.contentPhase(ctx, workers)
 }
 
 // walkableRoots returns the operands the walk can start from: regular
@@ -449,11 +399,9 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 // about, counted as skipped, and dropped. A dropped operand is no
 // longer a root, so the records stored beneath it are treated as if it
 // had not been given: those under no other root count as outside the
-// roots and are not deleted as unverified. An operand on a filesystem
+// roots and are not deleted as unverified. An operand that fails lstat
-// in rules that is not persistent is dropped the same way, with a line
+// here is kept, and the walk warns about it.
-// saying so that is not a warning, and is not counted as skipped. An
+func (s *scanState) walkableRoots(roots []string) []string {
-// operand that fails lstat here is kept, and the walk warns about it.
-func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
 	kept := make([]string, 0, len(roots))
 
 	for _, root := range roots {
@@ -467,13 +415,6 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
 
 				continue
 			}
-
-			line := rules.nonPersistentLine(root, fi)
-			if line != "" {
-				fmt.Fprintln(os.Stderr, escapePath(line))
-
-				continue
-			}
 		}
 
 		kept = append(kept, root)
@@ -489,21 +430,10 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
 // separately into the same database.
 func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 	// Indexing tens of millions of records takes real time; without a
-	// display the scan looks hung before the walk begins. Counting them
+	// display the scan looks hung before the walk begins.
-	// first gives the display its total.
+	prog := newProgress("load", -1)
-	prog := newProgress(progressParams{
-		label: scanPhase("load"), unit: scanUnit, total: -1,
-		waitingFor: countingRecords, stalledOn: "db",
-	})
 	defer prog.finish()
 
-	total, err := countRecords(ctx, s.db)
-	if err != nil {
-		return err
-	}
-
-	prog.setTotal(total)
-
 	s.existing = make(map[string]fileMeta)
 
 	return loadFileMeta(ctx, s.db,
@@ -528,48 +458,23 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 // (the content phase may still fill in its content hash). It returns
 // the new-or-changed files and the unchanged files whose records lack
 // hashes; both remain candidates until the census decides whether
-// their sizes are shared. A record at or below a path the walk left
+// their sizes are shared. A record below a directory -x did not enter
-// out, because -x stopped at it, it is a default exclude, or it is not
+// counts as outside the roots: it is not deleted, and its size joins
-// on a persistent filesystem, counts as outside the roots: it is not
+// the census. That does not hold below a root at or inside that
-// deleted, and its size joins the census. That does not hold below a
+// directory, since that root's own walk verifies the records below it.
-// root at or inside that path, since that root's own walk verifies the
-// records below it; nor is a line printed for such a root.
 func (s *scanState) walkPhase(
 	events <-chan walkEvent, roots []string,
 ) ([]fileRec, []fileRec) {
-	// The records the last scan left under the roots estimate how many
+	prog := newProgress("walk", -1)
-	// files the walk will find. A first scan has none, and no estimate.
-	estimate := int64(len(s.existing))
-	if estimate == 0 {
-		estimate = -1
-	}
-
-	prog := newProgress(progressParams{
-		label: scanPhase("walk"), unit: scanUnit, total: estimate,
-		estimate: true, stalledOn: "disk",
-	})
 
 	var (
 		changed, unhashed []fileRec
-		leftOut           []string
+		otherFSDirs       []string
 	)
 
 	for ev := range events {
 		if ev.otherFSDir != "" {
-			leftOut = append(leftOut, ev.otherFSDir)
+			otherFSDirs = append(otherFSDirs, ev.otherFSDir)
-
-			continue
-		}
-
-		// The line saying so is printed as a warning is, but what the
-		// walk leaves out is not counted as skipped. A default exclude
-		// that is itself a root is walked as one, so it gets no line.
-		if ev.leftOut != "" {
-			leftOut = append(leftOut, ev.leftOut)
-
-			if !slices.Contains(roots, ev.leftOut) {
-				prog.warnf("%s", ev.warn)
-			}
 
 			continue
 		}
@@ -604,13 +509,12 @@ func (s *scanState) walkPhase(
 	prog.finish()
 
 	for path, old := range s.existing {
-		// Kept only when a path the walk left out lies below the deepest
+		// Kept only when a directory -x did not enter lies below the
-		// root above the record, or is the record's own. A root at or
+		// deepest root above the record. A root at or below that
-		// below that path was walked on its own and did not find the
+		// directory was walked on its own and did not find the file.
-		// file. Both contain the record, so the longer path is the
+		// Both contain the record, so the longer path is the deeper.
-		// deeper.
+		otherFSDir := deepestRoot(path, otherFSDirs)
-		left := deepestRoot(path, leftOut)
+		if len(otherFSDir) > len(deepestRoot(path, roots)) {
-		if len(left) > len(deepestRoot(path, roots)) {
 			delete(s.existing, path)
 
 			s.sizes = append(s.sizes, old.size)
@@ -716,15 +620,6 @@ func sameInode(a, b fileRec) bool {
 	return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
 }
 
-// isDatabaseFile reports whether f, a file the walk reached, is one of
-// the scan's own database files in rules, which the walk passes over
-// without a line and without counting it.
-func isDatabaseFile(f fileRec, rules walkRules) bool {
-	return slices.ContainsFunc(rules.databaseFiles, func(d fileRec) bool {
-		return sameInode(d, f)
-	})
-}
-
 // hashPhase hashes every queued file with hashSignature — the head and
 // tail of a file of headTailMin or more, the whole file below that —
 // committing completed records to the database in batches as results
@@ -741,11 +636,10 @@ func (s *scanState) hashPhase(ctx context.Context, workers int) error {
 }
 
 // readRuns reads runs with the worker pool, one read per inode run, in
-// the order given, under the progress display of the phase named label.
+// the order given, under a progress display named label. The workers
-// The workers compute each run's hashes with hash, and each result goes
+// compute each run's hashes with hash, and each result goes to record;
-// to record; a run that fails to read is warned about and counted as
+// a run that fails to read is warned about and counted as skipped
-// skipped instead. The total counts actual reads, so the display shows
+// instead. The total counts actual reads, so the bar shows a real ETA.
-// a real ETA.
 //
 // Returning early — a failed database write, or a cancelled scan — must
 // not strand the pool: the feeder would park forever on a full jobs
@@ -759,12 +653,7 @@ func (s *scanState) readRuns(ctx context.Context, workers int,
 	pool := startHashPool(ctx, runs, workers, hash)
 	defer pool.stop()
 
-	// Between results the phase may wait on a file being read or on a
+	prog := newProgress(label, int64(len(runs)))
-	// batch of records being written.
-	prog := newProgress(progressParams{
-		label: scanPhase(label), unit: scanUnit, total: int64(len(runs)),
-		stalledOn: "disk/db",
-	})
 	defer prog.finish()
 
 	for range runs {
@@ -851,10 +740,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 	s.st.removed = len(deletes)
 
 	total := len(s.batch) + len(s.sentinels) + len(deletes)
-	prog := newProgress(progressParams{
+	prog := newProgress("update", int64(total))
-		label: scanPhase("update"), unit: scanUnit, total: int64(total),
-		stalledOn: "db",
-	})
 
 	defer prog.finish()
 
@@ -893,12 +779,9 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 // hash phase's worker pool and their records written back in batches. A
 // failed read is warned about and counted as skipped; the record keeps
 // its empty content, so it is never grouped, and a later scan tries
-// again. No file on a filesystem in rules that is not persistent is
+// again.
-// read.
+func (s *scanState) contentPhase(ctx context.Context, workers int) error {
-func (s *scanState) contentPhase(ctx context.Context, workers int,
+	toRead, recs, err := s.contentCandidates(ctx)
-	rules walkRules,
-) error {
-	toRead, recs, err := s.contentCandidates(ctx, rules)
 	if err != nil {
 		return err
 	}
@@ -919,53 +802,31 @@ func (s *scanState) contentPhase(ctx context.Context, workers int,
 		return err
 	}
 
-	// The last partial batch is written under a display of its own, as
+	return applyChanges(ctx, s.db, s.batch, nil, nil)
-	// the update phase writes the hash phase's.
-	prog := newProgress(progressParams{
-		label: scanPhase("content"), unit: scanUnit,
-		total: int64(len(s.batch)), stalledOn: "db",
-	})
-	defer prog.finish()
-
-	return applyChanges(ctx, s.db, s.batch, nil, prog)
 }
 
 // contentCandidates returns the files the content phase reads, and
 // their records by path. Every record contentCandidatesSQL returns has
 // its file checked with lstat, whether or not it already has a content
-// hash: a file that is gone, is no longer a regular file, has changed
+// hash: a file that is gone, is no longer a regular file, or has
-// by the walk's rule, or is on a filesystem in rules that is not
+// changed by the walk's rule keeps its record as it is and does not
-// persistent keeps its record as it is and does not count as a match
+// count as a match for the others, and any other lstat error is warned
-// for the others, and any other lstat error is warned about and counted
+// about and counted as skipped, with the same result. If such a record
-// as skipped, with the same result. If such a record has no content
+// has no content hash, it stays out of duplicate groups; if it has one,
-// hash, it stays out of duplicate groups; if it has one, it is still
+// it is still reported until a scan covering its own tree updates or
-// reported until a scan covering its own tree updates or removes it.
+// removes it. The files of a group that pass and have no content hash
-// The files of a group that pass and have no content hash are read only
+// are read only if at least minGroupSize of the group's files pass, so
-// if at least minGroupSize of the group's files pass, so a group whose
+// a group whose other members are all stale costs no reads. Only the
-// other members are all stale costs no reads. Only the records to be
+// records to be read are kept.
-// read are kept.
 func (s *scanState) contentCandidates(
-	ctx context.Context, rules walkRules,
+	ctx context.Context,
 ) ([]fileRec, map[string]scanRec, error) {
-	// The search and the checks take real time on a large database;
+	// The query and the checks take real time on a large database;
 	// without a display the scan looks hung before the reads begin.
-	// The display says it is searching until SQLite returns the first
+	prog := newProgress("content", -1)
-	// record. The checks start there, against a count of the records
-	// the search returns, and so do their rate and ETA. Between records
-	// they may wait on a file's lstat or on the search.
-	prog := newProgress(progressParams{
-		label: scanPhase("content"), unit: scanUnit, total: -1,
-		waitingFor: "searching the database", stalledOn: "disk/db",
-	})
 	defer prog.finish()
 
-	total, err := countContentCandidates(ctx, s.db)
-	if err != nil {
-		return nil, nil, err
-	}
-
 	var (
-		searching = true // no record has been returned yet
 		toRead []fileRec
 		first  scanRec   // the current group's first record
 		passed int       // the current group's files that passed the check
@@ -988,13 +849,7 @@ func (s *scanState) contentCandidates(
 		passed, unread = 0, nil
 	}
 
-	err = loadContentCandidates(ctx, s.db, func(r scanRec, hashed bool) {
+	err := loadContentCandidates(ctx, s.db, func(r scanRec, hashed bool) {
-		if searching {
-			searching = false
-
-			prog.setTotal(total)
-		}
-
 		prog.increment()
 
 		if r.size != first.size || r.head != first.head || r.tail != first.tail {
@@ -1003,7 +858,7 @@ func (s *scanState) contentCandidates(
 			first = r
 		}
 
-		f, ok, err := unchangedFile(r, rules)
+		f, ok, err := unchangedFile(r)
 		if err != nil {
 			s.st.skipped++
 
@@ -1025,11 +880,6 @@ func (s *scanState) contentCandidates(
 		return nil, nil, err
 	}
 
-	// A search that returned no record still ends showing its count.
-	if searching {
-		prog.setTotal(total)
-	}
-
 	endGroup()
 
 	return toRead, recs, nil
@@ -1037,10 +887,9 @@ func (s *scanState) contentCandidates(
 
 // unchangedFile lstats the file r names and returns it for reading if
 // it is still the regular file r records: the same size, and an mtime
-// no newer than recorded (the walk's change rule). A file that is gone,
+// no newer than recorded (the walk's change rule). A file that is gone
-// has changed, or is on a filesystem in rules that is not persistent
+// or has changed reports false; any other lstat error is returned.
-// reports false; any other lstat error is returned.
+func unchangedFile(r scanRec) (fileRec, bool, error) {
-func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
 	fi, err := os.Lstat(r.path)
 	if errors.Is(err, fs.ErrNotExist) {
 		return fileRec{}, false, nil
@@ -1057,12 +906,6 @@ func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
 
 	dev, ino := inodeOfInfo(fi)
 
-	// The scan reads nothing on a filesystem that is not persistent,
-	// whatever record an earlier scan left for the file.
-	if _, found := rules.nonPersistent[dev]; found {
-		return fileRec{}, false, nil
-	}
-
 	return fileRec{
 		path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
 	}, true, nil
@@ -1129,36 +972,32 @@ type dirJob struct {
 }
 
 // walkEvent is one walk result delivered to the walk phase: a regular
-// file's statted record, a warning when fail is set, or a path the walk
+// file's statted record, a warning when fail is set, or, when
-// left out: when otherFSDir is set, a directory -x did not enter
+// otherFSDir is set, a directory -x did not enter because it is on
-// because it is on another filesystem than its operand, and when
+// another filesystem than its operand.
-// leftOut is set, a default exclude or a directory or file on a
-// filesystem that is not persistent, with warn holding the line that
-// says so.
 type walkEvent struct {
 	rec        fileRec
 	warn       string
 	fail       bool
 	otherFSDir string
-	leftOut    string
 }
 
 // startWalk seeds every root into the shared walk worker pool and
 // returns the event stream: one record per regular file, one warning
-// event per per-path error, and one event per path -x or rules kept the
+// event per per-path error, and one event per directory -x did not
-// walk from entering or recording. The channel is closed when the walk
+// enter. The channel is closed when the walk completes, and also when
-// completes, and also when ctx is cancelled — every goroutine in the
+// ctx is cancelled — every goroutine in the pool abandons its blocking
-// pool abandons its blocking send in that case, so the consumer sees a
+// send in that case, so the consumer sees a truncated but properly
-// truncated but properly terminated stream instead of a stalled one.
+// terminated stream instead of a stalled one.
 func startWalk(ctx context.Context, roots []string, oneFS bool,
-	rules walkRules, workers int,
+	workers int,
 ) <-chan walkEvent {
-	jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS, rules)
+	jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS)
 
 	go func() {
 		initial := make([]dirJob, 0, len(roots))
 		for _, root := range roots {
-			initial = append(initial, seedRoot(ctx, root, rules, events)...)
+			initial = append(initial, seedRoot(ctx, root, events)...)
 		}
 
 		dispatchDirs(ctx, initial, jobs, subdirs)
@@ -1212,14 +1051,13 @@ func operandWarning(root string, fi fs.FileInfo) string {
 }
 
 // seedRoot turns one PATH operand into the walk's starting state: a
-// regular-file operand is statted and emitted directly, unless it is
+// regular-file operand is statted and emitted directly, and a directory
-// one of the scan's own database files in rules, and a directory
 // operand becomes an initial job. walkableRoots has already dropped
 // every other operand. One that has changed into something else since
 // is warned about and skipped here; it is still a root, so the records
 // stored beneath it, other than those under another root, are deleted
 // as unverified.
-func seedRoot(ctx context.Context, root string, rules walkRules,
+func seedRoot(ctx context.Context, root string,
 	events chan<- walkEvent,
 ) []dirJob {
 	fi, err := os.Lstat(root)
@@ -1246,17 +1084,14 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
 	}
 
 	dev, ino := inodeOfInfo(fi)
-	rec := fileRec{
+
+	sendEvent(ctx, events, walkEvent{rec: fileRec{
 		path:  root,
 		size:  fi.Size(),
 		mtime: fi.ModTime(),
 		dev:   dev,
 		ino:   ino,
-	}
+	}})
-
-	if !isDatabaseFile(rec, rules) {
-		sendEvent(ctx, events, walkEvent{rec: rec})
-	}
 
 	return nil
 }
@@ -1268,7 +1103,7 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
 // workers drop the directories still queued rather than stop reading
 // jobs, so the range always runs out and the pool always tears down.
 func startWalkWorkers(ctx context.Context, workers int,
-	oneFS bool, rules walkRules,
+	oneFS bool,
 ) (chan dirJob, chan []dirJob, chan walkEvent) {
 	jobs := make(chan dirJob, workQueueDepth)
 	subdirs := make(chan []dirJob, workers)
@@ -1284,7 +1119,7 @@ func startWalkWorkers(ctx context.Context, workers int,
 				}
 
 				select {
-				case subdirs <- walkOneDir(ctx, job, oneFS, rules, events):
+				case subdirs <- walkOneDir(ctx, job, oneFS, events):
 				case <-ctx.Done():
 				}
 			}
@@ -1342,7 +1177,7 @@ func dispatchDirs(ctx context.Context, initial []dirJob,
 // entry and a warning event per unreadable one, and returns the
 // subdirectories to descend into.
 func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
-	rules walkRules, events chan<- walkEvent,
+	events chan<- walkEvent,
 ) []dirJob {
 	entries, err := os.ReadDir(job.path)
 	if err != nil {
@@ -1366,8 +1201,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 		p := filepath.Join(job.path, e.Name())
 
 		if e.IsDir() {
-			sub, ok := subdirJob(ctx, p, e, job, oneFS, rules, events)
+			if sub, ok := subdirJob(ctx, p, e, job, oneFS, events); ok {
-			if ok {
 				subs = append(subs, sub)
 			}
 
@@ -1379,7 +1213,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 			continue
 		}
 
-		emitFile(ctx, p, e, rules, events)
+		emitFile(ctx, p, e, events)
 	}
 
 	return subs
@@ -1389,12 +1223,9 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 // The lstat happens here in the walk worker, while the directory's
 // metadata is still hot; a path that fails to stat (or stops being a
 // regular file) between the directory read and the lstat is warned
-// about and skipped. A file on a filesystem in rules that is not
+// about and skipped.
-// persistent, as through a bind mount of one file, is not recorded: it
-// gets an event naming it instead. One of the scan's own database files
-// in rules gets no event at all.
 func emitFile(ctx context.Context, p string, e fs.DirEntry,
-	rules walkRules, events chan<- walkEvent,
+	events chan<- walkEvent,
 ) {
 	info, err := e.Info()
 	if err != nil {
@@ -1410,43 +1241,35 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
 		return
 	}
 
-	line := rules.nonPersistentLine(p, info)
-	if line != "" {
-		sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
-
-		return
-	}
-
 	dev, ino := inodeOfInfo(info)
-	rec := fileRec{
+
+	sendEvent(ctx, events, walkEvent{rec: fileRec{
 		path:  p,
 		size:  info.Size(),
 		mtime: info.ModTime(),
 		dev:   dev,
 		ino:   ino,
-	}
+	}})
-
-	if !isDatabaseFile(rec, rules) {
-		sendEvent(ctx, events, walkEvent{rec: rec})
-	}
 }
 
 // subdirJob applies the descent rules to directory p: never enter
 // .zfs (ZFS snapshot pseudo-dirs would list every file once per
-// snapshot), never enter a directory on a filesystem in rules that is
+// snapshot), and with -x never enter a directory on a different
-// not persistent, nor a default exclude in rules, and with -x never
+// filesystem than its operand, sending an event that names it so the
-// enter a directory on a different filesystem than its operand. For the
+// scan keeps the records below it, except those below an operand at or
-// last three it sends an event that names the directory, so the scan
-// keeps the records below it, except those below an operand at or
 // inside it.
 func subdirJob(ctx context.Context, p string, e fs.DirEntry,
-	parent dirJob, oneFS bool, rules walkRules,
+	parent dirJob, oneFS bool, events chan<- walkEvent,
-	events chan<- walkEvent,
 ) (dirJob, bool) {
 	if e.Name() == ".zfs" {
 		return dirJob{}, false
 	}
 
+	job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
+	if !oneFS || !parent.rootDevOK {
+		return job, true
+	}
+
 	info, err := e.Info()
 	if err != nil {
 		sendEvent(ctx, events, walkEvent{
@@ -1457,27 +1280,6 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
 		return dirJob{}, false
 	}
 
-	line := rules.nonPersistentLine(p, info)
-	if line != "" {
-		sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
-
-		return dirJob{}, false
-	}
-
-	if isDefaultExclude(p, rules.excludes) {
-		sendEvent(ctx, events, walkEvent{
-			leftOut: p,
-			warn:    fmt.Sprintf("scan: skipped %s (default exclude)", p),
-		})
-
-		return dirJob{}, false
-	}
-
-	job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
-	if !oneFS || !parent.rootDevOK {
-		return job, true
-	}
-
 	if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
 		sendEvent(ctx, events, walkEvent{otherFSDir: p})
 

scan_test.go

@@ -782,8 +782,7 @@ func collectWalk(t *testing.T, roots []string, oneFS bool,
 		errs int
 	)
 
-	for ev := range startWalk(t.Context(), roots, oneFS, walkRules{},
+	for ev := range startWalk(t.Context(), roots, oneFS, workers) {
-		workers) {
 		if ev.fail {
 			errs++
 
@@ -1005,8 +1004,7 @@ func callSubdirJob(t *testing.T, p string, e fs.DirEntry,
 	t.Helper()
 
 	events := make(chan walkEvent, 1)
-	job, ok := subdirJob(t.Context(), p, e, parent, oneFS, walkRules{},
+	job, ok := subdirJob(t.Context(), p, e, parent, oneFS, events)
-		events)
 	close(events)
 
 	var evs []walkEvent
@@ -1154,7 +1152,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
 		return dev, ok
 	}
 
-	roots = pruneRoots(roots, true, walkRules{}, device)
+	roots = pruneRoots(roots, true, device)
 
 	ctx := t.Context()
 	s := &scanState{db: db}
@@ -1164,14 +1162,14 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
 		t.Fatal(err)
 	}
 
-	jobs, subdirs, events := startWalkWorkers(ctx, 2, true, walkRules{})
+	jobs, subdirs, events := startWalkWorkers(ctx, 2, true)
 
 	// seedRoot emits a regular-file operand itself and returns a job
 	// only for a directory operand.
 	var initial []dirJob
 
 	for _, root := range roots {
-		for _, job := range seedRoot(ctx, root, walkRules{}, events) {
+		for _, job := range seedRoot(ctx, root, events) {
 			job.rootDev = devices[root]
 			initial = append(initial, job)
 		}
@@ -1465,7 +1463,7 @@ const smokeTreeFiles = 15
 func syncTree(t *testing.T, db *sql.DB, roots ...string) scanStats {
 	t.Helper()
 
-	st, err := syncScan(t.Context(), db, roots, 4, false, walkRules{})
+	st, err := syncScan(t.Context(), db, roots, 4, false)
 	if err != nil {
 		t.Fatalf("syncScan: %v", err)
 	}
@@ -1480,7 +1478,7 @@ func dbRecords(t *testing.T, db *sql.DB) []scanRec {
 
 	var recs []scanRec
 
-	err := loadFileRows(t.Context(), db, func(int64) {}, func(r scanRec) {
+	err := loadFileRows(t.Context(), db, func(r scanRec) {
 		recs = append(recs, r)
 	})
 	if err != nil {
@@ -1552,7 +1550,7 @@ func assertSmokeTreeGroups(t *testing.T, dir string, db *sql.DB) {
 
 	super, dirs := dbTree(t, db)
 
-	tg := collectTreeGroups(dirs, super, nil)
+	tg := collectTreeGroups(dirs, super)
 	if len(tg) != 1 {
 		t.Fatalf("len(tree groups) = %d, want 1", len(tg))
 	}
@@ -1588,69 +1586,6 @@ func TestScanPipeline(t *testing.T) {
 	assertSmokeTreeGroups(t, dir, db)
 }
 
-// TestScanProgressNamesEachPhase checks the plain progress lines of a
-// first scan, and of a rescan of one operand holding two of the
-// database's records. Each phase's first line names it with its place
-// among the five: load counts the records before reading them, the
-// content phase searches the database, and the walk estimates its total
-// from the records under its operand, or says its ETA is unknown when
-// there are none.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestScanProgressNamesEachPhase(t *testing.T) {
-	stderr := captureStderr(t)
-	dir := buildSmokeTree(t)
-	db := openTestDB(t)
-
-	syncTree(t, db, dir)
-
-	first := stderr()
-
-	syncTree(t, db, filepath.Join(dir, "a"))
-
-	rescan := stderr()[len(first):]
-	load := "[1/5] load: counting the records, elapsed 0s, eta unknown"
-	content := "[5/5] content: searching the database, elapsed 0s, eta unknown"
-	cases := map[string][]string{
-		first: {
-			load, "[2/5] walk: 0 files, 0 files/s, elapsed 0s, eta unknown",
-			"[3/5] hash: [0/", "[4/5] update: [0/", content,
-		},
-		rescan: {
-			load, "[2/5] walk: [0/~2] 0%, 0 files/s, elapsed 0s, eta unknown",
-			"[3/5] hash: [0/", "[4/5] update: [0/", content,
-		},
-	}
-
-	for out, want := range cases {
-		var got []string
-
-		for line := range strings.Lines(out) {
-			label, _, _ := strings.Cut(line, ":")
-			if len(got) == 0 || !strings.HasPrefix(got[len(got)-1], label) {
-				got = append(got, line)
-			}
-		}
-
-		if len(got) != len(want) {
-			t.Fatalf("first lines of the phases = %q, want %q", got, want)
-		}
-
-		for i := range want {
-			if !strings.HasPrefix(got[i], want[i]) {
-				t.Errorf("phase %d starts %q, want %q", i+1, got[i], want[i])
-			}
-		}
-	}
-
-	// The load phase's total is the count of the records.
-	loaded := fmt.Sprintf("\n[1/5] load: [%d/%d] 100%%, ",
-		smokeTreeFiles, smokeTreeFiles)
-	if !strings.Contains(rescan, loaded) {
-		t.Errorf("rescan stderr = %q, want %q", rescan, loaded)
-	}
-}
-
 func TestSyncScanUnchangedReuse(t *testing.T) {
 	t.Parallel()
 
@@ -2094,7 +2029,7 @@ func TestTreesUnhashedNeverEqual(t *testing.T) {
 	for name, unknown := range cases {
 		super, dirs := treeOf(t, append(slices.Clone(shared), unknown...))
 
-		if tg := collectTreeGroups(dirs, super, nil); len(tg) != 0 {
+		if tg := collectTreeGroups(dirs, super); len(tg) != 0 {
 			t.Errorf("%s: tree groups = %d, want 0 (the files may differ)",
 				name, len(tg))
 		}
@@ -2414,7 +2349,7 @@ func TestPruneRoots(t *testing.T) {
 	// Duplicates and operands under other operands are dropped; /cc is
 	// not under /c (sibling with a shared prefix).
 	got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
-		false, walkRules{}, device)
+		false, device)
 
 	want := []string{"/a", "/c", "/cc"}
 	if !slices.Equal(got, want) {
@@ -2423,7 +2358,7 @@ func TestPruneRoots(t *testing.T) {
 
 	nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
 
-	got = pruneRoots(nested, false, walkRules{}, device)
+	got = pruneRoots(nested, false, device)
 
 	want = []string{"/p"}
 	if !slices.Equal(got, want) {
@@ -2431,31 +2366,12 @@ func TestPruneRoots(t *testing.T) {
 	}
 
 	// With -x, /p/q, on another filesystem than /p, is kept as well.
-	got = pruneRoots(nested, true, walkRules{}, device)
+	got = pruneRoots(nested, true, device)
 
 	want = []string{"/p", pqRoot}
 	if !slices.Equal(got, want) {
 		t.Fatalf("pruneRoots with -x = %q, want %q", got, want)
 	}
-
-	// The walk of /p does not enter /p/q when it is a default exclude, so
-	// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
-	got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
-		device)
-	if !slices.Equal(got, want) {
-		t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
-	}
-
-	// Nor does it enter /p/q when the filesystem of /p/q is not
-	// persistent, so /p/q/r/m, of no known device, is kept.
-	got = pruneRoots([]string{"/p", "/p/q/r/m"}, false,
-		walkRules{nonPersistent: map[uint64]string{2: "tmpfs"}}, device)
-
-	want = []string{"/p", "/p/q/r/m"}
-	if !slices.Equal(got, want) {
-		t.Fatalf("pruneRoots with %s not persistent = %q, want %q",
-			pqRoot, got, want)
-	}
 }
 
 func TestReportsNeverTouchFilesystem(t *testing.T) {
@@ -2501,99 +2417,3 @@ func TestUnderRoot(t *testing.T) {
 		}
 	}
 }
-
-// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
-// database, reached directly and through a symlinked directory. The
-// walk passes over the database file, its -wal and -shm files and its
-// .lock file with no line and no count, and the record an earlier scan
-// left for the .lock file is deleted. A rescan of the unchanged tree
-// updates nothing, and report lists none of the database's files.
-func TestRunScanPassesOverDatabase(t *testing.T) {
-	for _, viaSymlink := range []bool{false, true} {
-		t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
-			base := t.TempDir()
-			tree := filepath.Join(base, "real", "tree")
-			operand := tree
-
-			writeFile(t, tree, "a", pattern(1, 100))
-			writeFile(t, tree, "b", pattern(1, 100))
-			writeFile(t, tree, "empty1", nil)
-			writeFile(t, tree, "empty2", nil)
-
-			if viaSymlink {
-				err := os.Symlink(filepath.Join(base, "real"),
-					filepath.Join(base, "link"))
-				if err != nil {
-					t.Fatal(err)
-				}
-
-				operand = filepath.Join(base, "link", "tree")
-			}
-
-			dbPath := filepath.Join(tree, "db.sqlite")
-			t.Setenv(databaseEnv, dbPath)
-
-			db, err := openScanDatabase(t.Context(), dbPath)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			err = applyChanges(t.Context(), db, []scanRec{
-				{path: filepath.Join(operand, "db.sqlite.lock")},
-			}, nil, nil)
-			if err != nil {
-				t.Fatal(err)
-			}
-
-			closeScanDatabase(t.Context(), db, dbPath)
-
-			assertScanSummary(t,
-				"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
-				operand)
-			assertScanSummary(t,
-				"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
-				operand)
-
-			report := runStdout(t, cmdReport)
-			if strings.Contains(report, "db.sqlite") ||
-				!strings.Contains(report, filepath.Join(operand, "empty2")) {
-				t.Errorf("report = %q, want the empty files and none of "+
-					"the database's files", report)
-			}
-		})
-	}
-}
-
-// TestRunScanPassesOverDatabaseOperands scans the database file and its
-// .lock file given as operands themselves: neither is recorded or
-// counted.
-func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
-	dbPath := testDBPath(t)
-	t.Setenv(databaseEnv, dbPath)
-
-	// The first scan creates both files.
-	scanOK(t, t.TempDir())
-
-	assertScanSummary(t,
-		"0 files seen (0 added, 0 updated, 0 removed, 0 unchanged)",
-		dbPath, dbPath+".lock")
-}
-
-// assertScanSummary scans operands and fails the test unless the
-// summary line gives counts with nothing skipped, and the line naming
-// the database is the only one that names one of its files.
-func assertScanSummary(t *testing.T, counts string, operands ...string) {
-	t.Helper()
-
-	stderr := scanOK(t, operands...)
-
-	want := "scan: " + counts + ", 0 skipped\n"
-	if !strings.Contains(stderr, want) {
-		t.Errorf("scan %q: stderr = %q, want %q", operands, stderr, want)
-	}
-
-	if strings.Count(stderr, "db.sqlite") != 1 {
-		t.Errorf("scan %q: stderr = %q, want only the line naming the "+
-			"database to name one of its files", operands, stderr)
-	}
-}

trees.go

@@ -1,9 +1,9 @@
 package main
 
 import (
+	"bufio"
 	"context"
 	"crypto/sha256"
-	"database/sql"
 	"fmt"
 	"io"
 	"os"
@@ -24,13 +24,6 @@ type treeNode struct {
 	totalSize int64
 }
 
-// treesPhase names one phase of trees with its place among all of them,
-// as "[2/4] read", for the phase's progress display. The list here is
-// the one place the phases are counted.
-func treesPhase(name string) string {
-	return phaseLabel([]string{openPhase, "read", "group", "write"}, name)
-}
-
 // runTrees implements the trees subcommand: it reads every record from
 // the database in path order, reconstructs the directory hierarchy from
 // the record paths, computes a Merkle-style digest per directory, and
@@ -38,13 +31,6 @@ func treesPhase(name string) string {
 // touches the scanned filesystem; its only I/O is the database, stdout,
 // and stderr. It names the database on stderr before it opens it. Any
 // database problem, including a missing database, is fatal.
-//
-// The work runs in four phases, each under a progress display of its
-// own on stderr: open (open the database), read (count the records,
-// then read them into the hierarchy), group (group the directories by
-// digest) and write (write the groups to stdout). Each display ends
-// before the next phase starts, and before an error is returned to be
-// printed.
 func runTrees(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {
@@ -54,102 +40,34 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
 	fmt.Fprintf(os.Stderr, "trees: database %s (%s)\n",
 		escapePath(dbPath), source)
 
-	// Opening can wait on a busy database for as long as the busy
-	// timeout.
-	prog := newProgress(progressParams{
-		label: treesPhase(openPhase), total: -1,
-		waitingFor: openingDatabase,
-	})
 	db, err := openReportDatabase(ctx, dbPath)
-
-	prog.finish()
-
 	if err != nil {
 		return err
 	}
 
 	defer func() { _ = db.Close() }()
 
-	prog = newProgress(progressParams{
+	records := 0
-		label: treesPhase("read"), unit: "records", total: -1,
+	tree := newTreeBuilder()
-		waitingFor: countingRecords, stalledOn: "db",
+
+	err = loadFileRows(ctx, db, func(r scanRec) {
+		records++
+
+		tree.add(r)
 	})
-	records, super, allDirs, err := readTree(ctx, db, prog)
-
-	prog.finish()
-
 	if err != nil {
 		return fmt.Errorf("database %s: %w", dbPath, err)
 	}
 
-	// Grouping works in memory, so it waits on nothing but the
-	// processor.
-	prog = newProgress(progressParams{
-		label: treesPhase("group"), unit: "directories",
-		total: int64(len(allDirs)), stalledOn: "cpu",
-	})
-	dupes := collectTreeGroups(allDirs, super, prog)
-
-	prog.finish()
-
-	prog = newProgress(progressParams{
-		label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
-		stalledOn: "out",
-	})
-	dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
-
-	prog.finish()
-
-	if err != nil {
-		return err
-	}
-
-	fmt.Fprintf(os.Stderr,
-		"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
-			"%s reclaimable\n",
-		records, len(dupes), dupeTrees, humanBytes(reclaimable))
-
-	return nil
-}
-
-// readTree gives prog the number of records in db as its total, then
-// reads every record in path order into the directory hierarchy,
-// counting each on prog. It returns the number of records read, the
-// super-root and every directory.
-func readTree(ctx context.Context, db *sql.DB,
-	prog *progress,
-) (int, *treeNode, []*treeNode, error) {
-	records := 0
-	tree := newTreeBuilder()
-
-	err := loadFileRows(ctx, db, prog.setTotal, func(r scanRec) {
-		records++
-
-		prog.increment()
-		tree.add(r)
-	})
-	if err != nil {
-		return 0, nil, nil, err
-	}
-
 	super, allDirs := tree.finish()
 
-	return records, super, allDirs, nil
+	dupes := collectTreeGroups(allDirs, super)
-}
 
-// writeTreeGroups writes dupes to stdout as TSV, a header line and then
+	out := bufio.NewWriterSize(stdout, ioBufSize)
-// a row for every member of each group but its first, counting each
-// group on prog once its rows are written. stdout may be the terminal
-// prog is drawn on, so the lines go through prog's writer. It returns
-// the number of rows (dupe trees) and the bytes they could reclaim.
-func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
-	prog *progress,
-) (int, int64, error) {
-	out := prog.writer(stdout)
 
-	_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
+	_, err = fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
 	if err != nil {
-		return 0, 0, fmt.Errorf("write stdout: %w", err)
+		return fmt.Errorf("write stdout: %w", err)
 	}
 
 	dupeTrees := 0
@@ -163,22 +81,25 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
 				escapePath(first.path), escapePath(n.path),
 				first.fileCount, first.totalSize)
 			if err != nil {
-				return 0, 0, fmt.Errorf("write stdout: %w", err)
+				return fmt.Errorf("write stdout: %w", err)
 			}
 
 			dupeTrees++
 			reclaimable += first.totalSize
 		}
-
-		prog.increment()
 	}
 
 	err = out.Flush()
 	if err != nil {
-		return 0, 0, fmt.Errorf("write stdout: %w", err)
+		return fmt.Errorf("write stdout: %w", err)
 	}
 
-	return dupeTrees, reclaimable, nil
+	fmt.Fprintf(os.Stderr,
+		"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
+			"%s reclaimable\n",
+		records, len(dupes), dupeTrees, humanBytes(reclaimable))
+
+	return nil
 }
 
 // treeBuilder reconstructs the directory hierarchy from records added
@@ -300,18 +221,14 @@ func fileEntry(name string, r scanRec) string {
 		"\x00" + r.head + "\x00" + r.tail + "\x00" + content
 }
 
-// collectTreeGroups groups directories by digest, counting each one on
+// collectTreeGroups groups directories by digest and returns every
-// prog as it is grouped, and returns every maximal group with two or
+// maximal group with two or more members, each group's members sorted
-// more members, each group's members sorted by path, groups ordered by
+// by path, groups ordered by tree size descending then by first path
-// tree size descending then by first path ascending.
+// ascending.
-func collectTreeGroups(allDirs []*treeNode, super *treeNode,
+func collectTreeGroups(allDirs []*treeNode, super *treeNode) [][]*treeNode {
-	prog *progress,
-) [][]*treeNode {
 	groups := make(map[[sha256.Size]byte][]*treeNode)
 	for _, d := range allDirs {
 		groups[d.digest] = append(groups[d.digest], d)
-
-		prog.increment()
 	}
 
 	var dupes [][]*treeNode

trees_test.go

@@ -3,11 +3,8 @@ package main
 import (
 	"bytes"
 	"database/sql"
-	"os"
 	"slices"
-	"strings"
 	"testing"
-	"time"
 )
 
 // Signature hashes shared by the smoke-test records.
@@ -20,11 +17,6 @@ const (
 	f2Content = "f2c"
 )
 
-// treesOpenLine is the plain line of trees' open phase when it opens
-// the database at once; it prints it as the phase starts and as it ends.
-const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
-	"eta unknown"
-
 // smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
 // /d/t2 are identical, /d/t3 differs from them only by one filename.
 func smokeTreeRecs() []scanRec {
@@ -46,7 +38,7 @@ func dbTree(t *testing.T, db *sql.DB) (*treeNode, []*treeNode) {
 
 	tree := newTreeBuilder()
 
-	err := loadFileRows(t.Context(), db, func(int64) {}, tree.add)
+	err := loadFileRows(t.Context(), db, tree.add)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -178,7 +170,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
 		t.Fatalf("directory paths = %q, want %q", got, want)
 	}
 
-	groups := collectTreeGroups(dirs, super, nil)
+	groups := collectTreeGroups(dirs, super)
 
 	gotGroups := groupPaths(groups)
 	wantGroups := [][]string{{"/a", "/c"}}
@@ -211,98 +203,6 @@ func TestRunTreesEscapesPaths(t *testing.T) {
 	}
 }
 
-// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
-// records with stderr not a terminal, and checks every line it prints
-// there. After the line naming the database, each phase prints a line
-// naming it with its place among the four as it starts and another as
-// it ends. read counts the 6 records before it reads them, group counts
-// the 8 directories, and write the 1 group. The summary comes last, and
-// stdout holds the report alone.
-func TestRunTreesProgressNamesEachPhase(t *testing.T) {
-	path := seedDatabase(t, smokeTreeRecs())
-	t.Setenv(databaseEnv, path)
-
-	stderr := captureStderr(t)
-
-	var stdout bytes.Buffer
-
-	code := run([]string{cmdTrees}, &stdout, os.Stderr)
-	if code != exitOK {
-		t.Fatalf("run(trees) = %d, want %d; stderr: %s",
-			code, exitOK, stderr())
-	}
-
-	report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
-	if got := stdout.String(); got != report {
-		t.Errorf("stdout = %q, want %q", got, report)
-	}
-
-	// How each line starts; the rate at the end of a phase varies.
-	want := []string{
-		"trees: database " + path + " (from " + databaseEnv + ")",
-		treesOpenLine,
-		treesOpenLine,
-		"[2/4] read: counting the records, elapsed 0s, eta unknown",
-		"[2/4] read: [6/6] 100%, ",
-		"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown",
-		"[3/4] group: [8/8] 100%, ",
-		"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown",
-		"[4/4] write: [1/1] 100%, ",
-		"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
-	}
-
-	lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
-	if len(lines) != len(want) {
-		t.Fatalf("stderr lines = %q, want %d lines starting %q",
-			lines, len(want), want)
-	}
-
-	for i := range want {
-		if !strings.HasPrefix(lines[i], want[i]) {
-			t.Errorf("stderr line %d = %q, want it to start %q",
-				i+1, lines[i], want[i])
-		}
-	}
-}
-
-// TestWriteTreeGroupsOnTerminal writes the smoke-test report to the
-// terminal its write phase's display is drawn on, as a plain "sfdupes
-// trees" does: the terminal must show the header and the row whole,
-// each on a line of its own.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestWriteTreeGroupsOnTerminal(t *testing.T) {
-	stderr := captureStderr(t)
-	super, dirs := treeOf(t, smokeTreeRecs())
-	dupes := collectTreeGroups(dirs, super, nil)
-
-	// No timed draws: the display is drawn as it starts and as it ends.
-	p := startProgress(progressParams{
-		label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
-	}, terminalWidth, time.Hour, time.Now)
-
-	_, _, err := writeTreeGroups(os.Stderr, dupes, p)
-
-	p.finish()
-
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	// A terminal shows each line as the text after its last carriage
-	// return.
-	var shown []string
-
-	for line := range strings.SplitSeq(stderr(), "\n") {
-		shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
-	}
-
-	want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
-	if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
-		t.Errorf("terminal shows lines %q, want %q first", shown, want)
-	}
-}
-
 func TestTreeDigests(t *testing.T) {
 	t.Parallel()
 
@@ -354,7 +254,7 @@ func TestCollectTreeGroupsMaximal(t *testing.T) {
 
 	super, dirs := treeOf(t, smokeTreeRecs())
 
-	groups := collectTreeGroups(dirs, super, nil)
+	groups := collectTreeGroups(dirs, super)
 
 	want := [][]string{{"/d/t1", "/d/t2"}}
 	if got := groupPaths(groups); !slices.EqualFunc(got, want,
@@ -378,14 +278,14 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
 
 	super, dirs := treeOf(t, recs)
 
-	forward := groupPaths(collectTreeGroups(dirs, super, nil))
+	forward := groupPaths(collectTreeGroups(dirs, super))
 
 	reversed := slices.Clone(recs)
 	slices.Reverse(reversed)
 
 	superR, dirsR := treeOf(t, reversed)
 
-	backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
+	backward := groupPaths(collectTreeGroups(dirsR, superR))
 	if !slices.EqualFunc(forward, backward, slices.Equal) {
 		t.Fatalf("output depends on record order: %v vs %v",
 			forward, backward)
@@ -404,7 +304,7 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
 
 	super, dirs := treeOf(t, recs)
 
-	got := groupPaths(collectTreeGroups(dirs, super, nil))
+	got := groupPaths(collectTreeGroups(dirs, super))
 
 	want := [][]string{{"/p/x1", "/p/x2"}}
 	if !slices.EqualFunc(got, want, slices.Equal) {
@@ -426,7 +326,7 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
 
 	super, dirs := treeOf(t, recs)
 
-	got := groupPaths(collectTreeGroups(dirs, super, nil))
+	got := groupPaths(collectTreeGroups(dirs, super))
 
 	want := [][]string{{"/p/a/x", "/q/b/x"}}
 	if !slices.EqualFunc(got, want, slices.Equal) {