Install from @latest, retract v0.0.1, print a go-installed binary's version (closes #116)

README "Install" now leads with go install ...@latest: until 1.0 each
release is a v0.0.N tag on main's tip, so @latest gives that tip once
it has passed the build. It also says what @main and v0.0.1 give and
how to check an installed binary. go.mod retracts v0.0.1, the early
build without a database.

sfdupes --version printed "dev" for every go-installed binary, since
only the linker set Version. When the linker sets none, it now prints
the main module's version from runtime/debug.ReadBuildInfo; "dev" is
left only when neither is known. Version is now empty by default.

The retraction takes effect only once a release carrying it is the
newest.

Model: opus-5-5

README.md

@@ -38,8 +38,9 @@ database persists between runs; a rescan only hashes files that are new or
 changed, or that may have gained a duplicate since the last scan, and removes
 records for files that no longer exist. `report` reads the database and prints
 the file-level duplicates report. `trees` reads the same database and prints the
-duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
-with no `PATH` operand — prints a usage message and exits 2.
+duplicate-tree report. `info` prints the database's details: its path, the sizes
+of its files and the number of its records. A missing/invalid subcommand — or a
+`scan` invocation with no `PATH` operand — prints a usage message and exits 2.
 
 The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
 by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
@@ -48,15 +49,40 @@ results are as fresh as the last completed scan.
 
 ### Install
 
-With Go installed, this builds and installs the current `main` branch:
+With Go installed, this builds and installs the newest release:
+
+```sh
+go install sneak.berlin/go/sfdupes@latest
+```
+
+Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time
+`main` moves and its tip passes `script/cibuild`, that tip is tagged with the
+next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build.
+`@main` installs the tip before it is tagged:
 
 ```sh
 go install sneak.berlin/go/sfdupes@main
 ```
 
-The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. A
-binary installed this way reports its version as `dev`; one built from a clone
-or into the Docker image carries the git tag or commit it was built from.
+`v0.0.1` is an early build without a database. It does not work as this README
+describes, and `go.mod` retracts it.
+
+The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To
+see which version an installed binary is:
+
+```sh
+sfdupes --version
+go version -m "$(command -v sfdupes)"
+```
+
+`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`,
+`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`,
+or for `@main` a pseudo-version ending in the first 12 characters of the
+commit's hash. A binary built from a clone or into the Docker image carries the
+git tag or commit it was built from instead, and `VERSION` is `dev` only when
+the binary carries no version at all. `go version -m` prints what Go recorded in
+the binary; its `mod` line names the module's version. Binaries from `v0.0.2`
+and earlier print `dev`, so check those with `go version -m`.
 
 From a clone, `make build` writes the binary to `./sfdupes`:
 
@@ -99,9 +125,11 @@ As a file `/etc/cron.d/sfdupes`:
   `/var/log/sfdupes.log`, holds a first line naming the database (see
   "Database"), plain progress lines naming each phase and its place among the
   five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
-  runs, and one as it ends), a warning for each path it skips, and the summary
-  line (see "Progress" and "`scan` mode"). The log grows with every scan; rotate
-  it like any other.
+  runs, and one as it ends), a warning for each path it skips, a line for each
+  default exclude its walk does not enter and for each path it leaves out
+  because it is not on a persistent filesystem (see "Rules for the walk"), and
+  the summary line (see "Progress" and "`scan` mode"). The log grows with every
+  scan; rotate it like any other.
 - Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
   scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
   reason among the last lines of the log; `tail` prints them, and cron mails
@@ -109,9 +137,9 @@ As a file `/etc/cron.d/sfdupes`:
 - A scan still running when the next one starts carries on. The new one fails at
   once, and the lines cron mails include
   `sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
-- `report` and `trees` need only read access to the database (see "Database").
-  Under the usual umask of `022` the first scan creates it readable by every
-  user, so an unprivileged user can run them against root's database.
+- `report`, `trees` and `info` need only read access to the database (see
+  "Database"). Under the usual umask of `022` the first scan creates it readable
+  by every user, so an unprivileged user can run them against root's database.
 
 ### Reading the reports
 
@@ -199,12 +227,13 @@ Goals, in order:
     under resilver). Holding one small record (path, size, mtime) per file in
     memory during a scan is acceptable; holding every file's hashes is not (they
     stay in the database). The reporting commands do not hold every file's
-    hashes either: `report` lets SQLite group and order the records and writes
-    each row as it reads it, so its memory does not grow with the database, and
-    `trees` reads the records in path order and keeps each directory's path,
-    digest and totals, plus the hashes of only the files in the directories
-    holding the record being read, so its memory grows with the number of
-    directories and with the size of the largest directory.
+    hashes either: `report` reads the records in signature order and holds the
+    duplicate paths of one size at a time, so its memory grows with the
+    duplicates of the size that has the most of them, and `trees` reads the
+    records in path order and keeps each directory's path, digest and totals,
+    plus the hashes of only the files in the directories holding the record
+    being read, so its memory grows with the number of directories and with the
+    size of the largest directory.
 3.  **Scan incrementally, analyze offline.** The expensive filesystem scan
     maintains a persistent database; an unchanged file is never read again on a
     rescan, except to compute its content hash once a file of 10 MiB or more
@@ -234,7 +263,7 @@ Goals, in order:
 
 ### Subcommands
 
-Three subcommands, all implemented:
+Four subcommands, all implemented:
 
 1.  `scan` — walk the filesystem and synchronize the database: one signature
     record per regular file.
@@ -242,26 +271,30 @@ Three subcommands, all implemented:
 3.  `trees` — tree-level duplicate report: reconstruct the directory hierarchy
     from the database records, compute a Merkle-style digest per directory, and
     report maximal groups of identical trees.
+4.  `info` — the database's details: its path, the sizes of its files, its
+    schema version and the counts of its records.
 
 ```
-sfdupes scan [--workers N] [-x] PATH...
+sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
 sfdupes report > dupes.tsv
 sfdupes trees > dupetrees.tsv
+sfdupes info
 sfdupes --version
 sfdupes [command] --help
 ```
 
 `--workers N` sets the size of each `scan` worker pool (default: the number of
-CPUs), and `-x` (`--one-file-system`) keeps the walk of each operand on that
-operand's filesystem; both are described under "`scan` mode".
-`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
-exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
-subcommand, prints the help text to stderr and exits 0, writing nothing to
-stdout.
+CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
+operand's filesystem, and `--no-default-excludes` lets the walk enter the
+default excludes, a list of paths; all three are described under "`scan` mode".
+`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see
+"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`,
+alone or after a subcommand, prints the help text to stderr and exits 0, writing
+nothing to stdout.
 
 ### Database
 
-All three subcommands operate on a single SQLite database file:
+All four subcommands operate on a single SQLite database file:
 
 - Location: the value of the `SFDUPES_DATABASE` environment variable when set
   and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
@@ -284,13 +317,13 @@ All three subcommands operate on a single SQLite database file:
     symlinks resolved. After a `cd` through a symlink to a directory, a path
     starting `../` therefore names a file beside the directory the symlink leads
     to, as it does for any other program. `scan` says `new database` when no
-    file is at that path as it starts. `report` and `trees` print the line
-    before they open the database, so the error for a missing database comes
-    after it.
+    file is at that path as it starts. `report`, `trees` and `info` print the
+    line before they open the database, so the error for a missing database
+    comes after it.
 
-- `scan` creates the database (and its parent directory) on first use. `report`
-  and `trees` require an existing database; a missing database file is a fatal
-  error (exit 1) telling the user to run `scan` first.
+- `scan` creates the database (and its parent directory) on first use. `report`,
+  `trees` and `info` require an existing database; a missing database file is a
+  fatal error (exit 1) telling the user to run `scan` first.
 - Only one `scan` runs against a database at a time. For its whole run, `scan`
   holds an exclusive `flock(2)` lock on a lock file beside the database, named
   by appending `.lock` to the database's path with symlinks resolved
@@ -306,8 +339,8 @@ All three subcommands operate on a single SQLite database file:
   on. The lock file is created on first use, open to its owner only, and left in
   place: a leftover file blocks nothing, because the lock ends with the process
   holding it however it ends, a fatal error or an interrupt included, and
-  deleting the file while a scan runs would let a second scan start. `report`
-  and `trees` never take the lock, so they run during a scan.
+  deleting the file while a scan runs would let a second scan start. `report`,
+  `trees` and `info` never take the lock, so they run during a scan.
 - While `scan` runs, the database is in WAL journal mode with a busy timeout, so
   running a report while a cron `scan` is in progress is safe. The filesystem is
   authoritative; the database is an eventually-consistent reflection of it.
@@ -323,9 +356,9 @@ All three subcommands operate on a single SQLite database file:
   needs the database to itself: a `scan` that starts while a report is still
   reading waits for it up to the 10-second busy timeout, then fails; a `scan`
   that ends while a report has the database open warns and leaves the database
-  in WAL mode until the next scan. `report` writes each row as it reads it, so
-  it is still reading while its output is paused (a pager, a stalled pipe), and
-  a `scan` started then fails after the busy timeout.
+  in WAL mode until the next scan. `report` writes its rows while it reads the
+  records, so it is still reading while its output is paused (a pager, a stalled
+  pipe), and a `scan` started then fails after the busy timeout.
 - A `scan` writes to the database and the files beside it at these points (its
   phases are described under "`scan` mode"):
     - As it starts, before the walk: the directory, an empty database file and
@@ -340,10 +373,10 @@ All three subcommands operate on a single SQLite database file:
     - In the content phase: content hashes, again committed 10,000 at a time.
     - At the end, as it closes the database: the `-wal` and `-shm` files go
       away, unless a report still has the database open (see above).
-- `report` and `trees` open the database read-only and need only read access to
-  the database file, and no write access to its directory. While the database is
-  in WAL mode they also read the `-wal` and `-shm` files beside it, which SQLite
-  creates with the database file's permissions.
+- `report`, `trees` and `info` open the database read-only and need only read
+  access to the database file, and no write access to its directory. While the
+  database is in WAL mode they also read the `-wal` and `-shm` files beside it,
+  which SQLite creates with the database file's permissions.
 - Schema (`PRAGMA user_version` is the schema version, currently 1; a database
   with any other version is a fatal error. `scan` creates the schema and sets
   the version in one transaction, so a first scan stopped while doing so leaves
@@ -379,8 +412,8 @@ All three subcommands operate on a single SQLite database file:
     a file of 10 MiB or more, `content` stays empty until the content phase of a
     scan (see "`scan` mode" below) has read the file. A record with an empty
     `content` is never part of a duplicate group, though it still defines the
-    file for tree reconstruction. The `files_signature` index lets SQLite group
-    the records by signature for `report` without sorting the whole table.
+    file for tree reconstruction. The `files_signature` index lets `report` read
+    the records in signature order without sorting the table.
 
 ### Duplicate detection
 
@@ -434,11 +467,12 @@ results do not depend on the working directory. All operands belong to a single
 scan and are enumerated concurrently: every operand seeds the shared walk worker
 pool. Overlapping operands are harmless — an operand that duplicates another or
 lies under another is dropped before walking, so every file is reached exactly
-once and produces one database record. With `-x`, though, an operand under
-another one is kept and walked on its own when any directory from it (from its
-directory, for a regular file) up to but not including the operand covering it
-is on a different filesystem than that operand, because that operand's walk does
-not enter such a directory and so does not reach it.
+once and produces one database record. An operand under another one is kept,
+though, and walked on its own when any directory from it (from its directory,
+for a regular file) up to but not including the operand covering it is a default
+exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
+`-x`, is on a different filesystem than that operand, because that operand's
+walk does not enter such a directory and so does not reach it.
 
 An operand that is a symlink (never followed, not even as an operand), socket,
 FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -448,7 +482,10 @@ beneath it is still scanned. The records stored beneath it are not deleted: they
 are treated like any other record outside the scanned operands, including the
 content-phase exception below. If it lies under another operand, they are under
 that operand instead, and are treated like any other record there. This is not
-an error: a scan whose every operand is dropped walks nothing and exits 0.
+an error: a scan whose every operand is dropped walks nothing and exits 0. An
+operand that is not on a persistent filesystem (see "Rules for the walk") is
+dropped the same way, except that the line `scan` prints for it is not a
+warning, and it is not counted as skipped.
 
 `scan` synchronizes the database with the filesystem state under the scanned
 operands:
@@ -484,14 +521,16 @@ operands:
   can be scanned on different schedules into the same database. The one
   exception is the content phase below: a stored file of 10 MiB or more without
   a `content` hash is read for one, wherever it lies, once its size, `head`, and
-  `tail` match another record's. If that file is gone or has changed since its
-  record was written, the record is left as it is.
-- With `-x`, a record below a directory the walk did not enter because that
-  directory is on another filesystem than its operand counts as outside the
-  scanned operands: it is not deleted, and its size joins the size census. A
-  filesystem mounted under an operand can therefore be scanned on its own
-  schedule into the same database. Records below an operand at or inside such a
-  directory are not kept this way, since that operand's own walk verifies them.
+  `tail` match another record's. If that file is gone, has changed since its
+  record was written, or is not on a persistent filesystem, it is not read and
+  the record is left as it is.
+- A record at or below a path the walk left out, because that path is a default
+  exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
+  another filesystem than its operand, counts as outside the scanned operands:
+  it is not deleted, and its size joins the size census. A filesystem mounted
+  under an operand can therefore be scanned on its own schedule into the same
+  database. Records below an operand at or inside such a path are not kept this
+  way, since that operand's own walk verifies them.
 
 `scan` runs **five sequential phases over the whole scan**. Parallelism lives
 inside the walk, hash and content phases; batched database writes begin during
@@ -537,19 +576,20 @@ the hash phase:
     be read are kept in memory, never every file's hashes. Every record sharing
     their size, `head`, and `tail`, including one that already has a `content`
     hash, has its file checked with `lstat` first. A file that is gone, is no
-    longer a regular file, or has changed (a different size, or an mtime newer
-    than recorded) keeps its record as it is and does not count as a match for
-    the others. Any other `lstat` error is warned about and counted as skipped,
-    with the same result. If such a record has no `content` hash, it stays out
-    of duplicate groups; if it has one, it is still reported until a scan
-    covering its own tree updates or removes it. The files that pass and have no
-    `content` hash are read only if at least two of those records pass, so a
-    file whose only matches are stale costs no read; a file that already has a
-    `content` hash is never read again. They are read by a worker pool as in the
-    hash phase, in inode order and once per inode, and their content hashes are
-    committed in batches. A failed read is warned about and counted as skipped;
-    its record keeps an empty `content`, so it is not a duplicate, and a later
-    scan tries again.
+    longer a regular file, has changed (a different size, or an mtime newer than
+    recorded), or is not on a persistent filesystem (see "Rules for the walk")
+    keeps its record as it is and does not count as a match for the others. Any
+    other `lstat` error is warned about and counted as skipped, with the same
+    result. If such a record has no `content` hash, it stays out of duplicate
+    groups; if it has one, it is still reported until a scan covering its own
+    tree updates or removes it. The files that pass and have no `content` hash
+    are read only if at least two of those records pass, so a file whose only
+    matches are stale costs no read; a file that already has a `content` hash is
+    never read again. They are read by a worker pool as in the hash phase, in
+    inode order and once per inode, and their content hashes are committed in
+    batches. A failed read is warned about and counted as skipped; its record
+    keeps an empty `content`, so it is not a duplicate, and a later scan tries
+    again.
 
 Rules for the walk:
 
@@ -560,6 +600,15 @@ Rules for the walk:
 - Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
   them would list every file once per snapshot), not even when it is an operand;
   such an operand is dropped the same way.
+- Never record the scan's own database. Once the database and its `.lock` file
+  are open, `scan` notes the device and inode of the database file, its `-wal`
+  and `-shm` files and its `.lock` file, and the walk passes over a file with
+  one of them, however it reaches it: as an operand, below one, or through a
+  symlinked directory. It prints no line for such a file and does not count it
+  as skipped. A record that an earlier scan left for one of them is deleted,
+  like the record of any file the scan no longer verifies. There is no
+  `-journal` file during the walk, because the database is in WAL mode then (see
+  "Database").
 - Filesystem boundaries are crossed by default. With `-x` (long form
   `--one-file-system`, following the GNU `du`/`rsync` convention), never descend
   into a directory on a different filesystem than its `PATH` operand; each
@@ -567,6 +616,74 @@ Rules for the walk:
   unless they lie below an operand at or inside that directory, count as outside
   the scanned operands, as described in "`scan` mode" above: they are not
   deleted, and their sizes join the size census.
+- Never record anything that is not on a persistent filesystem. A filesystem is
+  not persistent when the kernel makes it up or keeps it in memory: one of type
+  `proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
+  `debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
+  `mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
+  `rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
+  it names the database and before it takes the lock, `scan` lists the mounted
+  filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
+  of mounted filesystems on macOS (on any other system it lists none), and notes
+  the device ID of each one of those types. On Linux that is the device ID the
+  list gives, so a filesystem that a later mount hides still gives its own. On
+  macOS it is read with `lstat` on the mount point, and where one filesystem is
+  mounted over another on the same mount point, the one mounted last counts. If
+  `scan` cannot list them, it fails (exit 1). The walk then never enters a
+  directory, and never records a file, on one of those devices, however it
+  reaches it: as an operand, below one, or through a bind mount, which keeps the
+  device ID of what it binds. The content phase never reads a file on one of
+  them either. No flag turns this off. For each operand, directory or file it
+  leaves out this way, `scan` prints a line on stderr naming it and the type of
+  its filesystem, as it prints a warning, so it never lands inside a progress
+  display:
+
+    ```
+    scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
+    ```
+
+    That line is not a warning, and the path is not counted as skipped. Records
+    that an earlier scan left at or below it, unless they lie below an operand
+    at or inside it, count as outside the scanned operands, as described in
+    "`scan` mode" above: they are not deleted, and their sizes join the size
+    census.
+
+- Below an operand, never descend into a default exclude: a path that holds the
+  kernel's view of the running system, or temporary files, so that
+  `sfdupes scan /` walks only the files worth comparing. The default excludes
+  depend on the system, and `sfdupes scan --help` lists those of the system it
+  runs on:
+    - On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
+    - On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
+      `/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
+      everything in `/System/Volumes/Data` through the firmlinks, so walking it
+      as well would record each of those files twice.
+    - On Windows, relative to the root of the drive an operand lies on: the
+      page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
+      `hiberfil.sys`, and the directories `System Volume Information`,
+      `$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
+      matches one path component, as it does in any default exclude. sfdupes
+      does not build for Windows yet, so this list is not used.
+    - On any other system there are none.
+
+    `--no-default-excludes` turns them all off; what is not on a persistent
+    filesystem stays left out. An operand at or inside a default exclude is
+    still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
+    persistent filesystem. For each default exclude its walk does not enter,
+    `scan` prints a line on stderr naming it, as it prints a warning:
+
+    ```
+    scan: skipped /var/tmp (default exclude)
+    ```
+
+    That line is not a warning, and the directory is not counted as skipped. A
+    default exclude that is not on a persistent filesystem, as `/proc` is, gets
+    the line for its filesystem instead, and one that is itself an operand gets
+    no line, since it is walked as that operand. The records below a default
+    exclude the walk does not enter, unless they lie below an operand at or
+    inside it, count as outside the scanned operands, as described in "`scan`
+    mode" above: they are not deleted, and their sizes join the size census.
+
 - On any per-path error (permission denied, file vanished between passes,
   unreadable): print a one-line warning to stderr, skip the path, and continue.
   Per-file errors never abort the run; the final summary reports how many were
@@ -604,11 +721,8 @@ total.)
 
 **`report` must never touch the filesystem being analyzed.** It does not stat,
 open, or otherwise access any path that appears in the records; its only I/O is
-reading the database, writing stdout/stderr, and the temporary file SQLite sorts
-in when the duplicate rows do not fit in memory. SQLite puts that file in
-`$SQLITE_TMPDIR` or `$TMPDIR` when set, otherwise in `/var/tmp` (or `/tmp`), and
-deletes it as soon as it has opened it. `report` must produce identical output
-whether or not the scanned filesystem is still mounted.
+reading the database and writing stdout/stderr. `report` must produce identical
+output whether or not the scanned filesystem is still mounted.
 
 Processing:
 
@@ -622,6 +736,16 @@ Processing:
   by `first` path ascending. Output must be fully deterministic for a given
   database state.
 
+`report` runs in two phases, each with its own progress display on stderr (see
+"Progress"): `open` opens the database; `read` counts the records, then reads
+them in signature order, largest size first. That is the order of the
+`files_signature` index read backwards, so SQLite sorts nothing and returns the
+first record at once. The records sharing a signature come together, and so do
+the records of one size, so `report` holds only the duplicate groups of the size
+it is reading, and writes them, ordered by `first` path, once it reaches a
+smaller size. Its memory holds at most the duplicate paths of one size at a
+time, never every record.
+
 #### Report output format
 
 TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
@@ -719,15 +843,57 @@ Summary to stderr: records read, number of duplicate-tree groups, number of dupe
 trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
 units.
 
+### `info` mode
+
+`info` takes no positional arguments and prints the database's details on
+stdout, one per line: a name, a tab, and the value. It opens the database the
+way `report` does, read-only and without the scan lock, so it runs during a scan
+and needs only read access (see "Database"). Like `report`, it never touches the
+scanned filesystem.
+
+```
+database	/var/lib/sfdupes/db.sqlite (default)
+database file size	2147483648 (2.0 GiB)
+-wal file size	4124152 (3.9 MiB)
+modified	2026-10-16T12:02:50Z
+schema version	1
+records	12345678
+records with head and tail hashes	1234567
+records with content hash	123456
+size of recorded files	150000000000000 (136.4 TiB)
+```
+
+- `database` is the path and where it came from, as on the line every subcommand
+  prints first on stderr (see "Database").
+- `database file size` and `-wal file size` are in bytes, then in human units.
+  The `-wal` line is there only when a `-wal` file is beside the database, as
+  while a scan runs.
+- `modified` is the time the database file was last changed, in UTC. While the
+  database is in WAL mode, the newest changes may be in the `-wal` file alone.
+- `schema version` is the database's `PRAGMA user_version`. A database with
+  another version than this build's is refused, as by `report`.
+- `records` is the number of records. `records with head and tail hashes` and
+  `records with content hash` count the records that have those hashes (see
+  "Database"), and `size of recorded files` adds up the sizes of all records, in
+  bytes, then in human units.
+
+`info` runs in two phases, each with its own progress display on stderr (see
+"Progress"): `open` opens the database; `count` reads the schema version and
+counts the records with one query, a single pass over the table, both in one
+read-only transaction, so the numbers agree with each other while a scan is
+committing. `info` writes its lines once the `count` display has ended, so on a
+terminal they come after the displays.
+
 ### Progress
 
-Use the progress-bar library for all progress of `scan` and `trees`; rendering
-in the style of `pv` is the model. All progress goes to stderr.
+Use the progress-bar library for all progress of `scan`, `report`, `trees` and
+`info`; rendering in the style of `pv` is the model. All progress goes to
+stderr.
 
-`scan` and `trees` must always show that they are working and not hung. Each
-phase gets its own display from the moment it starts, naming the phase and its
-place among the command's phases and showing its progress, its elapsed time and
-its estimated time remaining (ETA).
+`scan`, `report`, `trees` and `info` must always show that they are working and
+not hung. Each phase gets its own display from the moment it starts, naming the
+phase and its place among the command's phases and showing its progress, its
+elapsed time and its estimated time remaining (ETA).
 
 A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
 `[4/5] update`, `[5/5] content`):
@@ -745,7 +911,8 @@ A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
   `head`, and `tail`, without fetching the records. It then says it is searching
   the database until SQLite returns the first record; the checks, and their rate
   and ETA, start there. The files to read then get a display of their own, with
-  their count as its total.
+  their count as its total, and so does the writing of the last of their records
+  to the database, with the count of those records as its total.
 
 `trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
 `[4/4] write`, see "`trees` mode"):
@@ -757,42 +924,83 @@ A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
 - `group` has the directories `read` found as its total, and `write` the
   duplicate-tree groups.
 
+`report` has two phases (`[1/2] open`, `[2/2] read`, see "`report` mode"):
+
+- `open` has no total. While it opens the database, waiting on a busy one
+  included, the display says it is opening the database.
+- `read` counts the records first, then reads them against that count, writing
+  the rows to stdout as it goes. While the count runs, the display says it is
+  counting the records.
+
+`info` has two phases (`[1/2] open`, `[2/2] count`, see "`info` mode"):
+
+- `open` has no total. While it opens the database, waiting on a busy one
+  included, the display says it is opening the database.
+- `count` has no total either, because one query counts everything: the display
+  says it is counting the records until the query returns, and its ETA is
+  unknown.
+
 Each display is one line holding:
 
 - the phase and its place among the phases
 - a `[m/n] x%` display (items processed / total items, percent); before a phase
   has a total, the items processed, or what the phase is waiting for
 - current rate (items/s), naming the items the phase counts: files, records,
-  directories or groups
+  directories or groups. It covers the items completed over the last 5 seconds,
+  or, when stderr is not a terminal, since the line before, so it falls to 0
+  while nothing completes, and a burst of items after a wait shows as a burst
+  rather than as a steady rate. Once no item has completed for 10 seconds, the
+  word `stalled` and where the phase waits take the rate's place (see below).
 - elapsed time
-- estimated time remaining, or `unknown`
+- estimated time remaining, or `unknown`. It comes from the phase's average rate
+  since it started counting, so that it does not swing with each burst.
 
 Example shape (exact layout is flexible, content is not):
 
 ```
 [3/5] hash: [12345/98765] 12% |████      | 92 files/s elapsed 2:32 eta 17:54
 [2/5] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
+[3/5] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
 ```
 
+When no item has completed for 10 seconds, as while one large file is read, a
+disk is slow or the database is busy, the line says `stalled` and where the
+phase waits in place of the rate, which would read 0: `disk` for files, `db` for
+the database, `out` for stdout, and two of them, as `disk/db`, for a phase that
+may wait on either. `walk` says `disk`. `hash` and the content phase's reads say
+`disk/db`, as they read files and write their records to the database between
+them, and so do the content phase's checks, which `lstat` each record's file as
+the database search returns it. `load`, `update`, the content phase's last write
+and the `read` phase of `trees` say `db`. `trees`' `write` says `out`, and its
+`group`, which works in memory, says `cpu`. The `read` phase of `report` writes
+the report while it reads the records, so it says `db/out`. A phase still
+waiting for its total already says what it waits for, and says nothing more.
+
 Additional requirements:
 
-- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`
-  and `trees`' `open`, once it has counted what it will do) is drawn as its line
-  followed by the progress-bar library's bar, 10 columns wide, which shows the
-  percent in place of the line's own. Any other phase, and a phase whose bar has
-  no room, is drawn as the library's spinner followed by its line. Each display
-  is drawn again several times a second, so its elapsed time keeps moving while
-  a phase waits for its next item.
+- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
+  the `open` phase of `report` and `trees`, and both phases of `info`, once it
+  has counted what it will do) is drawn as its line followed by the progress-bar
+  library's bar, 10 columns wide, which shows the percent in place of the line's
+  own. Any other phase, and a phase whose bar has no room, is drawn as the
+  library's spinner followed by its line. Each display is drawn again several
+  times a second, so its elapsed time keeps moving while a phase waits for its
+  next item.
 - On a terminal no line reaches the terminal's last column, however narrow the
-  terminal is. The phase and its place, the progress, the elapsed time and the
-  ETA are kept whole: a line too wide for the terminal leaves out its rate first
-  (or what the phase is waiting for), and a bar with no room beside those four
-  gives way to the spinner. Only when those four alone are too wide, as on a
-  terminal narrower than about 70 columns hours into a scan of millions of
-  files, is the line cut short at the terminal's edge, so the end of the ETA is
-  lost first. A terminal made narrower while a phase runs gets the same: the
-  next draw erases the line within the new width and fits the line to it, and
-  the display goes on redrawing in place.
+  terminal is. A line too wide for the terminal first loses its bar, which gives
+  way to the spinner, then its rate, the words of a stalled phase, or what the
+  phase is waiting for. The phase and its place, the progress, the elapsed time
+  and the ETA are kept whole. On an 80-column terminal, at counts like those of
+  the stalled line in the example above, a stalled line keeps its words whenever
+  the elapsed time and the ETA are both in minutes, except in the content
+  phase's checks and reads. Their lines are the widest, and at most times in
+  minutes they lose the words. Where the words and the ETA do not both fit, as
+  at millions of files and times in hours, the words give way. Only when those
+  four alone are too wide, as on a terminal narrower than about 70 columns hours
+  into a scan of millions of files, is the line cut short at the terminal's
+  edge, so the end of the ETA is lost first. A terminal made narrower while a
+  phase runs gets the same: the next draw erases the line within the new width
+  and fits the line to it, and the display goes on redrawing in place.
 - When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
   redraws: print the line the moment each phase starts, every 5 seconds while it
   runs (whether or not an item has completed, and never more often), and once
@@ -802,20 +1010,25 @@ Additional requirements:
   its own.
 - A warning printed during a phase always lands on a line of its own, never
   inside the progress display. So does every line written to stdout when stdout
-  is the same terminal, as in a plain `sfdupes trees`.
+  and stderr are the same file (the same device and inode): one terminal, as in
+  a plain `sfdupes report` or `sfdupes trees`, or one pipe or file that both are
+  sent to. When they are not, stdout is written without waiting on the display,
+  and the display keeps being drawn while a slow reader of stdout holds up a
+  write.
 - A phase that stops short of its total, as an interrupted one does, is left
   showing the count it stopped at rather than filled up.
-- `report` mode needs no progress display, only its stderr summary.
 
 ### Error handling and exit codes
 
 - `0`: success, even if individual files were skipped with warnings.
 - `1`: fatal error (e.g., a `PATH` operand does not exist, another `scan` is
   already running against the same database, the database cannot be
-  created/opened/read/written, a missing database for `report`/`trees`, stdout
-  write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see below).
+  created/opened/read/written, a missing database for `report`/`trees`/`info`,
+  stdout write failure), or a `scan` stopped by `SIGINT` or `SIGTERM` (see
+  below).
 - `2`: usage error (including `scan` with no `PATH` operand, `scan` with
-  `--workers` below 1, and `report`/`trees` with any positional argument).
+  `--workers` below 1, and `report`/`trees`/`info` with any positional
+  argument).
 
 A stdout write failure, such as a full disk, is reported in one line on stderr
 and exits 1. Two cases never reach sfdupes as a failed write:
@@ -982,10 +1195,6 @@ All of the following, run in this directory, must pass:
     ./sfdupes report
     ```
 
-    (The database lives in a temp directory of its own: inside `$d`, the scan
-    would record it, and its empty lock file would join the `empty1`/`empty2`
-    group.)
-
     Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
     group (two dupe rows, `first` is the lexicographically smallest path);
     `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/

TODO.md

@@ -29,6 +29,86 @@
 
 # Completed Steps
 
+- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so
+  `go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has
+  passed the build; README "Install" says so, says what `@main` and `v0.0.1`
+  give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the
+  early build without a database. When the linker set no version,
+  `sfdupes --version` prints the main module's version that Go recorded in the
+  binary, the tag or the pseudo-version of a `go install` build, and `dev` only
+  when neither is known (2026-10-17,
+  https://git.eeqj.de/sneak/sfdupes/issues/116)
+
+- `scan` creates the database's directory where the database path leads: the
+  path with its last element cut off as text, never cleaned, so a `..` after a
+  symlinked directory is followed through the symlink, as when the file is
+  opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
+  symlink to `/data/x`, it creates `/data/new` and the database in it, where it
+  used to create `/srv/new` and then fail (2026-10-17,
+  https://git.eeqj.de/sneak/sfdupes/issues/119)
+
+- `scan` never records its own database: once the database and its `.lock` file
+  are open, it notes the device and inode of the database file, its `-wal` and
+  `-shm` files and its `.lock` file, and the walk passes over a file with one of
+  them wherever the operands reach it, through a symlinked directory included,
+  printing no line and counting nothing. A record an earlier scan left for one
+  of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
+
+- `scan` never records or reads anything that is not on a persistent filesystem
+  (`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
+  matched by the device ID of each such filesystem, listed as the scan starts
+  from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
+  however the walk or the content phase reaches it; no flag turns that off.
+  Below an operand, its walk does not enter the default excludes either, so
+  `sfdupes scan /` skips the kernel's view of the running system and the
+  temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
+  `/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
+  `/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
+  Windows list, not used yet. An operand at or inside a default exclude is still
+  walked. The scan prints a line for each path it leaves out, without counting
+  it as skipped, and the records there count as outside the scanned operands, as
+  for `-x`. `--no-default-excludes` turns off the default excludes only, and
+  `scan --help` lists them (2026-10-17,
+  https://git.eeqj.de/sneak/sfdupes/issues/124)
+
+- the rate a progress display shows covers the items completed over the last 5
+  seconds, or since the line before when stderr is not a terminal, so it falls
+  to 0 while nothing completes, and a burst after a wait no longer reads as a
+  steady rate of over 1,000 files a second; the ETA still comes from the phase's
+  average. Once no item has completed for 10 seconds, the word `stalled` and
+  where the phase waits take the rate's place, as `stalled disk/db`: `disk`,
+  `db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
+  either, each phase giving its word where it is defined. A line too wide for
+  the terminal loses its bar first, then its rate or those words, so the phase,
+  progress, elapsed time and ETA stay whole. On an 80-column terminal, at counts
+  like those of README's example, the content phase's checks and reads lose
+  those words at most times in minutes, and every other phase keeps them
+  whenever the elapsed time and the ETA are both in minutes. The content phase's
+  last partial batch is now written under a display of its own (2026-10-16,
+  https://git.eeqj.de/sneak/sfdupes/issues/125)
+
+- new subcommand `info` prints the database's details on stdout, one per line as
+  a name, a tab and the value: its path and where it came from, the sizes of the
+  database file and of its `-wal` file when there is one, when the database file
+  was last changed, the schema version, the number of records, how many have
+  head and tail hashes and how many a content hash, and the sizes of all records
+  added up. It opens the database as `report` does, and shows two phases on
+  stderr, `[1/2] open` and `[2/2] count`; the schema version and the counts come
+  from one read-only transaction, the counts from one query (2026-10-16,
+  https://git.eeqj.de/sneak/sfdupes/issues/117)
+
+- `report` shows its work on stderr in two phases, `[1/2] open` and
+  `[2/2] read`, with the progress display `scan` and `trees` use, from the
+  moment it starts until the summary line: `open` says it is opening the
+  database, also while it waits on a busy one; `read` counts the records, then
+  reads them against that count in signature order, which SQLite takes from the
+  `files_signature` index without sorting, so the first record comes at once.
+  `report` forms the duplicate groups itself, holding only those of the size it
+  is reading, and writes them ordered by first path before it reads the next
+  size. stdout and the summary line are unchanged, and SQLite no longer needs a
+  temporary file to sort in (2026-10-16,
+  https://git.eeqj.de/sneak/sfdupes/issues/114)
+
 - `trees` shows its work on stderr in four phases, `[1/4] open`, `[2/4] read`,
   `[3/4] group` and `[4/4] write`, each with the progress display `scan` uses,
   from the moment it starts until the summary line: `open` says it is opening
@@ -36,8 +116,9 @@
   then reads them against that count; `group` counts the directories it groups,
   and `write` the groups it writes. A display's rate names what its phase
   counts: files, records, directories or groups. stdout and the summary line are
-  unchanged, and when stdout is the same terminal, none of its lines shares a
-  line with the display (2026-10-16,
+  unchanged. When stdout and stderr are the same file, as one terminal is, none
+  of stdout's lines shares a line with the display; when they are not, the
+  display keeps being drawn while stdout waits on a slow reader (2026-10-16,
   https://git.eeqj.de/sneak/sfdupes/issues/113)
 
 - each `scan` phase's progress display names the phase and its place among the

cancel_test.go

@@ -196,7 +196,7 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
 	base := baselineGoroutines(t)
 
 	st, err := syncScan(newWalkClock(walkCancelAtDone), db,
-		[]string{dir}, walkCancelWorkers, false)
+		[]string{dir}, walkCancelWorkers, false, walkRules{})
 
 	assertWalkGuardAborted(t, st, err)
 	assertRecordsIntact(t, db, before)
@@ -268,7 +268,8 @@ func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
 	ctx, cancel := context.WithCancel(t.Context())
 	cancel()
 
-	st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false)
+	st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false,
+		walkRules{})
 	if !errors.Is(err, context.Canceled) {
 		t.Fatalf("syncScan on a cancelled context = %v, want %v",
 			err, context.Canceled)
@@ -301,7 +302,7 @@ func TestSyncScanCancelledMidHashKeepsHashedRecords(t *testing.T) {
 	db := openTestDB(t)
 
 	st, err := syncScan(newWalkClock(hashCancelAtDone), db,
-		[]string{dir}, walkCancelWorkers, false)
+		[]string{dir}, walkCancelWorkers, false, walkRules{})
 	if !errors.Is(err, context.Canceled) {
 		t.Fatalf("syncScan cancelled mid-hash = %v, want %v",
 			err, context.Canceled)
@@ -356,7 +357,8 @@ func TestRunScanInterrupted(t *testing.T) {
 	stderr := captureStderr(t)
 	dir := buildSmokeTree(t)
 
-	err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
+	err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
+		false)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -380,7 +382,7 @@ func TestRunScanInterrupted(t *testing.T) {
 		defer close(done)
 
 		err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
-			false)
+			false, false)
 	}()
 
 	awaitReturn(t, done, "runScan")
@@ -407,7 +409,8 @@ func TestRunScanInterrupted(t *testing.T) {
 			got, before)
 	}
 
-	err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
+	err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
+		false)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -433,7 +436,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
 	dir := buildWalkCancelTree(t)
 
 	err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
-		walkCancelWorkers, false)
+		walkCancelWorkers, false, false)
 	if !errors.Is(err, errInterrupted) {
 		t.Fatalf("runScan interrupted mid-hash = %v, want %v",
 			err, errInterrupted)
@@ -617,7 +620,8 @@ func TestWalkOneDirStopsWhenCancelled(t *testing.T) {
 	// Unbuffered and unread: on a cancelled scan every send gives up.
 	events := make(chan walkEvent)
 
-	subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false, events)
+	subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false,
+		walkRules{}, events)
 	if len(subs) != 0 {
 		t.Errorf("cancelled walkOneDir returned %+v to descend into, "+
 			"want none", subs)
@@ -637,7 +641,8 @@ func TestWalkWorkersDropQueuedDirs(t *testing.T) {
 
 	missing := filepath.Join(t.TempDir(), "missing")
 
-	jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false)
+	jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false,
+		walkRules{})
 
 	for range 64 {
 		jobs <- dirJob{path: missing}
@@ -668,7 +673,7 @@ func TestWalkWorkerAbandonsSubdirHandoff(t *testing.T) {
 	ctx, cancel := context.WithCancel(t.Context())
 	defer cancel()
 
-	jobs, subdirs, events := startWalkWorkers(ctx, 1, false)
+	jobs, subdirs, events := startWalkWorkers(ctx, 1, false, walkRules{})
 
 	// Fill the hand-back channel to its capacity — one slot per worker
 	// — so the worker's own hand-back is certain to block.

db.go

@@ -61,8 +61,8 @@ CREATE TABLE files (
 ) WITHOUT ROWID
 `
 
-// createIndexSQL indexes the records by signature, so report can have
-// SQLite group them without sorting the whole table.
+// createIndexSQL indexes the records by signature, so report can read
+// them in signature order without sorting the table.
 const createIndexSQL = `
 CREATE INDEX files_signature ON files (size, head, tail, content)
 `
@@ -79,7 +79,8 @@ ON CONFLICT (path) DO UPDATE SET
     content = excluded.content
 `
 
-// errNoDatabase reports a missing database file for report/trees.
+// errNoDatabase reports a missing database file for report, trees and
+// info.
 var errNoDatabase = errors.New(
 	"no database (run \"sfdupes scan\" first, or set " + databaseEnv + ")")
 
@@ -108,10 +109,8 @@ func databasePath() (string, string, error) {
 	}
 
 	// The working directory as the kernel reports it, with no symlink
-	// in it. os.Getwd may return PWD instead, which can name it through
-	// a symlink; a ".." after that symlink then leads somewhere else
-	// once the path is cleaned, as scan cleans it to create the
-	// database's directory.
+	// in it, as README "Database" says. os.Getwd may return PWD
+	// instead, which can name it through a symlink.
 	wd, err := unix.Getwd()
 	if err != nil {
 		return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
@@ -124,6 +123,24 @@ func databasePath() (string, string, error) {
 	return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
 }
 
+// createDatabaseDirectory creates the directory the database at path
+// goes in, and any directory above it, if missing. path is absolute, as
+// databasePath returns it. The directory is path with its last element
+// cut off as text, by filepath.Split. filepath.Dir would clean it,
+// dropping each ".." together with the directory name before it; when
+// that name is a symlink, the cleaned path names another directory than
+// the one the system reaches when it opens path.
+func createDatabaseDirectory(path string) error {
+	dir, _ := filepath.Split(path)
+
+	err := os.MkdirAll(dir, dbDirPerm)
+	if err != nil {
+		return fmt.Errorf("create database directory: %w", err)
+	}
+
+	return nil
+}
+
 // scanParams are the connection parameters for scan: read-write, with
 // WAL journaling and a busy timeout, so a report can run while a cron
 // scan is in progress. closeScanDatabase leaves WAL mode again.
@@ -131,9 +148,9 @@ const scanParams = "_pragma=busy_timeout(10000)" +
 	"&_pragma=journal_mode(WAL)" +
 	"&_pragma=synchronous(NORMAL)"
 
-// reportParams are the connection parameters for report and trees:
-// read-only, with the same busy timeout. They set no journal mode,
-// because setting one is a write.
+// reportParams are the connection parameters for report, trees and
+// info: read-only, with the same busy timeout. They set no journal
+// mode, because setting one is a write.
 const reportParams = "mode=ro" +
 	"&_pragma=busy_timeout(10000)" +
 	"&_pragma=query_only(1)"
@@ -178,9 +195,9 @@ func openDB(path, params string) (*sql.DB, error) {
 // that deleted it would let the next scan lock a new file while
 // another still holds the old one.
 func lockScanDatabase(path string) (*os.File, error) {
-	err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
+	err := createDatabaseDirectory(path)
 	if err != nil {
-		return nil, fmt.Errorf("create database directory: %w", err)
+		return nil, err
 	}
 
 	// Like SQLite, this follows a symlink at path and creates its
@@ -224,9 +241,9 @@ func lockScanDatabase(path string) (*os.File, error) {
 // openScanDatabase opens the database for the scan subcommand, creating
 // the file, its parent directory, and the schema as needed.
 func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
-	err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
+	err := createDatabaseDirectory(path)
 	if err != nil {
-		return nil, fmt.Errorf("create database directory: %w", err)
+		return nil, err
 	}
 
 	db, err := openDB(path, scanParams)
@@ -244,6 +261,42 @@ func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
 	return db, nil
 }
 
+// databaseFiles returns, by device and inode, the files of the database
+// at path that the scan keeps open: the database file, its -wal and
+// -shm files and its .lock file, each named from the database's path
+// with symlinks resolved, as SQLite and lockScanDatabase name them. The
+// scan calls it once the database and the lock file are open, so that
+// its walk passes over them however it reaches them, through a
+// symlinked directory included. A file that does not exist then is left
+// out, since no walk can reach it. There is no -journal file while the
+// database is in WAL mode.
+func databaseFiles(path string) ([]fileRec, error) {
+	resolved, err := filepath.EvalSymlinks(path)
+	if err != nil {
+		return nil, fmt.Errorf("resolve database path: %w", err)
+	}
+
+	var files []fileRec
+
+	for _, p := range []string{
+		resolved, resolved + "-wal", resolved + "-shm", resolved + ".lock",
+	} {
+		fi, err := os.Lstat(p)
+		if errors.Is(err, fs.ErrNotExist) {
+			continue
+		}
+
+		if err != nil {
+			return nil, fmt.Errorf("database: %w", err)
+		}
+
+		dev, ino := inodeOfInfo(fi)
+		files = append(files, fileRec{path: p, dev: dev, ino: ino})
+	}
+
+	return files, nil
+}
+
 // closeScanDatabase switches the database at path from WAL back to
 // rollback-journal mode and closes it. Out of WAL mode the database
 // file alone holds the whole database, so a reader needs no -wal or
@@ -262,9 +315,9 @@ func closeScanDatabase(ctx context.Context, db *sql.DB, path string) {
 	_ = db.Close()
 }
 
-// openReportDatabase opens an existing database for the report and
-// trees subcommands. A missing database file is an error directing the
-// user to run scan first; the schema version must match exactly.
+// openReportDatabase opens an existing database for the report, trees
+// and info subcommands. A missing database file is an error directing
+// the user to run scan first; the schema version must match exactly.
 func openReportDatabase(ctx context.Context,
 	path string,
 ) (*sql.DB, error) {
@@ -410,8 +463,8 @@ func userVersion(ctx context.Context, db *sql.DB) (int, error) {
 func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
 	fn func(r scanRec),
 ) error {
-	// Everything goes through tx: the report connection is the only
-	// one, so a query on db would wait for tx forever.
+	// Everything goes through tx: db has one connection, so a query on
+	// db would wait for tx forever.
 	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
 	if err != nil {
 		return fmt.Errorf("read records: %w", err)
@@ -463,77 +516,74 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
 	return nil
 }
 
-// dupeRowsSQL selects every record in a duplicate group, with the
-// group's first path. A group is the records with a content hash that
-// share a size, head, tail, and content, when there are two or more of
-// them. The rows come in report order: groups by size descending, then
-// by first path, and each group's paths ascending.
-const dupeRowsSQL = `
-SELECT g.first, f.path, f.size
-FROM files AS f
-JOIN (
-    SELECT size, head, tail, content, MIN(path) AS first
-    FROM files
-    WHERE content <> ''
-    GROUP BY size, head, tail, content
-    HAVING COUNT(*) > 1
-) AS g USING (size, head, tail, content)
-ORDER BY f.size DESC, g.first, f.path
+// signatureRowsSQL selects every record's signature and path in
+// signature order, largest size first. That is the order of the
+// files_signature index read backwards, so SQLite reads the index alone
+// and returns the first row at once, with nothing to sort.
+const signatureRowsSQL = `
+SELECT size, head, tail, content, path
+FROM files
+ORDER BY size DESC, head DESC, tail DESC, content DESC
 `
 
-// loadDupeRows streams the rows of dupeRowsSQL to fn and returns the
-// number of records in the database. The count and the rows are read
-// in one transaction, so they agree while a scan is committing. An
-// error from fn stops the reading and is returned as it is.
-func loadDupeRows(ctx context.Context, db *sql.DB,
-	fn func(first, path string, size int64) error,
-) (int, error) {
+// loadSignatureRows gives total the number of records, then streams
+// every record's signature and path to fn, without its mtime, in the
+// order of signatureRowsSQL. The count and the records are read in one
+// transaction, so they agree while a scan is committing. An error from
+// fn stops the reading and is returned as it is.
+func loadSignatureRows(ctx context.Context, db *sql.DB,
+	total func(n int64), fn func(r scanRec) error,
+) error {
 	// Everything goes through tx: the report connection is the only
 	// one, so a query on db would wait for tx forever.
 	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
 	if err != nil {
-		return 0, fmt.Errorf("read records: %w", err)
+		return fmt.Errorf("read records: %w", err)
 	}
 
 	defer func() { _ = tx.Rollback() }()
 
-	var records int
+	var n int64
 
-	err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&records)
+	err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
 	if err != nil {
-		return 0, fmt.Errorf("read records: %w", err)
+		return fmt.Errorf("count records: %w", err)
 	}
 
-	rows, err := tx.QueryContext(ctx, dupeRowsSQL)
+	total(n)
+
+	rows, err := tx.QueryContext(ctx, signatureRowsSQL)
 	if err != nil {
-		return 0, fmt.Errorf("read records: %w", err)
+		return fmt.Errorf("read records: %w", err)
 	}
 
 	defer func() { _ = rows.Close() }()
 
 	for rows.Next() {
 		var (
-			first, path []byte
-			size        int64
+			path []byte
+			r    scanRec
 		)
 
-		err = rows.Scan(&first, &path, &size)
+		err = rows.Scan(&r.size, &r.head, &r.tail, &r.content, &path)
 		if err != nil {
-			return 0, fmt.Errorf("read record: %w", err)
+			return fmt.Errorf("read record: %w", err)
 		}
 
-		err = fn(string(first), string(path), size)
+		r.path = string(path)
+
+		err = fn(r)
 		if err != nil {
-			return 0, err
+			return err
 		}
 	}
 
 	err = rows.Err()
 	if err != nil {
-		return 0, fmt.Errorf("read records: %w", err)
+		return fmt.Errorf("read records: %w", err)
 	}
 
-	return records, nil
+	return nil
 }
 
 // countRecords returns the number of records in the database.
@@ -548,6 +598,60 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
 	return n, nil
 }
 
+// databaseCountsSQL counts the records, those with head and tail
+// hashes and those with a content hash, and adds up their sizes, in one
+// pass over the table. A SUM over no rows is NULL, so on an empty table
+// each SUM falls back to 0.
+const databaseCountsSQL = `
+SELECT COUNT(*),
+       COALESCE(SUM(head <> '' AND tail <> ''), 0),
+       COALESCE(SUM(content <> ''), 0),
+       COALESCE(SUM(size), 0)
+FROM files
+`
+
+// databaseCounts is what info reads from the database: its schema
+// version and the counts of databaseCountsSQL.
+type databaseCounts struct {
+	schemaVersion int
+	records       int64 // every record
+	headTail      int64 // records with head and tail hashes
+	content       int64 // records with a content hash
+	size          int64 // the sizes of all records added up
+}
+
+// readDatabaseCounts reads the schema version, then the counts of
+// databaseCountsSQL, in one read-only transaction, so both describe the
+// database at one moment while a scan is committing.
+func readDatabaseCounts(ctx context.Context,
+	db *sql.DB,
+) (databaseCounts, error) {
+	var c databaseCounts
+
+	// Everything goes through tx: the report connection is the only
+	// one, so a query on db would wait for tx forever.
+	tx, err := db.BeginTx(ctx, &sql.TxOptions{ReadOnly: true})
+	if err != nil {
+		return c, fmt.Errorf("count records: %w", err)
+	}
+
+	defer func() { _ = tx.Rollback() }()
+
+	err = tx.QueryRowContext(ctx, "PRAGMA user_version").
+		Scan(&c.schemaVersion)
+	if err != nil {
+		return c, fmt.Errorf("read schema version: %w", err)
+	}
+
+	err = tx.QueryRowContext(ctx, databaseCountsSQL).
+		Scan(&c.records, &c.headTail, &c.content, &c.size)
+	if err != nil {
+		return c, fmt.Errorf("count records: %w", err)
+	}
+
+	return c, nil
+}
+
 // loadFileMeta streams every record's path, size, mtime, and whether
 // it carries hashes to fn. Scan change detection needs no hash
 // values, and skipping the hash columns keeps the scan's in-memory

db_test.go

@@ -5,6 +5,7 @@ import (
 	"database/sql"
 	"errors"
 	"fmt"
+	"io/fs"
 	"os"
 	"path/filepath"
 	"slices"
@@ -118,6 +119,116 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
 	}
 }
 
+// TestScanCreatesDatabaseDirectoryAfterSymlink has a database path go
+// through a symlink to a directory, then "..", to a missing directory.
+// With srv/link a symlink to data/x, srv/link/../new/db.sqlite leads to
+// data/new/db.sqlite. lockScanDatabase and openScanDatabase, the two
+// places scan creates the database's directory, must each create the
+// database there, and nothing at srv/new, where the path leads once
+// cleaned as text.
+func TestScanCreatesDatabaseDirectoryAfterSymlink(t *testing.T) {
+	t.Parallel()
+
+	creates := map[string]func(ctx context.Context, path string) error{
+		"lockScanDatabase": func(_ context.Context, path string) error {
+			lock, err := lockScanDatabase(path)
+			if err == nil {
+				_ = lock.Close()
+			}
+
+			return err
+		},
+		"openScanDatabase": func(ctx context.Context, path string) error {
+			db, err := openScanDatabase(ctx, path)
+			if err == nil {
+				_ = db.Close()
+			}
+
+			return err
+		},
+	}
+
+	for name, create := range creates {
+		t.Run(name, func(t *testing.T) {
+			t.Parallel()
+
+			dir, err := filepath.EvalSymlinks(t.TempDir())
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			err = os.MkdirAll(filepath.Join(dir, "data", "x"), 0o750)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			err = os.Mkdir(filepath.Join(dir, "srv"), 0o750)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			err = os.Symlink(filepath.Join(dir, "data", "x"),
+				filepath.Join(dir, "srv", "link"))
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			err = create(t.Context(), dir+"/srv/link/../new/db.sqlite")
+			if err != nil {
+				t.Fatalf("%s: %v", name, err)
+			}
+
+			_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
+			if err != nil {
+				t.Errorf("database the path leads to: %v", err)
+			}
+
+			_, err = os.Lstat(filepath.Join(dir, "srv", "new"))
+			if !errors.Is(err, fs.ErrNotExist) {
+				t.Errorf("srv/new: stat error %v, want it missing", err)
+			}
+		})
+	}
+}
+
+// TestDatabaseFiles opens a database through a symlink to its file:
+// databaseFiles finds the database file and its -wal and -shm files
+// beside the symlink's target, where SQLite keeps them, and leaves out
+// the .lock file, which only a scan creates.
+func TestDatabaseFiles(t *testing.T) {
+	t.Parallel()
+
+	path := testDBPath(t)
+	link := filepath.Join(t.TempDir(), "link.sqlite")
+
+	err := os.Symlink(path, link)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	db, err := openScanDatabase(t.Context(), link)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	defer func() { _ = db.Close() }()
+
+	files, err := databaseFiles(link)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	got := make([]string, 0, len(files))
+	for _, f := range files {
+		got = append(got, f.path)
+	}
+
+	want := []string{path, path + "-wal", path + "-shm"}
+	if !slices.Equal(got, want) {
+		t.Errorf("databaseFiles(%s) = %q, want %q", link, got, want)
+	}
+}
+
 func TestOpenDatabaseUnversionedForeign(t *testing.T) {
 	t.Parallel()
 
@@ -291,53 +402,118 @@ func TestCloseScanDatabaseWhileReportOpen(t *testing.T) {
 	}
 }
 
-// TestLoadFileRowsCountAgreesDuringScan has a scan commit records after
-// loadFileRows has counted the records and before it reads them, as
-// when trees runs during a scan: it must read exactly as many records
-// as it counted.
-func TestLoadFileRowsCountAgreesDuringScan(t *testing.T) {
+// TestLoadRowsCountAgreeDuringScan has a scan commit records after
+// loadFileRows, as trees calls it, or loadSignatureRows, as report
+// calls it, has counted the records and before it reads them, as when
+// trees or report runs during a scan: each must read exactly as many
+// records as it counted.
+func TestLoadRowsCountAgreeDuringScan(t *testing.T) {
 	t.Parallel()
 
-	path := testDBPath(t)
+	loads := map[string]func(context.Context, *sql.DB, func(int64),
+		func(scanRec)) error{
+		"loadFileRows": loadFileRows,
+		"loadSignatureRows": func(ctx context.Context, db *sql.DB,
+			total func(int64), fn func(scanRec),
+		) error {
+			return loadSignatureRows(ctx, db, total, func(r scanRec) error {
+				fn(r)
 
-	scanDB, err := openScanDatabase(t.Context(), path)
+				return nil
+			})
+		},
+	}
+
+	for name, load := range loads {
+		t.Run(name, func(t *testing.T) {
+			t.Parallel()
+
+			path := testDBPath(t)
+
+			scanDB, err := openScanDatabase(t.Context(), path)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			defer func() { _ = scanDB.Close() }()
+
+			recs := smokeTreeRecs()
+
+			err = applyChanges(t.Context(), scanDB, recs[:3], nil, nil)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			reportDB, err := openReportDatabase(t.Context(), path)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			defer func() { _ = reportDB.Close() }()
+
+			var counted, read int64
+
+			err = load(t.Context(), reportDB, func(n int64) {
+				counted = n
+
+				err := applyChanges(t.Context(), scanDB, recs[3:], nil, nil)
+				if err != nil {
+					t.Error(err)
+				}
+			}, func(scanRec) { read++ })
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			if counted != 3 || read != counted {
+				t.Errorf("counted %d records and read %d, want 3 and 3",
+					counted, read)
+			}
+		})
+	}
+}
+
+// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
+// from the files_signature index alone, with nothing to sort: the
+// first record comes at once, so report's read phase counts each
+// record as it comes rather than waiting on SQLite for all of them.
+func TestSignatureRowsNeedNoSort(t *testing.T) {
+	t.Parallel()
+
+	db := openTestDB(t)
+
+	rows, err := db.QueryContext(t.Context(),
+		"EXPLAIN QUERY PLAN "+signatureRowsSQL)
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	defer func() { _ = scanDB.Close() }()
+	defer func() { _ = rows.Close() }()
 
-	recs := smokeTreeRecs()
+	var plan []string
 
-	err = applyChanges(t.Context(), scanDB, recs[:3], nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
+	for rows.Next() {
+		var (
+			id, parent, unused int
+			detail             string
+		)
 
-	reportDB, err := openReportDatabase(t.Context(), path)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = reportDB.Close() }()
-
-	var counted, read int64
-
-	err = loadFileRows(t.Context(), reportDB, func(n int64) {
-		counted = n
-
-		err := applyChanges(t.Context(), scanDB, recs[3:], nil, nil)
+		err = rows.Scan(&id, &parent, &unused, &detail)
 		if err != nil {
-			t.Error(err)
+			t.Fatal(err)
 		}
-	}, func(scanRec) { read++ })
+
+		plan = append(plan, detail)
+	}
+
+	err = rows.Err()
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	if counted != 3 || read != counted {
-		t.Errorf("counted %d records and read %d, want 3 and 3",
-			counted, read)
+	want := []string{"SCAN files USING COVERING INDEX files_signature"}
+	if !slices.Equal(plan, want) {
+		t.Errorf("query plan = %q, want %q", plan, want)
 	}
 }
 

exclude.go

@@ -0,0 +1,195 @@
+package main
+
+// This file holds two rules for the walk (README "Rules for the walk"):
+// the default excludes, the paths a scan's walk does not enter below an
+// operand unless scan is given --no-default-excludes, and the
+// filesystems that are not persistent, on which the scan never records
+// or reads anything.
+
+import (
+	"bufio"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"path/filepath"
+	"slices"
+	"strings"
+
+	"golang.org/x/sys/unix"
+)
+
+// The systems that have default excludes, as runtime.GOOS names them.
+const (
+	goosLinux   = "linux"
+	goosDarwin  = "darwin"
+	goosWindows = "windows"
+)
+
+// The fields of a line of mountInfoPath that readMountInfo reads,
+// counted from 0 (see proc(5)): the device ID, as major:minor, is the
+// third field, and after the sixth come optional fields, then a field
+// holding "-", then the filesystem type.
+const (
+	deviceField         = 2
+	optionalFieldsStart = 6
+)
+
+// typesPerHelpLine is how many filesystem types the scan command's help
+// lists on each line.
+const typesPerHelpLine = 6
+
+// errMountInfoLine reports a line of mountInfoPath without the fields
+// readMountInfo reads.
+var errMountInfoLine = errors.New("unexpected line")
+
+// walkRules holds the rules for the walk that a scan sets up as it
+// starts. excludes are the default excludes, or none with
+// --no-default-excludes. nonPersistent maps the device ID of every
+// mounted filesystem that is not persistent to its type; the scan
+// records and reads nothing on one of those, whatever the flags.
+// databaseFiles are the scan's own database files, which the walk
+// passes over (see databaseFiles in db.go).
+type walkRules struct {
+	excludes      []string
+	nonPersistent map[uint64]string
+	databaseFiles []fileRec
+}
+
+// defaultExcludes returns the default excludes of the system goos
+// names, as runtime.GOOS does: a fixed list on Linux, macOS and
+// Windows, and none on any other system.
+func defaultExcludes(goos string) []string {
+	switch goos {
+	case goosLinux:
+		return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
+	case goosDarwin:
+		// A walk of / reaches everything in /System/Volumes/Data through
+		// the firmlinks, so walking it as well would record each of
+		// those files twice.
+		return []string{
+			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
+			"/System/Volumes/VM", "/System/Volumes/Data",
+		}
+	case goosWindows:
+		// Relative to the root of the drive an operand lies on. The
+		// first three are the page, swap and hibernation files.
+		return []string{
+			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
+			"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
+			`Users\*\AppData\Local\Temp`,
+		}
+	default:
+		return nil
+	}
+}
+
+// isDefaultExclude reports whether path is one of excludes, in which a
+// * matches one path component.
+func isDefaultExclude(path string, excludes []string) bool {
+	for _, pattern := range excludes {
+		// The patterns are fixed, so none is malformed.
+		if ok, _ := filepath.Match(pattern, path); ok {
+			return true
+		}
+	}
+
+	return false
+}
+
+// nonPersistentFilesystemTypes returns the filesystem types that are
+// not persistent: filesystems that the kernel makes up or that are kept
+// in memory. devfs is the macOS filesystem of /dev; the others are
+// Linux names.
+func nonPersistentFilesystemTypes() []string {
+	return []string{
+		"proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "ramfs",
+		"cgroup", "cgroup2", "debugfs", "tracefs", "securityfs", "pstore",
+		"bpf", "configfs", "fusectl", "mqueue", "hugetlbfs", "efivarfs",
+		"binfmt_misc", "autofs", "nsfs", "rpc_pipefs", "selinuxfs", "devfs",
+	}
+}
+
+// nonPersistentLine returns the line the scan prints for path when fi,
+// its lstat, puts it on a filesystem in rules.nonPersistent, and ""
+// when it is on any other.
+func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
+	dev, ok := deviceOfInfo(fi)
+	if !ok {
+		return ""
+	}
+
+	fsType, found := rules.nonPersistent[dev]
+	if !found {
+		return ""
+	}
+
+	return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
+		path, fsType)
+}
+
+// readMountInfo reads mountinfo, laid out as mountInfoPath is, and
+// returns the device ID of every mounted filesystem that is not
+// persistent, with its type. The device ID is the one each line gives,
+// not that of the file now at its mount point, so a mount that a later
+// mount hides never adds the device of another filesystem.
+func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
+	types := nonPersistentFilesystemTypes()
+	devices := make(map[uint64]string)
+	lines := bufio.NewScanner(mountinfo)
+
+	for lines.Scan() {
+		fields := strings.Fields(lines.Text())
+
+		sep := slices.Index(fields, "-")
+		if sep < optionalFieldsStart || sep+1 >= len(fields) {
+			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
+		}
+
+		var major, minor uint32
+
+		_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
+		if err != nil {
+			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
+		}
+
+		fsType := fields[sep+1]
+		if slices.Contains(types, fsType) {
+			devices[unix.Mkdev(major, minor)] = fsType
+		}
+	}
+
+	return devices, lines.Err()
+}
+
+// defaultExcludesHelp describes, for the scan command's help, the
+// default excludes of the system goos names and, where the scan lists
+// the mounted filesystems, the types it never records anything on.
+func defaultExcludesHelp(goos string) string {
+	var help strings.Builder
+
+	dirs := defaultExcludes(goos)
+	if len(dirs) == 0 {
+		help.WriteString("This system has no default excludes.\n")
+	} else {
+		help.WriteString("Below an operand, the walk does not enter these " +
+			"default excludes, unless\n--no-default-excludes is given:\n\n")
+
+		for _, dir := range dirs {
+			fmt.Fprintf(&help, "  %s\n", dir)
+		}
+	}
+
+	if goos == goosLinux || goos == goosDarwin {
+		help.WriteString("\nWith or without --no-default-excludes, the scan " +
+			"records nothing on a\nfilesystem that is not persistent, " +
+			"one of these types:\n\n")
+
+		types := nonPersistentFilesystemTypes()
+		for line := range slices.Chunk(types, typesPerHelpLine) {
+			fmt.Fprintf(&help, "  %s\n", strings.Join(line, ", "))
+		}
+	}
+
+	return help.String()
+}

exclude_darwin.go

@@ -0,0 +1,52 @@
+//go:build darwin
+
+package main
+
+import (
+	"slices"
+
+	"golang.org/x/sys/unix"
+)
+
+// nonPersistentFilesystems returns the device ID of every mounted
+// filesystem that is not persistent, with its type, from the system's
+// list of mounted filesystems. The device ID is read with lstat on the
+// mount point. Where one filesystem is mounted over another on the same
+// mount point, the later one in the list counts. A mount point that
+// lstat cannot reach is left out.
+func nonPersistentFilesystems() (map[uint64]string, error) {
+	// The first call counts the filesystems, the second lists them.
+	n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
+	if err != nil {
+		return nil, err
+	}
+
+	list := make([]unix.Statfs_t, n)
+
+	n, err = unix.Getfsstat(list, unix.MNT_NOWAIT)
+	if err != nil {
+		return nil, err
+	}
+
+	mounts := make(map[string]string, n)
+
+	for _, st := range list[:n] {
+		mounts[unix.ByteSliceToString(st.Mntonname[:])] =
+			unix.ByteSliceToString(st.Fstypename[:])
+	}
+
+	types := nonPersistentFilesystemTypes()
+	devices := make(map[uint64]string)
+
+	for mountPoint, fsType := range mounts {
+		if !slices.Contains(types, fsType) {
+			continue
+		}
+
+		if dev, ok := deviceOfPath(mountPoint); ok {
+			devices[dev] = fsType
+		}
+	}
+
+	return devices, nil
+}

exclude_other.go

@@ -0,0 +1,36 @@
+//go:build !darwin
+
+package main
+
+import (
+	"fmt"
+	"os"
+	"runtime"
+)
+
+// mountInfoPath is the file in which Linux lists the mounts the reading
+// process sees.
+const mountInfoPath = "/proc/self/mountinfo"
+
+// nonPersistentFilesystems returns the device ID of every mounted
+// filesystem that is not persistent, with its type: on Linux read from
+// mountInfoPath (see readMountInfo), and on any other system none.
+func nonPersistentFilesystems() (map[uint64]string, error) {
+	if runtime.GOOS != goosLinux {
+		return map[uint64]string{}, nil
+	}
+
+	f, err := os.Open(mountInfoPath)
+	if err != nil {
+		return nil, err
+	}
+
+	defer func() { _ = f.Close() }()
+
+	devices, err := readMountInfo(f)
+	if err != nil {
+		return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
+	}
+
+	return devices, nil
+}

exclude_test.go

@@ -0,0 +1,583 @@
+package main
+
+import (
+	"bytes"
+	"errors"
+	"maps"
+	"os"
+	"path/filepath"
+	"runtime"
+	"slices"
+	"strings"
+	"testing"
+
+	"golang.org/x/sys/unix"
+)
+
+func TestDefaultExcludes(t *testing.T) {
+	t.Parallel()
+
+	// README "Rules for the walk": each system's list.
+	cases := map[string][]string{
+		goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
+		goosDarwin: {
+			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
+			"/System/Volumes/VM", "/System/Volumes/Data",
+		},
+		goosWindows: {
+			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
+			"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
+			`Users\*\AppData\Local\Temp`,
+		},
+		"freebsd": nil,
+	}
+
+	for goos, want := range cases {
+		if got := defaultExcludes(goos); !slices.Equal(got, want) {
+			t.Errorf("defaultExcludes(%q) = %q, want %q", goos, got, want)
+		}
+	}
+}
+
+func TestIsDefaultExclude(t *testing.T) {
+	t.Parallel()
+
+	// A * matches one path component, no more and no fewer.
+	excludes := []string{"/scratch", "/home/*/cache"}
+
+	cases := map[string]bool{
+		"/scratch":        true,
+		"/scratch/x":      false,
+		"/home/u/cache":   true,
+		"/home/u/v/cache": false,
+		"/home/cache":     false,
+	}
+
+	for path, want := range cases {
+		if got := isDefaultExclude(path, excludes); got != want {
+			t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
+		}
+	}
+}
+
+func TestReadMountInfo(t *testing.T) {
+	t.Parallel()
+
+	// The second-to-last line has no optional fields. The tmpfs on
+	// /data/a/b is hidden by the root filesystem's /srv, mounted over
+	// /data/a after it: the tmpfs still gives its own device, and the
+	// root filesystem's device is not listed.
+	mountinfo := strings.Join([]string{
+		"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
+		"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
+		"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
+		"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw",
+		"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw",
+		"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
+	}, "\n")
+
+	got, err := readMountInfo(strings.NewReader(mountinfo))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	want := map[uint64]string{
+		unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs",
+		unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs",
+	}
+	if !maps.Equal(got, want) {
+		t.Errorf("readMountInfo = %v, want %v", got, want)
+	}
+
+	for _, line := range []string{
+		"22 1 8:1 / / rw,relatime\n",
+		"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n",
+	} {
+		_, err = readMountInfo(strings.NewReader(line))
+		if !errors.Is(err, errMountInfoLine) {
+			t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
+				errMountInfoLine)
+		}
+	}
+}
+
+func TestNonPersistentFilesystems(t *testing.T) {
+	t.Parallel()
+
+	if runtime.GOOS != goosLinux {
+		t.Skip("the test knows a mount only on Linux: /proc")
+	}
+
+	nonPersistent, err := nonPersistentFilesystems()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	dev, ok := deviceOfPath("/proc")
+	if !ok {
+		t.Fatal("no device for /proc")
+	}
+
+	if got := nonPersistent[dev]; got != "proc" {
+		t.Errorf("filesystem of /proc = %q, want %q", got, "proc")
+	}
+}
+
+func TestDefaultExcludesHelp(t *testing.T) {
+	t.Parallel()
+
+	for _, goos := range []string{goosLinux, goosDarwin, goosWindows} {
+		help := defaultExcludesHelp(goos)
+
+		for _, dir := range defaultExcludes(goos) {
+			if !strings.Contains(help, "\n  "+dir+"\n") {
+				t.Errorf("help for %s does not list %s:\n%s", goos, dir, help)
+			}
+		}
+
+		// Only Linux and macOS list their mounted filesystems.
+		for _, fsType := range nonPersistentFilesystemTypes() {
+			listed := strings.Contains(help, " "+fsType+",") ||
+				strings.Contains(help, " "+fsType+"\n")
+			if listed != (goos != goosWindows) {
+				t.Errorf("help for %s lists %s: %v", goos, fsType, listed)
+			}
+		}
+	}
+
+	want := "This system has no default excludes.\n"
+	if got := defaultExcludesHelp("freebsd"); got != want {
+		t.Errorf("help for freebsd = %q, want %q", got, want)
+	}
+
+	// scan's help shows those of the system it runs on.
+	var stdout, stderr bytes.Buffer
+
+	run([]string{cmdScan, "-h"}, &stdout, &stderr)
+
+	help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
+	if !strings.Contains(stderr.String(), help) {
+		t.Errorf("scan --help = %q, want it to hold %q", stderr.String(), help)
+	}
+}
+
+// TestScanDoesNotEnterDefaultExclude checks that the walk does not
+// enter a default exclude below an operand. It says so on a line of its
+// own and does not count it as skipped. The records below it count as
+// outside the operands: the record of a file gone from it is kept, and
+// its size joins the size census, while a file added to it is not
+// recorded.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
+	stderr := captureStderr(t)
+	dir := t.TempDir()
+	db := openTestDB(t)
+	excluded := filepath.Join(dir, "tmp")
+
+	// A scan with no default excludes records the file in excluded.
+	gone := writeFile(t, excluded, "gone", pattern(1, 100))
+	syncTree(t, db, dir)
+
+	err := os.Remove(gone)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	unseen := writeFile(t, excluded, "new", pattern(2, 10))
+
+	// The same size as gone, so it is hashed only if the census holds the
+	// size of gone's record.
+	added := writeFile(t, dir, "added", pattern(3, 100))
+
+	st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
+		walkRules{excludes: []string{excluded}})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if st.skipped != 0 {
+		t.Errorf("skipped = %d, want 0", st.skipped)
+	}
+
+	recs := dbRecords(t, db)
+	recordByPath(t, recs, gone)
+
+	if recordByPath(t, recs, added).head == "" {
+		t.Errorf("%s not hashed: the census lacks the size of %s",
+			added, gone)
+	}
+
+	if slices.Contains(recordPaths(recs), unseen) {
+		t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
+	}
+
+	line := "\nscan: skipped " + excluded + " (default exclude)\n"
+	if !strings.Contains(stderr(), line) {
+		t.Errorf("stderr = %q, want %q", stderr(), line)
+	}
+}
+
+// TestScanWalksOperandInDefaultExclude checks that an operand at or
+// inside a default exclude is walked, also when it lies under another
+// operand, whose walk does not enter the default exclude.
+func TestScanWalksOperandInDefaultExclude(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	excluded := filepath.Join(dir, "tmp")
+	f := writeFile(t, excluded, "sub/f", pattern(1, 10))
+	sub := filepath.Dir(f)
+
+	cases := map[string][]string{
+		"at it":                      {excluded},
+		"inside it":                  {sub},
+		"inside it under an operand": {dir, sub},
+	}
+
+	for name, roots := range cases {
+		t.Run(name, func(t *testing.T) {
+			t.Parallel()
+
+			db := openTestDB(t)
+
+			st, err := syncScan(t.Context(), db, roots, 1, false,
+				walkRules{excludes: []string{excluded}})
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			if st.added != 1 {
+				t.Errorf("added = %d, want %s recorded once", st.added, f)
+			}
+
+			recordByPath(t, dbRecords(t, db), f)
+		})
+	}
+}
+
+// TestScanNamesNoDefaultExcludeThatIsAnOperand checks that a default
+// exclude given as an operand under another operand, which the walk of
+// that other operand does not enter, gets no line saying it was
+// skipped: it is walked as an operand of its own.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
+	stderr := captureStderr(t)
+	dir := t.TempDir()
+	db := openTestDB(t)
+	excluded := filepath.Join(dir, "tmp")
+	f := writeFile(t, excluded, "f", pattern(1, 10))
+
+	st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
+		walkRules{excludes: []string{excluded}})
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if st.added != 1 {
+		t.Errorf("added = %d, want %s recorded once", st.added, f)
+	}
+
+	recordByPath(t, dbRecords(t, db), f)
+
+	if strings.Contains(stderr(), "scan: skipped") {
+		t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
+	}
+}
+
+// TestRunScanTempDirOperand scans a tree in the tests' temporary
+// directory, which on Linux lies inside the default exclude /tmp unless
+// TMPDIR names another place, with the system's own default excludes:
+// the scan walks the whole tree and reports no default exclude.
+func TestRunScanTempDirOperand(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := t.TempDir()
+	if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
+		t.Skip(dir + " is not inside a default exclude")
+	}
+
+	f := writeFile(t, dir, "sub/f", pattern(1, 10))
+
+	stderr := scanOK(t, dir)
+	if strings.Contains(stderr, "scan: skipped") {
+		t.Errorf("stderr = %q, want no line saying a path was skipped",
+			stderr)
+	}
+
+	assertRecorded(t, path, f, true)
+}
+
+// TestRunScanNoDefaultExcludes scans /var through the command. Its
+// /var/tmp, on a persistent filesystem, is a default exclude on Linux:
+// without --no-default-excludes the walk does not enter it and says so,
+// and with that flag the walk records a file the test puts there.
+func TestRunScanNoDefaultExcludes(t *testing.T) {
+	if runtime.GOOS != goosLinux {
+		t.Skip("only Linux has /var/tmp among its default excludes")
+	}
+
+	nonPersistent, err := nonPersistentFilesystems()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	rules := walkRules{nonPersistent: nonPersistent}
+
+	for _, dir := range []string{"/var", "/var/tmp"} {
+		fi, lstatErr := os.Lstat(dir)
+		if lstatErr != nil || !fi.IsDir() ||
+			rules.nonPersistentLine(dir, fi) != "" {
+			t.Skip(dir + " is not a directory on a persistent filesystem here")
+		}
+	}
+
+	//nolint:usetesting // the directory must be inside /var/tmp
+	tmp, err := os.MkdirTemp("/var/tmp", "sfdupes-test-")
+	if err != nil {
+		t.Skip("cannot write to /var/tmp here: " + err.Error())
+	}
+
+	t.Cleanup(func() { _ = os.RemoveAll(tmp) })
+
+	f := writeFile(t, tmp, "f", pattern(1, 10))
+
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	line := "scan: skipped /var/tmp (default exclude)\n"
+
+	if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
+		t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
+	}
+
+	assertRecorded(t, path, f, false)
+
+	stderr := scanOK(t, "--no-default-excludes", "/var")
+	if strings.Contains(stderr, line) {
+		t.Errorf("scan --no-default-excludes /var: stderr = %q, want no %q",
+			stderr, line)
+	}
+
+	assertRecorded(t, path, f, true)
+}
+
+// TestRunScanSkipsNonPersistentOperand scans, through the command, a
+// directory on a filesystem that is not persistent, with and without
+// --no-default-excludes: either way it is skipped, with a line saying
+// so, and nothing in it is recorded.
+func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
+	shm, fsType, _ := shmTempDir(t)
+	writeFile(t, shm, "f", pattern(1, 10))
+
+	t.Setenv(databaseEnv, testDBPath(t))
+
+	line := "scan: skipped " + shm + " (" + fsType +
+		", not a persistent filesystem)\n"
+	summary := "scan: 0 files seen"
+
+	for _, flags := range [][]string{nil, {"--no-default-excludes"}} {
+		stderr := scanOK(t, append(flags, shm)...)
+		if !strings.Contains(stderr, line) ||
+			!strings.Contains(stderr, summary) {
+			t.Errorf("scan %q: stderr = %q, want %q and %q", flags, stderr,
+				line, summary)
+		}
+	}
+}
+
+// TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem
+// that is not persistent beside one on the tests' temporary directory.
+// The records an earlier scan left in it count as outside the operands:
+// the record of a file gone from it is kept, and its size joins the size
+// census, while a file added to it is not recorded, and its size does
+// not join the census.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
+	shm, fsType, rules := shmTempDir(t)
+	stderr := captureStderr(t)
+	dir := t.TempDir()
+	db := openTestDB(t)
+
+	// A scan that, without the list of filesystems, records gone.
+	gone := writeFile(t, shm, "gone", pattern(1, 100))
+	syncTree(t, db, shm)
+
+	err := os.Remove(gone)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	unseen := writeFile(t, shm, "new", pattern(2, 10))
+
+	// added has the size of gone, and lonely that of unseen: each is
+	// hashed only if the census holds that size.
+	added := writeFile(t, dir, "added", pattern(3, 100))
+	lonely := writeFile(t, dir, "lonely", pattern(4, 10))
+
+	st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if st.skipped != 0 {
+		t.Errorf("skipped = %d, want 0", st.skipped)
+	}
+
+	recs := dbRecords(t, db)
+	recordByPath(t, recs, gone)
+
+	if recordByPath(t, recs, added).head == "" {
+		t.Errorf("%s not hashed: the census lacks the size of %s",
+			added, gone)
+	}
+
+	if recordByPath(t, recs, lonely).head != "" {
+		t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen)
+	}
+
+	if slices.Contains(recordPaths(recs), unseen) {
+		t.Errorf("%s recorded", unseen)
+	}
+
+	line := "scan: skipped " + shm + " (" + fsType +
+		", not a persistent filesystem)\n"
+	if !strings.Contains(stderr(), line) {
+		t.Errorf("stderr = %q, want %q", stderr(), line)
+	}
+}
+
+// TestScanContentSkipsNonPersistent checks that the content phase does
+// not read a file on a filesystem that is not persistent, though an
+// earlier scan left a record for it: it is treated as a changed file,
+// so its record is left as it is and its identical copy in another
+// tree, having no other match, is not read either.
+func TestScanContentSkipsNonPersistent(t *testing.T) {
+	t.Parallel()
+
+	shm, _, rules := shmTempDir(t)
+	db := openTestDB(t)
+
+	// A scan that, without the list of filesystems, records a with a
+	// head and tail but no content hash.
+	a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
+
+	syncTree(t, db, shm)
+
+	before := recordByPath(t, dbRecords(t, db), a)
+	if before.head == "" || before.content != "" {
+		t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
+	}
+
+	b := sparseFile(t, t.TempDir(), "b", headTailMin)
+
+	st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
+		rules)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if st.skipped != 0 {
+		t.Errorf("skipped = %d, want 0", st.skipped)
+	}
+
+	recs := dbRecords(t, db)
+	if r := recordByPath(t, recs, a); r != before {
+		t.Errorf("record = %+v, want it left as %+v", r, before)
+	}
+
+	if r := recordByPath(t, recs, b); r.content != "" {
+		t.Errorf("%s: content = %q, want none: its only match is on %s",
+			b, r.content, shm)
+	}
+}
+
+// TestWalkLeavesOutNonPersistent starts the walk itself in a directory
+// on a filesystem that is not persistent, which a scan never does: it
+// drops such an operand first. Below a directory on a persistent
+// filesystem, one on another takes a mount, which a test cannot make.
+// The walk enters no directory and records no file there, and names
+// each.
+func TestWalkLeavesOutNonPersistent(t *testing.T) {
+	t.Parallel()
+
+	shm, fsType, rules := shmTempDir(t)
+	f := writeFile(t, shm, "f", pattern(1, 10))
+	sub := filepath.Dir(writeFile(t, shm, "sub/g", pattern(1, 10)))
+
+	var leftOut []string
+
+	for ev := range startWalk(t.Context(), []string{shm}, false, rules, 1) {
+		if ev.rec.path != "" {
+			t.Errorf("walk recorded %s", ev.rec.path)
+		}
+
+		if ev.leftOut == "" {
+			continue
+		}
+
+		leftOut = append(leftOut, ev.leftOut)
+
+		want := "scan: skipped " + ev.leftOut + " (" + fsType +
+			", not a persistent filesystem)"
+		if ev.warn != want {
+			t.Errorf("line = %q, want %q", ev.warn, want)
+		}
+	}
+
+	slices.Sort(leftOut)
+
+	if want := []string{f, sub}; !slices.Equal(leftOut, want) {
+		t.Errorf("walk left out %q, want %q", leftOut, want)
+	}
+}
+
+// shmTempDir returns a new directory in /dev/shm, removed when the test
+// ends, the type of its filesystem, and the rules holding the real
+// filesystems that are not persistent. It skips the test where /dev/shm
+// is on a persistent filesystem, or cannot be written to.
+func shmTempDir(t *testing.T) (string, string, walkRules) {
+	t.Helper()
+
+	nonPersistent, err := nonPersistentFilesystems()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	dev, ok := deviceOfPath("/dev/shm")
+
+	fsType, found := nonPersistent[dev]
+	if !ok || !found {
+		t.Skip("/dev/shm is not a memory-backed filesystem here")
+	}
+
+	//nolint:usetesting // the directory must be inside /dev/shm
+	dir, err := os.MkdirTemp("/dev/shm", "sfdupes-test-")
+	if err != nil {
+		t.Skip("cannot write to /dev/shm here: " + err.Error())
+	}
+
+	t.Cleanup(func() { _ = os.RemoveAll(dir) })
+
+	return dir, fsType, walkRules{nonPersistent: nonPersistent}
+}
+
+// assertRecorded checks that the database at dbPath holds a record for
+// path when want is set, and none when it is not.
+func assertRecorded(t *testing.T, dbPath, path string, want bool) {
+	t.Helper()
+
+	db, err := openDB(dbPath, reportParams)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	defer func() { _ = db.Close() }()
+
+	if got := slices.Contains(recordPaths(dbRecords(t, db)), path); got != want {
+		t.Errorf("%s recorded: %v, want %v", path, got, want)
+	}
+}

go.mod

@@ -24,3 +24,5 @@ require (
 	modernc.org/mathutil v1.7.1 // indirect
 	modernc.org/memory v1.11.0 // indirect
 )
+
+retract v0.0.1 // early build without a database; does not work as the README describes

info.go

@@ -0,0 +1,128 @@
+package main
+
+import (
+	"context"
+	"errors"
+	"fmt"
+	"io"
+	"io/fs"
+	"os"
+	"path/filepath"
+	"strings"
+	"time"
+)
+
+// infoPhase names one phase of info with its place among all of them,
+// as "[2/2] count", for the phase's progress display. The list here is
+// the one place the phases are counted.
+func infoPhase(name string) string {
+	return phaseLabel([]string{openPhase, "count"}, name)
+}
+
+// runInfo implements the info subcommand: it prints the database's
+// details on stdout (see writeInfo). It opens the database as report
+// does, read-only and without the scan lock, and never touches the
+// scanned filesystem. It names the database on stderr before it opens
+// it. Any database problem, including a missing database, is fatal.
+//
+// The work runs in two phases, each under a progress display of its
+// own on stderr: open (open the database) and count (count the records,
+// with one query). Each display ends before the next phase starts, and
+// before an error is returned to be printed. The details are written
+// once the last display has ended, so no display is drawn while they
+// are written.
+func runInfo(ctx context.Context, stdout io.Writer) error {
+	dbPath, source, err := databasePath()
+	if err != nil {
+		return err
+	}
+
+	fmt.Fprintf(os.Stderr, "info: database %s (%s)\n",
+		escapePath(dbPath), source)
+
+	// Opening can wait on a busy database for as long as the busy
+	// timeout.
+	prog := newProgress(progressParams{
+		label: infoPhase(openPhase), total: -1,
+		waitingFor: openingDatabase,
+	})
+	db, err := openReportDatabase(ctx, dbPath)
+
+	prog.finish()
+
+	if err != nil {
+		return err
+	}
+
+	defer func() { _ = db.Close() }()
+
+	// One query counts everything, so the phase never has a total.
+	prog = newProgress(progressParams{
+		label: infoPhase("count"), total: -1,
+		waitingFor: countingRecords,
+	})
+	counts, err := readDatabaseCounts(ctx, db)
+
+	prog.finish()
+
+	if err != nil {
+		return fmt.Errorf("database %s: %w", dbPath, err)
+	}
+
+	return writeInfo(stdout, dbPath, source, counts)
+}
+
+// writeInfo writes the details of the database at dbPath to stdout,
+// one per line, as a name, a tab and the value (README "`info` mode"):
+// the path and source as databasePath returns them, the sizes of the
+// database file and of its -wal file when there is one, when the
+// database file was last changed, and the schema version and record
+// counts that readDatabaseCounts read.
+func writeInfo(stdout io.Writer, dbPath, source string,
+	counts databaseCounts,
+) error {
+	// SQLite keeps the -wal file beside the database file, named from
+	// its path with symlinks resolved.
+	file, err := filepath.EvalSymlinks(dbPath)
+	if err != nil {
+		return fmt.Errorf("database: %w", err)
+	}
+
+	dbFile, err := os.Stat(file)
+	if err != nil {
+		return fmt.Errorf("database: %w", err)
+	}
+
+	walFile, err := os.Stat(file + "-wal")
+	if err != nil && !errors.Is(err, fs.ErrNotExist) {
+		return fmt.Errorf("database: %w", err)
+	}
+
+	var b strings.Builder
+
+	fmt.Fprintf(&b, "database\t%s (%s)\n", escapePath(dbPath), source)
+	fmt.Fprintf(&b, "database file size\t%d (%s)\n",
+		dbFile.Size(), humanBytes(dbFile.Size()))
+
+	if walFile != nil {
+		fmt.Fprintf(&b, "-wal file size\t%d (%s)\n",
+			walFile.Size(), humanBytes(walFile.Size()))
+	}
+
+	fmt.Fprintf(&b, "modified\t%s\n",
+		dbFile.ModTime().UTC().Format(time.RFC3339))
+	fmt.Fprintf(&b, "schema version\t%d\n", counts.schemaVersion)
+	fmt.Fprintf(&b, "records\t%d\n", counts.records)
+	fmt.Fprintf(&b, "records with head and tail hashes\t%d\n",
+		counts.headTail)
+	fmt.Fprintf(&b, "records with content hash\t%d\n", counts.content)
+	fmt.Fprintf(&b, "size of recorded files\t%d (%s)\n",
+		counts.size, humanBytes(counts.size))
+
+	_, err = io.WriteString(stdout, b.String())
+	if err != nil {
+		return fmt.Errorf("write stdout: %w", err)
+	}
+
+	return nil
+}

info_test.go

@@ -0,0 +1,168 @@
+package main
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+)
+
+// infoOpenLine is the plain line of info's open phase when it opens the
+// database at once; it prints it as the phase starts and as it ends.
+const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
+	"eta unknown"
+
+// infoCountLine is the plain line of info's count phase when the count
+// takes under a second; it prints it as the phase starts and as it
+// ends.
+const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " +
+	"eta unknown"
+
+// infoRecs are four records: a duplicate pair with every hash, a file
+// of 20 MiB with head and tail hashes but no content hash yet, and a
+// file never hashed.
+func infoRecs() []scanRec {
+	return []scanRec{
+		{size: 5, head: "h", tail: "t", content: "c", path: "/d/a"},
+		{size: 5, head: "h", tail: "t", content: "c", path: "/d/b"},
+		{size: 20 << 20, head: "bh", tail: "bt", path: "/d/big"},
+		{size: 7, path: "/d/unique"},
+	}
+}
+
+// TestRunInfo runs info on an empty and on a filled database, with
+// stdout and stderr the same file that is not a terminal, and checks
+// everything written there: the line naming the database, each phase's
+// line as it starts and as it ends, and then the details, each on a
+// line of its own. No -wal file is beside a database scan has closed.
+func TestRunInfo(t *testing.T) {
+	cases := map[string]struct {
+		recs []scanRec
+		want string // the details after the database file's time
+	}{
+		"empty": {
+			want: "schema version\t1\n" +
+				"records\t0\n" +
+				"records with head and tail hashes\t0\n" +
+				"records with content hash\t0\n" +
+				"size of recorded files\t0 (0 B)\n",
+		},
+		"filled": {
+			recs: infoRecs(),
+			want: "schema version\t1\n" +
+				"records\t4\n" +
+				"records with head and tail hashes\t3\n" +
+				"records with content hash\t2\n" +
+				"size of recorded files\t20971537 (20.0 MiB)\n",
+		},
+	}
+
+	for name, tc := range cases {
+		t.Run(name, func(t *testing.T) {
+			path := seedDatabase(t, tc.recs)
+			t.Setenv(databaseEnv, path)
+
+			file, err := os.Stat(path)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			stderr := captureStderr(t)
+
+			code := run([]string{cmdInfo}, os.Stderr, os.Stderr)
+			if code != exitOK {
+				t.Fatalf("run(info) = %d, want %d; stderr: %s",
+					code, exitOK, stderr())
+			}
+
+			named := path + " (from " + databaseEnv + ")\n"
+			want := "info: database " + named +
+				infoOpenLine + "\n" + infoOpenLine + "\n" +
+				infoCountLine + "\n" + infoCountLine + "\n" +
+				"database\t" + named +
+				fmt.Sprintf("database file size\t%d (%s)\n",
+					file.Size(), humanBytes(file.Size())) +
+				"modified\t" + file.ModTime().UTC().Format(time.RFC3339) +
+				"\n" + tc.want
+
+			if got := stderr(); got != want {
+				t.Errorf("stdout and stderr = %q, want %q", got, want)
+			}
+		})
+	}
+}
+
+// TestRunInfoDuringScan runs info while a scan holds the lock and has
+// the database open in WAL mode, with records committed to the -wal
+// file only. info must count those records and show the -wal file's
+// size, also when the database path is a symlink: SQLite keeps the
+// -wal file beside the symlink's target.
+func TestRunInfoDuringScan(t *testing.T) {
+	path := testDBPath(t)
+	link := filepath.Join(t.TempDir(), "link.sqlite")
+
+	err := os.Symlink(path, link)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	db, err := openScanDatabase(t.Context(), path)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	t.Cleanup(func() { _ = db.Close() })
+
+	err = applyChanges(t.Context(), db, infoRecs(), nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	holdScanLock(t, path)
+
+	wal, err := os.Stat(path + "-wal")
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	for _, named := range []string{path, link} {
+		t.Run(filepath.Base(named), func(t *testing.T) {
+			t.Setenv(databaseEnv, named)
+
+			stdout := runStdout(t, cmdInfo)
+
+			for _, want := range []string{
+				"database\t" + named + " (from " + databaseEnv + ")\n",
+				fmt.Sprintf("\n-wal file size\t%d (%s)\n",
+					wal.Size(), humanBytes(wal.Size())),
+				"\nrecords\t4\n",
+			} {
+				if !strings.Contains(stdout, want) {
+					t.Errorf("stdout = %q, want it to hold %q", stdout, want)
+				}
+			}
+		})
+	}
+}
+
+func TestRunInfoNeedsOnlyReadAccess(t *testing.T) {
+	// README "`info` mode": like report, info needs only read access to
+	// the database file and none to its directory.
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	scanFixture(t)
+	makeReadOnly(t, path)
+
+	stdout := runStdout(t, cmdInfo)
+
+	if want := "\nrecords\t2\n"; !strings.Contains(stdout, want) {
+		t.Errorf("stdout = %q, want it to hold %q", stdout, want)
+	}
+
+	if strings.Contains(stdout, "-wal") {
+		t.Errorf("stdout = %q, want no -wal file between scans", stdout)
+	}
+}

main.go

@@ -8,13 +8,14 @@
 // under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
 // or larger. scan maintains a persistent SQLite database of file
 // signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
-// that the reporting subcommands read.
+// that the other subcommands read.
 //
 // Usage:
 //
-//	sfdupes scan [--workers N] [-x] PATH...
+//	sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
 //	sfdupes report > dupes.tsv
 //	sfdupes trees > dupetrees.tsv
+//	sfdupes info
 //	sfdupes --version
 //
 // See README.md for the complete specification.
@@ -27,6 +28,7 @@ import (
 	"io"
 	"os"
 	"runtime"
+	"runtime/debug"
 
 	"github.com/spf13/cobra"
 )
@@ -44,6 +46,7 @@ const (
 	cmdScan   = "scan"
 	cmdReport = "report"
 	cmdTrees  = "trees"
+	cmdInfo   = "info"
 )
 
 // errNoSubcommand is returned by the root command when it is invoked
@@ -56,10 +59,32 @@ var errNoSubcommand = errors.New("no subcommand")
 var errWorkersBelowOne = errors.New("--workers must be at least 1")
 
 // Version is the build version, injected at link time via -ldflags
-// (see the Makefile); "dev" for a plain go build.
+// (see the Makefile); empty when the linker set none, as for a binary
+// built by go install. buildVersion turns it into what --version prints.
 //
 //nolint:gochecknoglobals // written only by the linker
-var Version = "dev"
+var Version string
+
+// buildVersion returns the version that sfdupes --version prints
+// (README "Install"): the version the linker set, when it set one;
+// otherwise the main module's version that Go recorded in the binary,
+// a tag such as v0.0.2 or the pseudo-version of an @main build; and
+// "dev" when neither is known. info is what debug.ReadBuildInfo
+// returns, nil when the binary carries none. Go records "(devel)" for
+// a build that has no version, such as one inside a copy of the source
+// without git.
+func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
+	if linkerVersion != "" {
+		return linkerVersion
+	}
+
+	if info == nil || info.Main.Version == "" ||
+		info.Main.Version == "(devel)" {
+		return "dev"
+	}
+
+	return info.Main.Version
+}
 
 func main() {
 	// Once the reader of a stdout pipe has gone, as in "sfdupes report |
@@ -73,8 +98,8 @@ func main() {
 // exit code. It is the program's single exit point: the subcommands
 // return their errors instead of exiting, so every deferred cleanup —
 // above all closing the database, which checkpoints the SQLite WAL —
-// runs before the process ends. The report and trees subcommands write
-// their data to stdout.
+// runs before the process ends. The report, trees and info subcommands
+// write their data to stdout.
 func run(args []string, stdout, stderr io.Writer) int {
 	// A nil slice makes cobra fall back to os.Args, which would let a
 	// test binary's own flags reach the command tree.
@@ -114,7 +139,10 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 	var showVersion bool
 
 	printVersion := runE(func(context.Context, []string) error {
-		_, err := fmt.Fprintf(stdout, "sfdupes %s\n", Version)
+		info, _ := debug.ReadBuildInfo()
+
+		_, err := fmt.Fprintf(stdout, "sfdupes %s\n",
+			buildVersion(Version, info))
 		if err != nil {
 			return fmt.Errorf("write stdout: %w", err)
 		}
@@ -148,30 +176,6 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 	root.Flags().BoolVarP(&showVersion, "version", "v", false,
 		"print the version to stdout")
 
-	var (
-		scanWorkers int
-		scanOneFS   bool
-	)
-
-	scanCmd := &cobra.Command{
-		Use:   cmdScan + " [--workers N] [-x] PATH...",
-		Short: "Walk trees and synchronize the scan database",
-		Args:  cobra.MinimumNArgs(1),
-		PreRunE: func(cmd *cobra.Command, _ []string) error {
-			return checkScanWorkers(cmd, scanWorkers)
-		},
-		RunE: runE(func(ctx context.Context, args []string) error {
-			ctx, stop := interruptContext(ctx)
-			defer stop()
-
-			return runScan(ctx, args, scanWorkers, scanOneFS)
-		}),
-	}
-	scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
-		"concurrent workers for the walk, hash, and content phases")
-	scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
-		"do not cross filesystem boundaries")
-
 	reportCmd := &cobra.Command{
 		Use:   cmdReport,
 		Short: "Read the scan database and print the file-level duplicates report",
@@ -190,11 +194,59 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 		}),
 	}
 
-	root.AddCommand(scanCmd, reportCmd, treesCmd)
+	infoCmd := &cobra.Command{
+		Use:   cmdInfo,
+		Short: "Read the scan database and print its details",
+		Args:  cobra.NoArgs,
+		RunE: runE(func(ctx context.Context, _ []string) error {
+			return runInfo(ctx, stdout)
+		}),
+	}
+
+	root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
 
 	return root
 }
 
+// newScanCommand builds the scan subcommand, with its --workers, -x and
+// --no-default-excludes flags. Its help lists the default excludes of
+// the system it runs on, and the filesystem types it never records
+// anything on.
+func newScanCommand() *cobra.Command {
+	var (
+		scanWorkers           int
+		scanOneFS             bool
+		scanNoDefaultExcludes bool
+	)
+
+	short := "Walk trees and synchronize the scan database"
+
+	scanCmd := &cobra.Command{
+		Use:   cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
+		Short: short,
+		Long:  short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
+		Args:  cobra.MinimumNArgs(1),
+		PreRunE: func(cmd *cobra.Command, _ []string) error {
+			return checkScanWorkers(cmd, scanWorkers)
+		},
+		RunE: runE(func(ctx context.Context, args []string) error {
+			ctx, stop := interruptContext(ctx)
+			defer stop()
+
+			return runScan(ctx, args, scanWorkers, scanOneFS,
+				scanNoDefaultExcludes)
+		}),
+	}
+	scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
+		"concurrent workers for the walk, hash, and content phases")
+	scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
+		"do not cross filesystem boundaries")
+	scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
+		false, "enter the default excludes, the paths listed above")
+
+	return scanCmd
+}
+
 // checkScanWorkers rejects a scan --workers value below 1. That is a
 // usage error reported in one line: cobra prints the returned message
 // without the usage text, and run exits 2.

main_test.go

@@ -4,10 +4,12 @@ import (
 	"bytes"
 	"context"
 	"errors"
+	"fmt"
 	"io"
 	"io/fs"
 	"os"
 	"path/filepath"
+	"runtime/debug"
 	"slices"
 	"strconv"
 	"strings"
@@ -18,6 +20,36 @@ import (
 // error prints and a runtime failure must not.
 const usageMarker = "Usage:"
 
+// TestMain stops the tests at once, saying why, when their temporary
+// directory is not on a persistent filesystem: scan records nothing
+// there (README "Rules for the walk"), so most scan tests would fail
+// without saying why.
+func TestMain(m *testing.M) {
+	nonPersistent, err := nonPersistentFilesystems()
+	if err != nil {
+		fmt.Fprintf(os.Stderr, "list the mounted filesystems: %v\n", err)
+		os.Exit(1)
+	}
+
+	dir := os.TempDir()
+
+	fi, err := os.Stat(dir)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, err)
+		os.Exit(1)
+	}
+
+	dev, ok := deviceOfInfo(fi)
+	if fsType, found := nonPersistent[dev]; ok && found {
+		fmt.Fprintf(os.Stderr, "%s is on %s, not a persistent filesystem, "+
+			"and scan records nothing there: set TMPDIR to a directory on a "+
+			"persistent filesystem\n", dir, fsType)
+		os.Exit(1)
+	}
+
+	os.Exit(m.Run())
+}
+
 // walSuffixes are the SQLite sidecar files a WAL-mode database keeps
 // while it is open. A clean close checkpoints the WAL and removes
 // both; finding either afterwards means the database was never closed.
@@ -284,10 +316,16 @@ func assertFatalOutput(t *testing.T, stderr, stdout string) {
 }
 
 func TestRunMissingDatabaseIsFatal(t *testing.T) {
-	// README §Database: report and trees need an existing database; a
-	// missing one exits 1 with a message telling the user to run scan,
-	// after the line naming the database.
-	for _, name := range []string{cmdReport, cmdTrees} {
+	// README §Database: report, trees and info need an existing
+	// database; a missing one exits 1 with a message telling the user to
+	// run scan, after the line naming the database.
+	openLines := map[string]string{
+		cmdReport: reportOpenLine,
+		cmdTrees:  treesOpenLine,
+		cmdInfo:   infoOpenLine,
+	}
+
+	for name, openLine := range openLines {
 		t.Run(name, func(t *testing.T) {
 			path := testDBPath(t)
 			t.Setenv(databaseEnv, path)
@@ -303,15 +341,10 @@ func TestRunMissingDatabaseIsFatal(t *testing.T) {
 				t.Errorf("run(%s) = %d, want %d", name, code, exitFatal)
 			}
 
-			// trees shows its open phase, as it starts and as it ends,
+			// Each shows its open phase, as it starts and as it ends,
 			// before the error.
-			opening := ""
-			if name == cmdTrees {
-				opening = treesOpenLine + "\n" + treesOpenLine + "\n"
-			}
-
 			want := name + ": database " + path + " (from " + databaseEnv +
-				")\n" + opening +
+				")\n" + openLine + "\n" + openLine + "\n" +
 				"sfdupes: " + path + ": no database (run \"sfdupes " +
 				"scan\" first, or set " + databaseEnv + ")\n"
 			if got := stderr(); got != want {
@@ -512,8 +545,9 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
 	// exist at the default path on the machine running the test, so
 	// nothing here may create or read one: scan gets an operand that
 	// does not exist, which stops it before it takes the lock, and
-	// report and trees a cancelled context, which stops them before
-	// they read anything. trees shows its open phase after the line.
+	// report, trees and info a cancelled context, which stops them
+	// before they read anything. They show their open phase after the
+	// line.
 	t.Setenv(databaseEnv, "")
 
 	// Whether scan calls the database new depends on that machine too.
@@ -533,7 +567,7 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
 	}{
 		cmdScan: {
 			run: func() error {
-				return runScan(t.Context(), []string{missing}, 1, false)
+				return runScan(t.Context(), []string{missing}, 1, false, false)
 			},
 			want: scanWant,
 		},
@@ -541,7 +575,8 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
 			run: func() error {
 				return runReport(cancelledContext(t), io.Discard)
 			},
-			want: "report: database " + defaultDatabasePath + " (default)\n",
+			want: "report: database " + defaultDatabasePath + " (default)\n" +
+				reportOpenLine + "\n" + reportOpenLine + "\n",
 		},
 		cmdTrees: {
 			run: func() error {
@@ -550,6 +585,13 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
 			want: "trees: database " + defaultDatabasePath + " (default)\n" +
 				treesOpenLine + "\n" + treesOpenLine + "\n",
 		},
+		cmdInfo: {
+			run: func() error {
+				return runInfo(cancelledContext(t), io.Discard)
+			},
+			want: "info: database " + defaultDatabasePath + " (default)\n" +
+				infoOpenLine + "\n" + infoOpenLine + "\n",
+		},
 	}
 
 	for name, tc := range cases {
@@ -576,6 +618,7 @@ func TestRunUsageErrors(t *testing.T) {
 		"scan without paths": {[]string{cmdScan}, usageMarker},
 		"report with args":   {[]string{cmdReport, "x"}, usageMarker},
 		"trees with args":    {[]string{cmdTrees, "x"}, usageMarker},
+		"info with args":     {[]string{cmdInfo, "x"}, usageMarker},
 		"unknown flag":       {[]string{cmdScan, "--nope", "/"}, usageMarker},
 		"unknown subcommand": {[]string{"nope"}, "unknown command"},
 	}
@@ -672,6 +715,8 @@ func TestRunVersion(t *testing.T) {
 
 	// README §Subcommands: the version is one line on stdout, with
 	// nothing on stderr, and exits 0.
+	info, _ := debug.ReadBuildInfo()
+
 	for _, arg := range []string{"--version", "-v"} {
 		var stdout, stderr bytes.Buffer
 
@@ -680,7 +725,7 @@ func TestRunVersion(t *testing.T) {
 			t.Errorf("run(%s) = %d, want %d", arg, code, exitOK)
 		}
 
-		want := "sfdupes " + Version + "\n"
+		want := "sfdupes " + buildVersion(Version, info) + "\n"
 		if got := stdout.String(); got != want {
 			t.Errorf("run(%s) stdout = %q, want %q", arg, got, want)
 		}
@@ -691,6 +736,40 @@ func TestRunVersion(t *testing.T) {
 	}
 }
 
+func TestBuildVersion(t *testing.T) {
+	t.Parallel()
+
+	// README "Install": the linker's version wins; otherwise the main
+	// module's version that Go recorded; "dev" when neither is known.
+	tagged := &debug.BuildInfo{Main: debug.Module{Version: "v0.0.2"}}
+	pseudo := &debug.BuildInfo{Main: debug.Module{
+		Version: "v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1",
+	}}
+	devel := &debug.BuildInfo{Main: debug.Module{Version: "(devel)"}}
+
+	cases := []struct {
+		name   string
+		linker string
+		info   *debug.BuildInfo
+		want   string
+	}{
+		{"linker set", "v0.0.2-3-g4e4a94d", tagged, "v0.0.2-3-g4e4a94d"},
+		{"linker set, no build info", "4e4a94d", nil, "4e4a94d"},
+		{"build info with a tag", "", tagged, "v0.0.2"},
+		{"build info with a pseudo-version", "", pseudo,
+			"v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1"},
+		{"(devel)", "", devel, "dev"},
+		{"no build info", "", nil, "dev"},
+	}
+
+	for _, c := range cases {
+		if got := buildVersion(c.linker, c.info); got != c.want {
+			t.Errorf("%s: buildVersion(%q, ...) = %q, want %q",
+				c.name, c.linker, got, c.want)
+		}
+	}
+}
+
 func TestRunVersionWriteFailureIsFatal(t *testing.T) {
 	t.Parallel()
 
@@ -916,11 +995,16 @@ func TestRunReportSucceeds(t *testing.T) {
 		t.Errorf("stdout = %q, want %q", got, want)
 	}
 
-	want = "report: database " + path + " (from " + databaseEnv + ")\n" +
-		"report: 2 records read, 1 duplicate groups, 1 dupe files, " +
+	// Its phases come between the two lines; see
+	// TestRunReportProgressNamesEachPhase.
+	first := "report: database " + path + " (from " + databaseEnv + ")\n"
+	last := "\nreport: 2 records read, 1 duplicate groups, 1 dupe files, " +
 		"300 B reclaimable\n"
-	if got := stderr(); got != want {
-		t.Errorf("stderr = %q, want %q", got, want)
+
+	if got := stderr(); !strings.HasPrefix(got, first) ||
+		!strings.HasSuffix(got, last) {
+		t.Errorf("stderr = %q, want it to start %q and end %q",
+			got, first, last)
 	}
 
 	assertNoSidecars(t, path)
@@ -1251,7 +1335,7 @@ func TestRunReportsDuringScan(t *testing.T) {
 func TestRunStdoutClosedIsFatal(t *testing.T) {
 	// README §Error handling: a stdout write failure exits 1, reported
 	// in one line on stderr.
-	for _, name := range []string{cmdReport, cmdTrees} {
+	for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
 		t.Run(name, func(t *testing.T) {
 			t.Setenv(databaseEnv, testDBPath(t))
 
@@ -1301,6 +1385,7 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
 	cases := map[string]func(context.Context, io.Writer) error{
 		cmdReport: runReport,
 		cmdTrees:  runTrees,
+		cmdInfo:   runInfo,
 	}
 
 	for name, fn := range cases {

progress.go

@@ -22,6 +22,15 @@ const plainInterval = 5 * time.Second
 // a TTY.
 const redrawInterval = 100 * time.Millisecond
 
+// rateWindow is the time the rate a display shows covers: the items
+// completed over the last rateWindow, which is the time since the line
+// before when stderr is not a TTY.
+const rateWindow = 5 * time.Second
+
+// stallTime is how long a phase goes with no item completed before its
+// line says it has stalled, and where it waits, in place of the rate.
+const stallTime = 10 * time.Second
+
 // spinnerType selects the progressbar library's spinner style.
 const spinnerType = 14
 
@@ -47,6 +56,18 @@ const spinnerMargin = 5
 // around them, and the terminal's last column.
 const barMargin = barWidth + 11
 
+// countingRecords is what the display of a phase that reads the records
+// says while it counts them, before it has its total.
+const countingRecords = "counting the records"
+
+// openPhase is the name of the phase in which report, trees and info
+// open the database, and openingDatabase what its display says while
+// it does.
+const (
+	openPhase       = "open"
+	openingDatabase = "opening the database"
+)
+
 // stderrIsTTY reports whether stderr is attached to a terminal.
 func stderrIsTTY() bool {
 	return term.IsTerminal(int(os.Stderr.Fd()))
@@ -77,6 +98,14 @@ type progressParams struct {
 	total      int64  // the items the phase will complete; -1 when not known
 	estimate   bool   // total is an estimate, which the count may pass
 	waitingFor string // what the phase waits for until setTotal is called
+	stalledOn  string // where the phase waits, as "disk/db"; see stallTime
+}
+
+// drawnCount is the count of items completed as of one draw, and when
+// that draw was.
+type drawnCount struct {
+	at    time.Time
+	count int64
 }
 
 // progress renders one phase's progress on stderr. On a TTY the
@@ -95,9 +124,10 @@ type progressParams struct {
 // phase's helpers without rendering anything, and its writer writes
 // with no display to clear.
 type progress struct {
-	width func() int    // reads the terminal's width; nil when not a TTY
-	stop  chan struct{} // closed by finish to end the draws
-	done  chan struct{} // closed once the draws have ended
+	now   func() time.Time // reads the time
+	width func() int       // reads the terminal's width; nil when not a TTY
+	stop  chan struct{}    // closed by finish to end the draws
+	done  chan struct{}    // closed once the draws have ended
 
 	mu         sync.Mutex
 	bar        *progressbar.ProgressBar // what the library draws; nil until then
@@ -108,33 +138,42 @@ type progress struct {
 	total      int64
 	estimate   bool
 	waitingFor string
+	stalledOn  string
 	count      int64
 	start      time.Time // when the phase started, for the elapsed time
-	rateStart  time.Time // when the count started, for the rate
+	rateStart  time.Time // when the count started, for the ETA
+	lastItem   time.Time // when the last item was completed, or the count started
+	// drawn holds the count at each draw over the last rateWindow, oldest
+	// first; the rate counts from the first. It keeps the draw before the
+	// latest even when that is older, as the line before is when lines
+	// are plainInterval apart and a tick comes late.
+	drawn []drawnCount
 }
 
 // newProgress starts the display of one phase on stderr.
 func newProgress(params progressParams) *progress {
 	if stderrIsTTY() {
-		return startProgress(params, terminalWidth, redrawInterval)
+		return startProgress(params, terminalWidth, redrawInterval, time.Now)
 	}
 
-	return startProgress(params, nil, plainInterval)
+	return startProgress(params, nil, plainInterval, time.Now)
 }
 
 // startProgress shows the phase params describes at once, and again
 // every interval until finish: with the progressbar library on a
 // terminal, whose width each draw reads by calling width, or as a plain
-// line when width is nil.
+// line when width is nil. The display reads the time by calling now.
 func startProgress(params progressParams, width func() int,
-	interval time.Duration,
+	interval time.Duration, now func() time.Time,
 ) *progress {
-	now := time.Now()
+	start := now()
 	p := &progress{
-		width: width, stop: make(chan struct{}), done: make(chan struct{}),
+		now: now, width: width,
+		stop: make(chan struct{}), done: make(chan struct{}),
 		label: params.label, unit: params.unit, total: params.total,
 		estimate: params.estimate, waitingFor: params.waitingFor,
-		start: now, rateStart: now,
+		stalledOn: params.stalledOn, start: start, rateStart: start,
+		lastItem: start, drawn: []drawnCount{{at: start}},
 	}
 
 	// Show the zero state at once: the first item may take minutes,
@@ -161,32 +200,42 @@ func startProgress(params progressParams, width func() int,
 }
 
 // draw shows the phase's progress: on a TTY in place of the line
-// before, and otherwise as a plain line.
+// before, and otherwise as a plain line. It first adds the count to
+// drawn, then drops from drawn the draws older than rateWindow, except
+// the one before this draw.
 func (p *progress) draw() {
 	p.mu.Lock()
 	defer p.mu.Unlock()
 
+	now := p.now()
+
+	p.drawn = append(p.drawn, drawnCount{at: now, count: p.count})
+	for len(p.drawn) > 2 && now.Sub(p.drawn[0].at) > rateWindow {
+		p.drawn = p.drawn[1:]
+	}
+
 	if p.width == nil {
-		text, _ := p.line(math.MaxInt, true)
-		fmt.Fprintln(os.Stderr, text)
+		fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true))
 
 		return
 	}
 
 	width := p.width()
 
-	// A phase with an exact total gets the library's bar when its line,
-	// without the percent the bar shows, fits whole beside the bar. A
-	// total of 0 leaves the bar nothing to fill.
+	// A phase with an exact total gets the library's bar when its whole
+	// line, without the percent the bar shows, fits beside the bar: the
+	// bar gives way before any part of the line. A total of 0 leaves the
+	// bar nothing to fill.
 	text, withBar := "", false
 	if p.total > 0 && !p.estimate {
-		text, withBar = p.line(width-barMargin, false)
+		text = p.line(now, math.MaxInt, false)
+		withBar = len(text) <= width-barMargin
 	}
 
 	margin := barMargin
 	if !withBar {
 		margin = spinnerMargin
-		text, _ = p.line(width-margin, true)
+		text = p.line(now, width-margin, true)
 	}
 
 	// The library makes a bar or a spinner, and before each draw erases
@@ -243,18 +292,22 @@ func (p *progress) newBar(withBar bool, width int) {
 }
 
 // setTotal gives a phase that started without a total its total, once
-// it is known. The rate, and so the ETA, count from then on: the time
-// spent waiting for the total says nothing about how fast items go.
+// it is known. The rate, the ETA and the time with no item completed
+// count from then on: the time spent waiting for the total says nothing
+// about how fast items go.
 func (p *progress) setTotal(total int64) {
 	if p == nil {
 		return
 	}
 
+	now := p.now()
+
 	p.mu.Lock()
 	defer p.mu.Unlock()
 
 	p.total = total
-	p.rateStart = time.Now()
+	p.rateStart, p.lastItem = now, now
+	p.drawn = []drawnCount{{at: now, count: p.count}}
 }
 
 // increment records one completed item, which the display shows at its
@@ -264,10 +317,13 @@ func (p *progress) increment() {
 		return
 	}
 
+	now := p.now()
+
 	p.mu.Lock()
 	defer p.mu.Unlock()
 
 	p.count++
+	p.lastItem = now
 }
 
 // warnf prints a one-line warning to stderr without corrupting the
@@ -292,25 +348,42 @@ func (p *progress) warnf(format string, args ...any) {
 	fmt.Fprintln(os.Stderr, msg)
 }
 
-// progressWriter writes a phase's output to out, which may be the
-// terminal the phase's display is drawn on, as stdout is in a plain
-// "sfdupes trees". It holds what is written to it and writes that to
-// out in pieces of about ioBufSize bytes, each under the display's
-// lock with the display cleared first, as warnf does for a warning: no
-// draw lands in the middle of a piece, and the next one shows the
-// display again, on the line below. Every write to it must end with a
-// newline, as a row written with one fmt.Fprintf does, so that each
-// piece ends a line. Flush writes what it holds once the last row is
-// written.
+// progressWriter writes a phase's output to out, which may be the file
+// the phase's display is drawn on, as stdout is in a plain "sfdupes
+// trees" on a terminal. It holds what is written to it and writes that
+// to out in pieces of about ioBufSize bytes. When out is the file
+// stderr is, each piece is written under the display's lock with the
+// display cleared first, as warnf does for a warning: no draw lands in
+// the middle of a piece, and the next draw shows the display again, on
+// the line below. Every write to it must end with a newline, as a row
+// written with one fmt.Fprintf does, so that each piece ends a line.
+// Flush writes what it holds once the last row is written.
 type progressWriter struct {
+	// p is the display to clear, nil unless out is the file stderr is.
 	p   *progress
 	out io.Writer
 	buf []byte
 }
 
 // writer returns a progressWriter writing to out around p's display.
+// It clears the display and takes its lock only when out is the file
+// stderr is: the same device and inode, as os.SameFile compares them,
+// which is one terminal, or one pipe or file that both are sent to.
+// Otherwise it writes to out without the lock, so the display keeps
+// being drawn while a write to out waits for a slow reader.
 func (p *progress) writer(out io.Writer) *progressWriter {
-	return &progressWriter{p: p, out: out}
+	w := &progressWriter{out: out}
+
+	if f, ok := out.(*os.File); ok {
+		outInfo, outErr := f.Stat()
+		errInfo, errErr := os.Stderr.Stat()
+
+		if outErr == nil && errErr == nil && os.SameFile(outInfo, errInfo) {
+			w.p = p
+		}
+	}
+
+	return w
 }
 
 // Write holds b, and writes what it holds to out once that comes to
@@ -324,8 +397,8 @@ func (w *progressWriter) Write(b []byte) (int, error) {
 	return len(b), w.Flush()
 }
 
-// Flush writes what w holds to out, with the display cleared first and
-// not drawn again until it is written.
+// Flush writes what w holds to out. When out is the file stderr is, the
+// display is cleared first and not drawn again until it is written.
 func (w *progressWriter) Flush() error {
 	if w.p != nil {
 		w.p.mu.Lock()
@@ -362,50 +435,68 @@ func (p *progress) finish() {
 	}
 }
 
-// line formats the phase's progress in at most room columns, as
+// line formats the phase's progress as of now in at most room columns,
+// as
 //
 //	[3/5] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s
 //
 // where "files" is the phase's unit, without the percent when percent
 // is false, as before the library's bar, which shows its own. A phase
-// with no total shows the items done instead, or what it waits for.
+// with no total shows the items done instead, or what it waits for. The
+// rate is that since the first draw in drawn, and the ETA comes from
+// the rate since the count started. Once no item has been completed for
+// stallTime, the rate, which would read 0, gives way to the word
+// "stalled" and where the phase waits, as "stalled disk/db".
 //
 // The phase and its place, the progress, the elapsed time and the ETA
-// are kept whole. A line wider than room leaves out the rate, or what
-// the phase waits for; only if it is still wider is it cut at room, and
-// line reports that it does not fit. Every part is ASCII, so its length
-// is its width in columns. p.mu must be held.
-func (p *progress) line(room int, percent bool) (string, bool) {
-	elapsed := time.Since(p.start).Round(time.Second)
+// are kept whole. A line wider than room leaves out the rate, the words
+// of a stalled phase, or what the phase waits for; only if it is still
+// wider is it cut at room. Every part is ASCII, so its length is its
+// width in columns. p.mu must be held.
+func (p *progress) line(now time.Time, room int, percent bool) string {
+	elapsed := now.Sub(p.start).Round(time.Second)
 
 	// head and tail hold the parts kept whole; extra, between them, is
 	// the part left out.
 	var head, extra, tail string
 
 	if p.total < 0 && p.waitingFor != "" {
-		head, extra = p.label+": ", p.waitingFor
+		head, extra = p.label+": ", p.waitingFor+", "
 		tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed)
 	} else {
-		rate := 0.0
-		if s := time.Since(p.rateStart).Seconds(); s > 0 {
-			rate = float64(p.count) / s
-		}
+		since := p.drawn[0]
+		rate := perSecond(p.count-since.count, now.Sub(since.at))
+		average := perSecond(p.count, now.Sub(p.rateStart))
 
 		head = p.label + ": " + p.itemsDone(percent) + ", "
-		extra = fmt.Sprintf("%.0f %s/s", rate, p.unit)
-		tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(rate))
+		extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit)
+		tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average))
+
+		if now.Sub(p.lastItem) >= stallTime {
+			extra = "stalled " + p.stalledOn + ", "
+		}
 	}
 
-	full, kept := head+extra+", "+tail, head+tail
+	full, kept := head+extra+tail, head+tail
 
 	switch {
 	case len(full) <= room:
-		return full, true
+		return full
 	case len(kept) <= room:
-		return kept, true
+		return kept
 	}
 
-	return kept[:max(room, 0)], false
+	return kept[:max(room, 0)]
+}
+
+// perSecond returns count items over d as items a second, or 0 when no
+// time has passed.
+func perSecond(count int64, d time.Duration) float64 {
+	if d <= 0 {
+		return 0
+	}
+
+	return float64(count) / d.Seconds()
 }
 
 // itemsDone formats the items done, as "[1234/5000] 25%", with "~"

progress_test.go

@@ -2,11 +2,13 @@ package main
 
 import (
 	"fmt"
+	"io"
 	"math"
 	"os"
 	"path/filepath"
 	"slices"
 	"strings"
+	"sync"
 	"testing"
 	"time"
 	"unicode/utf8"
@@ -20,6 +22,33 @@ const testRedraw = time.Millisecond
 // it expects.
 const testDeadline = 5 * time.Second
 
+// hashStalledOn is where the hash phase says it waits once it has
+// stalled, as readRuns gives it.
+const hashStalledOn = "disk/db"
+
+// testClock is a clock a test moves by hand, for a display that reads
+// it from its own goroutine too.
+type testClock struct {
+	mu  sync.Mutex
+	now time.Time
+}
+
+// Now returns the clock's time.
+func (c *testClock) Now() time.Time {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	return c.now
+}
+
+// Add moves the clock on by d.
+func (c *testClock) Add(d time.Duration) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+
+	c.now = c.now.Add(d)
+}
+
 //nolint:paralleltest // replaces the process-wide os.Stderr
 func TestStderrIsTTYFalseForNonTerminals(t *testing.T) {
 	r, pipe, err := os.Pipe()
@@ -72,11 +101,13 @@ func TestPhaseLabel(t *testing.T) {
 }
 
 // TestProgressLine checks the line a display shows in each state, ten
-// seconds into the phase.
+// seconds into the phase, with no draw since the first and the last
+// item just completed.
 func TestProgressLine(t *testing.T) {
 	t.Parallel()
 
-	ago := time.Now().Add(-10 * time.Second)
+	now := time.Now()
+	ago := now.Add(-10 * time.Second)
 
 	cases := []struct {
 		p    *progress
@@ -84,7 +115,7 @@ func TestProgressLine(t *testing.T) {
 	}{{
 		p: &progress{
 			label: scanPhase("load"), total: -1,
-			waitingFor: "counting the records",
+			waitingFor: countingRecords,
 		},
 		want: "[1/5] load: counting the records, elapsed 10s, eta unknown",
 	}, {
@@ -129,90 +160,120 @@ func TestProgressLine(t *testing.T) {
 	}}
 
 	for _, tc := range cases {
-		tc.p.start, tc.p.rateStart = ago, ago
+		tc.p.start, tc.p.rateStart, tc.p.lastItem = ago, ago, now
+		tc.p.drawn = []drawnCount{{at: ago}}
 
-		if got, _ := tc.p.line(math.MaxInt, true); got != tc.want {
+		if got := tc.p.line(now, math.MaxInt, true); got != tc.want {
 			t.Errorf("line = %q, want %q", got, tc.want)
 		}
 	}
 }
 
 // TestLineFitsRoom checks how a line is fitted to the room it has, ten
-// seconds into the phase: it leaves out the rate, or what the phase
-// waits for, and is cut only when what is left is still too wide.
+// seconds into the phase, and fifty into a phase with no file read for
+// the last 42: it leaves out the rate, what the phase waits for, or the
+// words of a stalled phase, and is cut only when what is left is still
+// too wide.
 func TestLineFitsRoom(t *testing.T) {
 	t.Parallel()
 
-	ago := time.Now().Add(-10 * time.Second)
+	now := time.Now()
+	ago := now.Add(-10 * time.Second)
 	hash := &progress{
 		label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
-		start: ago, rateStart: ago,
+		start: ago, rateStart: ago, lastItem: now,
+		drawn: []drawnCount{{at: ago}},
 	}
 	content := &progress{
 		label: scanPhase("content"), total: -1,
 		waitingFor: "searching the database", start: ago, rateStart: ago,
 	}
 
+	started := now.Add(-50 * time.Second)
+	stalled := &progress{
+		label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
+		stalledOn: hashStalledOn, start: started, rateStart: started,
+		lastItem: now.Add(-42 * time.Second),
+		drawn:    []drawnCount{{at: now.Add(-rateWindow), count: 20}},
+	}
+
 	cases := []struct {
 		p       *progress
 		room    int
 		percent bool
 		want    string
-		fits    bool
 	}{{
 		p: hash, room: 57, percent: true,
 		want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
-		fits: true,
 	}, {
 		p: hash, room: 56, percent: true,
 		want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40s",
-		fits: true,
 	}, {
 		p: hash, room: 45, percent: true,
 		want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40",
-		fits: false,
 	}, {
 		// Before the library's bar, which shows the percent.
 		p: hash, room: 57, percent: false,
 		want: "[3/5] hash: [20/100], 2 files/s, elapsed 10s, eta 40s",
-		fits: true,
 	}, {
 		p: content, room: 40, percent: true,
 		want: "[5/5] content: elapsed 10s, eta unknown",
-		fits: true,
+	}, {
+		p: stalled, room: 65, percent: true,
+		want: "[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " +
+			"eta 3m20s",
+	}, {
+		p: stalled, room: 64, percent: true,
+		want: "[3/5] hash: [20/100] 20%, elapsed 50s, eta 3m20s",
 	}}
 
 	for _, tc := range cases {
-		got, fits := tc.p.line(tc.room, tc.percent)
-		if got != tc.want || fits != tc.fits {
-			t.Errorf("line(%d, %t) = %q, %t, want %q, %t",
-				tc.room, tc.percent, got, fits, tc.want, tc.fits)
+		if got := tc.p.line(now, tc.room, tc.percent); got != tc.want {
+			t.Errorf("line(%d, %t) = %q, want %q",
+				tc.room, tc.percent, got, tc.want)
 		}
 	}
 }
 
 // TestSetTotalStartsRate checks that setTotal starts a phase's rate,
-// and so its ETA, from when the total becomes known, not from when the
-// phase started waiting for it.
+// its ETA and its time with no record read from when the total becomes
+// known, not from when the phase started waiting for it: after 20
+// seconds of waiting for the total, the phase has not stalled 5 seconds
+// after it, and has 10 seconds after it with no record read.
 func TestSetTotalStartsRate(t *testing.T) {
 	t.Parallel()
 
-	waited := time.Now().Add(-20 * time.Second)
+	clock := &testClock{now: time.Now()}
+	waited := clock.Now().Add(-20 * time.Second)
 	p := &progress{
-		label: scanPhase("load"), unit: scanUnit, total: -1, start: waited,
-		rateStart: waited,
+		now: clock.Now, label: scanPhase("load"), unit: scanUnit, total: -1,
+		stalledOn: "db", start: waited, rateStart: waited,
+		lastItem: waited, drawn: []drawnCount{{at: waited}},
 	}
 
 	p.setTotal(100)
 
-	// Ten seconds after the total came, without waiting ten seconds,
-	// 20 records are read.
-	p.start = p.start.Add(-10 * time.Second)
-	p.rateStart = p.rateStart.Add(-10 * time.Second)
-	p.count = 20
+	clock.Add(5 * time.Second)
 
-	want := "[1/5] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s"
-	if got, _ := p.line(math.MaxInt, true); got != want {
+	want := "[1/5] load: [0/100] 0%, 0 files/s, elapsed 25s, eta unknown"
+	if got := p.line(clock.Now(), math.MaxInt, true); got != want {
+		t.Errorf("line = %q, want %q", got, want)
+	}
+
+	clock.Add(5 * time.Second)
+
+	want = "[1/5] load: [0/100] 0%, stalled db, elapsed 30s, eta unknown"
+	if got := p.line(clock.Now(), math.MaxInt, true); got != want {
+		t.Errorf("line = %q, want %q", got, want)
+	}
+
+	// Then 20 records are read.
+	for range 20 {
+		p.increment()
+	}
+
+	want = "[1/5] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s"
+	if got := p.line(clock.Now(), math.MaxInt, true); got != want {
 		t.Errorf("line = %q, want %q", got, want)
 	}
 }
@@ -271,14 +332,138 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) {
 	stderr := captureStderr(t)
 	p := startProgress(progressParams{
 		label: scanPhase("load"), total: -1,
-		waitingFor: "counting the records",
-	}, nil, testRedraw)
+		waitingFor: countingRecords,
+	}, nil, testRedraw, time.Now)
 
 	awaitOutput(t, stderr,
 		"[1/5] load: counting the records, elapsed 0s, eta unknown\n", 3)
 	p.finish()
 }
 
+// TestPlainProgressStalledProducer checks that a phase whose files stop
+// coming, as when one large file is being read or a disk is slow, goes
+// on printing its line every interval: the rate falls to 0 once no file
+// has been read for rateWindow, and once none has for stallTime the
+// line says the phase has stalled, and where it waits, in place of the
+// rate. The test moves the display's clock by hand.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestPlainProgressStalledProducer(t *testing.T) {
+	stderr := captureStderr(t)
+	clock := &testClock{now: time.Now()}
+	p := startProgress(progressParams{
+		label: scanPhase("hash"), unit: scanUnit, total: 100,
+		stalledOn: hashStalledOn,
+	}, nil, testRedraw, clock.Now)
+
+	// Pass or fail, the display ends before stderr is put back.
+	t.Cleanup(p.finish)
+
+	// 20 files read as the phase starts, then none.
+	for range 20 {
+		p.increment()
+	}
+
+	clock.Add(5 * time.Second)
+	awaitOutput(t, stderr,
+		"[3/5] hash: [20/100] 20%, 4 files/s, elapsed 5s, eta 20s\n", 3)
+
+	clock.Add(time.Second)
+	awaitOutput(t, stderr,
+		"[3/5] hash: [20/100] 20%, 0 files/s, elapsed 6s, eta 24s\n", 3)
+
+	clock.Add(4 * time.Second)
+	awaitOutput(t, stderr,
+		"[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 10s, eta 40s\n",
+		3)
+}
+
+// TestPlainProgressRateAfterStall draws a phase's plain line by hand, a
+// little over plainInterval after the line before, as a late tick
+// does, through a stall and a burst of files after it: the rate covers
+// the files read since the line before, while the ETA comes from the
+// phase's average.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestPlainProgressRateAfterStall(t *testing.T) {
+	stderr := captureStderr(t)
+	clock := &testClock{now: time.Now()}
+
+	// No timed draws: the test draws.
+	p := startProgress(progressParams{
+		label: scanPhase("hash"), unit: scanUnit, total: 1000,
+		stalledOn: hashStalledOn,
+	}, nil, time.Hour, clock.Now)
+
+	late := plainInterval + time.Millisecond
+
+	clock.Add(late)
+	p.draw()
+	clock.Add(late)
+	p.draw()
+
+	for range 100 {
+		p.increment()
+	}
+
+	clock.Add(late)
+	p.finish()
+
+	want := []string{
+		"[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown",
+		"[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 5s, eta unknown",
+		"[3/5] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, eta unknown",
+		"[3/5] hash: [100/1000] 10%, 20 files/s, elapsed 15s, eta 2m15s",
+	}
+
+	lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
+	if !slices.Equal(lines, want) {
+		t.Errorf("stderr lines = %q, want %q", lines, want)
+	}
+}
+
+// TestProgressRateAfterStallRedrawn draws a phase by hand every
+// redrawInterval, as a terminal does, through a stall of stallTime and
+// a burst of files after it: the rate covers the files read over the
+// last rateWindow, and falls to 0 once the burst is older than that,
+// while the ETA comes from the phase's average. Plain lines show each
+// draw.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestProgressRateAfterStallRedrawn(t *testing.T) {
+	stderr := captureStderr(t)
+	clock := &testClock{now: time.Now()}
+
+	// No timed draws: the test draws.
+	p := startProgress(progressParams{
+		label: scanPhase("hash"), unit: scanUnit, total: 1000,
+		stalledOn: hashStalledOn,
+	}, nil, time.Hour, clock.Now)
+
+	// redraw draws the display every redrawInterval for d.
+	redraw := func(d time.Duration) {
+		for range d / redrawInterval {
+			clock.Add(redrawInterval)
+			p.draw()
+		}
+	}
+
+	redraw(stallTime)
+
+	for range 50 {
+		p.increment()
+	}
+
+	redraw(rateWindow + redrawInterval)
+	p.finish()
+
+	want := "[3/5] hash: [50/1000] 5%, 10 files/s, elapsed 15s, eta 4m45s\n" +
+		"[3/5] hash: [50/1000] 5%, 0 files/s, elapsed 15s, eta 4m47s\n"
+	if !strings.Contains(stderr(), want) {
+		t.Errorf("stderr = %q, want the lines %q", stderr(), want)
+	}
+}
+
 // lastFrame returns what a terminal shows of the frames a display drew:
 // the last one. The library starts each frame with a carriage return
 // and erases the previous one with spaces first.
@@ -321,7 +506,7 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) {
 	for _, tc := range cases {
 		before := len(stderr())
 		since := func() string { return stderr()[before:] }
-		p := startProgress(tc.params, terminalWidth, testRedraw)
+		p := startProgress(tc.params, terminalWidth, testRedraw, time.Now)
 
 		for range 50 {
 			p.increment()
@@ -343,21 +528,23 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) {
 
 // drawLate draws the display params describes on a terminal width
 // columns wide as it starts and once more as it ends, with count items
-// done two and a half hours into the phase, and returns what it wrote
-// to stderr.
+// done two and a half hours into the phase, the last just now, and
+// returns what it wrote to stderr.
 func drawLate(stderr func() string, params progressParams, width int,
 	count int64,
 ) string {
 	before := len(stderr())
+	clock := &testClock{now: time.Now()}
 
 	// No timed draws come between the two.
-	p := startProgress(params, func() int { return width }, time.Hour)
+	p := startProgress(params, func() int { return width }, time.Hour,
+		clock.Now)
 
 	// Without counting the items one by one or waiting the hours.
+	clock.Add(150 * time.Minute)
+
 	p.mu.Lock()
-	p.count = count
-	p.start = p.start.Add(-150 * time.Minute)
-	p.rateStart = p.start
+	p.count, p.lastItem = count, clock.Now()
 	p.mu.Unlock()
 
 	p.finish()
@@ -413,6 +600,85 @@ func TestTerminalLinesKeepETA(t *testing.T) {
 	}
 }
 
+// TestTerminalStalledLines draws every phase's line on an 80-column
+// terminal once the phase has stalled, its last item done 42 seconds
+// before. At an ordinary scan's counts and times, 12345 of 45678 items
+// done four and a half minutes in, the line shows where the phase waits
+// whole, beside the phase, progress, elapsed time and ETA. At a large
+// scan's, 1234567 of 9876543 two and a half hours in, those four still
+// show whole, as the words give way first where they do not fit.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestTerminalStalledLines(t *testing.T) {
+	stderr := captureStderr(t)
+
+	// drawStalled draws the display params describes as it starts and
+	// once more as it ends, with count of total items done elapsed into
+	// the phase, and returns what the terminal shows.
+	drawStalled := func(params progressParams, total, count int64,
+		elapsed time.Duration,
+	) string {
+		before := len(stderr())
+		clock := &testClock{now: time.Now()}
+		params.total = total
+
+		// No timed draws come between the two.
+		p := startProgress(params, func() int { return 80 }, time.Hour,
+			clock.Now)
+
+		clock.Add(elapsed - 42*time.Second)
+
+		p.mu.Lock()
+		p.count, p.lastItem = count, clock.Now()
+		p.mu.Unlock()
+
+		clock.Add(42 * time.Second)
+		p.finish()
+
+		return lastFrame(stderr()[before:])
+	}
+
+	phases := []progressParams{
+		{label: scanPhase("load"), stalledOn: "db"},
+		{label: scanPhase("walk"), estimate: true, stalledOn: "disk"},
+		{label: scanPhase("hash"), stalledOn: hashStalledOn},
+		{label: scanPhase("update"), stalledOn: "db"},
+		// The content phase's checks and reads, then its last write.
+		{label: scanPhase("content"), stalledOn: "disk/db"},
+		{label: scanPhase("content"), stalledOn: "db"},
+		{label: treesPhase("read"), stalledOn: "db"},
+		{label: treesPhase("group"), stalledOn: "cpu"},
+		{label: treesPhase("write"), stalledOn: "out"},
+		{label: reportPhase("read"), stalledOn: "db/out"},
+	}
+
+	for _, params := range phases {
+		approx := ""
+		if params.estimate {
+			approx = "~"
+		}
+
+		shown := drawStalled(params, 45678, 12345,
+			4*time.Minute+30*time.Second)
+		want := params.label + ": [12345/" + approx + "45678] 27%, stalled " +
+			params.stalledOn + ", elapsed 4m30s, eta 12m9s"
+
+		if !strings.Contains(shown, want) {
+			t.Errorf("terminal shows %q, want %q", shown, want)
+		}
+
+		shown = drawStalled(params, 9876543, 1234567, 150*time.Minute)
+		for _, want := range []string{
+			params.label + ": [1234567/" + approx + "9876543] 12%, ",
+			", elapsed 2h30m0s, eta 17h30m0s",
+		} {
+			if !strings.Contains(shown, want) {
+				t.Errorf("terminal shows %q, want %q", shown, want)
+			}
+		}
+	}
+}
+
 // TestTerminalLinesFitWidth draws a large scan's bar display and a small
 // scan's, two and a half hours into the phase, on terminals from 120
 // columns down to 40, and checks that every line stays clear of the
@@ -455,7 +721,7 @@ func TestTerminalLinesFitWidth(t *testing.T) {
 // times. Nothing written after that, the erase before each redraw
 // included, may reach the narrower terminal's last column: it would
 // wrap, and every redraw would leave a line behind. At these widths the
-// spinner stays a spinner and the bar stays a bar.
+// spinner stays a spinner and the bar gives way to the spinner.
 //
 //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
 func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
@@ -479,14 +745,16 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
 
 	for _, tc := range cases {
 		width := 120
+		clock := &testClock{now: time.Now()}
 
 		// No timed draws: the test draws.
-		p := startProgress(tc.params, func() int { return width }, time.Hour)
+		p := startProgress(tc.params, func() int { return width }, time.Hour,
+			clock.Now)
+
+		clock.Add(150 * time.Minute)
 
 		p.mu.Lock()
-		p.count = 1234567
-		p.start = p.start.Add(-150 * time.Minute)
-		p.rateStart = p.start
+		p.count, p.lastItem = 1234567, clock.Now()
 		p.mu.Unlock()
 
 		p.draw()
@@ -528,7 +796,7 @@ func TestProgressWarningsOnOwnLines(t *testing.T) {
 	issued := 0
 
 	for _, params := range cases {
-		p := startProgress(params, terminalWidth, testRedraw)
+		p := startProgress(params, terminalWidth, testRedraw, time.Now)
 
 		// No pause between warnings: one written straight to stderr is
 		// garbled only if a redraw lands while it is being written.
@@ -581,7 +849,7 @@ func TestProgressWriterRowsOnOwnLines(t *testing.T) {
 	var want []string
 
 	for _, params := range cases {
-		p := startProgress(params, terminalWidth, testRedraw)
+		p := startProgress(params, terminalWidth, testRedraw, time.Now)
 		out := p.writer(os.Stderr)
 
 		// No pause between rows: one is garbled only if a redraw lands
@@ -613,9 +881,70 @@ func TestProgressWriterRowsOnOwnLines(t *testing.T) {
 		}
 	}
 
-	if !slices.Equal(shown, want) {
-		t.Errorf("terminal shows rows %q, want %q", shown, want)
+	if len(shown) != len(want) {
+		t.Fatalf("terminal shows %d rows, want %d", len(shown), len(want))
 	}
+
+	for i := range want {
+		if shown[i] != want[i] {
+			t.Fatalf("terminal shows %q, want %q", shown[i], want[i])
+		}
+	}
+}
+
+// TestProgressWriterStalledStdoutKeepsLines writes a row larger than a
+// pipe holds through the display's writer to a pipe nobody reads yet,
+// as stdout is when its reader has stopped while stderr goes to a file:
+// while the write waits, the display must go on printing its line.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestProgressWriterStalledStdoutKeepsLines(t *testing.T) {
+	stderr := captureStderr(t)
+
+	r, w, err := os.Pipe()
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	t.Cleanup(func() {
+		_ = w.Close()
+		_ = r.Close()
+	})
+
+	p := startProgress(progressParams{label: treesPhase("write"), total: 1},
+		nil, testRedraw, time.Now)
+	out := p.writer(w)
+	written := make(chan error, 1)
+
+	go func() {
+		_, writeErr := fmt.Fprintln(out, strings.Repeat("x", ioBufSize))
+		written <- writeErr
+	}()
+
+	// Pass or fail, the pipe is then read so that the write ends, and
+	// the display ends before stderr is put back.
+	t.Cleanup(func() {
+		go func() {
+			_, _ = io.Copy(io.Discard, r)
+		}()
+
+		writeErr := <-written
+		if writeErr != nil {
+			t.Error(writeErr)
+		}
+
+		p.finish()
+	})
+
+	// The first byte read shows that the write has started; the rest
+	// of the row waits until the pipe is read.
+	_, err = r.Read(make([]byte, 1))
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	line := "[4/4] write: "
+	awaitOutput(t, stderr, line, strings.Count(stderr(), line)+3)
 }
 
 // TestTerminalStoppedShortKeepsCount checks that the terminal display
@@ -626,7 +955,7 @@ func TestProgressWriterRowsOnOwnLines(t *testing.T) {
 func TestTerminalStoppedShortKeepsCount(t *testing.T) {
 	stderr := captureStderr(t)
 	p := startProgress(progressParams{label: scanPhase("hash"), total: 10},
-		terminalWidth, redrawInterval)
+		terminalWidth, redrawInterval, time.Now)
 
 	p.increment()
 	p.increment()
@@ -645,7 +974,7 @@ func TestTerminalStoppedShortKeepsCount(t *testing.T) {
 func TestWarningAfterLastItemKeepsBar(t *testing.T) {
 	stderr := captureStderr(t)
 	p := startProgress(progressParams{label: scanPhase("hash"), total: 2},
-		terminalWidth, testRedraw)
+		terminalWidth, testRedraw, time.Now)
 
 	p.increment()
 	p.increment()

report.go

@@ -1,11 +1,12 @@
 package main
 
 import (
-	"bufio"
 	"context"
+	"database/sql"
 	"fmt"
 	"io"
 	"os"
+	"slices"
 	"strings"
 	"time"
 )
@@ -29,13 +30,25 @@ type scanRec struct {
 	path    string
 }
 
+// reportPhase names one phase of report with its place among all of
+// them, as "[2/2] read", for the phase's progress display. The list
+// here is the one place the phases are counted.
+func reportPhase(name string) string {
+	return phaseLabel([]string{openPhase, "read"}, name)
+}
+
 // runReport implements the report subcommand: it prints the file-level
-// duplicates report as TSV on stdout. SQLite groups and orders the
-// records, and each row is written as it is read, so no group is held
-// in memory. It never touches the scanned filesystem; its only I/O is
-// the database (with SQLite's temporary sort file), stdout, and stderr.
-// It names the database on stderr before it opens it. Any database
+// duplicates report as TSV on stdout. It never touches the scanned
+// filesystem; its only I/O is the database, stdout, and stderr. It
+// names the database on stderr before it opens it. Any database
 // problem, including a missing database, is fatal.
+//
+// The work runs in two phases, each under a progress display of its
+// own on stderr: open (open the database) and read (count the records,
+// then read them in signature order, writing each size's duplicate
+// groups to stdout once that size is read; see readReport). Each
+// display ends before the next phase starts, and before an error is
+// returned to be printed.
 func runReport(ctx context.Context, stdout io.Writer) error {
 	dbPath, source, err := databasePath()
 	if err != nil {
@@ -45,65 +58,196 @@ func runReport(ctx context.Context, stdout io.Writer) error {
 	fmt.Fprintf(os.Stderr, "report: database %s (%s)\n",
 		escapePath(dbPath), source)
 
+	// Opening can wait on a busy database for as long as the busy
+	// timeout.
+	prog := newProgress(progressParams{
+		label: reportPhase(openPhase), total: -1,
+		waitingFor: openingDatabase,
+	})
 	db, err := openReportDatabase(ctx, dbPath)
+
+	prog.finish()
+
 	if err != nil {
 		return err
 	}
 
 	defer func() { _ = db.Close() }()
 
-	out := bufio.NewWriterSize(stdout, ioBufSize)
+	// The read phase writes the report to stdout while it reads the
+	// records, so it may wait on either.
+	prog = newProgress(progressParams{
+		label: reportPhase("read"), unit: "records", total: -1,
+		waitingFor: countingRecords, stalledOn: "db/out",
+	})
+	report, err := readReport(ctx, db, dbPath, stdout, prog)
 
-	_, err = fmt.Fprintln(out, "first\tdupe\tsize")
-	if err != nil {
-		return fmt.Errorf("write stdout: %w", err)
-	}
-
-	var (
-		groups, dupeFiles int
-		reclaimable       int64
-		writeErr          error
-	)
-
-	records, err := loadDupeRows(ctx, db,
-		func(first, path string, size int64) error {
-			// A group's first path is its first row; every other
-			// path is a dupe.
-			if path == first {
-				groups++
-
-				return nil
-			}
-
-			_, writeErr = fmt.Fprintf(out, "%s\t%s\t%d\n",
-				escapePath(first), escapePath(path), size)
-			dupeFiles++
-			reclaimable += size
-
-			return writeErr
-		})
-
-	if writeErr != nil {
-		return fmt.Errorf("write stdout: %w", writeErr)
-	}
+	prog.finish()
 
 	if err != nil {
-		return fmt.Errorf("database %s: %w", dbPath, err)
-	}
-
-	err = out.Flush()
-	if err != nil {
-		return fmt.Errorf("write stdout: %w", err)
+		return err
 	}
 
 	fmt.Fprintf(os.Stderr,
 		"report: %d records read, %d duplicate groups, %d dupe files, "+
 			"%s reclaimable\n",
-		records, groups, dupeFiles, humanBytes(reclaimable))
+		report.records, report.dupeGroups, report.dupeFiles,
+		humanBytes(report.reclaimable))
 
 	return nil
 }
 
+// readReport gives prog the number of records in db, the database at
+// dbPath, as its total, then reads every record in signature order,
+// counting each on prog, and writes the report to stdout: the header
+// line, then the rows of each size's duplicate groups once that size is
+// read. stdout may be the terminal prog is drawn on, so the lines go
+// through prog's writer. It returns what it wrote, with the totals for
+// the summary line.
+func readReport(ctx context.Context, db *sql.DB, dbPath string,
+	stdout io.Writer, prog *progress,
+) (*reportWriter, error) {
+	out := prog.writer(stdout)
+	report := &reportWriter{out: out}
+
+	_, err := fmt.Fprintln(out, "first\tdupe\tsize")
+	if err != nil {
+		return nil, fmt.Errorf("write stdout: %w", err)
+	}
+
+	var writeErr error
+
+	err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
+		prog.increment()
+
+		writeErr = report.add(r)
+
+		return writeErr
+	})
+
+	if writeErr != nil {
+		return nil, fmt.Errorf("write stdout: %w", writeErr)
+	}
+
+	if err != nil {
+		return nil, fmt.Errorf("database %s: %w", dbPath, err)
+	}
+
+	err = report.finish()
+	if err == nil {
+		err = out.Flush()
+	}
+
+	if err != nil {
+		return nil, fmt.Errorf("write stdout: %w", err)
+	}
+
+	return report, nil
+}
+
+// reportWriter writes the report's rows to out from the records, added
+// in signature order, largest size first, as loadSignatureRows reads
+// them. The records sharing a signature come together, and so do the
+// records of one size, so it holds only the duplicate groups of the
+// size being read, and writes them, ordered by first path, once a
+// record of a smaller size comes. It counts what the summary line
+// reports.
+type reportWriter struct {
+	out io.Writer
+
+	sig    scanRec    // the signature being read; its path is not used
+	paths  []string   // the paths read so far with sig
+	groups [][]string // the duplicate groups of sig.size, each sorted
+
+	records, dupeGroups, dupeFiles int
+	reclaimable                    int64
+}
+
+// add takes the next record. One with the signature being read joins
+// its paths; any other ends them, and one of a smaller size first
+// writes the groups of the size before.
+func (w *reportWriter) add(r scanRec) error {
+	w.records++
+
+	// A record without a content hash has unknown content (README
+	// "Database"), so it is never a duplicate.
+	if r.content == "" {
+		return nil
+	}
+
+	if r.size == w.sig.size && r.head == w.sig.head &&
+		r.tail == w.sig.tail && r.content == w.sig.content {
+		w.paths = append(w.paths, r.path)
+
+		return nil
+	}
+
+	w.endGroup()
+
+	if r.size != w.sig.size {
+		err := w.writeGroups()
+		if err != nil {
+			return err
+		}
+	}
+
+	w.sig = r
+	w.paths = append(w.paths, r.path)
+
+	return nil
+}
+
+// endGroup ends the paths read with w.sig: two or more are a duplicate
+// group, held with its paths sorted until writeGroups writes it.
+func (w *reportWriter) endGroup() {
+	if len(w.paths) < minGroupSize {
+		// A path with no duplicate is dropped, and its slice reused.
+		w.paths = w.paths[:0]
+
+		return
+	}
+
+	slices.Sort(w.paths)
+	w.groups = append(w.groups, w.paths)
+	w.paths = nil
+}
+
+// writeGroups writes the duplicate groups held, all of size w.sig.size,
+// ordered by first path: for each group, a row for every path but the
+// first. No two groups share a first path, since no two records share
+// a path.
+func (w *reportWriter) writeGroups() error {
+	slices.SortFunc(w.groups, func(a, b []string) int {
+		return strings.Compare(a[0], b[0])
+	})
+
+	for _, g := range w.groups {
+		for _, path := range g[1:] {
+			_, err := fmt.Fprintf(w.out, "%s\t%s\t%d\n",
+				escapePath(g[0]), escapePath(path), w.sig.size)
+			if err != nil {
+				return err
+			}
+
+			w.dupeFiles++
+			w.reclaimable += w.sig.size
+		}
+
+		w.dupeGroups++
+	}
+
+	w.groups = nil
+
+	return nil
+}
+
+// finish writes the groups of the last size read.
+func (w *reportWriter) finish() error {
+	w.endGroup()
+
+	return w.writeGroups()
+}
+
 // escapePath returns a path as it is written in a report column (README
 // "Report output format"): a backslash, tab, newline or carriage return
 // becomes \\, \t, \n or \r, and every other byte is kept as it is.

report_test.go

@@ -9,6 +9,7 @@ import (
 	"os"
 	"path/filepath"
 	"slices"
+	"strconv"
 	"strings"
 	"testing"
 	"time"
@@ -28,7 +29,8 @@ func awkwardPairRecs() []scanRec {
 	}
 }
 
-// seedDatabase writes recs into a fresh database and returns its path.
+// seedDatabase writes recs into a fresh database, closes it as scan
+// does, out of WAL mode, and returns its path.
 func seedDatabase(t *testing.T, recs []scanRec) string {
 	t.Helper()
 
@@ -44,10 +46,7 @@ func seedDatabase(t *testing.T, recs []scanRec) string {
 		t.Fatal(err)
 	}
 
-	err = db.Close()
-	if err != nil {
-		t.Fatal(err)
-	}
+	closeScanDatabase(t.Context(), db, path)
 
 	return path
 }
@@ -60,27 +59,41 @@ type dupeGroup struct {
 }
 
 // dupeGroups returns the duplicate groups report reads from db, in
-// report order.
+// report order, from the rows it writes: a group's rows come together,
+// each naming the group's first path.
 func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
 	t.Helper()
 
-	var groups []dupeGroup
+	var out bytes.Buffer
 
-	_, err := loadDupeRows(t.Context(), db,
-		func(first, path string, size int64) error {
-			if path == first {
-				groups = append(groups, dupeGroup{size: size})
-			}
-
-			g := &groups[len(groups)-1]
-			g.paths = append(g.paths, path)
-
-			return nil
-		})
+	_, err := readReport(t.Context(), db, "", &out, nil)
 	if err != nil {
 		t.Fatal(err)
 	}
 
+	var groups []dupeGroup
+
+	rows := strings.Split(strings.TrimSuffix(out.String(), "\n"), "\n")
+	for _, row := range rows[1:] { // after the header
+		fields := strings.Split(row, "\t")
+		if len(fields) != 3 {
+			t.Fatalf("row %q, want first, dupe and size", row)
+		}
+
+		size, err := strconv.ParseInt(fields[2], 10, 64)
+		if err != nil {
+			t.Fatal(err)
+		}
+
+		if len(groups) == 0 || groups[len(groups)-1].paths[0] != fields[0] {
+			groups = append(groups,
+				dupeGroup{size: size, paths: []string{fields[0]}})
+		}
+
+		g := &groups[len(groups)-1]
+		g.paths = append(g.paths, fields[1])
+	}
+
 	return groups
 }
 
@@ -118,26 +131,147 @@ func TestRunReportEscapesPaths(t *testing.T) {
 }
 
 func TestReportStdoutFailsWhileReading(t *testing.T) {
-	// Each row holds two paths longer than dir, so the report is more
-	// than twice the stdout buffer and stdout fails while rows are
-	// still being read, not at the final flush.
+	t.Parallel()
+
+	// Every two records are a duplicate pair of a size of their own, and
+	// a pair's row is written once a smaller size is read. Each row
+	// holds two paths longer than dir, so the rows come to more than
+	// twice the stdout buffer, and stdout fails while records are still
+	// being read, not once every record is.
 	dir := "/" + strings.Repeat("d", 4096)
 
-	recs := make([]scanRec, ioBufSize/len(dir))
+	recs := make([]scanRec, 2*ioBufSize/len(dir))
 	for i := range recs {
 		recs[i] = scanRec{
-			size: 1, head: "h", tail: "t", content: "c",
+			size: int64(i/2) + 1, head: "h", tail: "t", content: "c",
 			path: fmt.Sprintf("%s/%d", dir, i),
 		}
 	}
 
-	t.Setenv(databaseEnv, seedDatabase(t, recs))
+	db := openTestDB(t)
 
-	err := runReport(t.Context(), failingWriter{})
+	err := applyChanges(t.Context(), db, recs, nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// Counts the records read, and draws nothing.
+	prog := &progress{now: time.Now}
+
+	_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
 	if !errors.Is(err, errWriteFailed) ||
 		!strings.HasPrefix(err.Error(), "write stdout: ") {
 		t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
 	}
+
+	if prog.count >= int64(len(recs)) {
+		t.Errorf("stdout failed after all %d records were read, want it "+
+			"to stop the read", prog.count)
+	}
+}
+
+// reportOpenLine is the plain line of report's open phase when it opens
+// the database at once; it prints it as the phase starts and as it ends.
+const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
+	"eta unknown"
+
+// smokeReport is report's stdout for the smoke-test records.
+const smokeReport = "first\tdupe\tsize\n" +
+	"/d/t1/f1\t/d/t2/f1\t3000\n" +
+	"/d/t1/f1\t/d/t3/f1\t3000\n" +
+	"/d/t1/sub/f2\t/d/t2/sub/f2\t100\n" +
+	"/d/t1/sub/f2\t/d/t3/sub/f2renamed\t100\n"
+
+// TestRunReportProgressNamesEachPhase runs report on the smoke-test
+// records with stderr not a terminal, and checks every line it prints
+// there. After the line naming the database, each phase prints a line
+// naming it with its place among the two as it starts and another as
+// it ends; read counts the 6 records before it reads them. The summary
+// comes last, and stdout holds the report alone.
+func TestRunReportProgressNamesEachPhase(t *testing.T) {
+	path := seedDatabase(t, smokeTreeRecs())
+	t.Setenv(databaseEnv, path)
+
+	stderr := captureStderr(t)
+
+	var stdout bytes.Buffer
+
+	code := run([]string{cmdReport}, &stdout, os.Stderr)
+	if code != exitOK {
+		t.Fatalf("run(report) = %d, want %d; stderr: %s",
+			code, exitOK, stderr())
+	}
+
+	if got := stdout.String(); got != smokeReport {
+		t.Errorf("stdout = %q, want %q", got, smokeReport)
+	}
+
+	// How each line starts; the rate at the end of a phase varies.
+	want := []string{
+		"report: database " + path + " (from " + databaseEnv + ")",
+		reportOpenLine,
+		reportOpenLine,
+		"[2/2] read: counting the records, elapsed 0s, eta unknown",
+		"[2/2] read: [6/6] 100%, ",
+		"report: 6 records read, 2 duplicate groups, 4 dupe files, " +
+			"6.1 KiB reclaimable",
+	}
+
+	lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
+	if len(lines) != len(want) {
+		t.Fatalf("stderr lines = %q, want %d lines starting %q",
+			lines, len(want), want)
+	}
+
+	for i := range want {
+		if !strings.HasPrefix(lines[i], want[i]) {
+			t.Errorf("stderr line %d = %q, want it to start %q",
+				i+1, lines[i], want[i])
+		}
+	}
+}
+
+// TestReadReportOnTerminal writes the smoke-test report to the terminal
+// its read phase's display is drawn on, as a plain "sfdupes report"
+// does: the terminal must show the header and every row whole, each on
+// a line of its own.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestReadReportOnTerminal(t *testing.T) {
+	stderr := captureStderr(t)
+	db := openTestDB(t)
+
+	err := applyChanges(t.Context(), db, smokeTreeRecs(), nil, nil)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// No timed draws: the display is drawn as it starts and as it ends.
+	p := startProgress(progressParams{
+		label: reportPhase("read"), unit: "records", total: -1,
+		waitingFor: countingRecords,
+	}, terminalWidth, time.Hour, time.Now)
+
+	_, err = readReport(t.Context(), db, "", os.Stderr, p)
+
+	p.finish()
+
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	// A terminal shows each line as the text after its last carriage
+	// return.
+	var shown []string
+
+	for line := range strings.SplitSeq(stderr(), "\n") {
+		shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
+	}
+
+	want := strings.Split(strings.TrimSuffix(smokeReport, "\n"), "\n")
+	if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
+		t.Errorf("terminal shows lines %q, want %q first", shown, want)
+	}
 }
 
 func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
@@ -354,6 +488,37 @@ func TestDupeGroupsTieBreak(t *testing.T) {
 	}
 }
 
+func TestDupeGroupsOrderedByFirstPath(t *testing.T) {
+	t.Parallel()
+
+	// report reads the groups of size 50 in signature order, here /a,
+	// /z, /k, which is neither the order of their first paths nor its
+	// reverse, and must write them ordered by first path. The group of
+	// size 7 comes after them all, though its first path sorts first.
+	recs := []scanRec{
+		{size: 50, head: "a", tail: "a", content: "a", path: "/k/2"},
+		{size: 50, head: "a", tail: "a", content: "a", path: "/k/1"},
+		{size: 50, head: "b", tail: "b", content: "b", path: "/z/1"},
+		{size: 50, head: "b", tail: "b", content: "b", path: "/z/2"},
+		{size: 50, head: "c", tail: "c", content: "c", path: "/a/2"},
+		{size: 50, head: "c", tail: "c", content: "c", path: "/a/1"},
+		{size: 7, head: "d", tail: "d", content: "d", path: "/0/2"},
+		{size: 7, head: "d", tail: "d", content: "d", path: "/0/1"},
+	}
+
+	groups := dupeGroupsOf(t, recs)
+
+	got := make([]string, 0, len(groups))
+	for _, g := range groups {
+		got = append(got, g.paths[0])
+	}
+
+	want := []string{"/a/1", "/k/1", "/z/1", "/0/1"}
+	if !slices.Equal(got, want) {
+		t.Errorf("groups' first paths = %q, want %q", got, want)
+	}
+}
+
 func TestDupeGroupsDeterministic(t *testing.T) {
 	t.Parallel()
 

scan.go

@@ -13,6 +13,7 @@ import (
 	"os"
 	"os/signal"
 	"path/filepath"
+	"runtime"
 	"slices"
 	"strings"
 	"sync"
@@ -114,9 +115,13 @@ func scanPhase(name string) string {
 // SIGTERM that interruptContext catches, the scan keeps what it has
 // hashed (see syncScan), prints how many files its walk reached, and
 // returns errInterrupted. workers must be at least 1; the scan command
-// rejects anything less.
+// rejects anything less. Before the lock is taken, the scan lists the
+// filesystems that are not persistent, on which the walk records
+// nothing; unless noDefaultExcludes is set, the walk does not enter the
+// default excludes either. Once the database is open, the scan notes
+// its files, which the walk passes over.
 func runScan(ctx context.Context, roots []string, workers int,
-	oneFS bool,
+	oneFS, noDefaultExcludes bool,
 ) error {
 	dbPath, source, err := databasePath()
 	if err != nil {
@@ -139,6 +144,16 @@ func runScan(ctx context.Context, roots []string, workers int,
 		return err
 	}
 
+	nonPersistent, err := nonPersistentFilesystems()
+	if err != nil {
+		return fmt.Errorf("list the mounted filesystems: %w", err)
+	}
+
+	rules := walkRules{nonPersistent: nonPersistent}
+	if !noDefaultExcludes {
+		rules.excludes = defaultExcludes(runtime.GOOS)
+	}
+
 	lock, err := lockScanDatabase(dbPath)
 	if err != nil {
 		return err
@@ -159,7 +174,12 @@ func runScan(ctx context.Context, roots []string, workers int,
 
 	defer closeScanDatabase(ctx, db, dbPath)
 
-	st, err := syncScan(ctx, db, roots, workers, oneFS)
+	rules.databaseFiles, err = databaseFiles(dbPath)
+	if err != nil {
+		return err
+	}
+
+	st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
 	if errors.Is(err, context.Canceled) {
 		return interrupted(st.walked)
 	}
@@ -231,14 +251,12 @@ func resolveRoots(roots []string) ([]string, error) {
 }
 
 // pruneRoots drops operands already covered by another operand:
-// duplicates and any operand lying under another one. With -x, an
-// operand under another one is kept when the walk of the deepest kept
-// operand above it cannot reach it: when device puts any directory from
-// it (from its directory, for a regular file) up to but not including
-// that operand on a different filesystem than that operand. Every
-// remaining file is then reachable through exactly one root, so walked
-// paths are unique without keeping a scan-wide set of every path seen.
-func pruneRoots(roots []string, oneFS bool,
+// duplicates and any operand lying under another one. An operand under
+// another one is kept, though, when the walk of the deepest kept operand
+// above it does not reach it (see walkReaches). Every remaining file is
+// then reachable through exactly one root, so walked paths are unique
+// without keeping a scan-wide set of every path seen.
+func pruneRoots(roots []string, oneFS bool, rules walkRules,
 	device func(path string) (uint64, bool),
 ) []string {
 	// Shorter-first ordering guarantees an ancestor is kept before any
@@ -262,40 +280,57 @@ func pruneRoots(roots []string, oneFS bool,
 			continue
 		}
 
-		// Without -x, and always for a duplicate, r is dropped.
-		if !oneFS || above == r {
-			continue
-		}
-
-		aboveDev, aboveOK := device(above)
-		if !aboveOK {
-			continue
-		}
-
-		// The walk of above enters a directory only when it is on
-		// above's filesystem, and then reaches every regular file in it
-		// whatever device the file reports, so for a regular-file
-		// operand the directories checked start at its directory.
-		dir := r
-
-		fi, err := os.Lstat(r)
-		if err == nil && fi.Mode().IsRegular() {
-			dir = filepath.Dir(r)
-		}
-
-		for ; dir != above; dir = filepath.Dir(dir) {
-			dev, ok := device(dir)
-			if ok && dev != aboveDev {
-				kept = append(kept, r)
-
-				break
-			}
+		// A duplicate is dropped, and so is an operand the walk of above
+		// reaches.
+		if above != r && !walkReaches(above, r, oneFS, rules, device) {
+			kept = append(kept, r)
 		}
 	}
 
 	return kept
 }
 
+// walkReaches reports whether the walk of operand above reaches r, an
+// operand under it. That walk does not enter a default exclude in
+// rules, nor a directory that device puts on a filesystem in rules that
+// is not persistent, nor, with -x, one that device puts on a different
+// filesystem than above, when device knows above's. It reaches every
+// regular file in a directory it enters, so for a regular-file operand
+// the directories checked start at its directory; they end below above.
+func walkReaches(above, r string, oneFS bool, rules walkRules,
+	device func(path string) (uint64, bool),
+) bool {
+	aboveDev, aboveDevOK := device(above)
+
+	dir := r
+
+	fi, err := os.Lstat(r)
+	if err == nil && fi.Mode().IsRegular() {
+		dir = filepath.Dir(r)
+	}
+
+	for ; dir != above; dir = filepath.Dir(dir) {
+		if isDefaultExclude(dir, rules.excludes) {
+			return false
+		}
+
+		dev, ok := device(dir)
+		if !ok {
+			continue
+		}
+
+		if _, found := rules.nonPersistent[dev]; found {
+			return false
+		}
+
+		if oneFS && aboveDevOK && dev != aboveDev {
+			return false
+		}
+	}
+
+	return true
+}
+
 // deviceOfPath returns the filesystem device ID of the file at path,
 // when lstat succeeds and the platform exposes one.
 func deviceOfPath(path string) (uint64, bool) {
@@ -337,11 +372,11 @@ type scanState struct {
 // the hashed records still waiting in the batch, starts no other write
 // or deletion, and returns the cancellation.
 func syncScan(ctx context.Context, db *sql.DB, roots []string,
-	workers int, oneFS bool,
+	workers int, oneFS bool, rules walkRules,
 ) (scanStats, error) {
 	s := &scanState{db: db}
 
-	err := s.runPhases(ctx, roots, workers, oneFS)
+	err := s.runPhases(ctx, roots, workers, oneFS, rules)
 	if err == nil || ctx.Err() == nil {
 		return s.st, err
 	}
@@ -365,15 +400,18 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
 // and content (fill in the content hash of every record of headTailMin
 // or more whose size, head, and tail match another record's). Records
 // outside the roots are never touched, except that the content phase
-// fills in their content hash. Operands the walk cannot start from are
-// dropped first, so the records beneath them are treated as if those
-// operands had not been given.
+// fills in their content hash. Operands the walk cannot start from, or
+// that are not on a persistent filesystem, are dropped first, so the
+// records beneath them are treated as if those operands had not been
+// given. Below an operand, the walk follows rules.
 func (s *scanState) runPhases(ctx context.Context, roots []string,
-	workers int, oneFS bool,
+	workers int, oneFS bool, rules walkRules,
 ) error {
-	// Types are checked before pruning so that an operand under a
-	// dropped one is still scanned, not dropped as lying under it.
-	roots = pruneRoots(s.walkableRoots(roots), oneFS, deviceOfPath)
+	// Types and filesystems are checked before pruning so that an
+	// operand under a dropped one is still scanned, not dropped as lying
+	// under it.
+	roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
+		deviceOfPath)
 
 	err := s.loadIndex(ctx, roots)
 	if err != nil {
@@ -381,7 +419,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 	}
 
 	changed, unhashed := s.walkPhase(
-		startWalk(ctx, roots, oneFS, workers), roots)
+		startWalk(ctx, roots, oneFS, rules, workers), roots)
 
 	// A cancelled walk stops early, so its size census covers only part
 	// of the roots, and every file it never reached would look vanished
@@ -403,7 +441,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 		return err
 	}
 
-	return s.contentPhase(ctx, workers)
+	return s.contentPhase(ctx, workers, rules)
 }
 
 // walkableRoots returns the operands the walk can start from: regular
@@ -411,9 +449,11 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 // about, counted as skipped, and dropped. A dropped operand is no
 // longer a root, so the records stored beneath it are treated as if it
 // had not been given: those under no other root count as outside the
-// roots and are not deleted as unverified. An operand that fails lstat
-// here is kept, and the walk warns about it.
-func (s *scanState) walkableRoots(roots []string) []string {
+// roots and are not deleted as unverified. An operand on a filesystem
+// in rules that is not persistent is dropped the same way, with a line
+// saying so that is not a warning, and is not counted as skipped. An
+// operand that fails lstat here is kept, and the walk warns about it.
+func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
 	kept := make([]string, 0, len(roots))
 
 	for _, root := range roots {
@@ -427,6 +467,13 @@ func (s *scanState) walkableRoots(roots []string) []string {
 
 				continue
 			}
+
+			line := rules.nonPersistentLine(root, fi)
+			if line != "" {
+				fmt.Fprintln(os.Stderr, escapePath(line))
+
+				continue
+			}
 		}
 
 		kept = append(kept, root)
@@ -446,7 +493,7 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 	// first gives the display its total.
 	prog := newProgress(progressParams{
 		label: scanPhase("load"), unit: scanUnit, total: -1,
-		waitingFor: "counting the records",
+		waitingFor: countingRecords, stalledOn: "db",
 	})
 	defer prog.finish()
 
@@ -481,10 +528,12 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 // (the content phase may still fill in its content hash). It returns
 // the new-or-changed files and the unchanged files whose records lack
 // hashes; both remain candidates until the census decides whether
-// their sizes are shared. A record below a directory -x did not enter
-// counts as outside the roots: it is not deleted, and its size joins
-// the census. That does not hold below a root at or inside that
-// directory, since that root's own walk verifies the records below it.
+// their sizes are shared. A record at or below a path the walk left
+// out, because -x stopped at it, it is a default exclude, or it is not
+// on a persistent filesystem, counts as outside the roots: it is not
+// deleted, and its size joins the census. That does not hold below a
+// root at or inside that path, since that root's own walk verifies the
+// records below it; nor is a line printed for such a root.
 func (s *scanState) walkPhase(
 	events <-chan walkEvent, roots []string,
 ) ([]fileRec, []fileRec) {
@@ -497,17 +546,30 @@ func (s *scanState) walkPhase(
 
 	prog := newProgress(progressParams{
 		label: scanPhase("walk"), unit: scanUnit, total: estimate,
-		estimate: true,
+		estimate: true, stalledOn: "disk",
 	})
 
 	var (
 		changed, unhashed []fileRec
-		otherFSDirs       []string
+		leftOut           []string
 	)
 
 	for ev := range events {
 		if ev.otherFSDir != "" {
-			otherFSDirs = append(otherFSDirs, ev.otherFSDir)
+			leftOut = append(leftOut, ev.otherFSDir)
+
+			continue
+		}
+
+		// The line saying so is printed as a warning is, but what the
+		// walk leaves out is not counted as skipped. A default exclude
+		// that is itself a root is walked as one, so it gets no line.
+		if ev.leftOut != "" {
+			leftOut = append(leftOut, ev.leftOut)
+
+			if !slices.Contains(roots, ev.leftOut) {
+				prog.warnf("%s", ev.warn)
+			}
 
 			continue
 		}
@@ -542,12 +604,13 @@ func (s *scanState) walkPhase(
 	prog.finish()
 
 	for path, old := range s.existing {
-		// Kept only when a directory -x did not enter lies below the
-		// deepest root above the record. A root at or below that
-		// directory was walked on its own and did not find the file.
-		// Both contain the record, so the longer path is the deeper.
-		otherFSDir := deepestRoot(path, otherFSDirs)
-		if len(otherFSDir) > len(deepestRoot(path, roots)) {
+		// Kept only when a path the walk left out lies below the deepest
+		// root above the record, or is the record's own. A root at or
+		// below that path was walked on its own and did not find the
+		// file. Both contain the record, so the longer path is the
+		// deeper.
+		left := deepestRoot(path, leftOut)
+		if len(left) > len(deepestRoot(path, roots)) {
 			delete(s.existing, path)
 
 			s.sizes = append(s.sizes, old.size)
@@ -653,6 +716,15 @@ func sameInode(a, b fileRec) bool {
 	return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
 }
 
+// isDatabaseFile reports whether f, a file the walk reached, is one of
+// the scan's own database files in rules, which the walk passes over
+// without a line and without counting it.
+func isDatabaseFile(f fileRec, rules walkRules) bool {
+	return slices.ContainsFunc(rules.databaseFiles, func(d fileRec) bool {
+		return sameInode(d, f)
+	})
+}
+
 // hashPhase hashes every queued file with hashSignature — the head and
 // tail of a file of headTailMin or more, the whole file below that —
 // committing completed records to the database in batches as results
@@ -687,8 +759,11 @@ func (s *scanState) readRuns(ctx context.Context, workers int,
 	pool := startHashPool(ctx, runs, workers, hash)
 	defer pool.stop()
 
+	// Between results the phase may wait on a file being read or on a
+	// batch of records being written.
 	prog := newProgress(progressParams{
 		label: scanPhase(label), unit: scanUnit, total: int64(len(runs)),
+		stalledOn: "disk/db",
 	})
 	defer prog.finish()
 
@@ -778,6 +853,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 	total := len(s.batch) + len(s.sentinels) + len(deletes)
 	prog := newProgress(progressParams{
 		label: scanPhase("update"), unit: scanUnit, total: int64(total),
+		stalledOn: "db",
 	})
 
 	defer prog.finish()
@@ -817,9 +893,12 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 // hash phase's worker pool and their records written back in batches. A
 // failed read is warned about and counted as skipped; the record keeps
 // its empty content, so it is never grouped, and a later scan tries
-// again.
-func (s *scanState) contentPhase(ctx context.Context, workers int) error {
-	toRead, recs, err := s.contentCandidates(ctx)
+// again. No file on a filesystem in rules that is not persistent is
+// read.
+func (s *scanState) contentPhase(ctx context.Context, workers int,
+	rules walkRules,
+) error {
+	toRead, recs, err := s.contentCandidates(ctx, rules)
 	if err != nil {
 		return err
 	}
@@ -840,33 +919,43 @@ func (s *scanState) contentPhase(ctx context.Context, workers int) error {
 		return err
 	}
 
-	return applyChanges(ctx, s.db, s.batch, nil, nil)
+	// The last partial batch is written under a display of its own, as
+	// the update phase writes the hash phase's.
+	prog := newProgress(progressParams{
+		label: scanPhase("content"), unit: scanUnit,
+		total: int64(len(s.batch)), stalledOn: "db",
+	})
+	defer prog.finish()
+
+	return applyChanges(ctx, s.db, s.batch, nil, prog)
 }
 
 // contentCandidates returns the files the content phase reads, and
 // their records by path. Every record contentCandidatesSQL returns has
 // its file checked with lstat, whether or not it already has a content
-// hash: a file that is gone, is no longer a regular file, or has
-// changed by the walk's rule keeps its record as it is and does not
-// count as a match for the others, and any other lstat error is warned
-// about and counted as skipped, with the same result. If such a record
-// has no content hash, it stays out of duplicate groups; if it has one,
-// it is still reported until a scan covering its own tree updates or
-// removes it. The files of a group that pass and have no content hash
-// are read only if at least minGroupSize of the group's files pass, so
-// a group whose other members are all stale costs no reads. Only the
-// records to be read are kept.
+// hash: a file that is gone, is no longer a regular file, has changed
+// by the walk's rule, or is on a filesystem in rules that is not
+// persistent keeps its record as it is and does not count as a match
+// for the others, and any other lstat error is warned about and counted
+// as skipped, with the same result. If such a record has no content
+// hash, it stays out of duplicate groups; if it has one, it is still
+// reported until a scan covering its own tree updates or removes it.
+// The files of a group that pass and have no content hash are read only
+// if at least minGroupSize of the group's files pass, so a group whose
+// other members are all stale costs no reads. Only the records to be
+// read are kept.
 func (s *scanState) contentCandidates(
-	ctx context.Context,
+	ctx context.Context, rules walkRules,
 ) ([]fileRec, map[string]scanRec, error) {
 	// The search and the checks take real time on a large database;
 	// without a display the scan looks hung before the reads begin.
 	// The display says it is searching until SQLite returns the first
 	// record. The checks start there, against a count of the records
-	// the search returns, and so do their rate and ETA.
+	// the search returns, and so do their rate and ETA. Between records
+	// they may wait on a file's lstat or on the search.
 	prog := newProgress(progressParams{
 		label: scanPhase("content"), unit: scanUnit, total: -1,
-		waitingFor: "searching the database",
+		waitingFor: "searching the database", stalledOn: "disk/db",
 	})
 	defer prog.finish()
 
@@ -914,7 +1003,7 @@ func (s *scanState) contentCandidates(
 			first = r
 		}
 
-		f, ok, err := unchangedFile(r)
+		f, ok, err := unchangedFile(r, rules)
 		if err != nil {
 			s.st.skipped++
 
@@ -948,9 +1037,10 @@ func (s *scanState) contentCandidates(
 
 // unchangedFile lstats the file r names and returns it for reading if
 // it is still the regular file r records: the same size, and an mtime
-// no newer than recorded (the walk's change rule). A file that is gone
-// or has changed reports false; any other lstat error is returned.
-func unchangedFile(r scanRec) (fileRec, bool, error) {
+// no newer than recorded (the walk's change rule). A file that is gone,
+// has changed, or is on a filesystem in rules that is not persistent
+// reports false; any other lstat error is returned.
+func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
 	fi, err := os.Lstat(r.path)
 	if errors.Is(err, fs.ErrNotExist) {
 		return fileRec{}, false, nil
@@ -967,6 +1057,12 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
 
 	dev, ino := inodeOfInfo(fi)
 
+	// The scan reads nothing on a filesystem that is not persistent,
+	// whatever record an earlier scan left for the file.
+	if _, found := rules.nonPersistent[dev]; found {
+		return fileRec{}, false, nil
+	}
+
 	return fileRec{
 		path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
 	}, true, nil
@@ -1033,32 +1129,36 @@ type dirJob struct {
 }
 
 // walkEvent is one walk result delivered to the walk phase: a regular
-// file's statted record, a warning when fail is set, or, when
-// otherFSDir is set, a directory -x did not enter because it is on
-// another filesystem than its operand.
+// file's statted record, a warning when fail is set, or a path the walk
+// left out: when otherFSDir is set, a directory -x did not enter
+// because it is on another filesystem than its operand, and when
+// leftOut is set, a default exclude or a directory or file on a
+// filesystem that is not persistent, with warn holding the line that
+// says so.
 type walkEvent struct {
 	rec        fileRec
 	warn       string
 	fail       bool
 	otherFSDir string
+	leftOut    string
 }
 
 // startWalk seeds every root into the shared walk worker pool and
 // returns the event stream: one record per regular file, one warning
-// event per per-path error, and one event per directory -x did not
-// enter. The channel is closed when the walk completes, and also when
-// ctx is cancelled — every goroutine in the pool abandons its blocking
-// send in that case, so the consumer sees a truncated but properly
-// terminated stream instead of a stalled one.
+// event per per-path error, and one event per path -x or rules kept the
+// walk from entering or recording. The channel is closed when the walk
+// completes, and also when ctx is cancelled — every goroutine in the
+// pool abandons its blocking send in that case, so the consumer sees a
+// truncated but properly terminated stream instead of a stalled one.
 func startWalk(ctx context.Context, roots []string, oneFS bool,
-	workers int,
+	rules walkRules, workers int,
 ) <-chan walkEvent {
-	jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS)
+	jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS, rules)
 
 	go func() {
 		initial := make([]dirJob, 0, len(roots))
 		for _, root := range roots {
-			initial = append(initial, seedRoot(ctx, root, events)...)
+			initial = append(initial, seedRoot(ctx, root, rules, events)...)
 		}
 
 		dispatchDirs(ctx, initial, jobs, subdirs)
@@ -1112,13 +1212,14 @@ func operandWarning(root string, fi fs.FileInfo) string {
 }
 
 // seedRoot turns one PATH operand into the walk's starting state: a
-// regular-file operand is statted and emitted directly, and a directory
+// regular-file operand is statted and emitted directly, unless it is
+// one of the scan's own database files in rules, and a directory
 // operand becomes an initial job. walkableRoots has already dropped
 // every other operand. One that has changed into something else since
 // is warned about and skipped here; it is still a root, so the records
 // stored beneath it, other than those under another root, are deleted
 // as unverified.
-func seedRoot(ctx context.Context, root string,
+func seedRoot(ctx context.Context, root string, rules walkRules,
 	events chan<- walkEvent,
 ) []dirJob {
 	fi, err := os.Lstat(root)
@@ -1145,14 +1246,17 @@ func seedRoot(ctx context.Context, root string,
 	}
 
 	dev, ino := inodeOfInfo(fi)
-
-	sendEvent(ctx, events, walkEvent{rec: fileRec{
+	rec := fileRec{
 		path:  root,
 		size:  fi.Size(),
 		mtime: fi.ModTime(),
 		dev:   dev,
 		ino:   ino,
-	}})
+	}
+
+	if !isDatabaseFile(rec, rules) {
+		sendEvent(ctx, events, walkEvent{rec: rec})
+	}
 
 	return nil
 }
@@ -1164,7 +1268,7 @@ func seedRoot(ctx context.Context, root string,
 // workers drop the directories still queued rather than stop reading
 // jobs, so the range always runs out and the pool always tears down.
 func startWalkWorkers(ctx context.Context, workers int,
-	oneFS bool,
+	oneFS bool, rules walkRules,
 ) (chan dirJob, chan []dirJob, chan walkEvent) {
 	jobs := make(chan dirJob, workQueueDepth)
 	subdirs := make(chan []dirJob, workers)
@@ -1180,7 +1284,7 @@ func startWalkWorkers(ctx context.Context, workers int,
 				}
 
 				select {
-				case subdirs <- walkOneDir(ctx, job, oneFS, events):
+				case subdirs <- walkOneDir(ctx, job, oneFS, rules, events):
 				case <-ctx.Done():
 				}
 			}
@@ -1238,7 +1342,7 @@ func dispatchDirs(ctx context.Context, initial []dirJob,
 // entry and a warning event per unreadable one, and returns the
 // subdirectories to descend into.
 func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
-	events chan<- walkEvent,
+	rules walkRules, events chan<- walkEvent,
 ) []dirJob {
 	entries, err := os.ReadDir(job.path)
 	if err != nil {
@@ -1262,7 +1366,8 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 		p := filepath.Join(job.path, e.Name())
 
 		if e.IsDir() {
-			if sub, ok := subdirJob(ctx, p, e, job, oneFS, events); ok {
+			sub, ok := subdirJob(ctx, p, e, job, oneFS, rules, events)
+			if ok {
 				subs = append(subs, sub)
 			}
 
@@ -1274,7 +1379,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 			continue
 		}
 
-		emitFile(ctx, p, e, events)
+		emitFile(ctx, p, e, rules, events)
 	}
 
 	return subs
@@ -1284,9 +1389,12 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 // The lstat happens here in the walk worker, while the directory's
 // metadata is still hot; a path that fails to stat (or stops being a
 // regular file) between the directory read and the lstat is warned
-// about and skipped.
+// about and skipped. A file on a filesystem in rules that is not
+// persistent, as through a bind mount of one file, is not recorded: it
+// gets an event naming it instead. One of the scan's own database files
+// in rules gets no event at all.
 func emitFile(ctx context.Context, p string, e fs.DirEntry,
-	events chan<- walkEvent,
+	rules walkRules, events chan<- walkEvent,
 ) {
 	info, err := e.Info()
 	if err != nil {
@@ -1302,35 +1410,43 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
 		return
 	}
 
-	dev, ino := inodeOfInfo(info)
+	line := rules.nonPersistentLine(p, info)
+	if line != "" {
+		sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
 
-	sendEvent(ctx, events, walkEvent{rec: fileRec{
+		return
+	}
+
+	dev, ino := inodeOfInfo(info)
+	rec := fileRec{
 		path:  p,
 		size:  info.Size(),
 		mtime: info.ModTime(),
 		dev:   dev,
 		ino:   ino,
-	}})
+	}
+
+	if !isDatabaseFile(rec, rules) {
+		sendEvent(ctx, events, walkEvent{rec: rec})
+	}
 }
 
 // subdirJob applies the descent rules to directory p: never enter
 // .zfs (ZFS snapshot pseudo-dirs would list every file once per
-// snapshot), and with -x never enter a directory on a different
-// filesystem than its operand, sending an event that names it so the
-// scan keeps the records below it, except those below an operand at or
+// snapshot), never enter a directory on a filesystem in rules that is
+// not persistent, nor a default exclude in rules, and with -x never
+// enter a directory on a different filesystem than its operand. For the
+// last three it sends an event that names the directory, so the scan
+// keeps the records below it, except those below an operand at or
 // inside it.
 func subdirJob(ctx context.Context, p string, e fs.DirEntry,
-	parent dirJob, oneFS bool, events chan<- walkEvent,
+	parent dirJob, oneFS bool, rules walkRules,
+	events chan<- walkEvent,
 ) (dirJob, bool) {
 	if e.Name() == ".zfs" {
 		return dirJob{}, false
 	}
 
-	job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
-	if !oneFS || !parent.rootDevOK {
-		return job, true
-	}
-
 	info, err := e.Info()
 	if err != nil {
 		sendEvent(ctx, events, walkEvent{
@@ -1341,6 +1457,27 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
 		return dirJob{}, false
 	}
 
+	line := rules.nonPersistentLine(p, info)
+	if line != "" {
+		sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
+
+		return dirJob{}, false
+	}
+
+	if isDefaultExclude(p, rules.excludes) {
+		sendEvent(ctx, events, walkEvent{
+			leftOut: p,
+			warn:    fmt.Sprintf("scan: skipped %s (default exclude)", p),
+		})
+
+		return dirJob{}, false
+	}
+
+	job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
+	if !oneFS || !parent.rootDevOK {
+		return job, true
+	}
+
 	if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
 		sendEvent(ctx, events, walkEvent{otherFSDir: p})
 

scan_test.go

@@ -782,7 +782,8 @@ func collectWalk(t *testing.T, roots []string, oneFS bool,
 		errs int
 	)
 
-	for ev := range startWalk(t.Context(), roots, oneFS, workers) {
+	for ev := range startWalk(t.Context(), roots, oneFS, walkRules{},
+		workers) {
 		if ev.fail {
 			errs++
 
@@ -1004,7 +1005,8 @@ func callSubdirJob(t *testing.T, p string, e fs.DirEntry,
 	t.Helper()
 
 	events := make(chan walkEvent, 1)
-	job, ok := subdirJob(t.Context(), p, e, parent, oneFS, events)
+	job, ok := subdirJob(t.Context(), p, e, parent, oneFS, walkRules{},
+		events)
 	close(events)
 
 	var evs []walkEvent
@@ -1152,7 +1154,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
 		return dev, ok
 	}
 
-	roots = pruneRoots(roots, true, device)
+	roots = pruneRoots(roots, true, walkRules{}, device)
 
 	ctx := t.Context()
 	s := &scanState{db: db}
@@ -1162,14 +1164,14 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
 		t.Fatal(err)
 	}
 
-	jobs, subdirs, events := startWalkWorkers(ctx, 2, true)
+	jobs, subdirs, events := startWalkWorkers(ctx, 2, true, walkRules{})
 
 	// seedRoot emits a regular-file operand itself and returns a job
 	// only for a directory operand.
 	var initial []dirJob
 
 	for _, root := range roots {
-		for _, job := range seedRoot(ctx, root, events) {
+		for _, job := range seedRoot(ctx, root, walkRules{}, events) {
 			job.rootDev = devices[root]
 			initial = append(initial, job)
 		}
@@ -1463,7 +1465,7 @@ const smokeTreeFiles = 15
 func syncTree(t *testing.T, db *sql.DB, roots ...string) scanStats {
 	t.Helper()
 
-	st, err := syncScan(t.Context(), db, roots, 4, false)
+	st, err := syncScan(t.Context(), db, roots, 4, false, walkRules{})
 	if err != nil {
 		t.Fatalf("syncScan: %v", err)
 	}
@@ -2412,7 +2414,7 @@ func TestPruneRoots(t *testing.T) {
 	// Duplicates and operands under other operands are dropped; /cc is
 	// not under /c (sibling with a shared prefix).
 	got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
-		false, device)
+		false, walkRules{}, device)
 
 	want := []string{"/a", "/c", "/cc"}
 	if !slices.Equal(got, want) {
@@ -2421,7 +2423,7 @@ func TestPruneRoots(t *testing.T) {
 
 	nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
 
-	got = pruneRoots(nested, false, device)
+	got = pruneRoots(nested, false, walkRules{}, device)
 
 	want = []string{"/p"}
 	if !slices.Equal(got, want) {
@@ -2429,12 +2431,31 @@ func TestPruneRoots(t *testing.T) {
 	}
 
 	// With -x, /p/q, on another filesystem than /p, is kept as well.
-	got = pruneRoots(nested, true, device)
+	got = pruneRoots(nested, true, walkRules{}, device)
 
 	want = []string{"/p", pqRoot}
 	if !slices.Equal(got, want) {
 		t.Fatalf("pruneRoots with -x = %q, want %q", got, want)
 	}
+
+	// The walk of /p does not enter /p/q when it is a default exclude, so
+	// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
+	got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
+		device)
+	if !slices.Equal(got, want) {
+		t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
+	}
+
+	// Nor does it enter /p/q when the filesystem of /p/q is not
+	// persistent, so /p/q/r/m, of no known device, is kept.
+	got = pruneRoots([]string{"/p", "/p/q/r/m"}, false,
+		walkRules{nonPersistent: map[uint64]string{2: "tmpfs"}}, device)
+
+	want = []string{"/p", "/p/q/r/m"}
+	if !slices.Equal(got, want) {
+		t.Fatalf("pruneRoots with %s not persistent = %q, want %q",
+			pqRoot, got, want)
+	}
 }
 
 func TestReportsNeverTouchFilesystem(t *testing.T) {
@@ -2480,3 +2501,99 @@ func TestUnderRoot(t *testing.T) {
 		}
 	}
 }
+
+// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
+// database, reached directly and through a symlinked directory. The
+// walk passes over the database file, its -wal and -shm files and its
+// .lock file with no line and no count, and the record an earlier scan
+// left for the .lock file is deleted. A rescan of the unchanged tree
+// updates nothing, and report lists none of the database's files.
+func TestRunScanPassesOverDatabase(t *testing.T) {
+	for _, viaSymlink := range []bool{false, true} {
+		t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
+			base := t.TempDir()
+			tree := filepath.Join(base, "real", "tree")
+			operand := tree
+
+			writeFile(t, tree, "a", pattern(1, 100))
+			writeFile(t, tree, "b", pattern(1, 100))
+			writeFile(t, tree, "empty1", nil)
+			writeFile(t, tree, "empty2", nil)
+
+			if viaSymlink {
+				err := os.Symlink(filepath.Join(base, "real"),
+					filepath.Join(base, "link"))
+				if err != nil {
+					t.Fatal(err)
+				}
+
+				operand = filepath.Join(base, "link", "tree")
+			}
+
+			dbPath := filepath.Join(tree, "db.sqlite")
+			t.Setenv(databaseEnv, dbPath)
+
+			db, err := openScanDatabase(t.Context(), dbPath)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			err = applyChanges(t.Context(), db, []scanRec{
+				{path: filepath.Join(operand, "db.sqlite.lock")},
+			}, nil, nil)
+			if err != nil {
+				t.Fatal(err)
+			}
+
+			closeScanDatabase(t.Context(), db, dbPath)
+
+			assertScanSummary(t,
+				"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
+				operand)
+			assertScanSummary(t,
+				"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
+				operand)
+
+			report := runStdout(t, cmdReport)
+			if strings.Contains(report, "db.sqlite") ||
+				!strings.Contains(report, filepath.Join(operand, "empty2")) {
+				t.Errorf("report = %q, want the empty files and none of "+
+					"the database's files", report)
+			}
+		})
+	}
+}
+
+// TestRunScanPassesOverDatabaseOperands scans the database file and its
+// .lock file given as operands themselves: neither is recorded or
+// counted.
+func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
+	dbPath := testDBPath(t)
+	t.Setenv(databaseEnv, dbPath)
+
+	// The first scan creates both files.
+	scanOK(t, t.TempDir())
+
+	assertScanSummary(t,
+		"0 files seen (0 added, 0 updated, 0 removed, 0 unchanged)",
+		dbPath, dbPath+".lock")
+}
+
+// assertScanSummary scans operands and fails the test unless the
+// summary line gives counts with nothing skipped, and the line naming
+// the database is the only one that names one of its files.
+func assertScanSummary(t *testing.T, counts string, operands ...string) {
+	t.Helper()
+
+	stderr := scanOK(t, operands...)
+
+	want := "scan: " + counts + ", 0 skipped\n"
+	if !strings.Contains(stderr, want) {
+		t.Errorf("scan %q: stderr = %q, want %q", operands, stderr, want)
+	}
+
+	if strings.Count(stderr, "db.sqlite") != 1 {
+		t.Errorf("scan %q: stderr = %q, want only the line naming the "+
+			"database to name one of its files", operands, stderr)
+	}
+}

trees.go

@@ -28,7 +28,7 @@ type treeNode struct {
 // as "[2/4] read", for the phase's progress display. The list here is
 // the one place the phases are counted.
 func treesPhase(name string) string {
-	return phaseLabel([]string{"open", "read", "group", "write"}, name)
+	return phaseLabel([]string{openPhase, "read", "group", "write"}, name)
 }
 
 // runTrees implements the trees subcommand: it reads every record from
@@ -57,8 +57,8 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
 	// Opening can wait on a busy database for as long as the busy
 	// timeout.
 	prog := newProgress(progressParams{
-		label: treesPhase("open"), total: -1,
-		waitingFor: "opening the database",
+		label: treesPhase(openPhase), total: -1,
+		waitingFor: openingDatabase,
 	})
 	db, err := openReportDatabase(ctx, dbPath)
 
@@ -72,7 +72,7 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
 
 	prog = newProgress(progressParams{
 		label: treesPhase("read"), unit: "records", total: -1,
-		waitingFor: "counting the records",
+		waitingFor: countingRecords, stalledOn: "db",
 	})
 	records, super, allDirs, err := readTree(ctx, db, prog)
 
@@ -82,9 +82,11 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
 		return fmt.Errorf("database %s: %w", dbPath, err)
 	}
 
+	// Grouping works in memory, so it waits on nothing but the
+	// processor.
 	prog = newProgress(progressParams{
 		label: treesPhase("group"), unit: "directories",
-		total: int64(len(allDirs)),
+		total: int64(len(allDirs)), stalledOn: "cpu",
 	})
 	dupes := collectTreeGroups(allDirs, super, prog)
 
@@ -92,6 +94,7 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
 
 	prog = newProgress(progressParams{
 		label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
+		stalledOn: "out",
 	})
 	dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
 

trees_test.go

@@ -279,7 +279,7 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) {
 	// No timed draws: the display is drawn as it starts and as it ends.
 	p := startProgress(progressParams{
 		label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
-	}, terminalWidth, time.Hour)
+	}, terminalWidth, time.Hour, time.Now)
 
 	_, _, err := writeTreeGroups(os.Stderr, dupes, p)