scan: skip OS default excludes, and record nothing on filesystems that are not persistent (closes #124)

scan never records or reads anything on a filesystem that is not
persistent (tmpfs, proc, sysfs and the like), however it reaches it; no
flag turns this off. It notes each such filesystem's device ID as it
starts: on Linux from /proc/self/mountinfo, on macOS by lstat of each
listed mount point. Below an operand the walk also skips the default
excludes, a path list per system (Linux, macOS, and Windows, not used
yet); --no-default-excludes turns off only those. A default exclude that
is itself an operand gets no skip line. The tests stop at once, saying
why, when TMPDIR is not on a persistent filesystem.

Judgement call: an unreadable mount list fails the scan, and no flag gets round it.
Judgement call: a file on such a filesystem below an operand gets the same line as a directory.
Judgement call: such an operand that another operand's walk also reaches is named twice.
Unverified: whether macOS's mount list gives the device lstat reports, so macOS keeps lstat, and a hidden mount there can still add another filesystem's device.
Unverified: the macOS code is never built or run here; bind mounts need root to test.

Model: opus-5-5

README.md

@@ -100,9 +100,11 @@ As a file `/etc/cron.d/sfdupes`:
   `/var/log/sfdupes.log`, holds a first line naming the database (see
   "Database"), plain progress lines naming each phase and its place among the
   five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
-  runs, and one as it ends), a warning for each path it skips, and the summary
-  line (see "Progress" and "`scan` mode"). The log grows with every scan; rotate
-  it like any other.
+  runs, and one as it ends), a warning for each path it skips, a line for each
+  default exclude its walk does not enter and for each path it leaves out
+  because it is not on a persistent filesystem (see "Rules for the walk"), and
+  the summary line (see "Progress" and "`scan` mode"). The log grows with every
+  scan; rotate it like any other.
 - Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
   scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
   reason among the last lines of the log; `tail` prints them, and cron mails
@@ -248,7 +250,7 @@ Four subcommands, all implemented:
     schema version and the counts of its records.
 
 ```
-sfdupes scan [--workers N] [-x] PATH...
+sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
 sfdupes report > dupes.tsv
 sfdupes trees > dupetrees.tsv
 sfdupes info
@@ -257,8 +259,9 @@ sfdupes [command] --help
 ```
 
 `--workers N` sets the size of each `scan` worker pool (default: the number of
-CPUs), and `-x` (`--one-file-system`) keeps the walk of each operand on that
-operand's filesystem; both are described under "`scan` mode".
+CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
+operand's filesystem, and `--no-default-excludes` lets the walk enter the
+default excludes, a list of paths; all three are described under "`scan` mode".
 `sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
 exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
 subcommand, prints the help text to stderr and exits 0, writing nothing to
@@ -439,11 +442,12 @@ results do not depend on the working directory. All operands belong to a single
 scan and are enumerated concurrently: every operand seeds the shared walk worker
 pool. Overlapping operands are harmless — an operand that duplicates another or
 lies under another is dropped before walking, so every file is reached exactly
-once and produces one database record. With `-x`, though, an operand under
-another one is kept and walked on its own when any directory from it (from its
-directory, for a regular file) up to but not including the operand covering it
-is on a different filesystem than that operand, because that operand's walk does
-not enter such a directory and so does not reach it.
+once and produces one database record. An operand under another one is kept,
+though, and walked on its own when any directory from it (from its directory,
+for a regular file) up to but not including the operand covering it is a default
+exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
+`-x`, is on a different filesystem than that operand, because that operand's
+walk does not enter such a directory and so does not reach it.
 
 An operand that is a symlink (never followed, not even as an operand), socket,
 FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -453,7 +457,10 @@ beneath it is still scanned. The records stored beneath it are not deleted: they
 are treated like any other record outside the scanned operands, including the
 content-phase exception below. If it lies under another operand, they are under
 that operand instead, and are treated like any other record there. This is not
-an error: a scan whose every operand is dropped walks nothing and exits 0.
+an error: a scan whose every operand is dropped walks nothing and exits 0. An
+operand that is not on a persistent filesystem (see "Rules for the walk") is
+dropped the same way, except that the line `scan` prints for it is not a
+warning, and it is not counted as skipped.
 
 `scan` synchronizes the database with the filesystem state under the scanned
 operands:
@@ -489,14 +496,16 @@ operands:
   can be scanned on different schedules into the same database. The one
   exception is the content phase below: a stored file of 10 MiB or more without
   a `content` hash is read for one, wherever it lies, once its size, `head`, and
-  `tail` match another record's. If that file is gone or has changed since its
-  record was written, the record is left as it is.
-- With `-x`, a record below a directory the walk did not enter because that
-  directory is on another filesystem than its operand counts as outside the
-  scanned operands: it is not deleted, and its size joins the size census. A
-  filesystem mounted under an operand can therefore be scanned on its own
-  schedule into the same database. Records below an operand at or inside such a
-  directory are not kept this way, since that operand's own walk verifies them.
+  `tail` match another record's. If that file is gone, has changed since its
+  record was written, or is not on a persistent filesystem, it is not read and
+  the record is left as it is.
+- A record at or below a path the walk left out, because that path is a default
+  exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
+  another filesystem than its operand, counts as outside the scanned operands:
+  it is not deleted, and its size joins the size census. A filesystem mounted
+  under an operand can therefore be scanned on its own schedule into the same
+  database. Records below an operand at or inside such a path are not kept this
+  way, since that operand's own walk verifies them.
 
 `scan` runs **five sequential phases over the whole scan**. Parallelism lives
 inside the walk, hash and content phases; batched database writes begin during
@@ -542,19 +551,20 @@ the hash phase:
     be read are kept in memory, never every file's hashes. Every record sharing
     their size, `head`, and `tail`, including one that already has a `content`
     hash, has its file checked with `lstat` first. A file that is gone, is no
-    longer a regular file, or has changed (a different size, or an mtime newer
-    than recorded) keeps its record as it is and does not count as a match for
-    the others. Any other `lstat` error is warned about and counted as skipped,
-    with the same result. If such a record has no `content` hash, it stays out
-    of duplicate groups; if it has one, it is still reported until a scan
-    covering its own tree updates or removes it. The files that pass and have no
-    `content` hash are read only if at least two of those records pass, so a
-    file whose only matches are stale costs no read; a file that already has a
-    `content` hash is never read again. They are read by a worker pool as in the
-    hash phase, in inode order and once per inode, and their content hashes are
-    committed in batches. A failed read is warned about and counted as skipped;
-    its record keeps an empty `content`, so it is not a duplicate, and a later
-    scan tries again.
+    longer a regular file, has changed (a different size, or an mtime newer than
+    recorded), or is not on a persistent filesystem (see "Rules for the walk")
+    keeps its record as it is and does not count as a match for the others. Any
+    other `lstat` error is warned about and counted as skipped, with the same
+    result. If such a record has no `content` hash, it stays out of duplicate
+    groups; if it has one, it is still reported until a scan covering its own
+    tree updates or removes it. The files that pass and have no `content` hash
+    are read only if at least two of those records pass, so a file whose only
+    matches are stale costs no read; a file that already has a `content` hash is
+    never read again. They are read by a worker pool as in the hash phase, in
+    inode order and once per inode, and their content hashes are committed in
+    batches. A failed read is warned about and counted as skipped; its record
+    keeps an empty `content`, so it is not a duplicate, and a later scan tries
+    again.
 
 Rules for the walk:
 
@@ -572,6 +582,74 @@ Rules for the walk:
   unless they lie below an operand at or inside that directory, count as outside
   the scanned operands, as described in "`scan` mode" above: they are not
   deleted, and their sizes join the size census.
+- Never record anything that is not on a persistent filesystem. A filesystem is
+  not persistent when the kernel makes it up or keeps it in memory: one of type
+  `proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
+  `debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
+  `mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
+  `rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
+  it names the database and before it takes the lock, `scan` lists the mounted
+  filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
+  of mounted filesystems on macOS (on any other system it lists none), and notes
+  the device ID of each one of those types. On Linux that is the device ID the
+  list gives, so a filesystem that a later mount hides still gives its own. On
+  macOS it is read with `lstat` on the mount point, and where one filesystem is
+  mounted over another on the same mount point, the one mounted last counts. If
+  `scan` cannot list them, it fails (exit 1). The walk then never enters a
+  directory, and never records a file, on one of those devices, however it
+  reaches it: as an operand, below one, or through a bind mount, which keeps the
+  device ID of what it binds. The content phase never reads a file on one of
+  them either. No flag turns this off. For each operand, directory or file it
+  leaves out this way, `scan` prints a line on stderr naming it and the type of
+  its filesystem, as it prints a warning, so it never lands inside a progress
+  display:
+
+    ```
+    scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
+    ```
+
+    That line is not a warning, and the path is not counted as skipped. Records
+    that an earlier scan left at or below it, unless they lie below an operand
+    at or inside it, count as outside the scanned operands, as described in
+    "`scan` mode" above: they are not deleted, and their sizes join the size
+    census.
+
+- Below an operand, never descend into a default exclude: a path that holds the
+  kernel's view of the running system, or temporary files, so that
+  `sfdupes scan /` walks only the files worth comparing. The default excludes
+  depend on the system, and `sfdupes scan --help` lists those of the system it
+  runs on:
+    - On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
+    - On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
+      `/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
+      everything in `/System/Volumes/Data` through the firmlinks, so walking it
+      as well would record each of those files twice.
+    - On Windows, relative to the root of the drive an operand lies on: the
+      page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
+      `hiberfil.sys`, and the directories `System Volume Information`,
+      `$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
+      matches one path component, as it does in any default exclude. sfdupes
+      does not build for Windows yet, so this list is not used.
+    - On any other system there are none.
+
+    `--no-default-excludes` turns them all off; what is not on a persistent
+    filesystem stays left out. An operand at or inside a default exclude is
+    still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
+    persistent filesystem. For each default exclude its walk does not enter,
+    `scan` prints a line on stderr naming it, as it prints a warning:
+
+    ```
+    scan: skipped /var/tmp (default exclude)
+    ```
+
+    That line is not a warning, and the directory is not counted as skipped. A
+    default exclude that is not on a persistent filesystem, as `/proc` is, gets
+    the line for its filesystem instead, and one that is itself an operand gets
+    no line, since it is walked as that operand. The records below a default
+    exclude the walk does not enter, unless they lie below an operand at or
+    inside it, count as outside the scanned operands, as described in "`scan`
+    mode" above: they are not deleted, and their sizes join the size census.
+
 - On any per-path error (permission denied, file vanished between passes,
   unreadable): print a one-line warning to stderr, skip the path, and continue.
   Per-file errors never abort the run; the final summary reports how many were