scan: skip OS-appropriate default excludes (/proc, /sys, /tmp, /run, /var/tmp on Linux) (closes #124)
check / check (push) Canceled after 0s

Below an operand, the walk no longer enters the default excludes: on
Linux /proc, /sys, /dev, /run, /tmp, /var/tmp and every mount point of a
kernel or memory-backed filesystem, read once from /proc/self/mountinfo;
on macOS /dev, /private/tmp, /private/var/tmp, /private/var/vm,
/System/Volumes/VM and /System/Volumes/Data. The lists live in the new
exclude.go. An operand at or inside one is still walked, also under
another operand, as with -x. Each skipped directory gets one stderr line
through the warning path, is not counted as skipped, and its records
count as outside the scanned operands. --no-default-excludes turns them
off; scan --help lists them.

Judgement call: an unreadable /proc/self/mountinfo fails the scan.
Judgement call: where filesystems are mounted over one another, the last decides.

Model: opus-5-5
This commit is contained in:
2026-10-16 23:28:07 +00:00
parent 68f6bde5ee
commit 5af53914e8
9 changed files with 693 additions and 120 deletions
+57 -17
View File
@@ -100,9 +100,10 @@ As a file `/etc/cron.d/sfdupes`:
`/var/log/sfdupes.log`, holds a first line naming the database (see
"Database"), plain progress lines naming each phase and its place among the
five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
runs, and one as it ends), a warning for each path it skips, and the summary
line (see "Progress" and "`scan` mode"). The log grows with every scan; rotate
it like any other.
runs, and one as it ends), a warning for each path it skips, a line for each
default exclude its walk does not enter (see "Rules for the walk"), and the
summary line (see "Progress" and "`scan` mode"). The log grows with every
scan; rotate it like any other.
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
reason among the last lines of the log; `tail` prints them, and cron mails
@@ -248,7 +249,7 @@ Four subcommands, all implemented:
schema version and the counts of its records.
```
sfdupes scan [--workers N] [-x] PATH...
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
sfdupes info
@@ -257,8 +258,9 @@ sfdupes [command] --help
```
`--workers N` sets the size of each `scan` worker pool (default: the number of
CPUs), and `-x` (`--one-file-system`) keeps the walk of each operand on that
operand's filesystem; both are described under "`scan` mode".
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
operand's filesystem, and `--no-default-excludes` lets the walk enter the
default excludes; all three are described under "`scan` mode".
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
subcommand, prints the help text to stderr and exits 0, writing nothing to
@@ -439,11 +441,12 @@ results do not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared walk worker
pool. Overlapping operands are harmless — an operand that duplicates another or
lies under another is dropped before walking, so every file is reached exactly
once and produces one database record. With `-x`, though, an operand under
another one is kept and walked on its own when any directory from it (from its
directory, for a regular file) up to but not including the operand covering it
is on a different filesystem than that operand, because that operand's walk does
not enter such a directory and so does not reach it.
once and produces one database record. An operand under another one is kept,
though, and walked on its own when any directory from it (from its directory,
for a regular file) up to but not including the operand covering it is a default
exclude (see "Rules for the walk") or, with `-x`, is on a different filesystem
than that operand, because that operand's walk does not enter such a directory
and so does not reach it.
An operand that is a symlink (never followed, not even as an operand), socket,
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -491,12 +494,13 @@ operands:
a `content` hash is read for one, wherever it lies, once its size, `head`, and
`tail` match another record's. If that file is gone or has changed since its
record was written, the record is left as it is.
- With `-x`, a record below a directory the walk did not enter because that
directory is on another filesystem than its operand counts as outside the
scanned operands: it is not deleted, and its size joins the size census. A
filesystem mounted under an operand can therefore be scanned on its own
schedule into the same database. Records below an operand at or inside such a
directory are not kept this way, since that operand's own walk verifies them.
- A record below a directory the walk did not enter, because that directory is a
default exclude or, with `-x`, because it is on another filesystem than its
operand, counts as outside the scanned operands: it is not deleted, and its
size joins the size census. A filesystem mounted under an operand can
therefore be scanned on its own schedule into the same database. Records below
an operand at or inside such a directory are not kept this way, since that
operand's own walk verifies them.
`scan` runs **five sequential phases over the whole scan**. Parallelism lives
inside the walk, hash and content phases; batched database writes begin during
@@ -572,6 +576,42 @@ Rules for the walk:
unless they lie below an operand at or inside that directory, count as outside
the scanned operands, as described in "`scan` mode" above: they are not
deleted, and their sizes join the size census.
- Below an operand, never descend into a default exclude: a directory that holds
the kernel's view of the running system, or temporary files, so that
`sfdupes scan /` walks only the files worth comparing. The default excludes
depend on the system, and `sfdupes scan --help` lists those of the system it
runs on:
- On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`, and the
mount point of every filesystem of type `proc`, `sysfs`, `devtmpfs`,
`devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`, `debugfs`, `tracefs`,
`securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`, `mqueue`,
`hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`, `rpc_pipefs` or
`selinuxfs`, wherever it is mounted. Where one filesystem is mounted over
another, the one mounted last counts. `scan` reads the mount points from
`/proc/self/mountinfo` once, after it names the database and before it
takes the lock; if it cannot read them, it fails (exit 1).
- On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
`/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
everything in `/System/Volumes/Data` through the firmlinks, so walking it
as well would record each of those files twice.
- On any other system there are none.
`--no-default-excludes` turns them all off, and `scan` then does not read
`/proc/self/mountinfo`. An operand at or inside a default exclude is still
walked, so `sfdupes scan /tmp/x` works. For each default exclude its walk
does not enter, `scan` prints a line on stderr naming it and why, as it
prints a warning, so it never lands inside a progress display:
```
scan: skipped /proc (default exclude, proc filesystem)
scan: skipped /tmp (default exclude)
```
That line is not a warning, and the directory is not counted as skipped. The
records below it, unless they lie below an operand at or inside it, count as
outside the scanned operands, as described in "`scan` mode" above: they are
not deleted, and their sizes join the size census.
- On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were