scan: skip OS-appropriate default excludes (/proc, /sys, /tmp, /run, /var/tmp on Linux) (closes #124)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
Below an operand, the walk no longer enters the default excludes: on Linux /proc, /sys, /dev, /run, /tmp, /var/tmp and every mount point of a kernel or memory-backed filesystem, read once from /proc/self/mountinfo; on macOS /dev, /private/tmp, /private/var/tmp, /private/var/vm, /System/Volumes/VM and /System/Volumes/Data. The lists live in the new exclude.go. An operand at or inside one is still walked, also under another operand, as with -x. Each skipped directory gets one stderr line through the warning path, is not counted as skipped, and its records count as outside the scanned operands. --no-default-excludes turns them off; scan --help lists them. Judgement call: an unreadable /proc/self/mountinfo fails the scan. Judgement call: where filesystems are mounted over one another, the last decides. Model: opus-5-5
This commit is contained in:
@@ -100,9 +100,10 @@ As a file `/etc/cron.d/sfdupes`:
|
||||
`/var/log/sfdupes.log`, holds a first line naming the database (see
|
||||
"Database"), plain progress lines naming each phase and its place among the
|
||||
five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
|
||||
runs, and one as it ends), a warning for each path it skips, and the summary
|
||||
line (see "Progress" and "`scan` mode"). The log grows with every scan; rotate
|
||||
it like any other.
|
||||
runs, and one as it ends), a warning for each path it skips, a line for each
|
||||
default exclude its walk does not enter (see "Rules for the walk"), and the
|
||||
summary line (see "Progress" and "`scan` mode"). The log grows with every
|
||||
scan; rotate it like any other.
|
||||
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
|
||||
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
|
||||
reason among the last lines of the log; `tail` prints them, and cron mails
|
||||
@@ -248,7 +249,7 @@ Four subcommands, all implemented:
|
||||
schema version and the counts of its records.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] PATH...
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
@@ -257,8 +258,9 @@ sfdupes [command] --help
|
||||
```
|
||||
|
||||
`--workers N` sets the size of each `scan` worker pool (default: the number of
|
||||
CPUs), and `-x` (`--one-file-system`) keeps the walk of each operand on that
|
||||
operand's filesystem; both are described under "`scan` mode".
|
||||
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
|
||||
operand's filesystem, and `--no-default-excludes` lets the walk enter the
|
||||
default excludes; all three are described under "`scan` mode".
|
||||
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
|
||||
exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
|
||||
subcommand, prints the help text to stderr and exits 0, writing nothing to
|
||||
@@ -439,11 +441,12 @@ results do not depend on the working directory. All operands belong to a single
|
||||
scan and are enumerated concurrently: every operand seeds the shared walk worker
|
||||
pool. Overlapping operands are harmless — an operand that duplicates another or
|
||||
lies under another is dropped before walking, so every file is reached exactly
|
||||
once and produces one database record. With `-x`, though, an operand under
|
||||
another one is kept and walked on its own when any directory from it (from its
|
||||
directory, for a regular file) up to but not including the operand covering it
|
||||
is on a different filesystem than that operand, because that operand's walk does
|
||||
not enter such a directory and so does not reach it.
|
||||
once and produces one database record. An operand under another one is kept,
|
||||
though, and walked on its own when any directory from it (from its directory,
|
||||
for a regular file) up to but not including the operand covering it is a default
|
||||
exclude (see "Rules for the walk") or, with `-x`, is on a different filesystem
|
||||
than that operand, because that operand's walk does not enter such a directory
|
||||
and so does not reach it.
|
||||
|
||||
An operand that is a symlink (never followed, not even as an operand), socket,
|
||||
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
|
||||
@@ -491,12 +494,13 @@ operands:
|
||||
a `content` hash is read for one, wherever it lies, once its size, `head`, and
|
||||
`tail` match another record's. If that file is gone or has changed since its
|
||||
record was written, the record is left as it is.
|
||||
- With `-x`, a record below a directory the walk did not enter because that
|
||||
directory is on another filesystem than its operand counts as outside the
|
||||
scanned operands: it is not deleted, and its size joins the size census. A
|
||||
filesystem mounted under an operand can therefore be scanned on its own
|
||||
schedule into the same database. Records below an operand at or inside such a
|
||||
directory are not kept this way, since that operand's own walk verifies them.
|
||||
- A record below a directory the walk did not enter, because that directory is a
|
||||
default exclude or, with `-x`, because it is on another filesystem than its
|
||||
operand, counts as outside the scanned operands: it is not deleted, and its
|
||||
size joins the size census. A filesystem mounted under an operand can
|
||||
therefore be scanned on its own schedule into the same database. Records below
|
||||
an operand at or inside such a directory are not kept this way, since that
|
||||
operand's own walk verifies them.
|
||||
|
||||
`scan` runs **five sequential phases over the whole scan**. Parallelism lives
|
||||
inside the walk, hash and content phases; batched database writes begin during
|
||||
@@ -572,6 +576,42 @@ Rules for the walk:
|
||||
unless they lie below an operand at or inside that directory, count as outside
|
||||
the scanned operands, as described in "`scan` mode" above: they are not
|
||||
deleted, and their sizes join the size census.
|
||||
- Below an operand, never descend into a default exclude: a directory that holds
|
||||
the kernel's view of the running system, or temporary files, so that
|
||||
`sfdupes scan /` walks only the files worth comparing. The default excludes
|
||||
depend on the system, and `sfdupes scan --help` lists those of the system it
|
||||
runs on:
|
||||
- On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`, and the
|
||||
mount point of every filesystem of type `proc`, `sysfs`, `devtmpfs`,
|
||||
`devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`, `debugfs`, `tracefs`,
|
||||
`securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`, `mqueue`,
|
||||
`hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`, `rpc_pipefs` or
|
||||
`selinuxfs`, wherever it is mounted. Where one filesystem is mounted over
|
||||
another, the one mounted last counts. `scan` reads the mount points from
|
||||
`/proc/self/mountinfo` once, after it names the database and before it
|
||||
takes the lock; if it cannot read them, it fails (exit 1).
|
||||
- On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
|
||||
`/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
|
||||
everything in `/System/Volumes/Data` through the firmlinks, so walking it
|
||||
as well would record each of those files twice.
|
||||
- On any other system there are none.
|
||||
|
||||
`--no-default-excludes` turns them all off, and `scan` then does not read
|
||||
`/proc/self/mountinfo`. An operand at or inside a default exclude is still
|
||||
walked, so `sfdupes scan /tmp/x` works. For each default exclude its walk
|
||||
does not enter, `scan` prints a line on stderr naming it and why, as it
|
||||
prints a warning, so it never lands inside a progress display:
|
||||
|
||||
```
|
||||
scan: skipped /proc (default exclude, proc filesystem)
|
||||
scan: skipped /tmp (default exclude)
|
||||
```
|
||||
|
||||
That line is not a warning, and the directory is not counted as skipped. The
|
||||
records below it, unless they lie below an operand at or inside it, count as
|
||||
outside the scanned operands, as described in "`scan` mode" above: they are
|
||||
not deleted, and their sizes join the size census.
|
||||
|
||||
- On any per-path error (permission denied, file vanished between passes,
|
||||
unreadable): print a one-line warning to stderr, skip the path, and continue.
|
||||
Per-file errors never abort the run; the final summary reports how many were
|
||||
|
||||
Reference in New Issue
Block a user