1 Commits
Author SHA1 Message Date
sneak 3f4ff92dae Show the progress rate over the last 5 seconds, and where a phase waits after 10 seconds with no item done (closes #125)
check / check (push) Canceled after 0s
The rate was the phase's average, so a burst after a long wait read as
over 1,000 files a second. It now covers the items completed over the
last 5 seconds, or since the line before off a terminal; the ETA still
comes from the average. Once no item has completed for 10 seconds, the
rate gives way to "stalled" and where the phase waits, as "stalled
disk/db": disk, db, out (stdout) or cpu, or two of them. A line too
wide for the terminal loses its bar first, then its rate or those
words, so the phase, progress, elapsed time and ETA stay whole. The
content phase's last partial batch is now written under a display of
its own. The tests set the display's clock by hand.

Deviation: the stall words hold no time and name where the phase waits, not what it does.
Judgement call: trees' group phase, which works in memory, says "cpu".

Model: opus-5-5
2026-10-16 22:11:48 +00:00
14 changed files with 215 additions and 1749 deletions
+54 -164
View File
@@ -49,40 +49,15 @@ results are as fresh as the last completed scan.
### Install ### Install
With Go installed, this builds and installs the newest release: With Go installed, this builds and installs the current `main` branch:
```sh
go install sneak.berlin/go/sfdupes@latest
```
Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time
`main` moves and its tip passes `script/cibuild`, that tip is tagged with the
next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build.
`@main` installs the tip before it is tagged:
```sh ```sh
go install sneak.berlin/go/sfdupes@main go install sneak.berlin/go/sfdupes@main
``` ```
`v0.0.1` is an early build without a database. It does not work as this README The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. A
describes, and `go.mod` retracts it. binary installed this way reports its version as `dev`; one built from a clone
or into the Docker image carries the git tag or commit it was built from.
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To
see which version an installed binary is:
```sh
sfdupes --version
go version -m "$(command -v sfdupes)"
```
`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`,
`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`,
or for `@main` a pseudo-version ending in the first 12 characters of the
commit's hash. A binary built from a clone or into the Docker image carries the
git tag or commit it was built from instead, and `VERSION` is `dev` only when
the binary carries no version at all. `go version -m` prints what Go recorded in
the binary; its `mod` line names the module's version. Binaries from `v0.0.2`
and earlier print `dev`, so check those with `go version -m`.
From a clone, `make build` writes the binary to `./sfdupes`: From a clone, `make build` writes the binary to `./sfdupes`:
@@ -125,11 +100,9 @@ As a file `/etc/cron.d/sfdupes`:
`/var/log/sfdupes.log`, holds a first line naming the database (see `/var/log/sfdupes.log`, holds a first line naming the database (see
"Database"), plain progress lines naming each phase and its place among the "Database"), plain progress lines naming each phase and its place among the
five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
runs, and one as it ends), a warning for each path it skips, a line for each runs, and one as it ends), a warning for each path it skips, and the summary
default exclude its walk does not enter and for each path it leaves out line (see "Progress" and "`scan` mode"). The log grows with every scan; rotate
because it is not on a persistent filesystem (see "Rules for the walk"), and it like any other.
the summary line (see "Progress" and "`scan` mode"). The log grows with every
scan; rotate it like any other.
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A - Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
reason among the last lines of the log; `tail` prints them, and cron mails reason among the last lines of the log; `tail` prints them, and cron mails
@@ -275,7 +248,7 @@ Four subcommands, all implemented:
schema version and the counts of its records. schema version and the counts of its records.
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH... sfdupes scan [--workers N] [-x] PATH...
sfdupes report > dupes.tsv sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv sfdupes trees > dupetrees.tsv
sfdupes info sfdupes info
@@ -284,13 +257,12 @@ sfdupes [command] --help
``` ```
`--workers N` sets the size of each `scan` worker pool (default: the number of `--workers N` sets the size of each `scan` worker pool (default: the number of
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that CPUs), and `-x` (`--one-file-system`) keeps the walk of each operand on that
operand's filesystem, and `--no-default-excludes` lets the walk enter the operand's filesystem; both are described under "`scan` mode".
default excludes, a list of paths; all three are described under "`scan` mode". `sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`, subcommand, prints the help text to stderr and exits 0, writing nothing to
alone or after a subcommand, prints the help text to stderr and exits 0, writing stdout.
nothing to stdout.
### Database ### Database
@@ -467,12 +439,11 @@ results do not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared walk worker scan and are enumerated concurrently: every operand seeds the shared walk worker
pool. Overlapping operands are harmless — an operand that duplicates another or pool. Overlapping operands are harmless — an operand that duplicates another or
lies under another is dropped before walking, so every file is reached exactly lies under another is dropped before walking, so every file is reached exactly
once and produces one database record. An operand under another one is kept, once and produces one database record. With `-x`, though, an operand under
though, and walked on its own when any directory from it (from its directory, another one is kept and walked on its own when any directory from it (from its
for a regular file) up to but not including the operand covering it is a default directory, for a regular file) up to but not including the operand covering it
exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with is on a different filesystem than that operand, because that operand's walk does
`-x`, is on a different filesystem than that operand, because that operand's not enter such a directory and so does not reach it.
walk does not enter such a directory and so does not reach it.
An operand that is a symlink (never followed, not even as an operand), socket, An operand that is a symlink (never followed, not even as an operand), socket,
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -482,10 +453,7 @@ beneath it is still scanned. The records stored beneath it are not deleted: they
are treated like any other record outside the scanned operands, including the are treated like any other record outside the scanned operands, including the
content-phase exception below. If it lies under another operand, they are under content-phase exception below. If it lies under another operand, they are under
that operand instead, and are treated like any other record there. This is not that operand instead, and are treated like any other record there. This is not
an error: a scan whose every operand is dropped walks nothing and exits 0. An an error: a scan whose every operand is dropped walks nothing and exits 0.
operand that is not on a persistent filesystem (see "Rules for the walk") is
dropped the same way, except that the line `scan` prints for it is not a
warning, and it is not counted as skipped.
`scan` synchronizes the database with the filesystem state under the scanned `scan` synchronizes the database with the filesystem state under the scanned
operands: operands:
@@ -521,16 +489,14 @@ operands:
can be scanned on different schedules into the same database. The one can be scanned on different schedules into the same database. The one
exception is the content phase below: a stored file of 10 MiB or more without exception is the content phase below: a stored file of 10 MiB or more without
a `content` hash is read for one, wherever it lies, once its size, `head`, and a `content` hash is read for one, wherever it lies, once its size, `head`, and
`tail` match another record's. If that file is gone, has changed since its `tail` match another record's. If that file is gone or has changed since its
record was written, or is not on a persistent filesystem, it is not read and record was written, the record is left as it is.
the record is left as it is. - With `-x`, a record below a directory the walk did not enter because that
- A record at or below a path the walk left out, because that path is a default directory is on another filesystem than its operand counts as outside the
exclude, is not on a persistent filesystem, or, with `-x`, is a directory on scanned operands: it is not deleted, and its size joins the size census. A
another filesystem than its operand, counts as outside the scanned operands: filesystem mounted under an operand can therefore be scanned on its own
it is not deleted, and its size joins the size census. A filesystem mounted schedule into the same database. Records below an operand at or inside such a
under an operand can therefore be scanned on its own schedule into the same directory are not kept this way, since that operand's own walk verifies them.
database. Records below an operand at or inside such a path are not kept this
way, since that operand's own walk verifies them.
`scan` runs **five sequential phases over the whole scan**. Parallelism lives `scan` runs **five sequential phases over the whole scan**. Parallelism lives
inside the walk, hash and content phases; batched database writes begin during inside the walk, hash and content phases; batched database writes begin during
@@ -576,20 +542,19 @@ the hash phase:
be read are kept in memory, never every file's hashes. Every record sharing be read are kept in memory, never every file's hashes. Every record sharing
their size, `head`, and `tail`, including one that already has a `content` their size, `head`, and `tail`, including one that already has a `content`
hash, has its file checked with `lstat` first. A file that is gone, is no hash, has its file checked with `lstat` first. A file that is gone, is no
longer a regular file, has changed (a different size, or an mtime newer than longer a regular file, or has changed (a different size, or an mtime newer
recorded), or is not on a persistent filesystem (see "Rules for the walk") than recorded) keeps its record as it is and does not count as a match for
keeps its record as it is and does not count as a match for the others. Any the others. Any other `lstat` error is warned about and counted as skipped,
other `lstat` error is warned about and counted as skipped, with the same with the same result. If such a record has no `content` hash, it stays out
result. If such a record has no `content` hash, it stays out of duplicate of duplicate groups; if it has one, it is still reported until a scan
groups; if it has one, it is still reported until a scan covering its own covering its own tree updates or removes it. The files that pass and have no
tree updates or removes it. The files that pass and have no `content` hash `content` hash are read only if at least two of those records pass, so a
are read only if at least two of those records pass, so a file whose only file whose only matches are stale costs no read; a file that already has a
matches are stale costs no read; a file that already has a `content` hash is `content` hash is never read again. They are read by a worker pool as in the
never read again. They are read by a worker pool as in the hash phase, in hash phase, in inode order and once per inode, and their content hashes are
inode order and once per inode, and their content hashes are committed in committed in batches. A failed read is warned about and counted as skipped;
batches. A failed read is warned about and counted as skipped; its record its record keeps an empty `content`, so it is not a duplicate, and a later
keeps an empty `content`, so it is not a duplicate, and a later scan tries scan tries again.
again.
Rules for the walk: Rules for the walk:
@@ -600,15 +565,6 @@ Rules for the walk:
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking - Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
them would list every file once per snapshot), not even when it is an operand; them would list every file once per snapshot), not even when it is an operand;
such an operand is dropped the same way. such an operand is dropped the same way.
- Never record the scan's own database. Once the database and its `.lock` file
are open, `scan` notes the device and inode of the database file, its `-wal`
and `-shm` files and its `.lock` file, and the walk passes over a file with
one of them, however it reaches it: as an operand, below one, or through a
symlinked directory. It prints no line for such a file and does not count it
as skipped. A record that an earlier scan left for one of them is deleted,
like the record of any file the scan no longer verifies. There is no
`-journal` file during the walk, because the database is in WAL mode then (see
"Database").
- Filesystem boundaries are crossed by default. With `-x` (long form - Filesystem boundaries are crossed by default. With `-x` (long form
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend `--one-file-system`, following the GNU `du`/`rsync` convention), never descend
into a directory on a different filesystem than its `PATH` operand; each into a directory on a different filesystem than its `PATH` operand; each
@@ -616,74 +572,6 @@ Rules for the walk:
unless they lie below an operand at or inside that directory, count as outside unless they lie below an operand at or inside that directory, count as outside
the scanned operands, as described in "`scan` mode" above: they are not the scanned operands, as described in "`scan` mode" above: they are not
deleted, and their sizes join the size census. deleted, and their sizes join the size census.
- Never record anything that is not on a persistent filesystem. A filesystem is
not persistent when the kernel makes it up or keeps it in memory: one of type
`proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
`debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
`mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
`rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
it names the database and before it takes the lock, `scan` lists the mounted
filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
of mounted filesystems on macOS (on any other system it lists none), and notes
the device ID of each one of those types. On Linux that is the device ID the
list gives, so a filesystem that a later mount hides still gives its own. On
macOS it is read with `lstat` on the mount point, and where one filesystem is
mounted over another on the same mount point, the one mounted last counts. If
`scan` cannot list them, it fails (exit 1). The walk then never enters a
directory, and never records a file, on one of those devices, however it
reaches it: as an operand, below one, or through a bind mount, which keeps the
device ID of what it binds. The content phase never reads a file on one of
them either. No flag turns this off. For each operand, directory or file it
leaves out this way, `scan` prints a line on stderr naming it and the type of
its filesystem, as it prints a warning, so it never lands inside a progress
display:
```
scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
```
That line is not a warning, and the path is not counted as skipped. Records
that an earlier scan left at or below it, unless they lie below an operand
at or inside it, count as outside the scanned operands, as described in
"`scan` mode" above: they are not deleted, and their sizes join the size
census.
- Below an operand, never descend into a default exclude: a path that holds the
kernel's view of the running system, or temporary files, so that
`sfdupes scan /` walks only the files worth comparing. The default excludes
depend on the system, and `sfdupes scan --help` lists those of the system it
runs on:
- On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
- On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
`/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
everything in `/System/Volumes/Data` through the firmlinks, so walking it
as well would record each of those files twice.
- On Windows, relative to the root of the drive an operand lies on: the
page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
`hiberfil.sys`, and the directories `System Volume Information`,
`$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
matches one path component, as it does in any default exclude. sfdupes
does not build for Windows yet, so this list is not used.
- On any other system there are none.
`--no-default-excludes` turns them all off; what is not on a persistent
filesystem stays left out. An operand at or inside a default exclude is
still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
persistent filesystem. For each default exclude its walk does not enter,
`scan` prints a line on stderr naming it, as it prints a warning:
```
scan: skipped /var/tmp (default exclude)
```
That line is not a warning, and the directory is not counted as skipped. A
default exclude that is not on a persistent filesystem, as `/proc` is, gets
the line for its filesystem instead, and one that is itself an operand gets
no line, since it is walked as that operand. The records below a default
exclude the walk does not enter, unless they lie below an operand at or
inside it, count as outside the scanned operands, as described in "`scan`
mode" above: they are not deleted, and their sizes join the size census.
- On any per-path error (permission denied, file vanished between passes, - On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue. unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were Per-file errors never abort the run; the final summary reports how many were
@@ -990,17 +878,15 @@ Additional requirements:
terminal is. A line too wide for the terminal first loses its bar, which gives terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, at counts like those of and the ETA are kept whole. On an 80-column terminal, a stalled line at counts
the stalled line in the example above, a stalled line keeps its words whenever and times like those of the stalled line in the example above keeps its words
the elapsed time and the ETA are both in minutes, except in the content whole in every phase; where the words and the ETA do not both fit, as at
phase's checks and reads. Their lines are the widest, and at most times in millions of files and times in hours, the words give way. Only when those four
minutes they lose the words. Where the words and the ETA do not both fit, as alone are too wide, as on a terminal narrower than about 70 columns hours into
at millions of files and times in hours, the words give way. Only when those a scan of millions of files, is the line cut short at the terminal's edge, so
four alone are too wide, as on a terminal narrower than about 70 columns hours the end of the ETA is lost first. A terminal made narrower while a phase runs
into a scan of millions of files, is the line cut short at the terminal's gets the same: the next draw erases the line within the new width and fits the
edge, so the end of the ETA is lost first. A terminal made narrower while a line to it, and the display goes on redrawing in place.
phase runs gets the same: the next draw erases the line within the new width
and fits the line to it, and the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI - When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once runs (whether or not an item has completed, and never more often), and once
@@ -1195,6 +1081,10 @@ All of the following, run in this directory, must pass:
./sfdupes report ./sfdupes report
``` ```
(The database lives in a temp directory of its own: inside `$d`, the scan
would record it, and its empty lock file would join the `empty1`/`empty2`
group.)
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
group (two dupe rows, `first` is the lexicographically smallest path); group (two dupe rows, `first` is the lexicographically smallest path);
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/ `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
+3 -47
View File
@@ -29,48 +29,6 @@
# Completed Steps # Completed Steps
- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so
`go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has
passed the build; README "Install" says so, says what `@main` and `v0.0.1`
give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the
early build without a database. When the linker set no version,
`sfdupes --version` prints the main module's version that Go recorded in the
binary, the tag or the pseudo-version of a `go install` build, and `dev` only
when neither is known (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/116)
- `scan` creates the database's directory where the database path leads: the
path with its last element cut off as text, never cleaned, so a `..` after a
symlinked directory is followed through the symlink, as when the file is
opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
symlink to `/data/x`, it creates `/data/new` and the database in it, where it
used to create `/srv/new` and then fail (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/119)
- `scan` never records its own database: once the database and its `.lock` file
are open, it notes the device and inode of the database file, its `-wal` and
`-shm` files and its `.lock` file, and the walk passes over a file with one of
them wherever the operands reach it, through a symlinked directory included,
printing no line and counting nothing. A record an earlier scan left for one
of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
- `scan` never records or reads anything that is not on a persistent filesystem
(`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
matched by the device ID of each such filesystem, listed as the scan starts
from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
however the walk or the content phase reaches it; no flag turns that off.
Below an operand, its walk does not enter the default excludes either, so
`sfdupes scan /` skips the kernel's view of the running system and the
temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
`/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
`/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
Windows list, not used yet. An operand at or inside a default exclude is still
walked. The scan prints a line for each path it leaves out, without counting
it as skipped, and the records there count as outside the scanned operands, as
for `-x`. `--no-default-excludes` turns off the default excludes only, and
`scan --help` lists them (2026-10-17,
https://git.eeqj.de/sneak/sfdupes/issues/124)
- the rate a progress display shows covers the items completed over the last 5 - the rate a progress display shows covers the items completed over the last 5
seconds, or since the line before when stderr is not a terminal, so it falls seconds, or since the line before when stderr is not a terminal, so it falls
to 0 while nothing completes, and a burst after a wait no longer reads as a to 0 while nothing completes, and a burst after a wait no longer reads as a
@@ -80,11 +38,9 @@
`db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on `db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
either, each phase giving its word where it is defined. A line too wide for either, each phase giving its word where it is defined. A line too wide for
the terminal loses its bar first, then its rate or those words, so the phase, the terminal loses its bar first, then its rate or those words, so the phase,
progress, elapsed time and ETA stay whole. On an 80-column terminal, at counts progress, elapsed time and ETA stay whole; on an 80-column terminal the words
like those of README's example, the content phase's checks and reads lose fit beside them in an ordinary scan. The content phase's last partial batch is
those words at most times in minutes, and every other phase keeps them now written under a display of its own (2026-10-16,
whenever the elapsed time and the ETA are both in minutes. The content phase's
last partial batch is now written under a display of its own (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/125) https://git.eeqj.de/sneak/sfdupes/issues/125)
- new subcommand `info` prints the database's details on stdout, one per line as - new subcommand `info` prints the database's details on stdout, one per line as
+10 -15
View File
@@ -196,7 +196,7 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
base := baselineGoroutines(t) base := baselineGoroutines(t)
st, err := syncScan(newWalkClock(walkCancelAtDone), db, st, err := syncScan(newWalkClock(walkCancelAtDone), db,
[]string{dir}, walkCancelWorkers, false, walkRules{}) []string{dir}, walkCancelWorkers, false)
assertWalkGuardAborted(t, st, err) assertWalkGuardAborted(t, st, err)
assertRecordsIntact(t, db, before) assertRecordsIntact(t, db, before)
@@ -268,8 +268,7 @@ func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context()) ctx, cancel := context.WithCancel(t.Context())
cancel() cancel()
st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false, st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false)
walkRules{})
if !errors.Is(err, context.Canceled) { if !errors.Is(err, context.Canceled) {
t.Fatalf("syncScan on a cancelled context = %v, want %v", t.Fatalf("syncScan on a cancelled context = %v, want %v",
err, context.Canceled) err, context.Canceled)
@@ -302,7 +301,7 @@ func TestSyncScanCancelledMidHashKeepsHashedRecords(t *testing.T) {
db := openTestDB(t) db := openTestDB(t)
st, err := syncScan(newWalkClock(hashCancelAtDone), db, st, err := syncScan(newWalkClock(hashCancelAtDone), db,
[]string{dir}, walkCancelWorkers, false, walkRules{}) []string{dir}, walkCancelWorkers, false)
if !errors.Is(err, context.Canceled) { if !errors.Is(err, context.Canceled) {
t.Fatalf("syncScan cancelled mid-hash = %v, want %v", t.Fatalf("syncScan cancelled mid-hash = %v, want %v",
err, context.Canceled) err, context.Canceled)
@@ -357,8 +356,7 @@ func TestRunScanInterrupted(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
dir := buildSmokeTree(t) dir := buildSmokeTree(t)
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false, err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
false)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -382,7 +380,7 @@ func TestRunScanInterrupted(t *testing.T) {
defer close(done) defer close(done)
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers, err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
false, false) false)
}() }()
awaitReturn(t, done, "runScan") awaitReturn(t, done, "runScan")
@@ -409,8 +407,7 @@ func TestRunScanInterrupted(t *testing.T) {
got, before) got, before)
} }
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false, err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
false)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -436,7 +433,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
dir := buildWalkCancelTree(t) dir := buildWalkCancelTree(t)
err := runScan(newWalkClock(hashCancelAtDone), []string{dir}, err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
walkCancelWorkers, false, false) walkCancelWorkers, false)
if !errors.Is(err, errInterrupted) { if !errors.Is(err, errInterrupted) {
t.Fatalf("runScan interrupted mid-hash = %v, want %v", t.Fatalf("runScan interrupted mid-hash = %v, want %v",
err, errInterrupted) err, errInterrupted)
@@ -620,8 +617,7 @@ func TestWalkOneDirStopsWhenCancelled(t *testing.T) {
// Unbuffered and unread: on a cancelled scan every send gives up. // Unbuffered and unread: on a cancelled scan every send gives up.
events := make(chan walkEvent) events := make(chan walkEvent)
subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false, subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false, events)
walkRules{}, events)
if len(subs) != 0 { if len(subs) != 0 {
t.Errorf("cancelled walkOneDir returned %+v to descend into, "+ t.Errorf("cancelled walkOneDir returned %+v to descend into, "+
"want none", subs) "want none", subs)
@@ -641,8 +637,7 @@ func TestWalkWorkersDropQueuedDirs(t *testing.T) {
missing := filepath.Join(t.TempDir(), "missing") missing := filepath.Join(t.TempDir(), "missing")
jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false, jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false)
walkRules{})
for range 64 { for range 64 {
jobs <- dirJob{path: missing} jobs <- dirJob{path: missing}
@@ -673,7 +668,7 @@ func TestWalkWorkerAbandonsSubdirHandoff(t *testing.T) {
ctx, cancel := context.WithCancel(t.Context()) ctx, cancel := context.WithCancel(t.Context())
defer cancel() defer cancel()
jobs, subdirs, events := startWalkWorkers(ctx, 1, false, walkRules{}) jobs, subdirs, events := startWalkWorkers(ctx, 1, false)
// Fill the hand-back channel to its capacity — one slot per worker // Fill the hand-back channel to its capacity — one slot per worker
// — so the worker's own hand-back is certain to block. // — so the worker's own hand-back is certain to block.
+8 -60
View File
@@ -109,8 +109,10 @@ func databasePath() (string, string, error) {
} }
// The working directory as the kernel reports it, with no symlink // The working directory as the kernel reports it, with no symlink
// in it, as README "Database" says. os.Getwd may return PWD // in it. os.Getwd may return PWD instead, which can name it through
// instead, which can name it through a symlink. // a symlink; a ".." after that symlink then leads somewhere else
// once the path is cleaned, as scan cleans it to create the
// database's directory.
wd, err := unix.Getwd() wd, err := unix.Getwd()
if err != nil { if err != nil {
return "", "", fmt.Errorf("resolve database path %s: %w", path, err) return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
@@ -123,24 +125,6 @@ func databasePath() (string, string, error) {
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
} }
// createDatabaseDirectory creates the directory the database at path
// goes in, and any directory above it, if missing. path is absolute, as
// databasePath returns it. The directory is path with its last element
// cut off as text, by filepath.Split. filepath.Dir would clean it,
// dropping each ".." together with the directory name before it; when
// that name is a symlink, the cleaned path names another directory than
// the one the system reaches when it opens path.
func createDatabaseDirectory(path string) error {
dir, _ := filepath.Split(path)
err := os.MkdirAll(dir, dbDirPerm)
if err != nil {
return fmt.Errorf("create database directory: %w", err)
}
return nil
}
// scanParams are the connection parameters for scan: read-write, with // scanParams are the connection parameters for scan: read-write, with
// WAL journaling and a busy timeout, so a report can run while a cron // WAL journaling and a busy timeout, so a report can run while a cron
// scan is in progress. closeScanDatabase leaves WAL mode again. // scan is in progress. closeScanDatabase leaves WAL mode again.
@@ -195,9 +179,9 @@ func openDB(path, params string) (*sql.DB, error) {
// that deleted it would let the next scan lock a new file while // that deleted it would let the next scan lock a new file while
// another still holds the old one. // another still holds the old one.
func lockScanDatabase(path string) (*os.File, error) { func lockScanDatabase(path string) (*os.File, error) {
err := createDatabaseDirectory(path) err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("create database directory: %w", err)
} }
// Like SQLite, this follows a symlink at path and creates its // Like SQLite, this follows a symlink at path and creates its
@@ -241,9 +225,9 @@ func lockScanDatabase(path string) (*os.File, error) {
// openScanDatabase opens the database for the scan subcommand, creating // openScanDatabase opens the database for the scan subcommand, creating
// the file, its parent directory, and the schema as needed. // the file, its parent directory, and the schema as needed.
func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) { func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
err := createDatabaseDirectory(path) err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("create database directory: %w", err)
} }
db, err := openDB(path, scanParams) db, err := openDB(path, scanParams)
@@ -261,42 +245,6 @@ func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
return db, nil return db, nil
} }
// databaseFiles returns, by device and inode, the files of the database
// at path that the scan keeps open: the database file, its -wal and
// -shm files and its .lock file, each named from the database's path
// with symlinks resolved, as SQLite and lockScanDatabase name them. The
// scan calls it once the database and the lock file are open, so that
// its walk passes over them however it reaches them, through a
// symlinked directory included. A file that does not exist then is left
// out, since no walk can reach it. There is no -journal file while the
// database is in WAL mode.
func databaseFiles(path string) ([]fileRec, error) {
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return nil, fmt.Errorf("resolve database path: %w", err)
}
var files []fileRec
for _, p := range []string{
resolved, resolved + "-wal", resolved + "-shm", resolved + ".lock",
} {
fi, err := os.Lstat(p)
if errors.Is(err, fs.ErrNotExist) {
continue
}
if err != nil {
return nil, fmt.Errorf("database: %w", err)
}
dev, ino := inodeOfInfo(fi)
files = append(files, fileRec{path: p, dev: dev, ino: ino})
}
return files, nil
}
// closeScanDatabase switches the database at path from WAL back to // closeScanDatabase switches the database at path from WAL back to
// rollback-journal mode and closes it. Out of WAL mode the database // rollback-journal mode and closes it. Out of WAL mode the database
// file alone holds the whole database, so a reader needs no -wal or // file alone holds the whole database, so a reader needs no -wal or
-111
View File
@@ -5,7 +5,6 @@ import (
"database/sql" "database/sql"
"errors" "errors"
"fmt" "fmt"
"io/fs"
"os" "os"
"path/filepath" "path/filepath"
"slices" "slices"
@@ -119,116 +118,6 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
} }
} }
// TestScanCreatesDatabaseDirectoryAfterSymlink has a database path go
// through a symlink to a directory, then "..", to a missing directory.
// With srv/link a symlink to data/x, srv/link/../new/db.sqlite leads to
// data/new/db.sqlite. lockScanDatabase and openScanDatabase, the two
// places scan creates the database's directory, must each create the
// database there, and nothing at srv/new, where the path leads once
// cleaned as text.
func TestScanCreatesDatabaseDirectoryAfterSymlink(t *testing.T) {
t.Parallel()
creates := map[string]func(ctx context.Context, path string) error{
"lockScanDatabase": func(_ context.Context, path string) error {
lock, err := lockScanDatabase(path)
if err == nil {
_ = lock.Close()
}
return err
},
"openScanDatabase": func(ctx context.Context, path string) error {
db, err := openScanDatabase(ctx, path)
if err == nil {
_ = db.Close()
}
return err
},
}
for name, create := range creates {
t.Run(name, func(t *testing.T) {
t.Parallel()
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
err = os.MkdirAll(filepath.Join(dir, "data", "x"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Mkdir(filepath.Join(dir, "srv"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Symlink(filepath.Join(dir, "data", "x"),
filepath.Join(dir, "srv", "link"))
if err != nil {
t.Fatal(err)
}
err = create(t.Context(), dir+"/srv/link/../new/db.sqlite")
if err != nil {
t.Fatalf("%s: %v", name, err)
}
_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
if err != nil {
t.Errorf("database the path leads to: %v", err)
}
_, err = os.Lstat(filepath.Join(dir, "srv", "new"))
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("srv/new: stat error %v, want it missing", err)
}
})
}
}
// TestDatabaseFiles opens a database through a symlink to its file:
// databaseFiles finds the database file and its -wal and -shm files
// beside the symlink's target, where SQLite keeps them, and leaves out
// the .lock file, which only a scan creates.
func TestDatabaseFiles(t *testing.T) {
t.Parallel()
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
db, err := openScanDatabase(t.Context(), link)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
files, err := databaseFiles(link)
if err != nil {
t.Fatal(err)
}
got := make([]string, 0, len(files))
for _, f := range files {
got = append(got, f.path)
}
want := []string{path, path + "-wal", path + "-shm"}
if !slices.Equal(got, want) {
t.Errorf("databaseFiles(%s) = %q, want %q", link, got, want)
}
}
func TestOpenDatabaseUnversionedForeign(t *testing.T) { func TestOpenDatabaseUnversionedForeign(t *testing.T) {
t.Parallel() t.Parallel()
-195
View File
@@ -1,195 +0,0 @@
package main
// This file holds two rules for the walk (README "Rules for the walk"):
// the default excludes, the paths a scan's walk does not enter below an
// operand unless scan is given --no-default-excludes, and the
// filesystems that are not persistent, on which the scan never records
// or reads anything.
import (
"bufio"
"errors"
"fmt"
"io"
"io/fs"
"path/filepath"
"slices"
"strings"
"golang.org/x/sys/unix"
)
// The systems that have default excludes, as runtime.GOOS names them.
const (
goosLinux = "linux"
goosDarwin = "darwin"
goosWindows = "windows"
)
// The fields of a line of mountInfoPath that readMountInfo reads,
// counted from 0 (see proc(5)): the device ID, as major:minor, is the
// third field, and after the sixth come optional fields, then a field
// holding "-", then the filesystem type.
const (
deviceField = 2
optionalFieldsStart = 6
)
// typesPerHelpLine is how many filesystem types the scan command's help
// lists on each line.
const typesPerHelpLine = 6
// errMountInfoLine reports a line of mountInfoPath without the fields
// readMountInfo reads.
var errMountInfoLine = errors.New("unexpected line")
// walkRules holds the rules for the walk that a scan sets up as it
// starts. excludes are the default excludes, or none with
// --no-default-excludes. nonPersistent maps the device ID of every
// mounted filesystem that is not persistent to its type; the scan
// records and reads nothing on one of those, whatever the flags.
// databaseFiles are the scan's own database files, which the walk
// passes over (see databaseFiles in db.go).
type walkRules struct {
excludes []string
nonPersistent map[uint64]string
databaseFiles []fileRec
}
// defaultExcludes returns the default excludes of the system goos
// names, as runtime.GOOS does: a fixed list on Linux, macOS and
// Windows, and none on any other system.
func defaultExcludes(goos string) []string {
switch goos {
case goosLinux:
return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
case goosDarwin:
// A walk of / reaches everything in /System/Volumes/Data through
// the firmlinks, so walking it as well would record each of
// those files twice.
return []string{
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data",
}
case goosWindows:
// Relative to the root of the drive an operand lies on. The
// first three are the page, swap and hibernation files.
return []string{
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
`Users\*\AppData\Local\Temp`,
}
default:
return nil
}
}
// isDefaultExclude reports whether path is one of excludes, in which a
// * matches one path component.
func isDefaultExclude(path string, excludes []string) bool {
for _, pattern := range excludes {
// The patterns are fixed, so none is malformed.
if ok, _ := filepath.Match(pattern, path); ok {
return true
}
}
return false
}
// nonPersistentFilesystemTypes returns the filesystem types that are
// not persistent: filesystems that the kernel makes up or that are kept
// in memory. devfs is the macOS filesystem of /dev; the others are
// Linux names.
func nonPersistentFilesystemTypes() []string {
return []string{
"proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "ramfs",
"cgroup", "cgroup2", "debugfs", "tracefs", "securityfs", "pstore",
"bpf", "configfs", "fusectl", "mqueue", "hugetlbfs", "efivarfs",
"binfmt_misc", "autofs", "nsfs", "rpc_pipefs", "selinuxfs", "devfs",
}
}
// nonPersistentLine returns the line the scan prints for path when fi,
// its lstat, puts it on a filesystem in rules.nonPersistent, and ""
// when it is on any other.
func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
dev, ok := deviceOfInfo(fi)
if !ok {
return ""
}
fsType, found := rules.nonPersistent[dev]
if !found {
return ""
}
return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
path, fsType)
}
// readMountInfo reads mountinfo, laid out as mountInfoPath is, and
// returns the device ID of every mounted filesystem that is not
// persistent, with its type. The device ID is the one each line gives,
// not that of the file now at its mount point, so a mount that a later
// mount hides never adds the device of another filesystem.
func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
types := nonPersistentFilesystemTypes()
devices := make(map[uint64]string)
lines := bufio.NewScanner(mountinfo)
for lines.Scan() {
fields := strings.Fields(lines.Text())
sep := slices.Index(fields, "-")
if sep < optionalFieldsStart || sep+1 >= len(fields) {
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
}
var major, minor uint32
_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
if err != nil {
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
}
fsType := fields[sep+1]
if slices.Contains(types, fsType) {
devices[unix.Mkdev(major, minor)] = fsType
}
}
return devices, lines.Err()
}
// defaultExcludesHelp describes, for the scan command's help, the
// default excludes of the system goos names and, where the scan lists
// the mounted filesystems, the types it never records anything on.
func defaultExcludesHelp(goos string) string {
var help strings.Builder
dirs := defaultExcludes(goos)
if len(dirs) == 0 {
help.WriteString("This system has no default excludes.\n")
} else {
help.WriteString("Below an operand, the walk does not enter these " +
"default excludes, unless\n--no-default-excludes is given:\n\n")
for _, dir := range dirs {
fmt.Fprintf(&help, " %s\n", dir)
}
}
if goos == goosLinux || goos == goosDarwin {
help.WriteString("\nWith or without --no-default-excludes, the scan " +
"records nothing on a\nfilesystem that is not persistent, " +
"one of these types:\n\n")
types := nonPersistentFilesystemTypes()
for line := range slices.Chunk(types, typesPerHelpLine) {
fmt.Fprintf(&help, " %s\n", strings.Join(line, ", "))
}
}
return help.String()
}
-52
View File
@@ -1,52 +0,0 @@
//go:build darwin
package main
import (
"slices"
"golang.org/x/sys/unix"
)
// nonPersistentFilesystems returns the device ID of every mounted
// filesystem that is not persistent, with its type, from the system's
// list of mounted filesystems. The device ID is read with lstat on the
// mount point. Where one filesystem is mounted over another on the same
// mount point, the later one in the list counts. A mount point that
// lstat cannot reach is left out.
func nonPersistentFilesystems() (map[uint64]string, error) {
// The first call counts the filesystems, the second lists them.
n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
if err != nil {
return nil, err
}
list := make([]unix.Statfs_t, n)
n, err = unix.Getfsstat(list, unix.MNT_NOWAIT)
if err != nil {
return nil, err
}
mounts := make(map[string]string, n)
for _, st := range list[:n] {
mounts[unix.ByteSliceToString(st.Mntonname[:])] =
unix.ByteSliceToString(st.Fstypename[:])
}
types := nonPersistentFilesystemTypes()
devices := make(map[uint64]string)
for mountPoint, fsType := range mounts {
if !slices.Contains(types, fsType) {
continue
}
if dev, ok := deviceOfPath(mountPoint); ok {
devices[dev] = fsType
}
}
return devices, nil
}
-36
View File
@@ -1,36 +0,0 @@
//go:build !darwin
package main
import (
"fmt"
"os"
"runtime"
)
// mountInfoPath is the file in which Linux lists the mounts the reading
// process sees.
const mountInfoPath = "/proc/self/mountinfo"
// nonPersistentFilesystems returns the device ID of every mounted
// filesystem that is not persistent, with its type: on Linux read from
// mountInfoPath (see readMountInfo), and on any other system none.
func nonPersistentFilesystems() (map[uint64]string, error) {
if runtime.GOOS != goosLinux {
return map[uint64]string{}, nil
}
f, err := os.Open(mountInfoPath)
if err != nil {
return nil, err
}
defer func() { _ = f.Close() }()
devices, err := readMountInfo(f)
if err != nil {
return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
}
return devices, nil
}
-583
View File
@@ -1,583 +0,0 @@
package main
import (
"bytes"
"errors"
"maps"
"os"
"path/filepath"
"runtime"
"slices"
"strings"
"testing"
"golang.org/x/sys/unix"
)
func TestDefaultExcludes(t *testing.T) {
t.Parallel()
// README "Rules for the walk": each system's list.
cases := map[string][]string{
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
goosDarwin: {
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data",
},
goosWindows: {
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
`Users\*\AppData\Local\Temp`,
},
"freebsd": nil,
}
for goos, want := range cases {
if got := defaultExcludes(goos); !slices.Equal(got, want) {
t.Errorf("defaultExcludes(%q) = %q, want %q", goos, got, want)
}
}
}
func TestIsDefaultExclude(t *testing.T) {
t.Parallel()
// A * matches one path component, no more and no fewer.
excludes := []string{"/scratch", "/home/*/cache"}
cases := map[string]bool{
"/scratch": true,
"/scratch/x": false,
"/home/u/cache": true,
"/home/u/v/cache": false,
"/home/cache": false,
}
for path, want := range cases {
if got := isDefaultExclude(path, excludes); got != want {
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
}
}
}
func TestReadMountInfo(t *testing.T) {
t.Parallel()
// The second-to-last line has no optional fields. The tmpfs on
// /data/a/b is hidden by the root filesystem's /srv, mounted over
// /data/a after it: the tmpfs still gives its own device, and the
// root filesystem's device is not listed.
mountinfo := strings.Join([]string{
"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw",
"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw",
"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
}, "\n")
got, err := readMountInfo(strings.NewReader(mountinfo))
if err != nil {
t.Fatal(err)
}
want := map[uint64]string{
unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs",
unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs",
}
if !maps.Equal(got, want) {
t.Errorf("readMountInfo = %v, want %v", got, want)
}
for _, line := range []string{
"22 1 8:1 / / rw,relatime\n",
"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n",
} {
_, err = readMountInfo(strings.NewReader(line))
if !errors.Is(err, errMountInfoLine) {
t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
errMountInfoLine)
}
}
}
func TestNonPersistentFilesystems(t *testing.T) {
t.Parallel()
if runtime.GOOS != goosLinux {
t.Skip("the test knows a mount only on Linux: /proc")
}
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfPath("/proc")
if !ok {
t.Fatal("no device for /proc")
}
if got := nonPersistent[dev]; got != "proc" {
t.Errorf("filesystem of /proc = %q, want %q", got, "proc")
}
}
func TestDefaultExcludesHelp(t *testing.T) {
t.Parallel()
for _, goos := range []string{goosLinux, goosDarwin, goosWindows} {
help := defaultExcludesHelp(goos)
for _, dir := range defaultExcludes(goos) {
if !strings.Contains(help, "\n "+dir+"\n") {
t.Errorf("help for %s does not list %s:\n%s", goos, dir, help)
}
}
// Only Linux and macOS list their mounted filesystems.
for _, fsType := range nonPersistentFilesystemTypes() {
listed := strings.Contains(help, " "+fsType+",") ||
strings.Contains(help, " "+fsType+"\n")
if listed != (goos != goosWindows) {
t.Errorf("help for %s lists %s: %v", goos, fsType, listed)
}
}
}
want := "This system has no default excludes.\n"
if got := defaultExcludesHelp("freebsd"); got != want {
t.Errorf("help for freebsd = %q, want %q", got, want)
}
// scan's help shows those of the system it runs on.
var stdout, stderr bytes.Buffer
run([]string{cmdScan, "-h"}, &stdout, &stderr)
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
if !strings.Contains(stderr.String(), help) {
t.Errorf("scan --help = %q, want it to hold %q", stderr.String(), help)
}
}
// TestScanDoesNotEnterDefaultExclude checks that the walk does not
// enter a default exclude below an operand. It says so on a line of its
// own and does not count it as skipped. The records below it count as
// outside the operands: the record of a file gone from it is kept, and
// its size joins the size census, while a file added to it is not
// recorded.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
db := openTestDB(t)
excluded := filepath.Join(dir, "tmp")
// A scan with no default excludes records the file in excluded.
gone := writeFile(t, excluded, "gone", pattern(1, 100))
syncTree(t, db, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
unseen := writeFile(t, excluded, "new", pattern(2, 10))
// The same size as gone, so it is hashed only if the census holds the
// size of gone's record.
added := writeFile(t, dir, "added", pattern(3, 100))
st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
walkRules{excludes: []string{excluded}})
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
recordByPath(t, recs, gone)
if recordByPath(t, recs, added).head == "" {
t.Errorf("%s not hashed: the census lacks the size of %s",
added, gone)
}
if slices.Contains(recordPaths(recs), unseen) {
t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
}
line := "\nscan: skipped " + excluded + " (default exclude)\n"
if !strings.Contains(stderr(), line) {
t.Errorf("stderr = %q, want %q", stderr(), line)
}
}
// TestScanWalksOperandInDefaultExclude checks that an operand at or
// inside a default exclude is walked, also when it lies under another
// operand, whose walk does not enter the default exclude.
func TestScanWalksOperandInDefaultExclude(t *testing.T) {
t.Parallel()
dir := t.TempDir()
excluded := filepath.Join(dir, "tmp")
f := writeFile(t, excluded, "sub/f", pattern(1, 10))
sub := filepath.Dir(f)
cases := map[string][]string{
"at it": {excluded},
"inside it": {sub},
"inside it under an operand": {dir, sub},
}
for name, roots := range cases {
t.Run(name, func(t *testing.T) {
t.Parallel()
db := openTestDB(t)
st, err := syncScan(t.Context(), db, roots, 1, false,
walkRules{excludes: []string{excluded}})
if err != nil {
t.Fatal(err)
}
if st.added != 1 {
t.Errorf("added = %d, want %s recorded once", st.added, f)
}
recordByPath(t, dbRecords(t, db), f)
})
}
}
// TestScanNamesNoDefaultExcludeThatIsAnOperand checks that a default
// exclude given as an operand under another operand, which the walk of
// that other operand does not enter, gets no line saying it was
// skipped: it is walked as an operand of its own.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
db := openTestDB(t)
excluded := filepath.Join(dir, "tmp")
f := writeFile(t, excluded, "f", pattern(1, 10))
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
walkRules{excludes: []string{excluded}})
if err != nil {
t.Fatal(err)
}
if st.added != 1 {
t.Errorf("added = %d, want %s recorded once", st.added, f)
}
recordByPath(t, dbRecords(t, db), f)
if strings.Contains(stderr(), "scan: skipped") {
t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
}
}
// TestRunScanTempDirOperand scans a tree in the tests' temporary
// directory, which on Linux lies inside the default exclude /tmp unless
// TMPDIR names another place, with the system's own default excludes:
// the scan walks the whole tree and reports no default exclude.
func TestRunScanTempDirOperand(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
t.Skip(dir + " is not inside a default exclude")
}
f := writeFile(t, dir, "sub/f", pattern(1, 10))
stderr := scanOK(t, dir)
if strings.Contains(stderr, "scan: skipped") {
t.Errorf("stderr = %q, want no line saying a path was skipped",
stderr)
}
assertRecorded(t, path, f, true)
}
// TestRunScanNoDefaultExcludes scans /var through the command. Its
// /var/tmp, on a persistent filesystem, is a default exclude on Linux:
// without --no-default-excludes the walk does not enter it and says so,
// and with that flag the walk records a file the test puts there.
func TestRunScanNoDefaultExcludes(t *testing.T) {
if runtime.GOOS != goosLinux {
t.Skip("only Linux has /var/tmp among its default excludes")
}
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
t.Fatal(err)
}
rules := walkRules{nonPersistent: nonPersistent}
for _, dir := range []string{"/var", "/var/tmp"} {
fi, lstatErr := os.Lstat(dir)
if lstatErr != nil || !fi.IsDir() ||
rules.nonPersistentLine(dir, fi) != "" {
t.Skip(dir + " is not a directory on a persistent filesystem here")
}
}
//nolint:usetesting // the directory must be inside /var/tmp
tmp, err := os.MkdirTemp("/var/tmp", "sfdupes-test-")
if err != nil {
t.Skip("cannot write to /var/tmp here: " + err.Error())
}
t.Cleanup(func() { _ = os.RemoveAll(tmp) })
f := writeFile(t, tmp, "f", pattern(1, 10))
path := testDBPath(t)
t.Setenv(databaseEnv, path)
line := "scan: skipped /var/tmp (default exclude)\n"
if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
}
assertRecorded(t, path, f, false)
stderr := scanOK(t, "--no-default-excludes", "/var")
if strings.Contains(stderr, line) {
t.Errorf("scan --no-default-excludes /var: stderr = %q, want no %q",
stderr, line)
}
assertRecorded(t, path, f, true)
}
// TestRunScanSkipsNonPersistentOperand scans, through the command, a
// directory on a filesystem that is not persistent, with and without
// --no-default-excludes: either way it is skipped, with a line saying
// so, and nothing in it is recorded.
func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
shm, fsType, _ := shmTempDir(t)
writeFile(t, shm, "f", pattern(1, 10))
t.Setenv(databaseEnv, testDBPath(t))
line := "scan: skipped " + shm + " (" + fsType +
", not a persistent filesystem)\n"
summary := "scan: 0 files seen"
for _, flags := range [][]string{nil, {"--no-default-excludes"}} {
stderr := scanOK(t, append(flags, shm)...)
if !strings.Contains(stderr, line) ||
!strings.Contains(stderr, summary) {
t.Errorf("scan %q: stderr = %q, want %q and %q", flags, stderr,
line, summary)
}
}
}
// TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem
// that is not persistent beside one on the tests' temporary directory.
// The records an earlier scan left in it count as outside the operands:
// the record of a file gone from it is kept, and its size joins the size
// census, while a file added to it is not recorded, and its size does
// not join the census.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
shm, fsType, rules := shmTempDir(t)
stderr := captureStderr(t)
dir := t.TempDir()
db := openTestDB(t)
// A scan that, without the list of filesystems, records gone.
gone := writeFile(t, shm, "gone", pattern(1, 100))
syncTree(t, db, shm)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
unseen := writeFile(t, shm, "new", pattern(2, 10))
// added has the size of gone, and lonely that of unseen: each is
// hashed only if the census holds that size.
added := writeFile(t, dir, "added", pattern(3, 100))
lonely := writeFile(t, dir, "lonely", pattern(4, 10))
st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules)
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
recordByPath(t, recs, gone)
if recordByPath(t, recs, added).head == "" {
t.Errorf("%s not hashed: the census lacks the size of %s",
added, gone)
}
if recordByPath(t, recs, lonely).head != "" {
t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen)
}
if slices.Contains(recordPaths(recs), unseen) {
t.Errorf("%s recorded", unseen)
}
line := "scan: skipped " + shm + " (" + fsType +
", not a persistent filesystem)\n"
if !strings.Contains(stderr(), line) {
t.Errorf("stderr = %q, want %q", stderr(), line)
}
}
// TestScanContentSkipsNonPersistent checks that the content phase does
// not read a file on a filesystem that is not persistent, though an
// earlier scan left a record for it: it is treated as a changed file,
// so its record is left as it is and its identical copy in another
// tree, having no other match, is not read either.
func TestScanContentSkipsNonPersistent(t *testing.T) {
t.Parallel()
shm, _, rules := shmTempDir(t)
db := openTestDB(t)
// A scan that, without the list of filesystems, records a with a
// head and tail but no content hash.
a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
syncTree(t, db, shm)
before := recordByPath(t, dbRecords(t, db), a)
if before.head == "" || before.content != "" {
t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
}
b := sparseFile(t, t.TempDir(), "b", headTailMin)
st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
rules)
if err != nil {
t.Fatal(err)
}
if st.skipped != 0 {
t.Errorf("skipped = %d, want 0", st.skipped)
}
recs := dbRecords(t, db)
if r := recordByPath(t, recs, a); r != before {
t.Errorf("record = %+v, want it left as %+v", r, before)
}
if r := recordByPath(t, recs, b); r.content != "" {
t.Errorf("%s: content = %q, want none: its only match is on %s",
b, r.content, shm)
}
}
// TestWalkLeavesOutNonPersistent starts the walk itself in a directory
// on a filesystem that is not persistent, which a scan never does: it
// drops such an operand first. Below a directory on a persistent
// filesystem, one on another takes a mount, which a test cannot make.
// The walk enters no directory and records no file there, and names
// each.
func TestWalkLeavesOutNonPersistent(t *testing.T) {
t.Parallel()
shm, fsType, rules := shmTempDir(t)
f := writeFile(t, shm, "f", pattern(1, 10))
sub := filepath.Dir(writeFile(t, shm, "sub/g", pattern(1, 10)))
var leftOut []string
for ev := range startWalk(t.Context(), []string{shm}, false, rules, 1) {
if ev.rec.path != "" {
t.Errorf("walk recorded %s", ev.rec.path)
}
if ev.leftOut == "" {
continue
}
leftOut = append(leftOut, ev.leftOut)
want := "scan: skipped " + ev.leftOut + " (" + fsType +
", not a persistent filesystem)"
if ev.warn != want {
t.Errorf("line = %q, want %q", ev.warn, want)
}
}
slices.Sort(leftOut)
if want := []string{f, sub}; !slices.Equal(leftOut, want) {
t.Errorf("walk left out %q, want %q", leftOut, want)
}
}
// shmTempDir returns a new directory in /dev/shm, removed when the test
// ends, the type of its filesystem, and the rules holding the real
// filesystems that are not persistent. It skips the test where /dev/shm
// is on a persistent filesystem, or cannot be written to.
func shmTempDir(t *testing.T) (string, string, walkRules) {
t.Helper()
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfPath("/dev/shm")
fsType, found := nonPersistent[dev]
if !ok || !found {
t.Skip("/dev/shm is not a memory-backed filesystem here")
}
//nolint:usetesting // the directory must be inside /dev/shm
dir, err := os.MkdirTemp("/dev/shm", "sfdupes-test-")
if err != nil {
t.Skip("cannot write to /dev/shm here: " + err.Error())
}
t.Cleanup(func() { _ = os.RemoveAll(dir) })
return dir, fsType, walkRules{nonPersistent: nonPersistent}
}
// assertRecorded checks that the database at dbPath holds a record for
// path when want is set, and none when it is not.
func assertRecorded(t *testing.T, dbPath, path string, want bool) {
t.Helper()
db, err := openDB(dbPath, reportParams)
if err != nil {
t.Fatal(err)
}
defer func() { _ = db.Close() }()
if got := slices.Contains(recordPaths(dbRecords(t, db)), path); got != want {
t.Errorf("%s recorded: %v, want %v", path, got, want)
}
}
-2
View File
@@ -24,5 +24,3 @@ require (
modernc.org/mathutil v1.7.1 // indirect modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect modernc.org/memory v1.11.0 // indirect
) )
retract v0.0.1 // early build without a database; does not work as the README describes
+11 -46
View File
@@ -12,7 +12,7 @@
// //
// Usage: // Usage:
// //
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH... // sfdupes scan [--workers N] [-x] PATH...
// sfdupes report > dupes.tsv // sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv // sfdupes trees > dupetrees.tsv
// sfdupes info // sfdupes info
@@ -28,7 +28,6 @@ import (
"io" "io"
"os" "os"
"runtime" "runtime"
"runtime/debug"
"github.com/spf13/cobra" "github.com/spf13/cobra"
) )
@@ -59,32 +58,10 @@ var errNoSubcommand = errors.New("no subcommand")
var errWorkersBelowOne = errors.New("--workers must be at least 1") var errWorkersBelowOne = errors.New("--workers must be at least 1")
// Version is the build version, injected at link time via -ldflags // Version is the build version, injected at link time via -ldflags
// (see the Makefile); empty when the linker set none, as for a binary // (see the Makefile); "dev" for a plain go build.
// built by go install. buildVersion turns it into what --version prints.
// //
//nolint:gochecknoglobals // written only by the linker //nolint:gochecknoglobals // written only by the linker
var Version string var Version = "dev"
// buildVersion returns the version that sfdupes --version prints
// (README "Install"): the version the linker set, when it set one;
// otherwise the main module's version that Go recorded in the binary,
// a tag such as v0.0.2 or the pseudo-version of an @main build; and
// "dev" when neither is known. info is what debug.ReadBuildInfo
// returns, nil when the binary carries none. Go records "(devel)" for
// a build that has no version, such as one inside a copy of the source
// without git.
func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
if linkerVersion != "" {
return linkerVersion
}
if info == nil || info.Main.Version == "" ||
info.Main.Version == "(devel)" {
return "dev"
}
return info.Main.Version
}
func main() { func main() {
// Once the reader of a stdout pipe has gone, as in "sfdupes report | // Once the reader of a stdout pipe has gone, as in "sfdupes report |
@@ -139,10 +116,7 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
var showVersion bool var showVersion bool
printVersion := runE(func(context.Context, []string) error { printVersion := runE(func(context.Context, []string) error {
info, _ := debug.ReadBuildInfo() _, err := fmt.Fprintf(stdout, "sfdupes %s\n", Version)
_, err := fmt.Fprintf(stdout, "sfdupes %s\n",
buildVersion(Version, info))
if err != nil { if err != nil {
return fmt.Errorf("write stdout: %w", err) return fmt.Errorf("write stdout: %w", err)
} }
@@ -208,23 +182,17 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
return root return root
} }
// newScanCommand builds the scan subcommand, with its --workers, -x and // newScanCommand builds the scan subcommand, with its --workers and -x
// --no-default-excludes flags. Its help lists the default excludes of // flags.
// the system it runs on, and the filesystem types it never records
// anything on.
func newScanCommand() *cobra.Command { func newScanCommand() *cobra.Command {
var ( var (
scanWorkers int scanWorkers int
scanOneFS bool scanOneFS bool
scanNoDefaultExcludes bool
) )
short := "Walk trees and synchronize the scan database"
scanCmd := &cobra.Command{ scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...", Use: cmdScan + " [--workers N] [-x] PATH...",
Short: short, Short: "Walk trees and synchronize the scan database",
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1), Args: cobra.MinimumNArgs(1),
PreRunE: func(cmd *cobra.Command, _ []string) error { PreRunE: func(cmd *cobra.Command, _ []string) error {
return checkScanWorkers(cmd, scanWorkers) return checkScanWorkers(cmd, scanWorkers)
@@ -233,16 +201,13 @@ func newScanCommand() *cobra.Command {
ctx, stop := interruptContext(ctx) ctx, stop := interruptContext(ctx)
defer stop() defer stop()
return runScan(ctx, args, scanWorkers, scanOneFS, return runScan(ctx, args, scanWorkers, scanOneFS)
scanNoDefaultExcludes)
}), }),
} }
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(), scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
"concurrent workers for the walk, hash, and content phases") "concurrent workers for the walk, hash, and content phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false, scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries") "do not cross filesystem boundaries")
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
return scanCmd return scanCmd
} }
+2 -70
View File
@@ -4,12 +4,10 @@ import (
"bytes" "bytes"
"context" "context"
"errors" "errors"
"fmt"
"io" "io"
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"runtime/debug"
"slices" "slices"
"strconv" "strconv"
"strings" "strings"
@@ -20,36 +18,6 @@ import (
// error prints and a runtime failure must not. // error prints and a runtime failure must not.
const usageMarker = "Usage:" const usageMarker = "Usage:"
// TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail
// without saying why.
func TestMain(m *testing.M) {
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
fmt.Fprintf(os.Stderr, "list the mounted filesystems: %v\n", err)
os.Exit(1)
}
dir := os.TempDir()
fi, err := os.Stat(dir)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
dev, ok := deviceOfInfo(fi)
if fsType, found := nonPersistent[dev]; ok && found {
fmt.Fprintf(os.Stderr, "%s is on %s, not a persistent filesystem, "+
"and scan records nothing there: set TMPDIR to a directory on a "+
"persistent filesystem\n", dir, fsType)
os.Exit(1)
}
os.Exit(m.Run())
}
// walSuffixes are the SQLite sidecar files a WAL-mode database keeps // walSuffixes are the SQLite sidecar files a WAL-mode database keeps
// while it is open. A clean close checkpoints the WAL and removes // while it is open. A clean close checkpoints the WAL and removes
// both; finding either afterwards means the database was never closed. // both; finding either afterwards means the database was never closed.
@@ -567,7 +535,7 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}{ }{
cmdScan: { cmdScan: {
run: func() error { run: func() error {
return runScan(t.Context(), []string{missing}, 1, false, false) return runScan(t.Context(), []string{missing}, 1, false)
}, },
want: scanWant, want: scanWant,
}, },
@@ -715,8 +683,6 @@ func TestRunVersion(t *testing.T) {
// README §Subcommands: the version is one line on stdout, with // README §Subcommands: the version is one line on stdout, with
// nothing on stderr, and exits 0. // nothing on stderr, and exits 0.
info, _ := debug.ReadBuildInfo()
for _, arg := range []string{"--version", "-v"} { for _, arg := range []string{"--version", "-v"} {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -725,7 +691,7 @@ func TestRunVersion(t *testing.T) {
t.Errorf("run(%s) = %d, want %d", arg, code, exitOK) t.Errorf("run(%s) = %d, want %d", arg, code, exitOK)
} }
want := "sfdupes " + buildVersion(Version, info) + "\n" want := "sfdupes " + Version + "\n"
if got := stdout.String(); got != want { if got := stdout.String(); got != want {
t.Errorf("run(%s) stdout = %q, want %q", arg, got, want) t.Errorf("run(%s) stdout = %q, want %q", arg, got, want)
} }
@@ -736,40 +702,6 @@ func TestRunVersion(t *testing.T) {
} }
} }
func TestBuildVersion(t *testing.T) {
t.Parallel()
// README "Install": the linker's version wins; otherwise the main
// module's version that Go recorded; "dev" when neither is known.
tagged := &debug.BuildInfo{Main: debug.Module{Version: "v0.0.2"}}
pseudo := &debug.BuildInfo{Main: debug.Module{
Version: "v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1",
}}
devel := &debug.BuildInfo{Main: debug.Module{Version: "(devel)"}}
cases := []struct {
name string
linker string
info *debug.BuildInfo
want string
}{
{"linker set", "v0.0.2-3-g4e4a94d", tagged, "v0.0.2-3-g4e4a94d"},
{"linker set, no build info", "4e4a94d", nil, "4e4a94d"},
{"build info with a tag", "", tagged, "v0.0.2"},
{"build info with a pseudo-version", "", pseudo,
"v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1"},
{"(devel)", "", devel, "dev"},
{"no build info", "", nil, "dev"},
}
for _, c := range cases {
if got := buildVersion(c.linker, c.info); got != c.want {
t.Errorf("%s: buildVersion(%q, ...) = %q, want %q",
c.name, c.linker, got, c.want)
}
}
}
func TestRunVersionWriteFailureIsFatal(t *testing.T) { func TestRunVersionWriteFailureIsFatal(t *testing.T) {
t.Parallel() t.Parallel()
+118 -242
View File
@@ -13,7 +13,6 @@ import (
"os" "os"
"os/signal" "os/signal"
"path/filepath" "path/filepath"
"runtime"
"slices" "slices"
"strings" "strings"
"sync" "sync"
@@ -115,13 +114,9 @@ func scanPhase(name string) string {
// SIGTERM that interruptContext catches, the scan keeps what it has // SIGTERM that interruptContext catches, the scan keeps what it has
// hashed (see syncScan), prints how many files its walk reached, and // hashed (see syncScan), prints how many files its walk reached, and
// returns errInterrupted. workers must be at least 1; the scan command // returns errInterrupted. workers must be at least 1; the scan command
// rejects anything less. Before the lock is taken, the scan lists the // rejects anything less.
// filesystems that are not persistent, on which the walk records
// nothing; unless noDefaultExcludes is set, the walk does not enter the
// default excludes either. Once the database is open, the scan notes
// its files, which the walk passes over.
func runScan(ctx context.Context, roots []string, workers int, func runScan(ctx context.Context, roots []string, workers int,
oneFS, noDefaultExcludes bool, oneFS bool,
) error { ) error {
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
@@ -144,16 +139,6 @@ func runScan(ctx context.Context, roots []string, workers int,
return err return err
} }
nonPersistent, err := nonPersistentFilesystems()
if err != nil {
return fmt.Errorf("list the mounted filesystems: %w", err)
}
rules := walkRules{nonPersistent: nonPersistent}
if !noDefaultExcludes {
rules.excludes = defaultExcludes(runtime.GOOS)
}
lock, err := lockScanDatabase(dbPath) lock, err := lockScanDatabase(dbPath)
if err != nil { if err != nil {
return err return err
@@ -174,12 +159,7 @@ func runScan(ctx context.Context, roots []string, workers int,
defer closeScanDatabase(ctx, db, dbPath) defer closeScanDatabase(ctx, db, dbPath)
rules.databaseFiles, err = databaseFiles(dbPath) st, err := syncScan(ctx, db, roots, workers, oneFS)
if err != nil {
return err
}
st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
if errors.Is(err, context.Canceled) { if errors.Is(err, context.Canceled) {
return interrupted(st.walked) return interrupted(st.walked)
} }
@@ -251,12 +231,14 @@ func resolveRoots(roots []string) ([]string, error) {
} }
// pruneRoots drops operands already covered by another operand: // pruneRoots drops operands already covered by another operand:
// duplicates and any operand lying under another one. An operand under // duplicates and any operand lying under another one. With -x, an
// another one is kept, though, when the walk of the deepest kept operand // operand under another one is kept when the walk of the deepest kept
// above it does not reach it (see walkReaches). Every remaining file is // operand above it cannot reach it: when device puts any directory from
// then reachable through exactly one root, so walked paths are unique // it (from its directory, for a regular file) up to but not including
// without keeping a scan-wide set of every path seen. // that operand on a different filesystem than that operand. Every
func pruneRoots(roots []string, oneFS bool, rules walkRules, // remaining file is then reachable through exactly one root, so walked
// paths are unique without keeping a scan-wide set of every path seen.
func pruneRoots(roots []string, oneFS bool,
device func(path string) (uint64, bool), device func(path string) (uint64, bool),
) []string { ) []string {
// Shorter-first ordering guarantees an ancestor is kept before any // Shorter-first ordering guarantees an ancestor is kept before any
@@ -280,57 +262,40 @@ func pruneRoots(roots []string, oneFS bool, rules walkRules,
continue continue
} }
// A duplicate is dropped, and so is an operand the walk of above // Without -x, and always for a duplicate, r is dropped.
// reaches. if !oneFS || above == r {
if above != r && !walkReaches(above, r, oneFS, rules, device) { continue
kept = append(kept, r) }
aboveDev, aboveOK := device(above)
if !aboveOK {
continue
}
// The walk of above enters a directory only when it is on
// above's filesystem, and then reaches every regular file in it
// whatever device the file reports, so for a regular-file
// operand the directories checked start at its directory.
dir := r
fi, err := os.Lstat(r)
if err == nil && fi.Mode().IsRegular() {
dir = filepath.Dir(r)
}
for ; dir != above; dir = filepath.Dir(dir) {
dev, ok := device(dir)
if ok && dev != aboveDev {
kept = append(kept, r)
break
}
} }
} }
return kept return kept
} }
// walkReaches reports whether the walk of operand above reaches r, an
// operand under it. That walk does not enter a default exclude in
// rules, nor a directory that device puts on a filesystem in rules that
// is not persistent, nor, with -x, one that device puts on a different
// filesystem than above, when device knows above's. It reaches every
// regular file in a directory it enters, so for a regular-file operand
// the directories checked start at its directory; they end below above.
func walkReaches(above, r string, oneFS bool, rules walkRules,
device func(path string) (uint64, bool),
) bool {
aboveDev, aboveDevOK := device(above)
dir := r
fi, err := os.Lstat(r)
if err == nil && fi.Mode().IsRegular() {
dir = filepath.Dir(r)
}
for ; dir != above; dir = filepath.Dir(dir) {
if isDefaultExclude(dir, rules.excludes) {
return false
}
dev, ok := device(dir)
if !ok {
continue
}
if _, found := rules.nonPersistent[dev]; found {
return false
}
if oneFS && aboveDevOK && dev != aboveDev {
return false
}
}
return true
}
// deviceOfPath returns the filesystem device ID of the file at path, // deviceOfPath returns the filesystem device ID of the file at path,
// when lstat succeeds and the platform exposes one. // when lstat succeeds and the platform exposes one.
func deviceOfPath(path string) (uint64, bool) { func deviceOfPath(path string) (uint64, bool) {
@@ -372,11 +337,11 @@ type scanState struct {
// the hashed records still waiting in the batch, starts no other write // the hashed records still waiting in the batch, starts no other write
// or deletion, and returns the cancellation. // or deletion, and returns the cancellation.
func syncScan(ctx context.Context, db *sql.DB, roots []string, func syncScan(ctx context.Context, db *sql.DB, roots []string,
workers int, oneFS bool, rules walkRules, workers int, oneFS bool,
) (scanStats, error) { ) (scanStats, error) {
s := &scanState{db: db} s := &scanState{db: db}
err := s.runPhases(ctx, roots, workers, oneFS, rules) err := s.runPhases(ctx, roots, workers, oneFS)
if err == nil || ctx.Err() == nil { if err == nil || ctx.Err() == nil {
return s.st, err return s.st, err
} }
@@ -400,18 +365,15 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
// and content (fill in the content hash of every record of headTailMin // and content (fill in the content hash of every record of headTailMin
// or more whose size, head, and tail match another record's). Records // or more whose size, head, and tail match another record's). Records
// outside the roots are never touched, except that the content phase // outside the roots are never touched, except that the content phase
// fills in their content hash. Operands the walk cannot start from, or // fills in their content hash. Operands the walk cannot start from are
// that are not on a persistent filesystem, are dropped first, so the // dropped first, so the records beneath them are treated as if those
// records beneath them are treated as if those operands had not been // operands had not been given.
// given. Below an operand, the walk follows rules.
func (s *scanState) runPhases(ctx context.Context, roots []string, func (s *scanState) runPhases(ctx context.Context, roots []string,
workers int, oneFS bool, rules walkRules, workers int, oneFS bool,
) error { ) error {
// Types and filesystems are checked before pruning so that an // Types are checked before pruning so that an operand under a
// operand under a dropped one is still scanned, not dropped as lying // dropped one is still scanned, not dropped as lying under it.
// under it. roots = pruneRoots(s.walkableRoots(roots), oneFS, deviceOfPath)
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
deviceOfPath)
err := s.loadIndex(ctx, roots) err := s.loadIndex(ctx, roots)
if err != nil { if err != nil {
@@ -419,7 +381,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
} }
changed, unhashed := s.walkPhase( changed, unhashed := s.walkPhase(
startWalk(ctx, roots, oneFS, rules, workers), roots) startWalk(ctx, roots, oneFS, workers), roots)
// A cancelled walk stops early, so its size census covers only part // A cancelled walk stops early, so its size census covers only part
// of the roots, and every file it never reached would look vanished // of the roots, and every file it never reached would look vanished
@@ -441,7 +403,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
return err return err
} }
return s.contentPhase(ctx, workers, rules) return s.contentPhase(ctx, workers)
} }
// walkableRoots returns the operands the walk can start from: regular // walkableRoots returns the operands the walk can start from: regular
@@ -449,11 +411,9 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
// about, counted as skipped, and dropped. A dropped operand is no // about, counted as skipped, and dropped. A dropped operand is no
// longer a root, so the records stored beneath it are treated as if it // longer a root, so the records stored beneath it are treated as if it
// had not been given: those under no other root count as outside the // had not been given: those under no other root count as outside the
// roots and are not deleted as unverified. An operand on a filesystem // roots and are not deleted as unverified. An operand that fails lstat
// in rules that is not persistent is dropped the same way, with a line // here is kept, and the walk warns about it.
// saying so that is not a warning, and is not counted as skipped. An func (s *scanState) walkableRoots(roots []string) []string {
// operand that fails lstat here is kept, and the walk warns about it.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
kept := make([]string, 0, len(roots)) kept := make([]string, 0, len(roots))
for _, root := range roots { for _, root := range roots {
@@ -467,13 +427,6 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
continue continue
} }
line := rules.nonPersistentLine(root, fi)
if line != "" {
fmt.Fprintln(os.Stderr, escapePath(line))
continue
}
} }
kept = append(kept, root) kept = append(kept, root)
@@ -528,12 +481,10 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// (the content phase may still fill in its content hash). It returns // (the content phase may still fill in its content hash). It returns
// the new-or-changed files and the unchanged files whose records lack // the new-or-changed files and the unchanged files whose records lack
// hashes; both remain candidates until the census decides whether // hashes; both remain candidates until the census decides whether
// their sizes are shared. A record at or below a path the walk left // their sizes are shared. A record below a directory -x did not enter
// out, because -x stopped at it, it is a default exclude, or it is not // counts as outside the roots: it is not deleted, and its size joins
// on a persistent filesystem, counts as outside the roots: it is not // the census. That does not hold below a root at or inside that
// deleted, and its size joins the census. That does not hold below a // directory, since that root's own walk verifies the records below it.
// root at or inside that path, since that root's own walk verifies the
// records below it; nor is a line printed for such a root.
func (s *scanState) walkPhase( func (s *scanState) walkPhase(
events <-chan walkEvent, roots []string, events <-chan walkEvent, roots []string,
) ([]fileRec, []fileRec) { ) ([]fileRec, []fileRec) {
@@ -551,25 +502,12 @@ func (s *scanState) walkPhase(
var ( var (
changed, unhashed []fileRec changed, unhashed []fileRec
leftOut []string otherFSDirs []string
) )
for ev := range events { for ev := range events {
if ev.otherFSDir != "" { if ev.otherFSDir != "" {
leftOut = append(leftOut, ev.otherFSDir) otherFSDirs = append(otherFSDirs, ev.otherFSDir)
continue
}
// The line saying so is printed as a warning is, but what the
// walk leaves out is not counted as skipped. A default exclude
// that is itself a root is walked as one, so it gets no line.
if ev.leftOut != "" {
leftOut = append(leftOut, ev.leftOut)
if !slices.Contains(roots, ev.leftOut) {
prog.warnf("%s", ev.warn)
}
continue continue
} }
@@ -604,13 +542,12 @@ func (s *scanState) walkPhase(
prog.finish() prog.finish()
for path, old := range s.existing { for path, old := range s.existing {
// Kept only when a path the walk left out lies below the deepest // Kept only when a directory -x did not enter lies below the
// root above the record, or is the record's own. A root at or // deepest root above the record. A root at or below that
// below that path was walked on its own and did not find the // directory was walked on its own and did not find the file.
// file. Both contain the record, so the longer path is the // Both contain the record, so the longer path is the deeper.
// deeper. otherFSDir := deepestRoot(path, otherFSDirs)
left := deepestRoot(path, leftOut) if len(otherFSDir) > len(deepestRoot(path, roots)) {
if len(left) > len(deepestRoot(path, roots)) {
delete(s.existing, path) delete(s.existing, path)
s.sizes = append(s.sizes, old.size) s.sizes = append(s.sizes, old.size)
@@ -716,15 +653,6 @@ func sameInode(a, b fileRec) bool {
return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
} }
// isDatabaseFile reports whether f, a file the walk reached, is one of
// the scan's own database files in rules, which the walk passes over
// without a line and without counting it.
func isDatabaseFile(f fileRec, rules walkRules) bool {
return slices.ContainsFunc(rules.databaseFiles, func(d fileRec) bool {
return sameInode(d, f)
})
}
// hashPhase hashes every queued file with hashSignature — the head and // hashPhase hashes every queued file with hashSignature — the head and
// tail of a file of headTailMin or more, the whole file below that — // tail of a file of headTailMin or more, the whole file below that —
// committing completed records to the database in batches as results // committing completed records to the database in batches as results
@@ -893,12 +821,9 @@ func (s *scanState) updatePhase(ctx context.Context) error {
// hash phase's worker pool and their records written back in batches. A // hash phase's worker pool and their records written back in batches. A
// failed read is warned about and counted as skipped; the record keeps // failed read is warned about and counted as skipped; the record keeps
// its empty content, so it is never grouped, and a later scan tries // its empty content, so it is never grouped, and a later scan tries
// again. No file on a filesystem in rules that is not persistent is // again.
// read. func (s *scanState) contentPhase(ctx context.Context, workers int) error {
func (s *scanState) contentPhase(ctx context.Context, workers int, toRead, recs, err := s.contentCandidates(ctx)
rules walkRules,
) error {
toRead, recs, err := s.contentCandidates(ctx, rules)
if err != nil { if err != nil {
return err return err
} }
@@ -933,19 +858,18 @@ func (s *scanState) contentPhase(ctx context.Context, workers int,
// contentCandidates returns the files the content phase reads, and // contentCandidates returns the files the content phase reads, and
// their records by path. Every record contentCandidatesSQL returns has // their records by path. Every record contentCandidatesSQL returns has
// its file checked with lstat, whether or not it already has a content // its file checked with lstat, whether or not it already has a content
// hash: a file that is gone, is no longer a regular file, has changed // hash: a file that is gone, is no longer a regular file, or has
// by the walk's rule, or is on a filesystem in rules that is not // changed by the walk's rule keeps its record as it is and does not
// persistent keeps its record as it is and does not count as a match // count as a match for the others, and any other lstat error is warned
// for the others, and any other lstat error is warned about and counted // about and counted as skipped, with the same result. If such a record
// as skipped, with the same result. If such a record has no content // has no content hash, it stays out of duplicate groups; if it has one,
// hash, it stays out of duplicate groups; if it has one, it is still // it is still reported until a scan covering its own tree updates or
// reported until a scan covering its own tree updates or removes it. // removes it. The files of a group that pass and have no content hash
// The files of a group that pass and have no content hash are read only // are read only if at least minGroupSize of the group's files pass, so
// if at least minGroupSize of the group's files pass, so a group whose // a group whose other members are all stale costs no reads. Only the
// other members are all stale costs no reads. Only the records to be // records to be read are kept.
// read are kept.
func (s *scanState) contentCandidates( func (s *scanState) contentCandidates(
ctx context.Context, rules walkRules, ctx context.Context,
) ([]fileRec, map[string]scanRec, error) { ) ([]fileRec, map[string]scanRec, error) {
// The search and the checks take real time on a large database; // The search and the checks take real time on a large database;
// without a display the scan looks hung before the reads begin. // without a display the scan looks hung before the reads begin.
@@ -1003,7 +927,7 @@ func (s *scanState) contentCandidates(
first = r first = r
} }
f, ok, err := unchangedFile(r, rules) f, ok, err := unchangedFile(r)
if err != nil { if err != nil {
s.st.skipped++ s.st.skipped++
@@ -1037,10 +961,9 @@ func (s *scanState) contentCandidates(
// unchangedFile lstats the file r names and returns it for reading if // unchangedFile lstats the file r names and returns it for reading if
// it is still the regular file r records: the same size, and an mtime // it is still the regular file r records: the same size, and an mtime
// no newer than recorded (the walk's change rule). A file that is gone, // no newer than recorded (the walk's change rule). A file that is gone
// has changed, or is on a filesystem in rules that is not persistent // or has changed reports false; any other lstat error is returned.
// reports false; any other lstat error is returned. func unchangedFile(r scanRec) (fileRec, bool, error) {
func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
fi, err := os.Lstat(r.path) fi, err := os.Lstat(r.path)
if errors.Is(err, fs.ErrNotExist) { if errors.Is(err, fs.ErrNotExist) {
return fileRec{}, false, nil return fileRec{}, false, nil
@@ -1057,12 +980,6 @@ func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
dev, ino := inodeOfInfo(fi) dev, ino := inodeOfInfo(fi)
// The scan reads nothing on a filesystem that is not persistent,
// whatever record an earlier scan left for the file.
if _, found := rules.nonPersistent[dev]; found {
return fileRec{}, false, nil
}
return fileRec{ return fileRec{
path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino, path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
}, true, nil }, true, nil
@@ -1129,36 +1046,32 @@ type dirJob struct {
} }
// walkEvent is one walk result delivered to the walk phase: a regular // walkEvent is one walk result delivered to the walk phase: a regular
// file's statted record, a warning when fail is set, or a path the walk // file's statted record, a warning when fail is set, or, when
// left out: when otherFSDir is set, a directory -x did not enter // otherFSDir is set, a directory -x did not enter because it is on
// because it is on another filesystem than its operand, and when // another filesystem than its operand.
// leftOut is set, a default exclude or a directory or file on a
// filesystem that is not persistent, with warn holding the line that
// says so.
type walkEvent struct { type walkEvent struct {
rec fileRec rec fileRec
warn string warn string
fail bool fail bool
otherFSDir string otherFSDir string
leftOut string
} }
// startWalk seeds every root into the shared walk worker pool and // startWalk seeds every root into the shared walk worker pool and
// returns the event stream: one record per regular file, one warning // returns the event stream: one record per regular file, one warning
// event per per-path error, and one event per path -x or rules kept the // event per per-path error, and one event per directory -x did not
// walk from entering or recording. The channel is closed when the walk // enter. The channel is closed when the walk completes, and also when
// completes, and also when ctx is cancelled — every goroutine in the // ctx is cancelled — every goroutine in the pool abandons its blocking
// pool abandons its blocking send in that case, so the consumer sees a // send in that case, so the consumer sees a truncated but properly
// truncated but properly terminated stream instead of a stalled one. // terminated stream instead of a stalled one.
func startWalk(ctx context.Context, roots []string, oneFS bool, func startWalk(ctx context.Context, roots []string, oneFS bool,
rules walkRules, workers int, workers int,
) <-chan walkEvent { ) <-chan walkEvent {
jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS, rules) jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS)
go func() { go func() {
initial := make([]dirJob, 0, len(roots)) initial := make([]dirJob, 0, len(roots))
for _, root := range roots { for _, root := range roots {
initial = append(initial, seedRoot(ctx, root, rules, events)...) initial = append(initial, seedRoot(ctx, root, events)...)
} }
dispatchDirs(ctx, initial, jobs, subdirs) dispatchDirs(ctx, initial, jobs, subdirs)
@@ -1212,14 +1125,13 @@ func operandWarning(root string, fi fs.FileInfo) string {
} }
// seedRoot turns one PATH operand into the walk's starting state: a // seedRoot turns one PATH operand into the walk's starting state: a
// regular-file operand is statted and emitted directly, unless it is // regular-file operand is statted and emitted directly, and a directory
// one of the scan's own database files in rules, and a directory
// operand becomes an initial job. walkableRoots has already dropped // operand becomes an initial job. walkableRoots has already dropped
// every other operand. One that has changed into something else since // every other operand. One that has changed into something else since
// is warned about and skipped here; it is still a root, so the records // is warned about and skipped here; it is still a root, so the records
// stored beneath it, other than those under another root, are deleted // stored beneath it, other than those under another root, are deleted
// as unverified. // as unverified.
func seedRoot(ctx context.Context, root string, rules walkRules, func seedRoot(ctx context.Context, root string,
events chan<- walkEvent, events chan<- walkEvent,
) []dirJob { ) []dirJob {
fi, err := os.Lstat(root) fi, err := os.Lstat(root)
@@ -1246,17 +1158,14 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
} }
dev, ino := inodeOfInfo(fi) dev, ino := inodeOfInfo(fi)
rec := fileRec{
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: root, path: root,
size: fi.Size(), size: fi.Size(),
mtime: fi.ModTime(), mtime: fi.ModTime(),
dev: dev, dev: dev,
ino: ino, ino: ino,
} }})
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
return nil return nil
} }
@@ -1268,7 +1177,7 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
// workers drop the directories still queued rather than stop reading // workers drop the directories still queued rather than stop reading
// jobs, so the range always runs out and the pool always tears down. // jobs, so the range always runs out and the pool always tears down.
func startWalkWorkers(ctx context.Context, workers int, func startWalkWorkers(ctx context.Context, workers int,
oneFS bool, rules walkRules, oneFS bool,
) (chan dirJob, chan []dirJob, chan walkEvent) { ) (chan dirJob, chan []dirJob, chan walkEvent) {
jobs := make(chan dirJob, workQueueDepth) jobs := make(chan dirJob, workQueueDepth)
subdirs := make(chan []dirJob, workers) subdirs := make(chan []dirJob, workers)
@@ -1284,7 +1193,7 @@ func startWalkWorkers(ctx context.Context, workers int,
} }
select { select {
case subdirs <- walkOneDir(ctx, job, oneFS, rules, events): case subdirs <- walkOneDir(ctx, job, oneFS, events):
case <-ctx.Done(): case <-ctx.Done():
} }
} }
@@ -1342,7 +1251,7 @@ func dispatchDirs(ctx context.Context, initial []dirJob,
// entry and a warning event per unreadable one, and returns the // entry and a warning event per unreadable one, and returns the
// subdirectories to descend into. // subdirectories to descend into.
func walkOneDir(ctx context.Context, job dirJob, oneFS bool, func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
rules walkRules, events chan<- walkEvent, events chan<- walkEvent,
) []dirJob { ) []dirJob {
entries, err := os.ReadDir(job.path) entries, err := os.ReadDir(job.path)
if err != nil { if err != nil {
@@ -1366,8 +1275,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
p := filepath.Join(job.path, e.Name()) p := filepath.Join(job.path, e.Name())
if e.IsDir() { if e.IsDir() {
sub, ok := subdirJob(ctx, p, e, job, oneFS, rules, events) if sub, ok := subdirJob(ctx, p, e, job, oneFS, events); ok {
if ok {
subs = append(subs, sub) subs = append(subs, sub)
} }
@@ -1379,7 +1287,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
continue continue
} }
emitFile(ctx, p, e, rules, events) emitFile(ctx, p, e, events)
} }
return subs return subs
@@ -1389,12 +1297,9 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
// The lstat happens here in the walk worker, while the directory's // The lstat happens here in the walk worker, while the directory's
// metadata is still hot; a path that fails to stat (or stops being a // metadata is still hot; a path that fails to stat (or stops being a
// regular file) between the directory read and the lstat is warned // regular file) between the directory read and the lstat is warned
// about and skipped. A file on a filesystem in rules that is not // about and skipped.
// persistent, as through a bind mount of one file, is not recorded: it
// gets an event naming it instead. One of the scan's own database files
// in rules gets no event at all.
func emitFile(ctx context.Context, p string, e fs.DirEntry, func emitFile(ctx context.Context, p string, e fs.DirEntry,
rules walkRules, events chan<- walkEvent, events chan<- walkEvent,
) { ) {
info, err := e.Info() info, err := e.Info()
if err != nil { if err != nil {
@@ -1410,43 +1315,35 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
return return
} }
line := rules.nonPersistentLine(p, info)
if line != "" {
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
return
}
dev, ino := inodeOfInfo(info) dev, ino := inodeOfInfo(info)
rec := fileRec{
sendEvent(ctx, events, walkEvent{rec: fileRec{
path: p, path: p,
size: info.Size(), size: info.Size(),
mtime: info.ModTime(), mtime: info.ModTime(),
dev: dev, dev: dev,
ino: ino, ino: ino,
} }})
if !isDatabaseFile(rec, rules) {
sendEvent(ctx, events, walkEvent{rec: rec})
}
} }
// subdirJob applies the descent rules to directory p: never enter // subdirJob applies the descent rules to directory p: never enter
// .zfs (ZFS snapshot pseudo-dirs would list every file once per // .zfs (ZFS snapshot pseudo-dirs would list every file once per
// snapshot), never enter a directory on a filesystem in rules that is // snapshot), and with -x never enter a directory on a different
// not persistent, nor a default exclude in rules, and with -x never // filesystem than its operand, sending an event that names it so the
// enter a directory on a different filesystem than its operand. For the // scan keeps the records below it, except those below an operand at or
// last three it sends an event that names the directory, so the scan
// keeps the records below it, except those below an operand at or
// inside it. // inside it.
func subdirJob(ctx context.Context, p string, e fs.DirEntry, func subdirJob(ctx context.Context, p string, e fs.DirEntry,
parent dirJob, oneFS bool, rules walkRules, parent dirJob, oneFS bool, events chan<- walkEvent,
events chan<- walkEvent,
) (dirJob, bool) { ) (dirJob, bool) {
if e.Name() == ".zfs" { if e.Name() == ".zfs" {
return dirJob{}, false return dirJob{}, false
} }
job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
if !oneFS || !parent.rootDevOK {
return job, true
}
info, err := e.Info() info, err := e.Info()
if err != nil { if err != nil {
sendEvent(ctx, events, walkEvent{ sendEvent(ctx, events, walkEvent{
@@ -1457,27 +1354,6 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
return dirJob{}, false return dirJob{}, false
} }
line := rules.nonPersistentLine(p, info)
if line != "" {
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
return dirJob{}, false
}
if isDefaultExclude(p, rules.excludes) {
sendEvent(ctx, events, walkEvent{
leftOut: p,
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
})
return dirJob{}, false
}
job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
if !oneFS || !parent.rootDevOK {
return job, true
}
if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev { if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
sendEvent(ctx, events, walkEvent{otherFSDir: p}) sendEvent(ctx, events, walkEvent{otherFSDir: p})
+9 -126
View File
@@ -782,8 +782,7 @@ func collectWalk(t *testing.T, roots []string, oneFS bool,
errs int errs int
) )
for ev := range startWalk(t.Context(), roots, oneFS, walkRules{}, for ev := range startWalk(t.Context(), roots, oneFS, workers) {
workers) {
if ev.fail { if ev.fail {
errs++ errs++
@@ -1005,8 +1004,7 @@ func callSubdirJob(t *testing.T, p string, e fs.DirEntry,
t.Helper() t.Helper()
events := make(chan walkEvent, 1) events := make(chan walkEvent, 1)
job, ok := subdirJob(t.Context(), p, e, parent, oneFS, walkRules{}, job, ok := subdirJob(t.Context(), p, e, parent, oneFS, events)
events)
close(events) close(events)
var evs []walkEvent var evs []walkEvent
@@ -1154,7 +1152,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
return dev, ok return dev, ok
} }
roots = pruneRoots(roots, true, walkRules{}, device) roots = pruneRoots(roots, true, device)
ctx := t.Context() ctx := t.Context()
s := &scanState{db: db} s := &scanState{db: db}
@@ -1164,14 +1162,14 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
t.Fatal(err) t.Fatal(err)
} }
jobs, subdirs, events := startWalkWorkers(ctx, 2, true, walkRules{}) jobs, subdirs, events := startWalkWorkers(ctx, 2, true)
// seedRoot emits a regular-file operand itself and returns a job // seedRoot emits a regular-file operand itself and returns a job
// only for a directory operand. // only for a directory operand.
var initial []dirJob var initial []dirJob
for _, root := range roots { for _, root := range roots {
for _, job := range seedRoot(ctx, root, walkRules{}, events) { for _, job := range seedRoot(ctx, root, events) {
job.rootDev = devices[root] job.rootDev = devices[root]
initial = append(initial, job) initial = append(initial, job)
} }
@@ -1465,7 +1463,7 @@ const smokeTreeFiles = 15
func syncTree(t *testing.T, db *sql.DB, roots ...string) scanStats { func syncTree(t *testing.T, db *sql.DB, roots ...string) scanStats {
t.Helper() t.Helper()
st, err := syncScan(t.Context(), db, roots, 4, false, walkRules{}) st, err := syncScan(t.Context(), db, roots, 4, false)
if err != nil { if err != nil {
t.Fatalf("syncScan: %v", err) t.Fatalf("syncScan: %v", err)
} }
@@ -2414,7 +2412,7 @@ func TestPruneRoots(t *testing.T) {
// Duplicates and operands under other operands are dropped; /cc is // Duplicates and operands under other operands are dropped; /cc is
// not under /c (sibling with a shared prefix). // not under /c (sibling with a shared prefix).
got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"}, got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
false, walkRules{}, device) false, device)
want := []string{"/a", "/c", "/cc"} want := []string{"/a", "/c", "/cc"}
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
@@ -2423,7 +2421,7 @@ func TestPruneRoots(t *testing.T) {
nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"} nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
got = pruneRoots(nested, false, walkRules{}, device) got = pruneRoots(nested, false, device)
want = []string{"/p"} want = []string{"/p"}
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
@@ -2431,31 +2429,12 @@ func TestPruneRoots(t *testing.T) {
} }
// With -x, /p/q, on another filesystem than /p, is kept as well. // With -x, /p/q, on another filesystem than /p, is kept as well.
got = pruneRoots(nested, true, walkRules{}, device) got = pruneRoots(nested, true, device)
want = []string{"/p", pqRoot} want = []string{"/p", pqRoot}
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Fatalf("pruneRoots with -x = %q, want %q", got, want) t.Fatalf("pruneRoots with -x = %q, want %q", got, want)
} }
// The walk of /p does not enter /p/q when it is a default exclude, so
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
device)
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
}
// Nor does it enter /p/q when the filesystem of /p/q is not
// persistent, so /p/q/r/m, of no known device, is kept.
got = pruneRoots([]string{"/p", "/p/q/r/m"}, false,
walkRules{nonPersistent: map[uint64]string{2: "tmpfs"}}, device)
want = []string{"/p", "/p/q/r/m"}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots with %s not persistent = %q, want %q",
pqRoot, got, want)
}
} }
func TestReportsNeverTouchFilesystem(t *testing.T) { func TestReportsNeverTouchFilesystem(t *testing.T) {
@@ -2501,99 +2480,3 @@ func TestUnderRoot(t *testing.T) {
} }
} }
} }
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
// database, reached directly and through a symlinked directory. The
// walk passes over the database file, its -wal and -shm files and its
// .lock file with no line and no count, and the record an earlier scan
// left for the .lock file is deleted. A rescan of the unchanged tree
// updates nothing, and report lists none of the database's files.
func TestRunScanPassesOverDatabase(t *testing.T) {
for _, viaSymlink := range []bool{false, true} {
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
base := t.TempDir()
tree := filepath.Join(base, "real", "tree")
operand := tree
writeFile(t, tree, "a", pattern(1, 100))
writeFile(t, tree, "b", pattern(1, 100))
writeFile(t, tree, "empty1", nil)
writeFile(t, tree, "empty2", nil)
if viaSymlink {
err := os.Symlink(filepath.Join(base, "real"),
filepath.Join(base, "link"))
if err != nil {
t.Fatal(err)
}
operand = filepath.Join(base, "link", "tree")
}
dbPath := filepath.Join(tree, "db.sqlite")
t.Setenv(databaseEnv, dbPath)
db, err := openScanDatabase(t.Context(), dbPath)
if err != nil {
t.Fatal(err)
}
err = applyChanges(t.Context(), db, []scanRec{
{path: filepath.Join(operand, "db.sqlite.lock")},
}, nil, nil)
if err != nil {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, dbPath)
assertScanSummary(t,
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
operand)
assertScanSummary(t,
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
operand)
report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") ||
!strings.Contains(report, filepath.Join(operand, "empty2")) {
t.Errorf("report = %q, want the empty files and none of "+
"the database's files", report)
}
})
}
}
// TestRunScanPassesOverDatabaseOperands scans the database file and its
// .lock file given as operands themselves: neither is recorded or
// counted.
func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
dbPath := testDBPath(t)
t.Setenv(databaseEnv, dbPath)
// The first scan creates both files.
scanOK(t, t.TempDir())
assertScanSummary(t,
"0 files seen (0 added, 0 updated, 0 removed, 0 unchanged)",
dbPath, dbPath+".lock")
}
// assertScanSummary scans operands and fails the test unless the
// summary line gives counts with nothing skipped, and the line naming
// the database is the only one that names one of its files.
func assertScanSummary(t *testing.T, counts string, operands ...string) {
t.Helper()
stderr := scanOK(t, operands...)
want := "scan: " + counts + ", 0 skipped\n"
if !strings.Contains(stderr, want) {
t.Errorf("scan %q: stderr = %q, want %q", operands, stderr, want)
}
if strings.Count(stderr, "db.sqlite") != 1 {
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
"database to name one of its files", operands, stderr)
}
}