Compare commits
1
Commits
next
..
3f4ff92dae
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3f4ff92dae |
@@ -49,40 +49,15 @@ results are as fresh as the last completed scan.
|
|||||||
|
|
||||||
### Install
|
### Install
|
||||||
|
|
||||||
With Go installed, this builds and installs the newest release:
|
With Go installed, this builds and installs the current `main` branch:
|
||||||
|
|
||||||
```sh
|
|
||||||
go install sneak.berlin/go/sfdupes@latest
|
|
||||||
```
|
|
||||||
|
|
||||||
Until 1.0, releases are `v0.0.N` tags on the tip of the `main` branch: each time
|
|
||||||
`main` moves and its tip passes `script/cibuild`, that tip is tagged with the
|
|
||||||
next `v0.0.N`. So `@latest` installs `main`'s tip once it has passed the build.
|
|
||||||
`@main` installs the tip before it is tagged:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go install sneak.berlin/go/sfdupes@main
|
go install sneak.berlin/go/sfdupes@main
|
||||||
```
|
```
|
||||||
|
|
||||||
`v0.0.1` is an early build without a database. It does not work as this README
|
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. A
|
||||||
describes, and `go.mod` retracts it.
|
binary installed this way reports its version as `dev`; one built from a clone
|
||||||
|
or into the Docker image carries the git tag or commit it was built from.
|
||||||
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. To
|
|
||||||
see which version an installed binary is:
|
|
||||||
|
|
||||||
```sh
|
|
||||||
sfdupes --version
|
|
||||||
go version -m "$(command -v sfdupes)"
|
|
||||||
```
|
|
||||||
|
|
||||||
`sfdupes --version` prints `sfdupes VERSION`. For a binary from `go install`,
|
|
||||||
`VERSION` is the version Go recorded in it: the release tag, such as `v0.0.3`,
|
|
||||||
or for `@main` a pseudo-version ending in the first 12 characters of the
|
|
||||||
commit's hash. A binary built from a clone or into the Docker image carries the
|
|
||||||
git tag or commit it was built from instead, and `VERSION` is `dev` only when
|
|
||||||
the binary carries no version at all. `go version -m` prints what Go recorded in
|
|
||||||
the binary; its `mod` line names the module's version. Binaries from `v0.0.2`
|
|
||||||
and earlier print `dev`, so check those with `go version -m`.
|
|
||||||
|
|
||||||
From a clone, `make build` writes the binary to `./sfdupes`:
|
From a clone, `make build` writes the binary to `./sfdupes`:
|
||||||
|
|
||||||
@@ -125,11 +100,9 @@ As a file `/etc/cron.d/sfdupes`:
|
|||||||
`/var/log/sfdupes.log`, holds a first line naming the database (see
|
`/var/log/sfdupes.log`, holds a first line naming the database (see
|
||||||
"Database"), plain progress lines naming each phase and its place among the
|
"Database"), plain progress lines naming each phase and its place among the
|
||||||
five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
|
five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
|
||||||
runs, and one as it ends), a warning for each path it skips, a line for each
|
runs, and one as it ends), a warning for each path it skips, and the summary
|
||||||
default exclude its walk does not enter and for each path it leaves out
|
line (see "Progress" and "`scan` mode"). The log grows with every scan; rotate
|
||||||
because it is not on a persistent filesystem (see "Rules for the walk"), and
|
it like any other.
|
||||||
the summary line (see "Progress" and "`scan` mode"). The log grows with every
|
|
||||||
scan; rotate it like any other.
|
|
||||||
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
|
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
|
||||||
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
|
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
|
||||||
reason among the last lines of the log; `tail` prints them, and cron mails
|
reason among the last lines of the log; `tail` prints them, and cron mails
|
||||||
@@ -275,7 +248,7 @@ Four subcommands, all implemented:
|
|||||||
schema version and the counts of its records.
|
schema version and the counts of its records.
|
||||||
|
|
||||||
```
|
```
|
||||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
sfdupes scan [--workers N] [-x] PATH...
|
||||||
sfdupes report > dupes.tsv
|
sfdupes report > dupes.tsv
|
||||||
sfdupes trees > dupetrees.tsv
|
sfdupes trees > dupetrees.tsv
|
||||||
sfdupes info
|
sfdupes info
|
||||||
@@ -284,13 +257,12 @@ sfdupes [command] --help
|
|||||||
```
|
```
|
||||||
|
|
||||||
`--workers N` sets the size of each `scan` worker pool (default: the number of
|
`--workers N` sets the size of each `scan` worker pool (default: the number of
|
||||||
CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
|
CPUs), and `-x` (`--one-file-system`) keeps the walk of each operand on that
|
||||||
operand's filesystem, and `--no-default-excludes` lets the walk enter the
|
operand's filesystem; both are described under "`scan` mode".
|
||||||
default excludes, a list of paths; all three are described under "`scan` mode".
|
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
|
||||||
`sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION` (see
|
exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
|
||||||
"Install"), to stdout and exits 0, writing nothing to stderr. `-h` or `--help`,
|
subcommand, prints the help text to stderr and exits 0, writing nothing to
|
||||||
alone or after a subcommand, prints the help text to stderr and exits 0, writing
|
stdout.
|
||||||
nothing to stdout.
|
|
||||||
|
|
||||||
### Database
|
### Database
|
||||||
|
|
||||||
@@ -467,12 +439,11 @@ results do not depend on the working directory. All operands belong to a single
|
|||||||
scan and are enumerated concurrently: every operand seeds the shared walk worker
|
scan and are enumerated concurrently: every operand seeds the shared walk worker
|
||||||
pool. Overlapping operands are harmless — an operand that duplicates another or
|
pool. Overlapping operands are harmless — an operand that duplicates another or
|
||||||
lies under another is dropped before walking, so every file is reached exactly
|
lies under another is dropped before walking, so every file is reached exactly
|
||||||
once and produces one database record. An operand under another one is kept,
|
once and produces one database record. With `-x`, though, an operand under
|
||||||
though, and walked on its own when any directory from it (from its directory,
|
another one is kept and walked on its own when any directory from it (from its
|
||||||
for a regular file) up to but not including the operand covering it is a default
|
directory, for a regular file) up to but not including the operand covering it
|
||||||
exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
|
is on a different filesystem than that operand, because that operand's walk does
|
||||||
`-x`, is on a different filesystem than that operand, because that operand's
|
not enter such a directory and so does not reach it.
|
||||||
walk does not enter such a directory and so does not reach it.
|
|
||||||
|
|
||||||
An operand that is a symlink (never followed, not even as an operand), socket,
|
An operand that is a symlink (never followed, not even as an operand), socket,
|
||||||
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
|
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
|
||||||
@@ -482,10 +453,7 @@ beneath it is still scanned. The records stored beneath it are not deleted: they
|
|||||||
are treated like any other record outside the scanned operands, including the
|
are treated like any other record outside the scanned operands, including the
|
||||||
content-phase exception below. If it lies under another operand, they are under
|
content-phase exception below. If it lies under another operand, they are under
|
||||||
that operand instead, and are treated like any other record there. This is not
|
that operand instead, and are treated like any other record there. This is not
|
||||||
an error: a scan whose every operand is dropped walks nothing and exits 0. An
|
an error: a scan whose every operand is dropped walks nothing and exits 0.
|
||||||
operand that is not on a persistent filesystem (see "Rules for the walk") is
|
|
||||||
dropped the same way, except that the line `scan` prints for it is not a
|
|
||||||
warning, and it is not counted as skipped.
|
|
||||||
|
|
||||||
`scan` synchronizes the database with the filesystem state under the scanned
|
`scan` synchronizes the database with the filesystem state under the scanned
|
||||||
operands:
|
operands:
|
||||||
@@ -521,16 +489,14 @@ operands:
|
|||||||
can be scanned on different schedules into the same database. The one
|
can be scanned on different schedules into the same database. The one
|
||||||
exception is the content phase below: a stored file of 10 MiB or more without
|
exception is the content phase below: a stored file of 10 MiB or more without
|
||||||
a `content` hash is read for one, wherever it lies, once its size, `head`, and
|
a `content` hash is read for one, wherever it lies, once its size, `head`, and
|
||||||
`tail` match another record's. If that file is gone, has changed since its
|
`tail` match another record's. If that file is gone or has changed since its
|
||||||
record was written, or is not on a persistent filesystem, it is not read and
|
record was written, the record is left as it is.
|
||||||
the record is left as it is.
|
- With `-x`, a record below a directory the walk did not enter because that
|
||||||
- A record at or below a path the walk left out, because that path is a default
|
directory is on another filesystem than its operand counts as outside the
|
||||||
exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
|
scanned operands: it is not deleted, and its size joins the size census. A
|
||||||
another filesystem than its operand, counts as outside the scanned operands:
|
filesystem mounted under an operand can therefore be scanned on its own
|
||||||
it is not deleted, and its size joins the size census. A filesystem mounted
|
schedule into the same database. Records below an operand at or inside such a
|
||||||
under an operand can therefore be scanned on its own schedule into the same
|
directory are not kept this way, since that operand's own walk verifies them.
|
||||||
database. Records below an operand at or inside such a path are not kept this
|
|
||||||
way, since that operand's own walk verifies them.
|
|
||||||
|
|
||||||
`scan` runs **five sequential phases over the whole scan**. Parallelism lives
|
`scan` runs **five sequential phases over the whole scan**. Parallelism lives
|
||||||
inside the walk, hash and content phases; batched database writes begin during
|
inside the walk, hash and content phases; batched database writes begin during
|
||||||
@@ -576,20 +542,19 @@ the hash phase:
|
|||||||
be read are kept in memory, never every file's hashes. Every record sharing
|
be read are kept in memory, never every file's hashes. Every record sharing
|
||||||
their size, `head`, and `tail`, including one that already has a `content`
|
their size, `head`, and `tail`, including one that already has a `content`
|
||||||
hash, has its file checked with `lstat` first. A file that is gone, is no
|
hash, has its file checked with `lstat` first. A file that is gone, is no
|
||||||
longer a regular file, has changed (a different size, or an mtime newer than
|
longer a regular file, or has changed (a different size, or an mtime newer
|
||||||
recorded), or is not on a persistent filesystem (see "Rules for the walk")
|
than recorded) keeps its record as it is and does not count as a match for
|
||||||
keeps its record as it is and does not count as a match for the others. Any
|
the others. Any other `lstat` error is warned about and counted as skipped,
|
||||||
other `lstat` error is warned about and counted as skipped, with the same
|
with the same result. If such a record has no `content` hash, it stays out
|
||||||
result. If such a record has no `content` hash, it stays out of duplicate
|
of duplicate groups; if it has one, it is still reported until a scan
|
||||||
groups; if it has one, it is still reported until a scan covering its own
|
covering its own tree updates or removes it. The files that pass and have no
|
||||||
tree updates or removes it. The files that pass and have no `content` hash
|
`content` hash are read only if at least two of those records pass, so a
|
||||||
are read only if at least two of those records pass, so a file whose only
|
file whose only matches are stale costs no read; a file that already has a
|
||||||
matches are stale costs no read; a file that already has a `content` hash is
|
`content` hash is never read again. They are read by a worker pool as in the
|
||||||
never read again. They are read by a worker pool as in the hash phase, in
|
hash phase, in inode order and once per inode, and their content hashes are
|
||||||
inode order and once per inode, and their content hashes are committed in
|
committed in batches. A failed read is warned about and counted as skipped;
|
||||||
batches. A failed read is warned about and counted as skipped; its record
|
its record keeps an empty `content`, so it is not a duplicate, and a later
|
||||||
keeps an empty `content`, so it is not a duplicate, and a later scan tries
|
scan tries again.
|
||||||
again.
|
|
||||||
|
|
||||||
Rules for the walk:
|
Rules for the walk:
|
||||||
|
|
||||||
@@ -600,15 +565,6 @@ Rules for the walk:
|
|||||||
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
|
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
|
||||||
them would list every file once per snapshot), not even when it is an operand;
|
them would list every file once per snapshot), not even when it is an operand;
|
||||||
such an operand is dropped the same way.
|
such an operand is dropped the same way.
|
||||||
- Never record the scan's own database. Once the database and its `.lock` file
|
|
||||||
are open, `scan` notes the device and inode of the database file, its `-wal`
|
|
||||||
and `-shm` files and its `.lock` file, and the walk passes over a file with
|
|
||||||
one of them, however it reaches it: as an operand, below one, or through a
|
|
||||||
symlinked directory. It prints no line for such a file and does not count it
|
|
||||||
as skipped. A record that an earlier scan left for one of them is deleted,
|
|
||||||
like the record of any file the scan no longer verifies. There is no
|
|
||||||
`-journal` file during the walk, because the database is in WAL mode then (see
|
|
||||||
"Database").
|
|
||||||
- Filesystem boundaries are crossed by default. With `-x` (long form
|
- Filesystem boundaries are crossed by default. With `-x` (long form
|
||||||
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
|
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
|
||||||
into a directory on a different filesystem than its `PATH` operand; each
|
into a directory on a different filesystem than its `PATH` operand; each
|
||||||
@@ -616,74 +572,6 @@ Rules for the walk:
|
|||||||
unless they lie below an operand at or inside that directory, count as outside
|
unless they lie below an operand at or inside that directory, count as outside
|
||||||
the scanned operands, as described in "`scan` mode" above: they are not
|
the scanned operands, as described in "`scan` mode" above: they are not
|
||||||
deleted, and their sizes join the size census.
|
deleted, and their sizes join the size census.
|
||||||
- Never record anything that is not on a persistent filesystem. A filesystem is
|
|
||||||
not persistent when the kernel makes it up or keeps it in memory: one of type
|
|
||||||
`proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
|
|
||||||
`debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
|
|
||||||
`mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
|
|
||||||
`rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
|
|
||||||
it names the database and before it takes the lock, `scan` lists the mounted
|
|
||||||
filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
|
|
||||||
of mounted filesystems on macOS (on any other system it lists none), and notes
|
|
||||||
the device ID of each one of those types. On Linux that is the device ID the
|
|
||||||
list gives, so a filesystem that a later mount hides still gives its own. On
|
|
||||||
macOS it is read with `lstat` on the mount point, and where one filesystem is
|
|
||||||
mounted over another on the same mount point, the one mounted last counts. If
|
|
||||||
`scan` cannot list them, it fails (exit 1). The walk then never enters a
|
|
||||||
directory, and never records a file, on one of those devices, however it
|
|
||||||
reaches it: as an operand, below one, or through a bind mount, which keeps the
|
|
||||||
device ID of what it binds. The content phase never reads a file on one of
|
|
||||||
them either. No flag turns this off. For each operand, directory or file it
|
|
||||||
leaves out this way, `scan` prints a line on stderr naming it and the type of
|
|
||||||
its filesystem, as it prints a warning, so it never lands inside a progress
|
|
||||||
display:
|
|
||||||
|
|
||||||
```
|
|
||||||
scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
|
|
||||||
```
|
|
||||||
|
|
||||||
That line is not a warning, and the path is not counted as skipped. Records
|
|
||||||
that an earlier scan left at or below it, unless they lie below an operand
|
|
||||||
at or inside it, count as outside the scanned operands, as described in
|
|
||||||
"`scan` mode" above: they are not deleted, and their sizes join the size
|
|
||||||
census.
|
|
||||||
|
|
||||||
- Below an operand, never descend into a default exclude: a path that holds the
|
|
||||||
kernel's view of the running system, or temporary files, so that
|
|
||||||
`sfdupes scan /` walks only the files worth comparing. The default excludes
|
|
||||||
depend on the system, and `sfdupes scan --help` lists those of the system it
|
|
||||||
runs on:
|
|
||||||
- On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
|
|
||||||
- On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
|
|
||||||
`/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
|
|
||||||
everything in `/System/Volumes/Data` through the firmlinks, so walking it
|
|
||||||
as well would record each of those files twice.
|
|
||||||
- On Windows, relative to the root of the drive an operand lies on: the
|
|
||||||
page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
|
|
||||||
`hiberfil.sys`, and the directories `System Volume Information`,
|
|
||||||
`$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
|
|
||||||
matches one path component, as it does in any default exclude. sfdupes
|
|
||||||
does not build for Windows yet, so this list is not used.
|
|
||||||
- On any other system there are none.
|
|
||||||
|
|
||||||
`--no-default-excludes` turns them all off; what is not on a persistent
|
|
||||||
filesystem stays left out. An operand at or inside a default exclude is
|
|
||||||
still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
|
|
||||||
persistent filesystem. For each default exclude its walk does not enter,
|
|
||||||
`scan` prints a line on stderr naming it, as it prints a warning:
|
|
||||||
|
|
||||||
```
|
|
||||||
scan: skipped /var/tmp (default exclude)
|
|
||||||
```
|
|
||||||
|
|
||||||
That line is not a warning, and the directory is not counted as skipped. A
|
|
||||||
default exclude that is not on a persistent filesystem, as `/proc` is, gets
|
|
||||||
the line for its filesystem instead, and one that is itself an operand gets
|
|
||||||
no line, since it is walked as that operand. The records below a default
|
|
||||||
exclude the walk does not enter, unless they lie below an operand at or
|
|
||||||
inside it, count as outside the scanned operands, as described in "`scan`
|
|
||||||
mode" above: they are not deleted, and their sizes join the size census.
|
|
||||||
|
|
||||||
- On any per-path error (permission denied, file vanished between passes,
|
- On any per-path error (permission denied, file vanished between passes,
|
||||||
unreadable): print a one-line warning to stderr, skip the path, and continue.
|
unreadable): print a one-line warning to stderr, skip the path, and continue.
|
||||||
Per-file errors never abort the run; the final summary reports how many were
|
Per-file errors never abort the run; the final summary reports how many were
|
||||||
@@ -990,17 +878,15 @@ Additional requirements:
|
|||||||
terminal is. A line too wide for the terminal first loses its bar, which gives
|
terminal is. A line too wide for the terminal first loses its bar, which gives
|
||||||
way to the spinner, then its rate, the words of a stalled phase, or what the
|
way to the spinner, then its rate, the words of a stalled phase, or what the
|
||||||
phase is waiting for. The phase and its place, the progress, the elapsed time
|
phase is waiting for. The phase and its place, the progress, the elapsed time
|
||||||
and the ETA are kept whole. On an 80-column terminal, at counts like those of
|
and the ETA are kept whole. On an 80-column terminal, a stalled line at counts
|
||||||
the stalled line in the example above, a stalled line keeps its words whenever
|
and times like those of the stalled line in the example above keeps its words
|
||||||
the elapsed time and the ETA are both in minutes, except in the content
|
whole in every phase; where the words and the ETA do not both fit, as at
|
||||||
phase's checks and reads. Their lines are the widest, and at most times in
|
millions of files and times in hours, the words give way. Only when those four
|
||||||
minutes they lose the words. Where the words and the ETA do not both fit, as
|
alone are too wide, as on a terminal narrower than about 70 columns hours into
|
||||||
at millions of files and times in hours, the words give way. Only when those
|
a scan of millions of files, is the line cut short at the terminal's edge, so
|
||||||
four alone are too wide, as on a terminal narrower than about 70 columns hours
|
the end of the ETA is lost first. A terminal made narrower while a phase runs
|
||||||
into a scan of millions of files, is the line cut short at the terminal's
|
gets the same: the next draw erases the line within the new width and fits the
|
||||||
edge, so the end of the ETA is lost first. A terminal made narrower while a
|
line to it, and the display goes on redrawing in place.
|
||||||
phase runs gets the same: the next draw erases the line within the new width
|
|
||||||
and fits the line to it, and the display goes on redrawing in place.
|
|
||||||
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
|
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
|
||||||
redraws: print the line the moment each phase starts, every 5 seconds while it
|
redraws: print the line the moment each phase starts, every 5 seconds while it
|
||||||
runs (whether or not an item has completed, and never more often), and once
|
runs (whether or not an item has completed, and never more often), and once
|
||||||
@@ -1195,6 +1081,10 @@ All of the following, run in this directory, must pass:
|
|||||||
./sfdupes report
|
./sfdupes report
|
||||||
```
|
```
|
||||||
|
|
||||||
|
(The database lives in a temp directory of its own: inside `$d`, the scan
|
||||||
|
would record it, and its empty lock file would join the `empty1`/`empty2`
|
||||||
|
group.)
|
||||||
|
|
||||||
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
|
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
|
||||||
group (two dupe rows, `first` is the lexicographically smallest path);
|
group (two dupe rows, `first` is the lexicographically smallest path);
|
||||||
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
|
`t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
|
||||||
|
|||||||
@@ -29,48 +29,6 @@
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- releases until 1.0 are `v0.0.N` tags on `main`'s tip, so
|
|
||||||
`go install sneak.berlin/go/sfdupes@latest` installs `main`'s tip once it has
|
|
||||||
passed the build; README "Install" says so, says what `@main` and `v0.0.1`
|
|
||||||
give, and how to check an installed binary. `go.mod` retracts `v0.0.1`, the
|
|
||||||
early build without a database. When the linker set no version,
|
|
||||||
`sfdupes --version` prints the main module's version that Go recorded in the
|
|
||||||
binary, the tag or the pseudo-version of a `go install` build, and `dev` only
|
|
||||||
when neither is known (2026-10-17,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/116)
|
|
||||||
|
|
||||||
- `scan` creates the database's directory where the database path leads: the
|
|
||||||
path with its last element cut off as text, never cleaned, so a `..` after a
|
|
||||||
symlinked directory is followed through the symlink, as when the file is
|
|
||||||
opened. With `SFDUPES_DATABASE=/srv/link/../new/db.sqlite` and `/srv/link` a
|
|
||||||
symlink to `/data/x`, it creates `/data/new` and the database in it, where it
|
|
||||||
used to create `/srv/new` and then fail (2026-10-17,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/119)
|
|
||||||
|
|
||||||
- `scan` never records its own database: once the database and its `.lock` file
|
|
||||||
are open, it notes the device and inode of the database file, its `-wal` and
|
|
||||||
`-shm` files and its `.lock` file, and the walk passes over a file with one of
|
|
||||||
them wherever the operands reach it, through a symlinked directory included,
|
|
||||||
printing no line and counting nothing. A record an earlier scan left for one
|
|
||||||
of them is deleted (2026-10-17, https://git.eeqj.de/sneak/sfdupes/issues/115)
|
|
||||||
|
|
||||||
- `scan` never records or reads anything that is not on a persistent filesystem
|
|
||||||
(`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
|
|
||||||
matched by the device ID of each such filesystem, listed as the scan starts
|
|
||||||
from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
|
|
||||||
however the walk or the content phase reaches it; no flag turns that off.
|
|
||||||
Below an operand, its walk does not enter the default excludes either, so
|
|
||||||
`sfdupes scan /` skips the kernel's view of the running system and the
|
|
||||||
temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
|
|
||||||
`/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
|
|
||||||
`/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
|
|
||||||
Windows list, not used yet. An operand at or inside a default exclude is still
|
|
||||||
walked. The scan prints a line for each path it leaves out, without counting
|
|
||||||
it as skipped, and the records there count as outside the scanned operands, as
|
|
||||||
for `-x`. `--no-default-excludes` turns off the default excludes only, and
|
|
||||||
`scan --help` lists them (2026-10-17,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/124)
|
|
||||||
|
|
||||||
- the rate a progress display shows covers the items completed over the last 5
|
- the rate a progress display shows covers the items completed over the last 5
|
||||||
seconds, or since the line before when stderr is not a terminal, so it falls
|
seconds, or since the line before when stderr is not a terminal, so it falls
|
||||||
to 0 while nothing completes, and a burst after a wait no longer reads as a
|
to 0 while nothing completes, and a burst after a wait no longer reads as a
|
||||||
@@ -80,11 +38,9 @@
|
|||||||
`db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
|
`db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
|
||||||
either, each phase giving its word where it is defined. A line too wide for
|
either, each phase giving its word where it is defined. A line too wide for
|
||||||
the terminal loses its bar first, then its rate or those words, so the phase,
|
the terminal loses its bar first, then its rate or those words, so the phase,
|
||||||
progress, elapsed time and ETA stay whole. On an 80-column terminal, at counts
|
progress, elapsed time and ETA stay whole; on an 80-column terminal the words
|
||||||
like those of README's example, the content phase's checks and reads lose
|
fit beside them in an ordinary scan. The content phase's last partial batch is
|
||||||
those words at most times in minutes, and every other phase keeps them
|
now written under a display of its own (2026-10-16,
|
||||||
whenever the elapsed time and the ETA are both in minutes. The content phase's
|
|
||||||
last partial batch is now written under a display of its own (2026-10-16,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||||
|
|
||||||
- new subcommand `info` prints the database's details on stdout, one per line as
|
- new subcommand `info` prints the database's details on stdout, one per line as
|
||||||
|
|||||||
+10
-15
@@ -196,7 +196,7 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
|
|||||||
base := baselineGoroutines(t)
|
base := baselineGoroutines(t)
|
||||||
|
|
||||||
st, err := syncScan(newWalkClock(walkCancelAtDone), db,
|
st, err := syncScan(newWalkClock(walkCancelAtDone), db,
|
||||||
[]string{dir}, walkCancelWorkers, false, walkRules{})
|
[]string{dir}, walkCancelWorkers, false)
|
||||||
|
|
||||||
assertWalkGuardAborted(t, st, err)
|
assertWalkGuardAborted(t, st, err)
|
||||||
assertRecordsIntact(t, db, before)
|
assertRecordsIntact(t, db, before)
|
||||||
@@ -268,8 +268,7 @@ func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
|
|||||||
ctx, cancel := context.WithCancel(t.Context())
|
ctx, cancel := context.WithCancel(t.Context())
|
||||||
cancel()
|
cancel()
|
||||||
|
|
||||||
st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false,
|
st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false)
|
||||||
walkRules{})
|
|
||||||
if !errors.Is(err, context.Canceled) {
|
if !errors.Is(err, context.Canceled) {
|
||||||
t.Fatalf("syncScan on a cancelled context = %v, want %v",
|
t.Fatalf("syncScan on a cancelled context = %v, want %v",
|
||||||
err, context.Canceled)
|
err, context.Canceled)
|
||||||
@@ -302,7 +301,7 @@ func TestSyncScanCancelledMidHashKeepsHashedRecords(t *testing.T) {
|
|||||||
db := openTestDB(t)
|
db := openTestDB(t)
|
||||||
|
|
||||||
st, err := syncScan(newWalkClock(hashCancelAtDone), db,
|
st, err := syncScan(newWalkClock(hashCancelAtDone), db,
|
||||||
[]string{dir}, walkCancelWorkers, false, walkRules{})
|
[]string{dir}, walkCancelWorkers, false)
|
||||||
if !errors.Is(err, context.Canceled) {
|
if !errors.Is(err, context.Canceled) {
|
||||||
t.Fatalf("syncScan cancelled mid-hash = %v, want %v",
|
t.Fatalf("syncScan cancelled mid-hash = %v, want %v",
|
||||||
err, context.Canceled)
|
err, context.Canceled)
|
||||||
@@ -357,8 +356,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
|||||||
stderr := captureStderr(t)
|
stderr := captureStderr(t)
|
||||||
dir := buildSmokeTree(t)
|
dir := buildSmokeTree(t)
|
||||||
|
|
||||||
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
|
||||||
false)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -382,7 +380,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
|||||||
defer close(done)
|
defer close(done)
|
||||||
|
|
||||||
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
|
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
|
||||||
false, false)
|
false)
|
||||||
}()
|
}()
|
||||||
|
|
||||||
awaitReturn(t, done, "runScan")
|
awaitReturn(t, done, "runScan")
|
||||||
@@ -409,8 +407,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
|||||||
got, before)
|
got, before)
|
||||||
}
|
}
|
||||||
|
|
||||||
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false)
|
||||||
false)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -436,7 +433,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
|
|||||||
dir := buildWalkCancelTree(t)
|
dir := buildWalkCancelTree(t)
|
||||||
|
|
||||||
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
|
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
|
||||||
walkCancelWorkers, false, false)
|
walkCancelWorkers, false)
|
||||||
if !errors.Is(err, errInterrupted) {
|
if !errors.Is(err, errInterrupted) {
|
||||||
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
|
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
|
||||||
err, errInterrupted)
|
err, errInterrupted)
|
||||||
@@ -620,8 +617,7 @@ func TestWalkOneDirStopsWhenCancelled(t *testing.T) {
|
|||||||
// Unbuffered and unread: on a cancelled scan every send gives up.
|
// Unbuffered and unread: on a cancelled scan every send gives up.
|
||||||
events := make(chan walkEvent)
|
events := make(chan walkEvent)
|
||||||
|
|
||||||
subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false,
|
subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false, events)
|
||||||
walkRules{}, events)
|
|
||||||
if len(subs) != 0 {
|
if len(subs) != 0 {
|
||||||
t.Errorf("cancelled walkOneDir returned %+v to descend into, "+
|
t.Errorf("cancelled walkOneDir returned %+v to descend into, "+
|
||||||
"want none", subs)
|
"want none", subs)
|
||||||
@@ -641,8 +637,7 @@ func TestWalkWorkersDropQueuedDirs(t *testing.T) {
|
|||||||
|
|
||||||
missing := filepath.Join(t.TempDir(), "missing")
|
missing := filepath.Join(t.TempDir(), "missing")
|
||||||
|
|
||||||
jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false,
|
jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false)
|
||||||
walkRules{})
|
|
||||||
|
|
||||||
for range 64 {
|
for range 64 {
|
||||||
jobs <- dirJob{path: missing}
|
jobs <- dirJob{path: missing}
|
||||||
@@ -673,7 +668,7 @@ func TestWalkWorkerAbandonsSubdirHandoff(t *testing.T) {
|
|||||||
ctx, cancel := context.WithCancel(t.Context())
|
ctx, cancel := context.WithCancel(t.Context())
|
||||||
defer cancel()
|
defer cancel()
|
||||||
|
|
||||||
jobs, subdirs, events := startWalkWorkers(ctx, 1, false, walkRules{})
|
jobs, subdirs, events := startWalkWorkers(ctx, 1, false)
|
||||||
|
|
||||||
// Fill the hand-back channel to its capacity — one slot per worker
|
// Fill the hand-back channel to its capacity — one slot per worker
|
||||||
// — so the worker's own hand-back is certain to block.
|
// — so the worker's own hand-back is certain to block.
|
||||||
|
|||||||
@@ -109,8 +109,10 @@ func databasePath() (string, string, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// The working directory as the kernel reports it, with no symlink
|
// The working directory as the kernel reports it, with no symlink
|
||||||
// in it, as README "Database" says. os.Getwd may return PWD
|
// in it. os.Getwd may return PWD instead, which can name it through
|
||||||
// instead, which can name it through a symlink.
|
// a symlink; a ".." after that symlink then leads somewhere else
|
||||||
|
// once the path is cleaned, as scan cleans it to create the
|
||||||
|
// database's directory.
|
||||||
wd, err := unix.Getwd()
|
wd, err := unix.Getwd()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
||||||
@@ -123,24 +125,6 @@ func databasePath() (string, string, error) {
|
|||||||
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
|
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// createDatabaseDirectory creates the directory the database at path
|
|
||||||
// goes in, and any directory above it, if missing. path is absolute, as
|
|
||||||
// databasePath returns it. The directory is path with its last element
|
|
||||||
// cut off as text, by filepath.Split. filepath.Dir would clean it,
|
|
||||||
// dropping each ".." together with the directory name before it; when
|
|
||||||
// that name is a symlink, the cleaned path names another directory than
|
|
||||||
// the one the system reaches when it opens path.
|
|
||||||
func createDatabaseDirectory(path string) error {
|
|
||||||
dir, _ := filepath.Split(path)
|
|
||||||
|
|
||||||
err := os.MkdirAll(dir, dbDirPerm)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("create database directory: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// scanParams are the connection parameters for scan: read-write, with
|
// scanParams are the connection parameters for scan: read-write, with
|
||||||
// WAL journaling and a busy timeout, so a report can run while a cron
|
// WAL journaling and a busy timeout, so a report can run while a cron
|
||||||
// scan is in progress. closeScanDatabase leaves WAL mode again.
|
// scan is in progress. closeScanDatabase leaves WAL mode again.
|
||||||
@@ -195,9 +179,9 @@ func openDB(path, params string) (*sql.DB, error) {
|
|||||||
// that deleted it would let the next scan lock a new file while
|
// that deleted it would let the next scan lock a new file while
|
||||||
// another still holds the old one.
|
// another still holds the old one.
|
||||||
func lockScanDatabase(path string) (*os.File, error) {
|
func lockScanDatabase(path string) (*os.File, error) {
|
||||||
err := createDatabaseDirectory(path)
|
err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, fmt.Errorf("create database directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Like SQLite, this follows a symlink at path and creates its
|
// Like SQLite, this follows a symlink at path and creates its
|
||||||
@@ -241,9 +225,9 @@ func lockScanDatabase(path string) (*os.File, error) {
|
|||||||
// openScanDatabase opens the database for the scan subcommand, creating
|
// openScanDatabase opens the database for the scan subcommand, creating
|
||||||
// the file, its parent directory, and the schema as needed.
|
// the file, its parent directory, and the schema as needed.
|
||||||
func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
|
func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
|
||||||
err := createDatabaseDirectory(path)
|
err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, fmt.Errorf("create database directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
db, err := openDB(path, scanParams)
|
db, err := openDB(path, scanParams)
|
||||||
@@ -261,42 +245,6 @@ func openScanDatabase(ctx context.Context, path string) (*sql.DB, error) {
|
|||||||
return db, nil
|
return db, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// databaseFiles returns, by device and inode, the files of the database
|
|
||||||
// at path that the scan keeps open: the database file, its -wal and
|
|
||||||
// -shm files and its .lock file, each named from the database's path
|
|
||||||
// with symlinks resolved, as SQLite and lockScanDatabase name them. The
|
|
||||||
// scan calls it once the database and the lock file are open, so that
|
|
||||||
// its walk passes over them however it reaches them, through a
|
|
||||||
// symlinked directory included. A file that does not exist then is left
|
|
||||||
// out, since no walk can reach it. There is no -journal file while the
|
|
||||||
// database is in WAL mode.
|
|
||||||
func databaseFiles(path string) ([]fileRec, error) {
|
|
||||||
resolved, err := filepath.EvalSymlinks(path)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("resolve database path: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var files []fileRec
|
|
||||||
|
|
||||||
for _, p := range []string{
|
|
||||||
resolved, resolved + "-wal", resolved + "-shm", resolved + ".lock",
|
|
||||||
} {
|
|
||||||
fi, err := os.Lstat(p)
|
|
||||||
if errors.Is(err, fs.ErrNotExist) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("database: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
dev, ino := inodeOfInfo(fi)
|
|
||||||
files = append(files, fileRec{path: p, dev: dev, ino: ino})
|
|
||||||
}
|
|
||||||
|
|
||||||
return files, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// closeScanDatabase switches the database at path from WAL back to
|
// closeScanDatabase switches the database at path from WAL back to
|
||||||
// rollback-journal mode and closes it. Out of WAL mode the database
|
// rollback-journal mode and closes it. Out of WAL mode the database
|
||||||
// file alone holds the whole database, so a reader needs no -wal or
|
// file alone holds the whole database, so a reader needs no -wal or
|
||||||
|
|||||||
-111
@@ -5,7 +5,6 @@ import (
|
|||||||
"database/sql"
|
"database/sql"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io/fs"
|
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"slices"
|
"slices"
|
||||||
@@ -119,116 +118,6 @@ func TestOpenScanDatabaseCreates(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestScanCreatesDatabaseDirectoryAfterSymlink has a database path go
|
|
||||||
// through a symlink to a directory, then "..", to a missing directory.
|
|
||||||
// With srv/link a symlink to data/x, srv/link/../new/db.sqlite leads to
|
|
||||||
// data/new/db.sqlite. lockScanDatabase and openScanDatabase, the two
|
|
||||||
// places scan creates the database's directory, must each create the
|
|
||||||
// database there, and nothing at srv/new, where the path leads once
|
|
||||||
// cleaned as text.
|
|
||||||
func TestScanCreatesDatabaseDirectoryAfterSymlink(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
creates := map[string]func(ctx context.Context, path string) error{
|
|
||||||
"lockScanDatabase": func(_ context.Context, path string) error {
|
|
||||||
lock, err := lockScanDatabase(path)
|
|
||||||
if err == nil {
|
|
||||||
_ = lock.Close()
|
|
||||||
}
|
|
||||||
|
|
||||||
return err
|
|
||||||
},
|
|
||||||
"openScanDatabase": func(ctx context.Context, path string) error {
|
|
||||||
db, err := openScanDatabase(ctx, path)
|
|
||||||
if err == nil {
|
|
||||||
_ = db.Close()
|
|
||||||
}
|
|
||||||
|
|
||||||
return err
|
|
||||||
},
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, create := range creates {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir, err := filepath.EvalSymlinks(t.TempDir())
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = os.MkdirAll(filepath.Join(dir, "data", "x"), 0o750)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = os.Mkdir(filepath.Join(dir, "srv"), 0o750)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = os.Symlink(filepath.Join(dir, "data", "x"),
|
|
||||||
filepath.Join(dir, "srv", "link"))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = create(t.Context(), dir+"/srv/link/../new/db.sqlite")
|
|
||||||
if err != nil {
|
|
||||||
t.Fatalf("%s: %v", name, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
|
|
||||||
if err != nil {
|
|
||||||
t.Errorf("database the path leads to: %v", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = os.Lstat(filepath.Join(dir, "srv", "new"))
|
|
||||||
if !errors.Is(err, fs.ErrNotExist) {
|
|
||||||
t.Errorf("srv/new: stat error %v, want it missing", err)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestDatabaseFiles opens a database through a symlink to its file:
|
|
||||||
// databaseFiles finds the database file and its -wal and -shm files
|
|
||||||
// beside the symlink's target, where SQLite keeps them, and leaves out
|
|
||||||
// the .lock file, which only a scan creates.
|
|
||||||
func TestDatabaseFiles(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
path := testDBPath(t)
|
|
||||||
link := filepath.Join(t.TempDir(), "link.sqlite")
|
|
||||||
|
|
||||||
err := os.Symlink(path, link)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
db, err := openScanDatabase(t.Context(), link)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
files, err := databaseFiles(link)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
got := make([]string, 0, len(files))
|
|
||||||
for _, f := range files {
|
|
||||||
got = append(got, f.path)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := []string{path, path + "-wal", path + "-shm"}
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Errorf("databaseFiles(%s) = %q, want %q", link, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestOpenDatabaseUnversionedForeign(t *testing.T) {
|
func TestOpenDatabaseUnversionedForeign(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
-195
@@ -1,195 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
// This file holds two rules for the walk (README "Rules for the walk"):
|
|
||||||
// the default excludes, the paths a scan's walk does not enter below an
|
|
||||||
// operand unless scan is given --no-default-excludes, and the
|
|
||||||
// filesystems that are not persistent, on which the scan never records
|
|
||||||
// or reads anything.
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bufio"
|
|
||||||
"errors"
|
|
||||||
"fmt"
|
|
||||||
"io"
|
|
||||||
"io/fs"
|
|
||||||
"path/filepath"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
|
|
||||||
"golang.org/x/sys/unix"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The systems that have default excludes, as runtime.GOOS names them.
|
|
||||||
const (
|
|
||||||
goosLinux = "linux"
|
|
||||||
goosDarwin = "darwin"
|
|
||||||
goosWindows = "windows"
|
|
||||||
)
|
|
||||||
|
|
||||||
// The fields of a line of mountInfoPath that readMountInfo reads,
|
|
||||||
// counted from 0 (see proc(5)): the device ID, as major:minor, is the
|
|
||||||
// third field, and after the sixth come optional fields, then a field
|
|
||||||
// holding "-", then the filesystem type.
|
|
||||||
const (
|
|
||||||
deviceField = 2
|
|
||||||
optionalFieldsStart = 6
|
|
||||||
)
|
|
||||||
|
|
||||||
// typesPerHelpLine is how many filesystem types the scan command's help
|
|
||||||
// lists on each line.
|
|
||||||
const typesPerHelpLine = 6
|
|
||||||
|
|
||||||
// errMountInfoLine reports a line of mountInfoPath without the fields
|
|
||||||
// readMountInfo reads.
|
|
||||||
var errMountInfoLine = errors.New("unexpected line")
|
|
||||||
|
|
||||||
// walkRules holds the rules for the walk that a scan sets up as it
|
|
||||||
// starts. excludes are the default excludes, or none with
|
|
||||||
// --no-default-excludes. nonPersistent maps the device ID of every
|
|
||||||
// mounted filesystem that is not persistent to its type; the scan
|
|
||||||
// records and reads nothing on one of those, whatever the flags.
|
|
||||||
// databaseFiles are the scan's own database files, which the walk
|
|
||||||
// passes over (see databaseFiles in db.go).
|
|
||||||
type walkRules struct {
|
|
||||||
excludes []string
|
|
||||||
nonPersistent map[uint64]string
|
|
||||||
databaseFiles []fileRec
|
|
||||||
}
|
|
||||||
|
|
||||||
// defaultExcludes returns the default excludes of the system goos
|
|
||||||
// names, as runtime.GOOS does: a fixed list on Linux, macOS and
|
|
||||||
// Windows, and none on any other system.
|
|
||||||
func defaultExcludes(goos string) []string {
|
|
||||||
switch goos {
|
|
||||||
case goosLinux:
|
|
||||||
return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
|
|
||||||
case goosDarwin:
|
|
||||||
// A walk of / reaches everything in /System/Volumes/Data through
|
|
||||||
// the firmlinks, so walking it as well would record each of
|
|
||||||
// those files twice.
|
|
||||||
return []string{
|
|
||||||
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
|
|
||||||
"/System/Volumes/VM", "/System/Volumes/Data",
|
|
||||||
}
|
|
||||||
case goosWindows:
|
|
||||||
// Relative to the root of the drive an operand lies on. The
|
|
||||||
// first three are the page, swap and hibernation files.
|
|
||||||
return []string{
|
|
||||||
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
|
|
||||||
"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
|
|
||||||
`Users\*\AppData\Local\Temp`,
|
|
||||||
}
|
|
||||||
default:
|
|
||||||
return nil
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// isDefaultExclude reports whether path is one of excludes, in which a
|
|
||||||
// * matches one path component.
|
|
||||||
func isDefaultExclude(path string, excludes []string) bool {
|
|
||||||
for _, pattern := range excludes {
|
|
||||||
// The patterns are fixed, so none is malformed.
|
|
||||||
if ok, _ := filepath.Match(pattern, path); ok {
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
// nonPersistentFilesystemTypes returns the filesystem types that are
|
|
||||||
// not persistent: filesystems that the kernel makes up or that are kept
|
|
||||||
// in memory. devfs is the macOS filesystem of /dev; the others are
|
|
||||||
// Linux names.
|
|
||||||
func nonPersistentFilesystemTypes() []string {
|
|
||||||
return []string{
|
|
||||||
"proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "ramfs",
|
|
||||||
"cgroup", "cgroup2", "debugfs", "tracefs", "securityfs", "pstore",
|
|
||||||
"bpf", "configfs", "fusectl", "mqueue", "hugetlbfs", "efivarfs",
|
|
||||||
"binfmt_misc", "autofs", "nsfs", "rpc_pipefs", "selinuxfs", "devfs",
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// nonPersistentLine returns the line the scan prints for path when fi,
|
|
||||||
// its lstat, puts it on a filesystem in rules.nonPersistent, and ""
|
|
||||||
// when it is on any other.
|
|
||||||
func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
|
|
||||||
dev, ok := deviceOfInfo(fi)
|
|
||||||
if !ok {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
fsType, found := rules.nonPersistent[dev]
|
|
||||||
if !found {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
|
|
||||||
path, fsType)
|
|
||||||
}
|
|
||||||
|
|
||||||
// readMountInfo reads mountinfo, laid out as mountInfoPath is, and
|
|
||||||
// returns the device ID of every mounted filesystem that is not
|
|
||||||
// persistent, with its type. The device ID is the one each line gives,
|
|
||||||
// not that of the file now at its mount point, so a mount that a later
|
|
||||||
// mount hides never adds the device of another filesystem.
|
|
||||||
func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
|
|
||||||
types := nonPersistentFilesystemTypes()
|
|
||||||
devices := make(map[uint64]string)
|
|
||||||
lines := bufio.NewScanner(mountinfo)
|
|
||||||
|
|
||||||
for lines.Scan() {
|
|
||||||
fields := strings.Fields(lines.Text())
|
|
||||||
|
|
||||||
sep := slices.Index(fields, "-")
|
|
||||||
if sep < optionalFieldsStart || sep+1 >= len(fields) {
|
|
||||||
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
|
|
||||||
}
|
|
||||||
|
|
||||||
var major, minor uint32
|
|
||||||
|
|
||||||
_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
|
|
||||||
}
|
|
||||||
|
|
||||||
fsType := fields[sep+1]
|
|
||||||
if slices.Contains(types, fsType) {
|
|
||||||
devices[unix.Mkdev(major, minor)] = fsType
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return devices, lines.Err()
|
|
||||||
}
|
|
||||||
|
|
||||||
// defaultExcludesHelp describes, for the scan command's help, the
|
|
||||||
// default excludes of the system goos names and, where the scan lists
|
|
||||||
// the mounted filesystems, the types it never records anything on.
|
|
||||||
func defaultExcludesHelp(goos string) string {
|
|
||||||
var help strings.Builder
|
|
||||||
|
|
||||||
dirs := defaultExcludes(goos)
|
|
||||||
if len(dirs) == 0 {
|
|
||||||
help.WriteString("This system has no default excludes.\n")
|
|
||||||
} else {
|
|
||||||
help.WriteString("Below an operand, the walk does not enter these " +
|
|
||||||
"default excludes, unless\n--no-default-excludes is given:\n\n")
|
|
||||||
|
|
||||||
for _, dir := range dirs {
|
|
||||||
fmt.Fprintf(&help, " %s\n", dir)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if goos == goosLinux || goos == goosDarwin {
|
|
||||||
help.WriteString("\nWith or without --no-default-excludes, the scan " +
|
|
||||||
"records nothing on a\nfilesystem that is not persistent, " +
|
|
||||||
"one of these types:\n\n")
|
|
||||||
|
|
||||||
types := nonPersistentFilesystemTypes()
|
|
||||||
for line := range slices.Chunk(types, typesPerHelpLine) {
|
|
||||||
fmt.Fprintf(&help, " %s\n", strings.Join(line, ", "))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return help.String()
|
|
||||||
}
|
|
||||||
@@ -1,52 +0,0 @@
|
|||||||
//go:build darwin
|
|
||||||
|
|
||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"slices"
|
|
||||||
|
|
||||||
"golang.org/x/sys/unix"
|
|
||||||
)
|
|
||||||
|
|
||||||
// nonPersistentFilesystems returns the device ID of every mounted
|
|
||||||
// filesystem that is not persistent, with its type, from the system's
|
|
||||||
// list of mounted filesystems. The device ID is read with lstat on the
|
|
||||||
// mount point. Where one filesystem is mounted over another on the same
|
|
||||||
// mount point, the later one in the list counts. A mount point that
|
|
||||||
// lstat cannot reach is left out.
|
|
||||||
func nonPersistentFilesystems() (map[uint64]string, error) {
|
|
||||||
// The first call counts the filesystems, the second lists them.
|
|
||||||
n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
list := make([]unix.Statfs_t, n)
|
|
||||||
|
|
||||||
n, err = unix.Getfsstat(list, unix.MNT_NOWAIT)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
mounts := make(map[string]string, n)
|
|
||||||
|
|
||||||
for _, st := range list[:n] {
|
|
||||||
mounts[unix.ByteSliceToString(st.Mntonname[:])] =
|
|
||||||
unix.ByteSliceToString(st.Fstypename[:])
|
|
||||||
}
|
|
||||||
|
|
||||||
types := nonPersistentFilesystemTypes()
|
|
||||||
devices := make(map[uint64]string)
|
|
||||||
|
|
||||||
for mountPoint, fsType := range mounts {
|
|
||||||
if !slices.Contains(types, fsType) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if dev, ok := deviceOfPath(mountPoint); ok {
|
|
||||||
devices[dev] = fsType
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return devices, nil
|
|
||||||
}
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
//go:build !darwin
|
|
||||||
|
|
||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"fmt"
|
|
||||||
"os"
|
|
||||||
"runtime"
|
|
||||||
)
|
|
||||||
|
|
||||||
// mountInfoPath is the file in which Linux lists the mounts the reading
|
|
||||||
// process sees.
|
|
||||||
const mountInfoPath = "/proc/self/mountinfo"
|
|
||||||
|
|
||||||
// nonPersistentFilesystems returns the device ID of every mounted
|
|
||||||
// filesystem that is not persistent, with its type: on Linux read from
|
|
||||||
// mountInfoPath (see readMountInfo), and on any other system none.
|
|
||||||
func nonPersistentFilesystems() (map[uint64]string, error) {
|
|
||||||
if runtime.GOOS != goosLinux {
|
|
||||||
return map[uint64]string{}, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
f, err := os.Open(mountInfoPath)
|
|
||||||
if err != nil {
|
|
||||||
return nil, err
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = f.Close() }()
|
|
||||||
|
|
||||||
devices, err := readMountInfo(f)
|
|
||||||
if err != nil {
|
|
||||||
return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
|
|
||||||
}
|
|
||||||
|
|
||||||
return devices, nil
|
|
||||||
}
|
|
||||||
-583
@@ -1,583 +0,0 @@
|
|||||||
package main
|
|
||||||
|
|
||||||
import (
|
|
||||||
"bytes"
|
|
||||||
"errors"
|
|
||||||
"maps"
|
|
||||||
"os"
|
|
||||||
"path/filepath"
|
|
||||||
"runtime"
|
|
||||||
"slices"
|
|
||||||
"strings"
|
|
||||||
"testing"
|
|
||||||
|
|
||||||
"golang.org/x/sys/unix"
|
|
||||||
)
|
|
||||||
|
|
||||||
func TestDefaultExcludes(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// README "Rules for the walk": each system's list.
|
|
||||||
cases := map[string][]string{
|
|
||||||
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
|
|
||||||
goosDarwin: {
|
|
||||||
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
|
|
||||||
"/System/Volumes/VM", "/System/Volumes/Data",
|
|
||||||
},
|
|
||||||
goosWindows: {
|
|
||||||
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
|
|
||||||
"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
|
|
||||||
`Users\*\AppData\Local\Temp`,
|
|
||||||
},
|
|
||||||
"freebsd": nil,
|
|
||||||
}
|
|
||||||
|
|
||||||
for goos, want := range cases {
|
|
||||||
if got := defaultExcludes(goos); !slices.Equal(got, want) {
|
|
||||||
t.Errorf("defaultExcludes(%q) = %q, want %q", goos, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestIsDefaultExclude(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// A * matches one path component, no more and no fewer.
|
|
||||||
excludes := []string{"/scratch", "/home/*/cache"}
|
|
||||||
|
|
||||||
cases := map[string]bool{
|
|
||||||
"/scratch": true,
|
|
||||||
"/scratch/x": false,
|
|
||||||
"/home/u/cache": true,
|
|
||||||
"/home/u/v/cache": false,
|
|
||||||
"/home/cache": false,
|
|
||||||
}
|
|
||||||
|
|
||||||
for path, want := range cases {
|
|
||||||
if got := isDefaultExclude(path, excludes); got != want {
|
|
||||||
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestReadMountInfo(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// The second-to-last line has no optional fields. The tmpfs on
|
|
||||||
// /data/a/b is hidden by the root filesystem's /srv, mounted over
|
|
||||||
// /data/a after it: the tmpfs still gives its own device, and the
|
|
||||||
// root filesystem's device is not listed.
|
|
||||||
mountinfo := strings.Join([]string{
|
|
||||||
"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
|
|
||||||
"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
|
|
||||||
"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
|
|
||||||
"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw",
|
|
||||||
"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw",
|
|
||||||
"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
|
|
||||||
}, "\n")
|
|
||||||
|
|
||||||
got, err := readMountInfo(strings.NewReader(mountinfo))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := map[uint64]string{
|
|
||||||
unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs",
|
|
||||||
unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs",
|
|
||||||
}
|
|
||||||
if !maps.Equal(got, want) {
|
|
||||||
t.Errorf("readMountInfo = %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, line := range []string{
|
|
||||||
"22 1 8:1 / / rw,relatime\n",
|
|
||||||
"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n",
|
|
||||||
} {
|
|
||||||
_, err = readMountInfo(strings.NewReader(line))
|
|
||||||
if !errors.Is(err, errMountInfoLine) {
|
|
||||||
t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
|
|
||||||
errMountInfoLine)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestNonPersistentFilesystems(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
if runtime.GOOS != goosLinux {
|
|
||||||
t.Skip("the test knows a mount only on Linux: /proc")
|
|
||||||
}
|
|
||||||
|
|
||||||
nonPersistent, err := nonPersistentFilesystems()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
dev, ok := deviceOfPath("/proc")
|
|
||||||
if !ok {
|
|
||||||
t.Fatal("no device for /proc")
|
|
||||||
}
|
|
||||||
|
|
||||||
if got := nonPersistent[dev]; got != "proc" {
|
|
||||||
t.Errorf("filesystem of /proc = %q, want %q", got, "proc")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDefaultExcludesHelp(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
for _, goos := range []string{goosLinux, goosDarwin, goosWindows} {
|
|
||||||
help := defaultExcludesHelp(goos)
|
|
||||||
|
|
||||||
for _, dir := range defaultExcludes(goos) {
|
|
||||||
if !strings.Contains(help, "\n "+dir+"\n") {
|
|
||||||
t.Errorf("help for %s does not list %s:\n%s", goos, dir, help)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Only Linux and macOS list their mounted filesystems.
|
|
||||||
for _, fsType := range nonPersistentFilesystemTypes() {
|
|
||||||
listed := strings.Contains(help, " "+fsType+",") ||
|
|
||||||
strings.Contains(help, " "+fsType+"\n")
|
|
||||||
if listed != (goos != goosWindows) {
|
|
||||||
t.Errorf("help for %s lists %s: %v", goos, fsType, listed)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
want := "This system has no default excludes.\n"
|
|
||||||
if got := defaultExcludesHelp("freebsd"); got != want {
|
|
||||||
t.Errorf("help for freebsd = %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
// scan's help shows those of the system it runs on.
|
|
||||||
var stdout, stderr bytes.Buffer
|
|
||||||
|
|
||||||
run([]string{cmdScan, "-h"}, &stdout, &stderr)
|
|
||||||
|
|
||||||
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
|
|
||||||
if !strings.Contains(stderr.String(), help) {
|
|
||||||
t.Errorf("scan --help = %q, want it to hold %q", stderr.String(), help)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestScanDoesNotEnterDefaultExclude checks that the walk does not
|
|
||||||
// enter a default exclude below an operand. It says so on a line of its
|
|
||||||
// own and does not count it as skipped. The records below it count as
|
|
||||||
// outside the operands: the record of a file gone from it is kept, and
|
|
||||||
// its size joins the size census, while a file added to it is not
|
|
||||||
// recorded.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
|
||||||
func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
|
|
||||||
stderr := captureStderr(t)
|
|
||||||
dir := t.TempDir()
|
|
||||||
db := openTestDB(t)
|
|
||||||
excluded := filepath.Join(dir, "tmp")
|
|
||||||
|
|
||||||
// A scan with no default excludes records the file in excluded.
|
|
||||||
gone := writeFile(t, excluded, "gone", pattern(1, 100))
|
|
||||||
syncTree(t, db, dir)
|
|
||||||
|
|
||||||
err := os.Remove(gone)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
unseen := writeFile(t, excluded, "new", pattern(2, 10))
|
|
||||||
|
|
||||||
// The same size as gone, so it is hashed only if the census holds the
|
|
||||||
// size of gone's record.
|
|
||||||
added := writeFile(t, dir, "added", pattern(3, 100))
|
|
||||||
|
|
||||||
st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
|
|
||||||
walkRules{excludes: []string{excluded}})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if st.skipped != 0 {
|
|
||||||
t.Errorf("skipped = %d, want 0", st.skipped)
|
|
||||||
}
|
|
||||||
|
|
||||||
recs := dbRecords(t, db)
|
|
||||||
recordByPath(t, recs, gone)
|
|
||||||
|
|
||||||
if recordByPath(t, recs, added).head == "" {
|
|
||||||
t.Errorf("%s not hashed: the census lacks the size of %s",
|
|
||||||
added, gone)
|
|
||||||
}
|
|
||||||
|
|
||||||
if slices.Contains(recordPaths(recs), unseen) {
|
|
||||||
t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
|
|
||||||
}
|
|
||||||
|
|
||||||
line := "\nscan: skipped " + excluded + " (default exclude)\n"
|
|
||||||
if !strings.Contains(stderr(), line) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr(), line)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestScanWalksOperandInDefaultExclude checks that an operand at or
|
|
||||||
// inside a default exclude is walked, also when it lies under another
|
|
||||||
// operand, whose walk does not enter the default exclude.
|
|
||||||
func TestScanWalksOperandInDefaultExclude(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
excluded := filepath.Join(dir, "tmp")
|
|
||||||
f := writeFile(t, excluded, "sub/f", pattern(1, 10))
|
|
||||||
sub := filepath.Dir(f)
|
|
||||||
|
|
||||||
cases := map[string][]string{
|
|
||||||
"at it": {excluded},
|
|
||||||
"inside it": {sub},
|
|
||||||
"inside it under an operand": {dir, sub},
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, roots := range cases {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
st, err := syncScan(t.Context(), db, roots, 1, false,
|
|
||||||
walkRules{excludes: []string{excluded}})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if st.added != 1 {
|
|
||||||
t.Errorf("added = %d, want %s recorded once", st.added, f)
|
|
||||||
}
|
|
||||||
|
|
||||||
recordByPath(t, dbRecords(t, db), f)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestScanNamesNoDefaultExcludeThatIsAnOperand checks that a default
|
|
||||||
// exclude given as an operand under another operand, which the walk of
|
|
||||||
// that other operand does not enter, gets no line saying it was
|
|
||||||
// skipped: it is walked as an operand of its own.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
|
||||||
func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
|
|
||||||
stderr := captureStderr(t)
|
|
||||||
dir := t.TempDir()
|
|
||||||
db := openTestDB(t)
|
|
||||||
excluded := filepath.Join(dir, "tmp")
|
|
||||||
f := writeFile(t, excluded, "f", pattern(1, 10))
|
|
||||||
|
|
||||||
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
|
|
||||||
walkRules{excludes: []string{excluded}})
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if st.added != 1 {
|
|
||||||
t.Errorf("added = %d, want %s recorded once", st.added, f)
|
|
||||||
}
|
|
||||||
|
|
||||||
recordByPath(t, dbRecords(t, db), f)
|
|
||||||
|
|
||||||
if strings.Contains(stderr(), "scan: skipped") {
|
|
||||||
t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanTempDirOperand scans a tree in the tests' temporary
|
|
||||||
// directory, which on Linux lies inside the default exclude /tmp unless
|
|
||||||
// TMPDIR names another place, with the system's own default excludes:
|
|
||||||
// the scan walks the whole tree and reports no default exclude.
|
|
||||||
func TestRunScanTempDirOperand(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
|
|
||||||
t.Skip(dir + " is not inside a default exclude")
|
|
||||||
}
|
|
||||||
|
|
||||||
f := writeFile(t, dir, "sub/f", pattern(1, 10))
|
|
||||||
|
|
||||||
stderr := scanOK(t, dir)
|
|
||||||
if strings.Contains(stderr, "scan: skipped") {
|
|
||||||
t.Errorf("stderr = %q, want no line saying a path was skipped",
|
|
||||||
stderr)
|
|
||||||
}
|
|
||||||
|
|
||||||
assertRecorded(t, path, f, true)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanNoDefaultExcludes scans /var through the command. Its
|
|
||||||
// /var/tmp, on a persistent filesystem, is a default exclude on Linux:
|
|
||||||
// without --no-default-excludes the walk does not enter it and says so,
|
|
||||||
// and with that flag the walk records a file the test puts there.
|
|
||||||
func TestRunScanNoDefaultExcludes(t *testing.T) {
|
|
||||||
if runtime.GOOS != goosLinux {
|
|
||||||
t.Skip("only Linux has /var/tmp among its default excludes")
|
|
||||||
}
|
|
||||||
|
|
||||||
nonPersistent, err := nonPersistentFilesystems()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
rules := walkRules{nonPersistent: nonPersistent}
|
|
||||||
|
|
||||||
for _, dir := range []string{"/var", "/var/tmp"} {
|
|
||||||
fi, lstatErr := os.Lstat(dir)
|
|
||||||
if lstatErr != nil || !fi.IsDir() ||
|
|
||||||
rules.nonPersistentLine(dir, fi) != "" {
|
|
||||||
t.Skip(dir + " is not a directory on a persistent filesystem here")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
//nolint:usetesting // the directory must be inside /var/tmp
|
|
||||||
tmp, err := os.MkdirTemp("/var/tmp", "sfdupes-test-")
|
|
||||||
if err != nil {
|
|
||||||
t.Skip("cannot write to /var/tmp here: " + err.Error())
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = os.RemoveAll(tmp) })
|
|
||||||
|
|
||||||
f := writeFile(t, tmp, "f", pattern(1, 10))
|
|
||||||
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
line := "scan: skipped /var/tmp (default exclude)\n"
|
|
||||||
|
|
||||||
if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
|
|
||||||
t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
|
|
||||||
}
|
|
||||||
|
|
||||||
assertRecorded(t, path, f, false)
|
|
||||||
|
|
||||||
stderr := scanOK(t, "--no-default-excludes", "/var")
|
|
||||||
if strings.Contains(stderr, line) {
|
|
||||||
t.Errorf("scan --no-default-excludes /var: stderr = %q, want no %q",
|
|
||||||
stderr, line)
|
|
||||||
}
|
|
||||||
|
|
||||||
assertRecorded(t, path, f, true)
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanSkipsNonPersistentOperand scans, through the command, a
|
|
||||||
// directory on a filesystem that is not persistent, with and without
|
|
||||||
// --no-default-excludes: either way it is skipped, with a line saying
|
|
||||||
// so, and nothing in it is recorded.
|
|
||||||
func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
|
|
||||||
shm, fsType, _ := shmTempDir(t)
|
|
||||||
writeFile(t, shm, "f", pattern(1, 10))
|
|
||||||
|
|
||||||
t.Setenv(databaseEnv, testDBPath(t))
|
|
||||||
|
|
||||||
line := "scan: skipped " + shm + " (" + fsType +
|
|
||||||
", not a persistent filesystem)\n"
|
|
||||||
summary := "scan: 0 files seen"
|
|
||||||
|
|
||||||
for _, flags := range [][]string{nil, {"--no-default-excludes"}} {
|
|
||||||
stderr := scanOK(t, append(flags, shm)...)
|
|
||||||
if !strings.Contains(stderr, line) ||
|
|
||||||
!strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("scan %q: stderr = %q, want %q and %q", flags, stderr,
|
|
||||||
line, summary)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem
|
|
||||||
// that is not persistent beside one on the tests' temporary directory.
|
|
||||||
// The records an earlier scan left in it count as outside the operands:
|
|
||||||
// the record of a file gone from it is kept, and its size joins the size
|
|
||||||
// census, while a file added to it is not recorded, and its size does
|
|
||||||
// not join the census.
|
|
||||||
//
|
|
||||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
|
||||||
func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
|
|
||||||
shm, fsType, rules := shmTempDir(t)
|
|
||||||
stderr := captureStderr(t)
|
|
||||||
dir := t.TempDir()
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
// A scan that, without the list of filesystems, records gone.
|
|
||||||
gone := writeFile(t, shm, "gone", pattern(1, 100))
|
|
||||||
syncTree(t, db, shm)
|
|
||||||
|
|
||||||
err := os.Remove(gone)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
unseen := writeFile(t, shm, "new", pattern(2, 10))
|
|
||||||
|
|
||||||
// added has the size of gone, and lonely that of unseen: each is
|
|
||||||
// hashed only if the census holds that size.
|
|
||||||
added := writeFile(t, dir, "added", pattern(3, 100))
|
|
||||||
lonely := writeFile(t, dir, "lonely", pattern(4, 10))
|
|
||||||
|
|
||||||
st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if st.skipped != 0 {
|
|
||||||
t.Errorf("skipped = %d, want 0", st.skipped)
|
|
||||||
}
|
|
||||||
|
|
||||||
recs := dbRecords(t, db)
|
|
||||||
recordByPath(t, recs, gone)
|
|
||||||
|
|
||||||
if recordByPath(t, recs, added).head == "" {
|
|
||||||
t.Errorf("%s not hashed: the census lacks the size of %s",
|
|
||||||
added, gone)
|
|
||||||
}
|
|
||||||
|
|
||||||
if recordByPath(t, recs, lonely).head != "" {
|
|
||||||
t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen)
|
|
||||||
}
|
|
||||||
|
|
||||||
if slices.Contains(recordPaths(recs), unseen) {
|
|
||||||
t.Errorf("%s recorded", unseen)
|
|
||||||
}
|
|
||||||
|
|
||||||
line := "scan: skipped " + shm + " (" + fsType +
|
|
||||||
", not a persistent filesystem)\n"
|
|
||||||
if !strings.Contains(stderr(), line) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr(), line)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestScanContentSkipsNonPersistent checks that the content phase does
|
|
||||||
// not read a file on a filesystem that is not persistent, though an
|
|
||||||
// earlier scan left a record for it: it is treated as a changed file,
|
|
||||||
// so its record is left as it is and its identical copy in another
|
|
||||||
// tree, having no other match, is not read either.
|
|
||||||
func TestScanContentSkipsNonPersistent(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
shm, _, rules := shmTempDir(t)
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
// A scan that, without the list of filesystems, records a with a
|
|
||||||
// head and tail but no content hash.
|
|
||||||
a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
|
|
||||||
|
|
||||||
syncTree(t, db, shm)
|
|
||||||
|
|
||||||
before := recordByPath(t, dbRecords(t, db), a)
|
|
||||||
if before.head == "" || before.content != "" {
|
|
||||||
t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
|
|
||||||
}
|
|
||||||
|
|
||||||
b := sparseFile(t, t.TempDir(), "b", headTailMin)
|
|
||||||
|
|
||||||
st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
|
|
||||||
rules)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if st.skipped != 0 {
|
|
||||||
t.Errorf("skipped = %d, want 0", st.skipped)
|
|
||||||
}
|
|
||||||
|
|
||||||
recs := dbRecords(t, db)
|
|
||||||
if r := recordByPath(t, recs, a); r != before {
|
|
||||||
t.Errorf("record = %+v, want it left as %+v", r, before)
|
|
||||||
}
|
|
||||||
|
|
||||||
if r := recordByPath(t, recs, b); r.content != "" {
|
|
||||||
t.Errorf("%s: content = %q, want none: its only match is on %s",
|
|
||||||
b, r.content, shm)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestWalkLeavesOutNonPersistent starts the walk itself in a directory
|
|
||||||
// on a filesystem that is not persistent, which a scan never does: it
|
|
||||||
// drops such an operand first. Below a directory on a persistent
|
|
||||||
// filesystem, one on another takes a mount, which a test cannot make.
|
|
||||||
// The walk enters no directory and records no file there, and names
|
|
||||||
// each.
|
|
||||||
func TestWalkLeavesOutNonPersistent(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
shm, fsType, rules := shmTempDir(t)
|
|
||||||
f := writeFile(t, shm, "f", pattern(1, 10))
|
|
||||||
sub := filepath.Dir(writeFile(t, shm, "sub/g", pattern(1, 10)))
|
|
||||||
|
|
||||||
var leftOut []string
|
|
||||||
|
|
||||||
for ev := range startWalk(t.Context(), []string{shm}, false, rules, 1) {
|
|
||||||
if ev.rec.path != "" {
|
|
||||||
t.Errorf("walk recorded %s", ev.rec.path)
|
|
||||||
}
|
|
||||||
|
|
||||||
if ev.leftOut == "" {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
leftOut = append(leftOut, ev.leftOut)
|
|
||||||
|
|
||||||
want := "scan: skipped " + ev.leftOut + " (" + fsType +
|
|
||||||
", not a persistent filesystem)"
|
|
||||||
if ev.warn != want {
|
|
||||||
t.Errorf("line = %q, want %q", ev.warn, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.Sort(leftOut)
|
|
||||||
|
|
||||||
if want := []string{f, sub}; !slices.Equal(leftOut, want) {
|
|
||||||
t.Errorf("walk left out %q, want %q", leftOut, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// shmTempDir returns a new directory in /dev/shm, removed when the test
|
|
||||||
// ends, the type of its filesystem, and the rules holding the real
|
|
||||||
// filesystems that are not persistent. It skips the test where /dev/shm
|
|
||||||
// is on a persistent filesystem, or cannot be written to.
|
|
||||||
func shmTempDir(t *testing.T) (string, string, walkRules) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
nonPersistent, err := nonPersistentFilesystems()
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
dev, ok := deviceOfPath("/dev/shm")
|
|
||||||
|
|
||||||
fsType, found := nonPersistent[dev]
|
|
||||||
if !ok || !found {
|
|
||||||
t.Skip("/dev/shm is not a memory-backed filesystem here")
|
|
||||||
}
|
|
||||||
|
|
||||||
//nolint:usetesting // the directory must be inside /dev/shm
|
|
||||||
dir, err := os.MkdirTemp("/dev/shm", "sfdupes-test-")
|
|
||||||
if err != nil {
|
|
||||||
t.Skip("cannot write to /dev/shm here: " + err.Error())
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = os.RemoveAll(dir) })
|
|
||||||
|
|
||||||
return dir, fsType, walkRules{nonPersistent: nonPersistent}
|
|
||||||
}
|
|
||||||
|
|
||||||
// assertRecorded checks that the database at dbPath holds a record for
|
|
||||||
// path when want is set, and none when it is not.
|
|
||||||
func assertRecorded(t *testing.T, dbPath, path string, want bool) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
db, err := openDB(dbPath, reportParams)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
defer func() { _ = db.Close() }()
|
|
||||||
|
|
||||||
if got := slices.Contains(recordPaths(dbRecords(t, db)), path); got != want {
|
|
||||||
t.Errorf("%s recorded: %v, want %v", path, got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
@@ -24,5 +24,3 @@ require (
|
|||||||
modernc.org/mathutil v1.7.1 // indirect
|
modernc.org/mathutil v1.7.1 // indirect
|
||||||
modernc.org/memory v1.11.0 // indirect
|
modernc.org/memory v1.11.0 // indirect
|
||||||
)
|
)
|
||||||
|
|
||||||
retract v0.0.1 // early build without a database; does not work as the README describes
|
|
||||||
|
|||||||
@@ -12,7 +12,7 @@
|
|||||||
//
|
//
|
||||||
// Usage:
|
// Usage:
|
||||||
//
|
//
|
||||||
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
// sfdupes scan [--workers N] [-x] PATH...
|
||||||
// sfdupes report > dupes.tsv
|
// sfdupes report > dupes.tsv
|
||||||
// sfdupes trees > dupetrees.tsv
|
// sfdupes trees > dupetrees.tsv
|
||||||
// sfdupes info
|
// sfdupes info
|
||||||
@@ -28,7 +28,6 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"runtime"
|
"runtime"
|
||||||
"runtime/debug"
|
|
||||||
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
)
|
)
|
||||||
@@ -59,32 +58,10 @@ var errNoSubcommand = errors.New("no subcommand")
|
|||||||
var errWorkersBelowOne = errors.New("--workers must be at least 1")
|
var errWorkersBelowOne = errors.New("--workers must be at least 1")
|
||||||
|
|
||||||
// Version is the build version, injected at link time via -ldflags
|
// Version is the build version, injected at link time via -ldflags
|
||||||
// (see the Makefile); empty when the linker set none, as for a binary
|
// (see the Makefile); "dev" for a plain go build.
|
||||||
// built by go install. buildVersion turns it into what --version prints.
|
|
||||||
//
|
//
|
||||||
//nolint:gochecknoglobals // written only by the linker
|
//nolint:gochecknoglobals // written only by the linker
|
||||||
var Version string
|
var Version = "dev"
|
||||||
|
|
||||||
// buildVersion returns the version that sfdupes --version prints
|
|
||||||
// (README "Install"): the version the linker set, when it set one;
|
|
||||||
// otherwise the main module's version that Go recorded in the binary,
|
|
||||||
// a tag such as v0.0.2 or the pseudo-version of an @main build; and
|
|
||||||
// "dev" when neither is known. info is what debug.ReadBuildInfo
|
|
||||||
// returns, nil when the binary carries none. Go records "(devel)" for
|
|
||||||
// a build that has no version, such as one inside a copy of the source
|
|
||||||
// without git.
|
|
||||||
func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
|
|
||||||
if linkerVersion != "" {
|
|
||||||
return linkerVersion
|
|
||||||
}
|
|
||||||
|
|
||||||
if info == nil || info.Main.Version == "" ||
|
|
||||||
info.Main.Version == "(devel)" {
|
|
||||||
return "dev"
|
|
||||||
}
|
|
||||||
|
|
||||||
return info.Main.Version
|
|
||||||
}
|
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
// Once the reader of a stdout pipe has gone, as in "sfdupes report |
|
// Once the reader of a stdout pipe has gone, as in "sfdupes report |
|
||||||
@@ -139,10 +116,7 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
|||||||
var showVersion bool
|
var showVersion bool
|
||||||
|
|
||||||
printVersion := runE(func(context.Context, []string) error {
|
printVersion := runE(func(context.Context, []string) error {
|
||||||
info, _ := debug.ReadBuildInfo()
|
_, err := fmt.Fprintf(stdout, "sfdupes %s\n", Version)
|
||||||
|
|
||||||
_, err := fmt.Fprintf(stdout, "sfdupes %s\n",
|
|
||||||
buildVersion(Version, info))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("write stdout: %w", err)
|
return fmt.Errorf("write stdout: %w", err)
|
||||||
}
|
}
|
||||||
@@ -208,23 +182,17 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
|||||||
return root
|
return root
|
||||||
}
|
}
|
||||||
|
|
||||||
// newScanCommand builds the scan subcommand, with its --workers, -x and
|
// newScanCommand builds the scan subcommand, with its --workers and -x
|
||||||
// --no-default-excludes flags. Its help lists the default excludes of
|
// flags.
|
||||||
// the system it runs on, and the filesystem types it never records
|
|
||||||
// anything on.
|
|
||||||
func newScanCommand() *cobra.Command {
|
func newScanCommand() *cobra.Command {
|
||||||
var (
|
var (
|
||||||
scanWorkers int
|
scanWorkers int
|
||||||
scanOneFS bool
|
scanOneFS bool
|
||||||
scanNoDefaultExcludes bool
|
|
||||||
)
|
)
|
||||||
|
|
||||||
short := "Walk trees and synchronize the scan database"
|
|
||||||
|
|
||||||
scanCmd := &cobra.Command{
|
scanCmd := &cobra.Command{
|
||||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
Use: cmdScan + " [--workers N] [-x] PATH...",
|
||||||
Short: short,
|
Short: "Walk trees and synchronize the scan database",
|
||||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
|
||||||
Args: cobra.MinimumNArgs(1),
|
Args: cobra.MinimumNArgs(1),
|
||||||
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
||||||
return checkScanWorkers(cmd, scanWorkers)
|
return checkScanWorkers(cmd, scanWorkers)
|
||||||
@@ -233,16 +201,13 @@ func newScanCommand() *cobra.Command {
|
|||||||
ctx, stop := interruptContext(ctx)
|
ctx, stop := interruptContext(ctx)
|
||||||
defer stop()
|
defer stop()
|
||||||
|
|
||||||
return runScan(ctx, args, scanWorkers, scanOneFS,
|
return runScan(ctx, args, scanWorkers, scanOneFS)
|
||||||
scanNoDefaultExcludes)
|
|
||||||
}),
|
}),
|
||||||
}
|
}
|
||||||
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
|
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
|
||||||
"concurrent workers for the walk, hash, and content phases")
|
"concurrent workers for the walk, hash, and content phases")
|
||||||
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
|
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
|
||||||
"do not cross filesystem boundaries")
|
"do not cross filesystem boundaries")
|
||||||
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
|
|
||||||
false, "enter the default excludes, the paths listed above")
|
|
||||||
|
|
||||||
return scanCmd
|
return scanCmd
|
||||||
}
|
}
|
||||||
|
|||||||
+2
-70
@@ -4,12 +4,10 @@ import (
|
|||||||
"bytes"
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
|
||||||
"io"
|
"io"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"runtime/debug"
|
|
||||||
"slices"
|
"slices"
|
||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -20,36 +18,6 @@ import (
|
|||||||
// error prints and a runtime failure must not.
|
// error prints and a runtime failure must not.
|
||||||
const usageMarker = "Usage:"
|
const usageMarker = "Usage:"
|
||||||
|
|
||||||
// TestMain stops the tests at once, saying why, when their temporary
|
|
||||||
// directory is not on a persistent filesystem: scan records nothing
|
|
||||||
// there (README "Rules for the walk"), so most scan tests would fail
|
|
||||||
// without saying why.
|
|
||||||
func TestMain(m *testing.M) {
|
|
||||||
nonPersistent, err := nonPersistentFilesystems()
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintf(os.Stderr, "list the mounted filesystems: %v\n", err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
dir := os.TempDir()
|
|
||||||
|
|
||||||
fi, err := os.Stat(dir)
|
|
||||||
if err != nil {
|
|
||||||
fmt.Fprintln(os.Stderr, err)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
dev, ok := deviceOfInfo(fi)
|
|
||||||
if fsType, found := nonPersistent[dev]; ok && found {
|
|
||||||
fmt.Fprintf(os.Stderr, "%s is on %s, not a persistent filesystem, "+
|
|
||||||
"and scan records nothing there: set TMPDIR to a directory on a "+
|
|
||||||
"persistent filesystem\n", dir, fsType)
|
|
||||||
os.Exit(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
os.Exit(m.Run())
|
|
||||||
}
|
|
||||||
|
|
||||||
// walSuffixes are the SQLite sidecar files a WAL-mode database keeps
|
// walSuffixes are the SQLite sidecar files a WAL-mode database keeps
|
||||||
// while it is open. A clean close checkpoints the WAL and removes
|
// while it is open. A clean close checkpoints the WAL and removes
|
||||||
// both; finding either afterwards means the database was never closed.
|
// both; finding either afterwards means the database was never closed.
|
||||||
@@ -567,7 +535,7 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
cmdScan: {
|
cmdScan: {
|
||||||
run: func() error {
|
run: func() error {
|
||||||
return runScan(t.Context(), []string{missing}, 1, false, false)
|
return runScan(t.Context(), []string{missing}, 1, false)
|
||||||
},
|
},
|
||||||
want: scanWant,
|
want: scanWant,
|
||||||
},
|
},
|
||||||
@@ -715,8 +683,6 @@ func TestRunVersion(t *testing.T) {
|
|||||||
|
|
||||||
// README §Subcommands: the version is one line on stdout, with
|
// README §Subcommands: the version is one line on stdout, with
|
||||||
// nothing on stderr, and exits 0.
|
// nothing on stderr, and exits 0.
|
||||||
info, _ := debug.ReadBuildInfo()
|
|
||||||
|
|
||||||
for _, arg := range []string{"--version", "-v"} {
|
for _, arg := range []string{"--version", "-v"} {
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
@@ -725,7 +691,7 @@ func TestRunVersion(t *testing.T) {
|
|||||||
t.Errorf("run(%s) = %d, want %d", arg, code, exitOK)
|
t.Errorf("run(%s) = %d, want %d", arg, code, exitOK)
|
||||||
}
|
}
|
||||||
|
|
||||||
want := "sfdupes " + buildVersion(Version, info) + "\n"
|
want := "sfdupes " + Version + "\n"
|
||||||
if got := stdout.String(); got != want {
|
if got := stdout.String(); got != want {
|
||||||
t.Errorf("run(%s) stdout = %q, want %q", arg, got, want)
|
t.Errorf("run(%s) stdout = %q, want %q", arg, got, want)
|
||||||
}
|
}
|
||||||
@@ -736,40 +702,6 @@ func TestRunVersion(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestBuildVersion(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// README "Install": the linker's version wins; otherwise the main
|
|
||||||
// module's version that Go recorded; "dev" when neither is known.
|
|
||||||
tagged := &debug.BuildInfo{Main: debug.Module{Version: "v0.0.2"}}
|
|
||||||
pseudo := &debug.BuildInfo{Main: debug.Module{
|
|
||||||
Version: "v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1",
|
|
||||||
}}
|
|
||||||
devel := &debug.BuildInfo{Main: debug.Module{Version: "(devel)"}}
|
|
||||||
|
|
||||||
cases := []struct {
|
|
||||||
name string
|
|
||||||
linker string
|
|
||||||
info *debug.BuildInfo
|
|
||||||
want string
|
|
||||||
}{
|
|
||||||
{"linker set", "v0.0.2-3-g4e4a94d", tagged, "v0.0.2-3-g4e4a94d"},
|
|
||||||
{"linker set, no build info", "4e4a94d", nil, "4e4a94d"},
|
|
||||||
{"build info with a tag", "", tagged, "v0.0.2"},
|
|
||||||
{"build info with a pseudo-version", "", pseudo,
|
|
||||||
"v1.0.0-rc2.0.20261016120000-4e4a94d0c2a1"},
|
|
||||||
{"(devel)", "", devel, "dev"},
|
|
||||||
{"no build info", "", nil, "dev"},
|
|
||||||
}
|
|
||||||
|
|
||||||
for _, c := range cases {
|
|
||||||
if got := buildVersion(c.linker, c.info); got != c.want {
|
|
||||||
t.Errorf("%s: buildVersion(%q, ...) = %q, want %q",
|
|
||||||
c.name, c.linker, got, c.want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRunVersionWriteFailureIsFatal(t *testing.T) {
|
func TestRunVersionWriteFailureIsFatal(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -13,7 +13,6 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"os/signal"
|
"os/signal"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"runtime"
|
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
@@ -115,13 +114,9 @@ func scanPhase(name string) string {
|
|||||||
// SIGTERM that interruptContext catches, the scan keeps what it has
|
// SIGTERM that interruptContext catches, the scan keeps what it has
|
||||||
// hashed (see syncScan), prints how many files its walk reached, and
|
// hashed (see syncScan), prints how many files its walk reached, and
|
||||||
// returns errInterrupted. workers must be at least 1; the scan command
|
// returns errInterrupted. workers must be at least 1; the scan command
|
||||||
// rejects anything less. Before the lock is taken, the scan lists the
|
// rejects anything less.
|
||||||
// filesystems that are not persistent, on which the walk records
|
|
||||||
// nothing; unless noDefaultExcludes is set, the walk does not enter the
|
|
||||||
// default excludes either. Once the database is open, the scan notes
|
|
||||||
// its files, which the walk passes over.
|
|
||||||
func runScan(ctx context.Context, roots []string, workers int,
|
func runScan(ctx context.Context, roots []string, workers int,
|
||||||
oneFS, noDefaultExcludes bool,
|
oneFS bool,
|
||||||
) error {
|
) error {
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -144,16 +139,6 @@ func runScan(ctx context.Context, roots []string, workers int,
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
nonPersistent, err := nonPersistentFilesystems()
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("list the mounted filesystems: %w", err)
|
|
||||||
}
|
|
||||||
|
|
||||||
rules := walkRules{nonPersistent: nonPersistent}
|
|
||||||
if !noDefaultExcludes {
|
|
||||||
rules.excludes = defaultExcludes(runtime.GOOS)
|
|
||||||
}
|
|
||||||
|
|
||||||
lock, err := lockScanDatabase(dbPath)
|
lock, err := lockScanDatabase(dbPath)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -174,12 +159,7 @@ func runScan(ctx context.Context, roots []string, workers int,
|
|||||||
|
|
||||||
defer closeScanDatabase(ctx, db, dbPath)
|
defer closeScanDatabase(ctx, db, dbPath)
|
||||||
|
|
||||||
rules.databaseFiles, err = databaseFiles(dbPath)
|
st, err := syncScan(ctx, db, roots, workers, oneFS)
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
|
|
||||||
st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
|
|
||||||
if errors.Is(err, context.Canceled) {
|
if errors.Is(err, context.Canceled) {
|
||||||
return interrupted(st.walked)
|
return interrupted(st.walked)
|
||||||
}
|
}
|
||||||
@@ -251,12 +231,14 @@ func resolveRoots(roots []string) ([]string, error) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// pruneRoots drops operands already covered by another operand:
|
// pruneRoots drops operands already covered by another operand:
|
||||||
// duplicates and any operand lying under another one. An operand under
|
// duplicates and any operand lying under another one. With -x, an
|
||||||
// another one is kept, though, when the walk of the deepest kept operand
|
// operand under another one is kept when the walk of the deepest kept
|
||||||
// above it does not reach it (see walkReaches). Every remaining file is
|
// operand above it cannot reach it: when device puts any directory from
|
||||||
// then reachable through exactly one root, so walked paths are unique
|
// it (from its directory, for a regular file) up to but not including
|
||||||
// without keeping a scan-wide set of every path seen.
|
// that operand on a different filesystem than that operand. Every
|
||||||
func pruneRoots(roots []string, oneFS bool, rules walkRules,
|
// remaining file is then reachable through exactly one root, so walked
|
||||||
|
// paths are unique without keeping a scan-wide set of every path seen.
|
||||||
|
func pruneRoots(roots []string, oneFS bool,
|
||||||
device func(path string) (uint64, bool),
|
device func(path string) (uint64, bool),
|
||||||
) []string {
|
) []string {
|
||||||
// Shorter-first ordering guarantees an ancestor is kept before any
|
// Shorter-first ordering guarantees an ancestor is kept before any
|
||||||
@@ -280,57 +262,40 @@ func pruneRoots(roots []string, oneFS bool, rules walkRules,
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
// A duplicate is dropped, and so is an operand the walk of above
|
// Without -x, and always for a duplicate, r is dropped.
|
||||||
// reaches.
|
if !oneFS || above == r {
|
||||||
if above != r && !walkReaches(above, r, oneFS, rules, device) {
|
continue
|
||||||
kept = append(kept, r)
|
}
|
||||||
|
|
||||||
|
aboveDev, aboveOK := device(above)
|
||||||
|
if !aboveOK {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// The walk of above enters a directory only when it is on
|
||||||
|
// above's filesystem, and then reaches every regular file in it
|
||||||
|
// whatever device the file reports, so for a regular-file
|
||||||
|
// operand the directories checked start at its directory.
|
||||||
|
dir := r
|
||||||
|
|
||||||
|
fi, err := os.Lstat(r)
|
||||||
|
if err == nil && fi.Mode().IsRegular() {
|
||||||
|
dir = filepath.Dir(r)
|
||||||
|
}
|
||||||
|
|
||||||
|
for ; dir != above; dir = filepath.Dir(dir) {
|
||||||
|
dev, ok := device(dir)
|
||||||
|
if ok && dev != aboveDev {
|
||||||
|
kept = append(kept, r)
|
||||||
|
|
||||||
|
break
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return kept
|
return kept
|
||||||
}
|
}
|
||||||
|
|
||||||
// walkReaches reports whether the walk of operand above reaches r, an
|
|
||||||
// operand under it. That walk does not enter a default exclude in
|
|
||||||
// rules, nor a directory that device puts on a filesystem in rules that
|
|
||||||
// is not persistent, nor, with -x, one that device puts on a different
|
|
||||||
// filesystem than above, when device knows above's. It reaches every
|
|
||||||
// regular file in a directory it enters, so for a regular-file operand
|
|
||||||
// the directories checked start at its directory; they end below above.
|
|
||||||
func walkReaches(above, r string, oneFS bool, rules walkRules,
|
|
||||||
device func(path string) (uint64, bool),
|
|
||||||
) bool {
|
|
||||||
aboveDev, aboveDevOK := device(above)
|
|
||||||
|
|
||||||
dir := r
|
|
||||||
|
|
||||||
fi, err := os.Lstat(r)
|
|
||||||
if err == nil && fi.Mode().IsRegular() {
|
|
||||||
dir = filepath.Dir(r)
|
|
||||||
}
|
|
||||||
|
|
||||||
for ; dir != above; dir = filepath.Dir(dir) {
|
|
||||||
if isDefaultExclude(dir, rules.excludes) {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
dev, ok := device(dir)
|
|
||||||
if !ok {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
if _, found := rules.nonPersistent[dev]; found {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
if oneFS && aboveDevOK && dev != aboveDev {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
// deviceOfPath returns the filesystem device ID of the file at path,
|
// deviceOfPath returns the filesystem device ID of the file at path,
|
||||||
// when lstat succeeds and the platform exposes one.
|
// when lstat succeeds and the platform exposes one.
|
||||||
func deviceOfPath(path string) (uint64, bool) {
|
func deviceOfPath(path string) (uint64, bool) {
|
||||||
@@ -372,11 +337,11 @@ type scanState struct {
|
|||||||
// the hashed records still waiting in the batch, starts no other write
|
// the hashed records still waiting in the batch, starts no other write
|
||||||
// or deletion, and returns the cancellation.
|
// or deletion, and returns the cancellation.
|
||||||
func syncScan(ctx context.Context, db *sql.DB, roots []string,
|
func syncScan(ctx context.Context, db *sql.DB, roots []string,
|
||||||
workers int, oneFS bool, rules walkRules,
|
workers int, oneFS bool,
|
||||||
) (scanStats, error) {
|
) (scanStats, error) {
|
||||||
s := &scanState{db: db}
|
s := &scanState{db: db}
|
||||||
|
|
||||||
err := s.runPhases(ctx, roots, workers, oneFS, rules)
|
err := s.runPhases(ctx, roots, workers, oneFS)
|
||||||
if err == nil || ctx.Err() == nil {
|
if err == nil || ctx.Err() == nil {
|
||||||
return s.st, err
|
return s.st, err
|
||||||
}
|
}
|
||||||
@@ -400,18 +365,15 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
|
|||||||
// and content (fill in the content hash of every record of headTailMin
|
// and content (fill in the content hash of every record of headTailMin
|
||||||
// or more whose size, head, and tail match another record's). Records
|
// or more whose size, head, and tail match another record's). Records
|
||||||
// outside the roots are never touched, except that the content phase
|
// outside the roots are never touched, except that the content phase
|
||||||
// fills in their content hash. Operands the walk cannot start from, or
|
// fills in their content hash. Operands the walk cannot start from are
|
||||||
// that are not on a persistent filesystem, are dropped first, so the
|
// dropped first, so the records beneath them are treated as if those
|
||||||
// records beneath them are treated as if those operands had not been
|
// operands had not been given.
|
||||||
// given. Below an operand, the walk follows rules.
|
|
||||||
func (s *scanState) runPhases(ctx context.Context, roots []string,
|
func (s *scanState) runPhases(ctx context.Context, roots []string,
|
||||||
workers int, oneFS bool, rules walkRules,
|
workers int, oneFS bool,
|
||||||
) error {
|
) error {
|
||||||
// Types and filesystems are checked before pruning so that an
|
// Types are checked before pruning so that an operand under a
|
||||||
// operand under a dropped one is still scanned, not dropped as lying
|
// dropped one is still scanned, not dropped as lying under it.
|
||||||
// under it.
|
roots = pruneRoots(s.walkableRoots(roots), oneFS, deviceOfPath)
|
||||||
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
|
|
||||||
deviceOfPath)
|
|
||||||
|
|
||||||
err := s.loadIndex(ctx, roots)
|
err := s.loadIndex(ctx, roots)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -419,7 +381,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
|||||||
}
|
}
|
||||||
|
|
||||||
changed, unhashed := s.walkPhase(
|
changed, unhashed := s.walkPhase(
|
||||||
startWalk(ctx, roots, oneFS, rules, workers), roots)
|
startWalk(ctx, roots, oneFS, workers), roots)
|
||||||
|
|
||||||
// A cancelled walk stops early, so its size census covers only part
|
// A cancelled walk stops early, so its size census covers only part
|
||||||
// of the roots, and every file it never reached would look vanished
|
// of the roots, and every file it never reached would look vanished
|
||||||
@@ -441,7 +403,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
return s.contentPhase(ctx, workers, rules)
|
return s.contentPhase(ctx, workers)
|
||||||
}
|
}
|
||||||
|
|
||||||
// walkableRoots returns the operands the walk can start from: regular
|
// walkableRoots returns the operands the walk can start from: regular
|
||||||
@@ -449,11 +411,9 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
|||||||
// about, counted as skipped, and dropped. A dropped operand is no
|
// about, counted as skipped, and dropped. A dropped operand is no
|
||||||
// longer a root, so the records stored beneath it are treated as if it
|
// longer a root, so the records stored beneath it are treated as if it
|
||||||
// had not been given: those under no other root count as outside the
|
// had not been given: those under no other root count as outside the
|
||||||
// roots and are not deleted as unverified. An operand on a filesystem
|
// roots and are not deleted as unverified. An operand that fails lstat
|
||||||
// in rules that is not persistent is dropped the same way, with a line
|
// here is kept, and the walk warns about it.
|
||||||
// saying so that is not a warning, and is not counted as skipped. An
|
func (s *scanState) walkableRoots(roots []string) []string {
|
||||||
// operand that fails lstat here is kept, and the walk warns about it.
|
|
||||||
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
|
||||||
kept := make([]string, 0, len(roots))
|
kept := make([]string, 0, len(roots))
|
||||||
|
|
||||||
for _, root := range roots {
|
for _, root := range roots {
|
||||||
@@ -467,13 +427,6 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
|||||||
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
line := rules.nonPersistentLine(root, fi)
|
|
||||||
if line != "" {
|
|
||||||
fmt.Fprintln(os.Stderr, escapePath(line))
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
kept = append(kept, root)
|
kept = append(kept, root)
|
||||||
@@ -528,12 +481,10 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
|||||||
// (the content phase may still fill in its content hash). It returns
|
// (the content phase may still fill in its content hash). It returns
|
||||||
// the new-or-changed files and the unchanged files whose records lack
|
// the new-or-changed files and the unchanged files whose records lack
|
||||||
// hashes; both remain candidates until the census decides whether
|
// hashes; both remain candidates until the census decides whether
|
||||||
// their sizes are shared. A record at or below a path the walk left
|
// their sizes are shared. A record below a directory -x did not enter
|
||||||
// out, because -x stopped at it, it is a default exclude, or it is not
|
// counts as outside the roots: it is not deleted, and its size joins
|
||||||
// on a persistent filesystem, counts as outside the roots: it is not
|
// the census. That does not hold below a root at or inside that
|
||||||
// deleted, and its size joins the census. That does not hold below a
|
// directory, since that root's own walk verifies the records below it.
|
||||||
// root at or inside that path, since that root's own walk verifies the
|
|
||||||
// records below it; nor is a line printed for such a root.
|
|
||||||
func (s *scanState) walkPhase(
|
func (s *scanState) walkPhase(
|
||||||
events <-chan walkEvent, roots []string,
|
events <-chan walkEvent, roots []string,
|
||||||
) ([]fileRec, []fileRec) {
|
) ([]fileRec, []fileRec) {
|
||||||
@@ -551,25 +502,12 @@ func (s *scanState) walkPhase(
|
|||||||
|
|
||||||
var (
|
var (
|
||||||
changed, unhashed []fileRec
|
changed, unhashed []fileRec
|
||||||
leftOut []string
|
otherFSDirs []string
|
||||||
)
|
)
|
||||||
|
|
||||||
for ev := range events {
|
for ev := range events {
|
||||||
if ev.otherFSDir != "" {
|
if ev.otherFSDir != "" {
|
||||||
leftOut = append(leftOut, ev.otherFSDir)
|
otherFSDirs = append(otherFSDirs, ev.otherFSDir)
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
// The line saying so is printed as a warning is, but what the
|
|
||||||
// walk leaves out is not counted as skipped. A default exclude
|
|
||||||
// that is itself a root is walked as one, so it gets no line.
|
|
||||||
if ev.leftOut != "" {
|
|
||||||
leftOut = append(leftOut, ev.leftOut)
|
|
||||||
|
|
||||||
if !slices.Contains(roots, ev.leftOut) {
|
|
||||||
prog.warnf("%s", ev.warn)
|
|
||||||
}
|
|
||||||
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -604,13 +542,12 @@ func (s *scanState) walkPhase(
|
|||||||
prog.finish()
|
prog.finish()
|
||||||
|
|
||||||
for path, old := range s.existing {
|
for path, old := range s.existing {
|
||||||
// Kept only when a path the walk left out lies below the deepest
|
// Kept only when a directory -x did not enter lies below the
|
||||||
// root above the record, or is the record's own. A root at or
|
// deepest root above the record. A root at or below that
|
||||||
// below that path was walked on its own and did not find the
|
// directory was walked on its own and did not find the file.
|
||||||
// file. Both contain the record, so the longer path is the
|
// Both contain the record, so the longer path is the deeper.
|
||||||
// deeper.
|
otherFSDir := deepestRoot(path, otherFSDirs)
|
||||||
left := deepestRoot(path, leftOut)
|
if len(otherFSDir) > len(deepestRoot(path, roots)) {
|
||||||
if len(left) > len(deepestRoot(path, roots)) {
|
|
||||||
delete(s.existing, path)
|
delete(s.existing, path)
|
||||||
|
|
||||||
s.sizes = append(s.sizes, old.size)
|
s.sizes = append(s.sizes, old.size)
|
||||||
@@ -716,15 +653,6 @@ func sameInode(a, b fileRec) bool {
|
|||||||
return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
|
return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
|
||||||
}
|
}
|
||||||
|
|
||||||
// isDatabaseFile reports whether f, a file the walk reached, is one of
|
|
||||||
// the scan's own database files in rules, which the walk passes over
|
|
||||||
// without a line and without counting it.
|
|
||||||
func isDatabaseFile(f fileRec, rules walkRules) bool {
|
|
||||||
return slices.ContainsFunc(rules.databaseFiles, func(d fileRec) bool {
|
|
||||||
return sameInode(d, f)
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// hashPhase hashes every queued file with hashSignature — the head and
|
// hashPhase hashes every queued file with hashSignature — the head and
|
||||||
// tail of a file of headTailMin or more, the whole file below that —
|
// tail of a file of headTailMin or more, the whole file below that —
|
||||||
// committing completed records to the database in batches as results
|
// committing completed records to the database in batches as results
|
||||||
@@ -893,12 +821,9 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
|||||||
// hash phase's worker pool and their records written back in batches. A
|
// hash phase's worker pool and their records written back in batches. A
|
||||||
// failed read is warned about and counted as skipped; the record keeps
|
// failed read is warned about and counted as skipped; the record keeps
|
||||||
// its empty content, so it is never grouped, and a later scan tries
|
// its empty content, so it is never grouped, and a later scan tries
|
||||||
// again. No file on a filesystem in rules that is not persistent is
|
// again.
|
||||||
// read.
|
func (s *scanState) contentPhase(ctx context.Context, workers int) error {
|
||||||
func (s *scanState) contentPhase(ctx context.Context, workers int,
|
toRead, recs, err := s.contentCandidates(ctx)
|
||||||
rules walkRules,
|
|
||||||
) error {
|
|
||||||
toRead, recs, err := s.contentCandidates(ctx, rules)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -933,19 +858,18 @@ func (s *scanState) contentPhase(ctx context.Context, workers int,
|
|||||||
// contentCandidates returns the files the content phase reads, and
|
// contentCandidates returns the files the content phase reads, and
|
||||||
// their records by path. Every record contentCandidatesSQL returns has
|
// their records by path. Every record contentCandidatesSQL returns has
|
||||||
// its file checked with lstat, whether or not it already has a content
|
// its file checked with lstat, whether or not it already has a content
|
||||||
// hash: a file that is gone, is no longer a regular file, has changed
|
// hash: a file that is gone, is no longer a regular file, or has
|
||||||
// by the walk's rule, or is on a filesystem in rules that is not
|
// changed by the walk's rule keeps its record as it is and does not
|
||||||
// persistent keeps its record as it is and does not count as a match
|
// count as a match for the others, and any other lstat error is warned
|
||||||
// for the others, and any other lstat error is warned about and counted
|
// about and counted as skipped, with the same result. If such a record
|
||||||
// as skipped, with the same result. If such a record has no content
|
// has no content hash, it stays out of duplicate groups; if it has one,
|
||||||
// hash, it stays out of duplicate groups; if it has one, it is still
|
// it is still reported until a scan covering its own tree updates or
|
||||||
// reported until a scan covering its own tree updates or removes it.
|
// removes it. The files of a group that pass and have no content hash
|
||||||
// The files of a group that pass and have no content hash are read only
|
// are read only if at least minGroupSize of the group's files pass, so
|
||||||
// if at least minGroupSize of the group's files pass, so a group whose
|
// a group whose other members are all stale costs no reads. Only the
|
||||||
// other members are all stale costs no reads. Only the records to be
|
// records to be read are kept.
|
||||||
// read are kept.
|
|
||||||
func (s *scanState) contentCandidates(
|
func (s *scanState) contentCandidates(
|
||||||
ctx context.Context, rules walkRules,
|
ctx context.Context,
|
||||||
) ([]fileRec, map[string]scanRec, error) {
|
) ([]fileRec, map[string]scanRec, error) {
|
||||||
// The search and the checks take real time on a large database;
|
// The search and the checks take real time on a large database;
|
||||||
// without a display the scan looks hung before the reads begin.
|
// without a display the scan looks hung before the reads begin.
|
||||||
@@ -1003,7 +927,7 @@ func (s *scanState) contentCandidates(
|
|||||||
first = r
|
first = r
|
||||||
}
|
}
|
||||||
|
|
||||||
f, ok, err := unchangedFile(r, rules)
|
f, ok, err := unchangedFile(r)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
s.st.skipped++
|
s.st.skipped++
|
||||||
|
|
||||||
@@ -1037,10 +961,9 @@ func (s *scanState) contentCandidates(
|
|||||||
|
|
||||||
// unchangedFile lstats the file r names and returns it for reading if
|
// unchangedFile lstats the file r names and returns it for reading if
|
||||||
// it is still the regular file r records: the same size, and an mtime
|
// it is still the regular file r records: the same size, and an mtime
|
||||||
// no newer than recorded (the walk's change rule). A file that is gone,
|
// no newer than recorded (the walk's change rule). A file that is gone
|
||||||
// has changed, or is on a filesystem in rules that is not persistent
|
// or has changed reports false; any other lstat error is returned.
|
||||||
// reports false; any other lstat error is returned.
|
func unchangedFile(r scanRec) (fileRec, bool, error) {
|
||||||
func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
|
|
||||||
fi, err := os.Lstat(r.path)
|
fi, err := os.Lstat(r.path)
|
||||||
if errors.Is(err, fs.ErrNotExist) {
|
if errors.Is(err, fs.ErrNotExist) {
|
||||||
return fileRec{}, false, nil
|
return fileRec{}, false, nil
|
||||||
@@ -1057,12 +980,6 @@ func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
|
|||||||
|
|
||||||
dev, ino := inodeOfInfo(fi)
|
dev, ino := inodeOfInfo(fi)
|
||||||
|
|
||||||
// The scan reads nothing on a filesystem that is not persistent,
|
|
||||||
// whatever record an earlier scan left for the file.
|
|
||||||
if _, found := rules.nonPersistent[dev]; found {
|
|
||||||
return fileRec{}, false, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
return fileRec{
|
return fileRec{
|
||||||
path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
|
path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
|
||||||
}, true, nil
|
}, true, nil
|
||||||
@@ -1129,36 +1046,32 @@ type dirJob struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// walkEvent is one walk result delivered to the walk phase: a regular
|
// walkEvent is one walk result delivered to the walk phase: a regular
|
||||||
// file's statted record, a warning when fail is set, or a path the walk
|
// file's statted record, a warning when fail is set, or, when
|
||||||
// left out: when otherFSDir is set, a directory -x did not enter
|
// otherFSDir is set, a directory -x did not enter because it is on
|
||||||
// because it is on another filesystem than its operand, and when
|
// another filesystem than its operand.
|
||||||
// leftOut is set, a default exclude or a directory or file on a
|
|
||||||
// filesystem that is not persistent, with warn holding the line that
|
|
||||||
// says so.
|
|
||||||
type walkEvent struct {
|
type walkEvent struct {
|
||||||
rec fileRec
|
rec fileRec
|
||||||
warn string
|
warn string
|
||||||
fail bool
|
fail bool
|
||||||
otherFSDir string
|
otherFSDir string
|
||||||
leftOut string
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// startWalk seeds every root into the shared walk worker pool and
|
// startWalk seeds every root into the shared walk worker pool and
|
||||||
// returns the event stream: one record per regular file, one warning
|
// returns the event stream: one record per regular file, one warning
|
||||||
// event per per-path error, and one event per path -x or rules kept the
|
// event per per-path error, and one event per directory -x did not
|
||||||
// walk from entering or recording. The channel is closed when the walk
|
// enter. The channel is closed when the walk completes, and also when
|
||||||
// completes, and also when ctx is cancelled — every goroutine in the
|
// ctx is cancelled — every goroutine in the pool abandons its blocking
|
||||||
// pool abandons its blocking send in that case, so the consumer sees a
|
// send in that case, so the consumer sees a truncated but properly
|
||||||
// truncated but properly terminated stream instead of a stalled one.
|
// terminated stream instead of a stalled one.
|
||||||
func startWalk(ctx context.Context, roots []string, oneFS bool,
|
func startWalk(ctx context.Context, roots []string, oneFS bool,
|
||||||
rules walkRules, workers int,
|
workers int,
|
||||||
) <-chan walkEvent {
|
) <-chan walkEvent {
|
||||||
jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS, rules)
|
jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS)
|
||||||
|
|
||||||
go func() {
|
go func() {
|
||||||
initial := make([]dirJob, 0, len(roots))
|
initial := make([]dirJob, 0, len(roots))
|
||||||
for _, root := range roots {
|
for _, root := range roots {
|
||||||
initial = append(initial, seedRoot(ctx, root, rules, events)...)
|
initial = append(initial, seedRoot(ctx, root, events)...)
|
||||||
}
|
}
|
||||||
|
|
||||||
dispatchDirs(ctx, initial, jobs, subdirs)
|
dispatchDirs(ctx, initial, jobs, subdirs)
|
||||||
@@ -1212,14 +1125,13 @@ func operandWarning(root string, fi fs.FileInfo) string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// seedRoot turns one PATH operand into the walk's starting state: a
|
// seedRoot turns one PATH operand into the walk's starting state: a
|
||||||
// regular-file operand is statted and emitted directly, unless it is
|
// regular-file operand is statted and emitted directly, and a directory
|
||||||
// one of the scan's own database files in rules, and a directory
|
|
||||||
// operand becomes an initial job. walkableRoots has already dropped
|
// operand becomes an initial job. walkableRoots has already dropped
|
||||||
// every other operand. One that has changed into something else since
|
// every other operand. One that has changed into something else since
|
||||||
// is warned about and skipped here; it is still a root, so the records
|
// is warned about and skipped here; it is still a root, so the records
|
||||||
// stored beneath it, other than those under another root, are deleted
|
// stored beneath it, other than those under another root, are deleted
|
||||||
// as unverified.
|
// as unverified.
|
||||||
func seedRoot(ctx context.Context, root string, rules walkRules,
|
func seedRoot(ctx context.Context, root string,
|
||||||
events chan<- walkEvent,
|
events chan<- walkEvent,
|
||||||
) []dirJob {
|
) []dirJob {
|
||||||
fi, err := os.Lstat(root)
|
fi, err := os.Lstat(root)
|
||||||
@@ -1246,17 +1158,14 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
|
|||||||
}
|
}
|
||||||
|
|
||||||
dev, ino := inodeOfInfo(fi)
|
dev, ino := inodeOfInfo(fi)
|
||||||
rec := fileRec{
|
|
||||||
|
sendEvent(ctx, events, walkEvent{rec: fileRec{
|
||||||
path: root,
|
path: root,
|
||||||
size: fi.Size(),
|
size: fi.Size(),
|
||||||
mtime: fi.ModTime(),
|
mtime: fi.ModTime(),
|
||||||
dev: dev,
|
dev: dev,
|
||||||
ino: ino,
|
ino: ino,
|
||||||
}
|
}})
|
||||||
|
|
||||||
if !isDatabaseFile(rec, rules) {
|
|
||||||
sendEvent(ctx, events, walkEvent{rec: rec})
|
|
||||||
}
|
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -1268,7 +1177,7 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
|
|||||||
// workers drop the directories still queued rather than stop reading
|
// workers drop the directories still queued rather than stop reading
|
||||||
// jobs, so the range always runs out and the pool always tears down.
|
// jobs, so the range always runs out and the pool always tears down.
|
||||||
func startWalkWorkers(ctx context.Context, workers int,
|
func startWalkWorkers(ctx context.Context, workers int,
|
||||||
oneFS bool, rules walkRules,
|
oneFS bool,
|
||||||
) (chan dirJob, chan []dirJob, chan walkEvent) {
|
) (chan dirJob, chan []dirJob, chan walkEvent) {
|
||||||
jobs := make(chan dirJob, workQueueDepth)
|
jobs := make(chan dirJob, workQueueDepth)
|
||||||
subdirs := make(chan []dirJob, workers)
|
subdirs := make(chan []dirJob, workers)
|
||||||
@@ -1284,7 +1193,7 @@ func startWalkWorkers(ctx context.Context, workers int,
|
|||||||
}
|
}
|
||||||
|
|
||||||
select {
|
select {
|
||||||
case subdirs <- walkOneDir(ctx, job, oneFS, rules, events):
|
case subdirs <- walkOneDir(ctx, job, oneFS, events):
|
||||||
case <-ctx.Done():
|
case <-ctx.Done():
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -1342,7 +1251,7 @@ func dispatchDirs(ctx context.Context, initial []dirJob,
|
|||||||
// entry and a warning event per unreadable one, and returns the
|
// entry and a warning event per unreadable one, and returns the
|
||||||
// subdirectories to descend into.
|
// subdirectories to descend into.
|
||||||
func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
||||||
rules walkRules, events chan<- walkEvent,
|
events chan<- walkEvent,
|
||||||
) []dirJob {
|
) []dirJob {
|
||||||
entries, err := os.ReadDir(job.path)
|
entries, err := os.ReadDir(job.path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1366,8 +1275,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
|||||||
p := filepath.Join(job.path, e.Name())
|
p := filepath.Join(job.path, e.Name())
|
||||||
|
|
||||||
if e.IsDir() {
|
if e.IsDir() {
|
||||||
sub, ok := subdirJob(ctx, p, e, job, oneFS, rules, events)
|
if sub, ok := subdirJob(ctx, p, e, job, oneFS, events); ok {
|
||||||
if ok {
|
|
||||||
subs = append(subs, sub)
|
subs = append(subs, sub)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1379,7 +1287,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
emitFile(ctx, p, e, rules, events)
|
emitFile(ctx, p, e, events)
|
||||||
}
|
}
|
||||||
|
|
||||||
return subs
|
return subs
|
||||||
@@ -1389,12 +1297,9 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
|||||||
// The lstat happens here in the walk worker, while the directory's
|
// The lstat happens here in the walk worker, while the directory's
|
||||||
// metadata is still hot; a path that fails to stat (or stops being a
|
// metadata is still hot; a path that fails to stat (or stops being a
|
||||||
// regular file) between the directory read and the lstat is warned
|
// regular file) between the directory read and the lstat is warned
|
||||||
// about and skipped. A file on a filesystem in rules that is not
|
// about and skipped.
|
||||||
// persistent, as through a bind mount of one file, is not recorded: it
|
|
||||||
// gets an event naming it instead. One of the scan's own database files
|
|
||||||
// in rules gets no event at all.
|
|
||||||
func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||||
rules walkRules, events chan<- walkEvent,
|
events chan<- walkEvent,
|
||||||
) {
|
) {
|
||||||
info, err := e.Info()
|
info, err := e.Info()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -1410,43 +1315,35 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
line := rules.nonPersistentLine(p, info)
|
|
||||||
if line != "" {
|
|
||||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
|
||||||
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
dev, ino := inodeOfInfo(info)
|
dev, ino := inodeOfInfo(info)
|
||||||
rec := fileRec{
|
|
||||||
|
sendEvent(ctx, events, walkEvent{rec: fileRec{
|
||||||
path: p,
|
path: p,
|
||||||
size: info.Size(),
|
size: info.Size(),
|
||||||
mtime: info.ModTime(),
|
mtime: info.ModTime(),
|
||||||
dev: dev,
|
dev: dev,
|
||||||
ino: ino,
|
ino: ino,
|
||||||
}
|
}})
|
||||||
|
|
||||||
if !isDatabaseFile(rec, rules) {
|
|
||||||
sendEvent(ctx, events, walkEvent{rec: rec})
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// subdirJob applies the descent rules to directory p: never enter
|
// subdirJob applies the descent rules to directory p: never enter
|
||||||
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
|
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
|
||||||
// snapshot), never enter a directory on a filesystem in rules that is
|
// snapshot), and with -x never enter a directory on a different
|
||||||
// not persistent, nor a default exclude in rules, and with -x never
|
// filesystem than its operand, sending an event that names it so the
|
||||||
// enter a directory on a different filesystem than its operand. For the
|
// scan keeps the records below it, except those below an operand at or
|
||||||
// last three it sends an event that names the directory, so the scan
|
|
||||||
// keeps the records below it, except those below an operand at or
|
|
||||||
// inside it.
|
// inside it.
|
||||||
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||||
parent dirJob, oneFS bool, rules walkRules,
|
parent dirJob, oneFS bool, events chan<- walkEvent,
|
||||||
events chan<- walkEvent,
|
|
||||||
) (dirJob, bool) {
|
) (dirJob, bool) {
|
||||||
if e.Name() == ".zfs" {
|
if e.Name() == ".zfs" {
|
||||||
return dirJob{}, false
|
return dirJob{}, false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
|
||||||
|
if !oneFS || !parent.rootDevOK {
|
||||||
|
return job, true
|
||||||
|
}
|
||||||
|
|
||||||
info, err := e.Info()
|
info, err := e.Info()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
sendEvent(ctx, events, walkEvent{
|
sendEvent(ctx, events, walkEvent{
|
||||||
@@ -1457,27 +1354,6 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
return dirJob{}, false
|
return dirJob{}, false
|
||||||
}
|
}
|
||||||
|
|
||||||
line := rules.nonPersistentLine(p, info)
|
|
||||||
if line != "" {
|
|
||||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
|
||||||
|
|
||||||
return dirJob{}, false
|
|
||||||
}
|
|
||||||
|
|
||||||
if isDefaultExclude(p, rules.excludes) {
|
|
||||||
sendEvent(ctx, events, walkEvent{
|
|
||||||
leftOut: p,
|
|
||||||
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
|
|
||||||
})
|
|
||||||
|
|
||||||
return dirJob{}, false
|
|
||||||
}
|
|
||||||
|
|
||||||
job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
|
|
||||||
if !oneFS || !parent.rootDevOK {
|
|
||||||
return job, true
|
|
||||||
}
|
|
||||||
|
|
||||||
if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
|
if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
|
||||||
sendEvent(ctx, events, walkEvent{otherFSDir: p})
|
sendEvent(ctx, events, walkEvent{otherFSDir: p})
|
||||||
|
|
||||||
|
|||||||
+9
-126
@@ -782,8 +782,7 @@ func collectWalk(t *testing.T, roots []string, oneFS bool,
|
|||||||
errs int
|
errs int
|
||||||
)
|
)
|
||||||
|
|
||||||
for ev := range startWalk(t.Context(), roots, oneFS, walkRules{},
|
for ev := range startWalk(t.Context(), roots, oneFS, workers) {
|
||||||
workers) {
|
|
||||||
if ev.fail {
|
if ev.fail {
|
||||||
errs++
|
errs++
|
||||||
|
|
||||||
@@ -1005,8 +1004,7 @@ func callSubdirJob(t *testing.T, p string, e fs.DirEntry,
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
events := make(chan walkEvent, 1)
|
events := make(chan walkEvent, 1)
|
||||||
job, ok := subdirJob(t.Context(), p, e, parent, oneFS, walkRules{},
|
job, ok := subdirJob(t.Context(), p, e, parent, oneFS, events)
|
||||||
events)
|
|
||||||
close(events)
|
close(events)
|
||||||
|
|
||||||
var evs []walkEvent
|
var evs []walkEvent
|
||||||
@@ -1154,7 +1152,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
|
|||||||
return dev, ok
|
return dev, ok
|
||||||
}
|
}
|
||||||
|
|
||||||
roots = pruneRoots(roots, true, walkRules{}, device)
|
roots = pruneRoots(roots, true, device)
|
||||||
|
|
||||||
ctx := t.Context()
|
ctx := t.Context()
|
||||||
s := &scanState{db: db}
|
s := &scanState{db: db}
|
||||||
@@ -1164,14 +1162,14 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
jobs, subdirs, events := startWalkWorkers(ctx, 2, true, walkRules{})
|
jobs, subdirs, events := startWalkWorkers(ctx, 2, true)
|
||||||
|
|
||||||
// seedRoot emits a regular-file operand itself and returns a job
|
// seedRoot emits a regular-file operand itself and returns a job
|
||||||
// only for a directory operand.
|
// only for a directory operand.
|
||||||
var initial []dirJob
|
var initial []dirJob
|
||||||
|
|
||||||
for _, root := range roots {
|
for _, root := range roots {
|
||||||
for _, job := range seedRoot(ctx, root, walkRules{}, events) {
|
for _, job := range seedRoot(ctx, root, events) {
|
||||||
job.rootDev = devices[root]
|
job.rootDev = devices[root]
|
||||||
initial = append(initial, job)
|
initial = append(initial, job)
|
||||||
}
|
}
|
||||||
@@ -1465,7 +1463,7 @@ const smokeTreeFiles = 15
|
|||||||
func syncTree(t *testing.T, db *sql.DB, roots ...string) scanStats {
|
func syncTree(t *testing.T, db *sql.DB, roots ...string) scanStats {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
st, err := syncScan(t.Context(), db, roots, 4, false, walkRules{})
|
st, err := syncScan(t.Context(), db, roots, 4, false)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("syncScan: %v", err)
|
t.Fatalf("syncScan: %v", err)
|
||||||
}
|
}
|
||||||
@@ -2414,7 +2412,7 @@ func TestPruneRoots(t *testing.T) {
|
|||||||
// Duplicates and operands under other operands are dropped; /cc is
|
// Duplicates and operands under other operands are dropped; /cc is
|
||||||
// not under /c (sibling with a shared prefix).
|
// not under /c (sibling with a shared prefix).
|
||||||
got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
|
got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
|
||||||
false, walkRules{}, device)
|
false, device)
|
||||||
|
|
||||||
want := []string{"/a", "/c", "/cc"}
|
want := []string{"/a", "/c", "/cc"}
|
||||||
if !slices.Equal(got, want) {
|
if !slices.Equal(got, want) {
|
||||||
@@ -2423,7 +2421,7 @@ func TestPruneRoots(t *testing.T) {
|
|||||||
|
|
||||||
nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
|
nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
|
||||||
|
|
||||||
got = pruneRoots(nested, false, walkRules{}, device)
|
got = pruneRoots(nested, false, device)
|
||||||
|
|
||||||
want = []string{"/p"}
|
want = []string{"/p"}
|
||||||
if !slices.Equal(got, want) {
|
if !slices.Equal(got, want) {
|
||||||
@@ -2431,31 +2429,12 @@ func TestPruneRoots(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// With -x, /p/q, on another filesystem than /p, is kept as well.
|
// With -x, /p/q, on another filesystem than /p, is kept as well.
|
||||||
got = pruneRoots(nested, true, walkRules{}, device)
|
got = pruneRoots(nested, true, device)
|
||||||
|
|
||||||
want = []string{"/p", pqRoot}
|
want = []string{"/p", pqRoot}
|
||||||
if !slices.Equal(got, want) {
|
if !slices.Equal(got, want) {
|
||||||
t.Fatalf("pruneRoots with -x = %q, want %q", got, want)
|
t.Fatalf("pruneRoots with -x = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
// The walk of /p does not enter /p/q when it is a default exclude, so
|
|
||||||
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
|
|
||||||
got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
|
|
||||||
device)
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Nor does it enter /p/q when the filesystem of /p/q is not
|
|
||||||
// persistent, so /p/q/r/m, of no known device, is kept.
|
|
||||||
got = pruneRoots([]string{"/p", "/p/q/r/m"}, false,
|
|
||||||
walkRules{nonPersistent: map[uint64]string{2: "tmpfs"}}, device)
|
|
||||||
|
|
||||||
want = []string{"/p", "/p/q/r/m"}
|
|
||||||
if !slices.Equal(got, want) {
|
|
||||||
t.Fatalf("pruneRoots with %s not persistent = %q, want %q",
|
|
||||||
pqRoot, got, want)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReportsNeverTouchFilesystem(t *testing.T) {
|
func TestReportsNeverTouchFilesystem(t *testing.T) {
|
||||||
@@ -2501,99 +2480,3 @@ func TestUnderRoot(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
|
|
||||||
// database, reached directly and through a symlinked directory. The
|
|
||||||
// walk passes over the database file, its -wal and -shm files and its
|
|
||||||
// .lock file with no line and no count, and the record an earlier scan
|
|
||||||
// left for the .lock file is deleted. A rescan of the unchanged tree
|
|
||||||
// updates nothing, and report lists none of the database's files.
|
|
||||||
func TestRunScanPassesOverDatabase(t *testing.T) {
|
|
||||||
for _, viaSymlink := range []bool{false, true} {
|
|
||||||
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
|
|
||||||
base := t.TempDir()
|
|
||||||
tree := filepath.Join(base, "real", "tree")
|
|
||||||
operand := tree
|
|
||||||
|
|
||||||
writeFile(t, tree, "a", pattern(1, 100))
|
|
||||||
writeFile(t, tree, "b", pattern(1, 100))
|
|
||||||
writeFile(t, tree, "empty1", nil)
|
|
||||||
writeFile(t, tree, "empty2", nil)
|
|
||||||
|
|
||||||
if viaSymlink {
|
|
||||||
err := os.Symlink(filepath.Join(base, "real"),
|
|
||||||
filepath.Join(base, "link"))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
operand = filepath.Join(base, "link", "tree")
|
|
||||||
}
|
|
||||||
|
|
||||||
dbPath := filepath.Join(tree, "db.sqlite")
|
|
||||||
t.Setenv(databaseEnv, dbPath)
|
|
||||||
|
|
||||||
db, err := openScanDatabase(t.Context(), dbPath)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
err = applyChanges(t.Context(), db, []scanRec{
|
|
||||||
{path: filepath.Join(operand, "db.sqlite.lock")},
|
|
||||||
}, nil, nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
closeScanDatabase(t.Context(), db, dbPath)
|
|
||||||
|
|
||||||
assertScanSummary(t,
|
|
||||||
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
|
|
||||||
operand)
|
|
||||||
assertScanSummary(t,
|
|
||||||
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
|
|
||||||
operand)
|
|
||||||
|
|
||||||
report := runStdout(t, cmdReport)
|
|
||||||
if strings.Contains(report, "db.sqlite") ||
|
|
||||||
!strings.Contains(report, filepath.Join(operand, "empty2")) {
|
|
||||||
t.Errorf("report = %q, want the empty files and none of "+
|
|
||||||
"the database's files", report)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanPassesOverDatabaseOperands scans the database file and its
|
|
||||||
// .lock file given as operands themselves: neither is recorded or
|
|
||||||
// counted.
|
|
||||||
func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
|
|
||||||
dbPath := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, dbPath)
|
|
||||||
|
|
||||||
// The first scan creates both files.
|
|
||||||
scanOK(t, t.TempDir())
|
|
||||||
|
|
||||||
assertScanSummary(t,
|
|
||||||
"0 files seen (0 added, 0 updated, 0 removed, 0 unchanged)",
|
|
||||||
dbPath, dbPath+".lock")
|
|
||||||
}
|
|
||||||
|
|
||||||
// assertScanSummary scans operands and fails the test unless the
|
|
||||||
// summary line gives counts with nothing skipped, and the line naming
|
|
||||||
// the database is the only one that names one of its files.
|
|
||||||
func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
stderr := scanOK(t, operands...)
|
|
||||||
|
|
||||||
want := "scan: " + counts + ", 0 skipped\n"
|
|
||||||
if !strings.Contains(stderr, want) {
|
|
||||||
t.Errorf("scan %q: stderr = %q, want %q", operands, stderr, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
if strings.Count(stderr, "db.sqlite") != 1 {
|
|
||||||
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
|
|
||||||
"database to name one of its files", operands, stderr)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|||||||
Reference in New Issue
Block a user