Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dc82531aec | ||
|
|
db56fd0e4c | ||
|
|
6caaff935a |
No files matched your search
@@ -22,9 +22,8 @@ records the files it finds in an SQLite database. SQLite keeps a whole database
|
|||||||
in one file. The database is kept between runs. A rescan reads only new and
|
in one file. The database is kept between runs. A rescan reads only new and
|
||||||
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
||||||
system's job scheduler. `report`, `trees` and `info` work from the database and
|
system's job scheduler. `report`, `trees` and `info` work from the database and
|
||||||
can run at any time. `report` and `trees` leave out a copy that was deleted
|
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
|
||||||
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first
|
first look at a new machine with one command.
|
||||||
look at a new machine with one command.
|
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
@@ -255,15 +254,14 @@ In order:
|
|||||||
- `report` prints the file-level duplicate report from the database.
|
- `report` prints the file-level duplicate report from the database.
|
||||||
- `trees` prints the duplicate-tree report from the database.
|
- `trees` prints the duplicate-tree report from the database.
|
||||||
- `info` prints the database's details.
|
- `info` prints the database's details.
|
||||||
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
|
- `dwim` scans `/`, then prints the duplicate-tree report.
|
||||||
report.
|
|
||||||
|
|
||||||
```
|
```
|
||||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
|
||||||
sfdupes report > dupes.tsv
|
sfdupes report > dupes.tsv
|
||||||
sfdupes trees > dupetrees.tsv
|
sfdupes trees > dupetrees.tsv
|
||||||
sfdupes info
|
sfdupes info
|
||||||
sfdupes dwim
|
sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]...
|
||||||
sfdupes --version
|
sfdupes --version
|
||||||
sfdupes [command] --help
|
sfdupes [command] --help
|
||||||
```
|
```
|
||||||
@@ -293,22 +291,34 @@ The reference below is also the program's specification.
|
|||||||
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
|
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
|
||||||
it. The path names the file exactly, whatever characters it holds. A relative
|
it. The path names the file exactly, whatever characters it holds. A relative
|
||||||
path is relative to the working directory.
|
path is relative to the working directory.
|
||||||
- `--workers N` (`scan`) sets how many directories `scan` reads at once while it
|
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once
|
||||||
lists the trees. It also sets how many files `scan` reads at once while it
|
while it lists the trees. It also sets how many files `scan` reads at once
|
||||||
hashes them. The default is the number of CPUs. `N` must be at least 1. A
|
while it hashes them. The default is the number of CPUs. `N` must be at
|
||||||
smaller value is a usage error, reported in one line on stderr with exit 2
|
least 1. A smaller value is a usage error, reported in one line on stderr with
|
||||||
before anything is scanned.
|
exit 2 before anything is scanned.
|
||||||
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
|
- `-x`, `--one-file-system` (`scan`, `dwim`) keeps the walk of each operand out
|
||||||
directories on other filesystems, as the same flag does for `du` and `rsync`.
|
of directories on other filesystems, as the same flag does for `du` and
|
||||||
By default the walk also enters directories on other filesystems.
|
`rsync`. By default the walk also enters directories on other filesystems.
|
||||||
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
|
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
|
||||||
Those are the system and temporary directories the walk otherwise leaves out,
|
excludes. Those are the system and temporary directories the walk otherwise
|
||||||
such as `/var/tmp` on Linux (see "Rules for the walk").
|
leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||||
|
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
|
||||||
|
below it. `PATH` may be a directory or a file. The flag may be given more than
|
||||||
|
once. The walk knows `PATH` by its device ID and inode number, as it knows the
|
||||||
|
database files (see "Rules for the walk"). So the walk leaves `PATH` out
|
||||||
|
whatever path leads to it. That includes a hard link to a file given as
|
||||||
|
`PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in
|
||||||
|
`--exclude ""`, is a usage error, reported in one line on stderr with exit 2
|
||||||
|
before anything is scanned. A `PATH` that does not exist, or whose real path
|
||||||
|
`scan` cannot find, is a fatal error. So is an operand at or below one (see
|
||||||
|
"Operands").
|
||||||
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
|
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
|
||||||
exits 0. It writes nothing to stderr.
|
exits 0. It writes nothing to stderr.
|
||||||
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
|
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
|
||||||
and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the
|
and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
|
||||||
default excludes of the system it runs on, and the filesystem types the walk
|
names the database. The help of `report`, `trees`, `info` and `dwim` says what
|
||||||
|
each prints. `sfdupes scan --help` and `sfdupes dwim --help` also list the
|
||||||
|
default excludes of the system they run on, and the filesystem types the walk
|
||||||
never records anything on.
|
never records anything on.
|
||||||
|
|
||||||
### Duplicate detection
|
### Duplicate detection
|
||||||
@@ -362,6 +372,10 @@ default. An operand may be a directory or a regular file.
|
|||||||
- An operand that does not exist is a fatal error, and so is a symlink whose
|
- An operand that does not exist is a fatal error, and so is a symlink whose
|
||||||
target does not exist. `scan` then stops before it walks anything, and exits
|
target does not exist. `scan` then stops before it walks anything, and exits
|
||||||
with code 1.
|
with code 1.
|
||||||
|
- An operand at or below an `--exclude` path is a fatal error too. The message
|
||||||
|
names the operand and the `--exclude` path. `scan` compares the operand and
|
||||||
|
each directory above it with the `--exclude` paths by device ID and inode
|
||||||
|
number, not by name.
|
||||||
- Each operand is resolved to its real path before the walk, as `realpath`
|
- Each operand is resolved to its real path before the walk, as `realpath`
|
||||||
prints it. A relative operand is joined to the working directory as `pwd -P`
|
prints it. A relative operand is joined to the working directory as `pwd -P`
|
||||||
prints it. Every symlink in the path is then followed. A `..` after a symlink
|
prints it. Every symlink in the path is then followed. A `..` after a symlink
|
||||||
@@ -437,15 +451,16 @@ operands:
|
|||||||
scanned on different schedules into one database.
|
scanned on different schedules into one database.
|
||||||
- The scan keeps every record at or below a path that the walk leaves out for
|
- The scan keeps every record at or below a path that the walk leaves out for
|
||||||
one of these reasons:
|
one of these reasons:
|
||||||
|
- it is an `--exclude` path;
|
||||||
- it is a default exclude;
|
- it is a default exclude;
|
||||||
- it is not on a persistent filesystem;
|
- it is not on a persistent filesystem;
|
||||||
- with `-x`, it is a directory on another filesystem than its operand.
|
- with `-x`, it is a directory on another filesystem than its operand.
|
||||||
|
|
||||||
Such records count as outside the scanned operands. So with `-x`, a
|
Such records count as outside the scanned operands. So with `-x`, a
|
||||||
filesystem mounted under an operand can be scanned on its own schedule into
|
filesystem mounted under an operand can be scanned on its own schedule into
|
||||||
the same database. An operand may itself lie at or inside such a path, as
|
the same database. An operand may itself lie at or inside such a path, other
|
||||||
"Operands" describes. Its walk verifies the records below it, so they are
|
than an `--exclude` path (see "Operands"). Its walk verifies the records
|
||||||
deleted when their files are gone.
|
below it, so they are deleted when their files are gone.
|
||||||
|
|
||||||
#### Phases
|
#### Phases
|
||||||
|
|
||||||
@@ -544,6 +559,17 @@ walk practical on large or busy storage pools.
|
|||||||
The path is not counted as skipped. The records below it are kept, as
|
The path is not counted as skipped. The records below it are kept, as
|
||||||
described under "What a scan does to the database".
|
described under "What a scan does to the database".
|
||||||
|
|
||||||
|
- The walk never enters or records an `--exclude` path, nor anything below it.
|
||||||
|
For each one it reaches, `scan` prints a line on stderr naming it:
|
||||||
|
|
||||||
|
```
|
||||||
|
scan: skipped /srv/cache (--exclude)
|
||||||
|
```
|
||||||
|
|
||||||
|
The path is not counted as skipped. The scan keeps the records at and below
|
||||||
|
it, as described under "What a scan does to the database".
|
||||||
|
`--no-default-excludes` does not change this rule.
|
||||||
|
|
||||||
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
||||||
default excludes hold temporary files and the kernel's view of the running
|
default excludes hold temporary files and the kernel's view of the running
|
||||||
system. They depend on the system, and `sfdupes scan --help` lists those of
|
system. They depend on the system, and `sfdupes scan --help` lists those of
|
||||||
@@ -618,7 +644,8 @@ from a thread of its own and never blocks the workers.
|
|||||||
it repeats another or lies under another;
|
it repeats another or lies under another;
|
||||||
- the progress display of each phase (see "Progress"). `scan` prints each
|
- the progress display of each phase (see "Progress"). `scan` prints each
|
||||||
warning for a skipped path as it happens. It prints a line the same way for
|
warning for a skipped path as it happens. It prints a line the same way for
|
||||||
each path left out as a default exclude or as not on a persistent filesystem;
|
each path left out as an `--exclude` path, as a default exclude or as not on a
|
||||||
|
persistent filesystem;
|
||||||
- the summary line;
|
- the summary line;
|
||||||
- a warning when a report still has the database open as the scan ends (see
|
- a warning when a report still has the database open as the scan ends (see
|
||||||
"Concurrent reads").
|
"Concurrent reads").
|
||||||
@@ -830,14 +857,25 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
|
|||||||
### `dwim` mode
|
### `dwim` mode
|
||||||
|
|
||||||
`dwim`, short for "do what I mean", scans a new machine and reports its
|
`dwim`, short for "do what I mean", scans a new machine and reports its
|
||||||
duplicate trees in one command. It takes no positional arguments and no flags of
|
duplicate trees in one command. It takes no positional arguments. Like `scan`,
|
||||||
its own. Like `scan`, it should run as root, so that it can read every file.
|
it should run as root, so that it can read every file.
|
||||||
|
|
||||||
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
|
`dwim` takes these flags of `scan`, with the same meaning (see "Settings"):
|
||||||
excludes and crosses into other filesystems. `--workers` keeps its default. The
|
|
||||||
|
- `--workers N`;
|
||||||
|
- `-x`, `--one-file-system`;
|
||||||
|
- `--no-default-excludes`;
|
||||||
|
- `--exclude PATH`, which may be given more than once.
|
||||||
|
|
||||||
|
`dwim` runs what `sfdupes scan /` runs with the same flags. With no flags, the
|
||||||
|
walk skips the default excludes and crosses into other filesystems. The
|
||||||
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
|
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
|
||||||
what `sfdupes trees` runs. The largest trees come first.
|
what `sfdupes trees` runs. The largest trees come first.
|
||||||
|
|
||||||
|
```
|
||||||
|
sfdupes dwim -x --exclude /srv/cache
|
||||||
|
```
|
||||||
|
|
||||||
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
|
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
|
||||||
`scan` prints, then what `trees` prints, then one last line:
|
`scan` prints, then what `trees` prints, then one last line:
|
||||||
|
|
||||||
@@ -1199,6 +1237,9 @@ showing the count it stopped at.
|
|||||||
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
|
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
|
||||||
stopped by either during its scan. A fatal error is one of these:
|
stopped by either during its scan. A fatal error is one of these:
|
||||||
- an operand that does not exist, or whose real path `scan` cannot find;
|
- an operand that does not exist, or whose real path `scan` cannot find;
|
||||||
|
- an `--exclude` path that does not exist, or whose real path `scan` cannot
|
||||||
|
find;
|
||||||
|
- an operand at or below an `--exclude` path;
|
||||||
- a list of mounted filesystems that `scan` cannot read;
|
- a list of mounted filesystems that `scan` cannot read;
|
||||||
- another `scan` holding the lock (see "The scan lock");
|
- another `scan` holding the lock (see "The scan lock");
|
||||||
- a database, its directory or its lock file that cannot be created or
|
- a database, its directory or its lock file that cannot be created or
|
||||||
@@ -1218,6 +1259,7 @@ showing the count it stopped at.
|
|||||||
`--workers abc`;
|
`--workers abc`;
|
||||||
- `scan` with no operand;
|
- `scan` with no operand;
|
||||||
- `--workers` below 1;
|
- `--workers` below 1;
|
||||||
|
- an empty `--exclude` value, as in `--exclude ""`;
|
||||||
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
||||||
|
|
||||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||||
|
|||||||
@@ -29,6 +29,20 @@
|
|||||||
Newest first. An entry ends with its date and its issue or branch, where those
|
Newest first. An entry ends with its date and its issue or branch, where those
|
||||||
are known.
|
are known.
|
||||||
|
|
||||||
|
- `dwim` takes the `scan` flags `--workers`, `-x`, `--no-default-excludes` and
|
||||||
|
`--exclude`, with the same meaning. Both commands register and check them with
|
||||||
|
the same code (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/162)
|
||||||
|
|
||||||
|
- The help of `sfdupes` and of `scan` names the database, its default path and
|
||||||
|
`SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
|
||||||
|
order of the rows and the escapes. The help of `info` lists its lines. The
|
||||||
|
help of `dwim` says that it scans `/`, should run as root and keeps the scan
|
||||||
|
in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
|
||||||
|
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
|
||||||
|
enters nor records an excluded path, and keeps the records below it. An
|
||||||
|
operand at or below one is a fatal error (2026-11-11,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/163)
|
||||||
|
|
||||||
- `report` and `trees` leave out a copy that was deleted after the scan. Each
|
- `report` and `trees` leave out a copy that was deleted after the scan. Each
|
||||||
checks only the copy itself with `lstat`: the file, or the directory at the
|
checks only the copy itself with `lstat`: the file, or the directory at the
|
||||||
top of a tree. A group left with one copy is not reported. A missing `scan`
|
top of a tree. A group left with one copy is not reported. A missing `scan`
|
||||||
|
|||||||
@@ -358,8 +358,9 @@ func TestRunScanInterrupted(t *testing.T) {
|
|||||||
stderr := captureStderr(t)
|
stderr := captureStderr(t)
|
||||||
dir := buildSmokeTree(t)
|
dir := buildSmokeTree(t)
|
||||||
|
|
||||||
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
opts := scanOptions{workers: walkCancelWorkers}
|
||||||
false)
|
|
||||||
|
err := runScan(t.Context(), []string{dir}, opts)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -382,8 +383,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
|||||||
go func() {
|
go func() {
|
||||||
defer close(done)
|
defer close(done)
|
||||||
|
|
||||||
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
|
err = runScan(cancelledContext(t), []string{dir}, opts)
|
||||||
false, false)
|
|
||||||
}()
|
}()
|
||||||
|
|
||||||
awaitReturn(t, done, "runScan")
|
awaitReturn(t, done, "runScan")
|
||||||
@@ -410,8 +410,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
|||||||
got, before)
|
got, before)
|
||||||
}
|
}
|
||||||
|
|
||||||
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
err = runScan(t.Context(), []string{dir}, opts)
|
||||||
false)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -437,7 +436,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
|
|||||||
dir := buildWalkCancelTree(t)
|
dir := buildWalkCancelTree(t)
|
||||||
|
|
||||||
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
|
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
|
||||||
walkCancelWorkers, false, false)
|
scanOptions{workers: walkCancelWorkers})
|
||||||
if !errors.Is(err, errInterrupted) {
|
if !errors.Is(err, errInterrupted) {
|
||||||
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
|
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
|
||||||
err, errInterrupted)
|
err, errInterrupted)
|
||||||
|
|||||||
+168
-37
@@ -10,6 +10,7 @@ import (
|
|||||||
"os"
|
"os"
|
||||||
"runtime"
|
"runtime"
|
||||||
"runtime/debug"
|
"runtime/debug"
|
||||||
|
"slices"
|
||||||
|
|
||||||
"github.com/spf13/cobra"
|
"github.com/spf13/cobra"
|
||||||
)
|
)
|
||||||
@@ -31,15 +32,82 @@ const (
|
|||||||
cmdDwim = "dwim"
|
cmdDwim = "dwim"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// databaseHelp is the paragraph of the root and scan help that names
|
||||||
|
// the database (README "Location").
|
||||||
|
const databaseHelp = "scan and dwim store what they find in a database " +
|
||||||
|
"file. report, trees\nand info read it. The file is $" + databaseEnv +
|
||||||
|
" when that is set and\nnot empty, and otherwise " +
|
||||||
|
defaultDatabasePath + ", where creating it\nneeds root.\n"
|
||||||
|
|
||||||
|
// escapesHelp is the paragraph of the report and trees help on how
|
||||||
|
// paths are escaped (README "Report output format").
|
||||||
|
const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
|
||||||
|
|
||||||
|
\\ a backslash
|
||||||
|
\t a tab
|
||||||
|
\n a newline
|
||||||
|
\r a carriage return
|
||||||
|
`
|
||||||
|
|
||||||
|
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
|
||||||
|
// summary in the help of their subcommands.
|
||||||
|
const (
|
||||||
|
reportHelp = `The report is tab-separated text on stdout: a header line, then one
|
||||||
|
row per duplicate file. The columns are:
|
||||||
|
|
||||||
|
first the path that sorts first, byte by byte, in its group of copies
|
||||||
|
dupe another copy of that file
|
||||||
|
size the file's size in bytes
|
||||||
|
|
||||||
|
Rows are ordered by size, largest first, then by path.
|
||||||
|
|
||||||
|
` + escapesHelp
|
||||||
|
|
||||||
|
treesHelp = `The report is tab-separated text on stdout: a header line, then one
|
||||||
|
row per duplicate directory tree. The columns are:
|
||||||
|
|
||||||
|
first the path that sorts first, byte by byte, in its group of copies
|
||||||
|
dupe another copy of that tree
|
||||||
|
files the number of regular files in one copy
|
||||||
|
size their total size in bytes
|
||||||
|
disk_size the bytes the dupe copy takes on disk
|
||||||
|
|
||||||
|
Rows are ordered by size, largest first, then by path.
|
||||||
|
|
||||||
|
` + escapesHelp
|
||||||
|
|
||||||
|
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
|
||||||
|
value, separated by a tab:
|
||||||
|
|
||||||
|
database the path, and where it came from
|
||||||
|
database file size in bytes, then rounded, as 2147483648 (2.0 GiB)
|
||||||
|
-wal file size in bytes, then rounded; only if a -wal file exists
|
||||||
|
modified when the database file last changed, in UTC
|
||||||
|
schema version the version of the database's layout
|
||||||
|
records the number of records
|
||||||
|
size of recorded files their total size, in bytes, then rounded
|
||||||
|
`
|
||||||
|
|
||||||
|
dwimHelp = `dwim runs what "sfdupes scan /" runs with the same flags, then what
|
||||||
|
"sfdupes trees" runs. Run it as root, so that the scan can read every
|
||||||
|
file. The scan is stored in the database, as scan stores it, so
|
||||||
|
"sfdupes trees" prints the report again later.
|
||||||
|
`
|
||||||
|
)
|
||||||
|
|
||||||
// errNoSubcommand is returned by the root command when it is invoked
|
// errNoSubcommand is returned by the root command when it is invoked
|
||||||
// without a subcommand. That is a usage error, and the usage text
|
// without a subcommand. That is a usage error, and the usage text
|
||||||
// cobra prints for it is the whole message.
|
// cobra prints for it is the whole message.
|
||||||
var errNoSubcommand = errors.New("no subcommand")
|
var errNoSubcommand = errors.New("no subcommand")
|
||||||
|
|
||||||
// errWorkersBelowOne is the usage error for a scan --workers value
|
// errWorkersBelowOne is the usage error for a --workers value below 1.
|
||||||
// below 1.
|
|
||||||
var errWorkersBelowOne = errors.New("--workers must be at least 1")
|
var errWorkersBelowOne = errors.New("--workers must be at least 1")
|
||||||
|
|
||||||
|
// errEmptyExcludeValue is the usage error for an empty --exclude value,
|
||||||
|
// as from --exclude "$DIR" with DIR unset. Taken as a path, it
|
||||||
|
// would name the working directory.
|
||||||
|
var errEmptyExcludeValue = errors.New("--exclude must not be empty")
|
||||||
|
|
||||||
// Version is the build version, injected at link time via -ldflags
|
// Version is the build version, injected at link time via -ldflags
|
||||||
// (see the Makefile); empty when the linker set none, as for a binary
|
// (see the Makefile); empty when the linker set none, as for a binary
|
||||||
// built by go install. buildVersion turns it into what --version prints.
|
// built by go install. buildVersion turns it into what --version prints.
|
||||||
@@ -187,65 +255,116 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
|||||||
}),
|
}),
|
||||||
}
|
}
|
||||||
|
|
||||||
dwimCmd := &cobra.Command{
|
root.Long = root.Short + ".\n\n" + databaseHelp
|
||||||
Use: cmdDwim,
|
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
|
||||||
Short: "Scan / with the default excludes, then print the duplicate-tree report",
|
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
|
||||||
Args: cobra.NoArgs,
|
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
|
||||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
|
||||||
return runDwim(ctx, "/", stdout)
|
|
||||||
}),
|
|
||||||
}
|
|
||||||
|
|
||||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
|
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd,
|
||||||
|
newDwimCommand(stdout))
|
||||||
|
|
||||||
return root
|
return root
|
||||||
}
|
}
|
||||||
|
|
||||||
// newScanCommand builds the scan subcommand, with its --workers, -x and
|
// scanOptions holds the values of the flags that scan and dwim share:
|
||||||
// --no-default-excludes flags. Its help lists the default excludes of
|
// --workers, -x, --no-default-excludes and --exclude.
|
||||||
// the system it runs on, and the filesystem types it never records
|
type scanOptions struct {
|
||||||
// anything on.
|
workers int
|
||||||
|
oneFS bool
|
||||||
|
noDefaultExcludes bool
|
||||||
|
excludePaths []string
|
||||||
|
}
|
||||||
|
|
||||||
|
// scanFlagsUsage names the flags that scan and dwim share, for their
|
||||||
|
// usage lines.
|
||||||
|
const scanFlagsUsage = "[--workers N] [-x] [--no-default-excludes] " +
|
||||||
|
"[--exclude PATH]..."
|
||||||
|
|
||||||
|
// addScanFlags adds the flags that scan and dwim share to cmd, which
|
||||||
|
// stores their values in opts.
|
||||||
|
func addScanFlags(cmd *cobra.Command, opts *scanOptions) {
|
||||||
|
cmd.Flags().IntVar(&opts.workers, "workers", runtime.NumCPU(),
|
||||||
|
"concurrent workers for the walk and hash phases")
|
||||||
|
cmd.Flags().BoolVarP(&opts.oneFS, "one-file-system", "x", false,
|
||||||
|
"do not cross filesystem boundaries")
|
||||||
|
cmd.Flags().BoolVar(&opts.noDefaultExcludes, "no-default-excludes",
|
||||||
|
false, "enter the default excludes, the paths listed above")
|
||||||
|
// A string array, not a slice, so a comma in a path is not a
|
||||||
|
// separator.
|
||||||
|
cmd.Flags().StringArrayVar(&opts.excludePaths, "exclude", nil,
|
||||||
|
"never enter or record `PATH` or anything below it; repeatable")
|
||||||
|
}
|
||||||
|
|
||||||
|
// check rejects the flag values that are usage errors, before the
|
||||||
|
// command runs. It is the PreRunE of scan and dwim.
|
||||||
|
func (o *scanOptions) check(cmd *cobra.Command, _ []string) error {
|
||||||
|
err := checkScanWorkers(cmd, o.workers)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
return rejectEmptyExcludeValue(cmd, o.excludePaths)
|
||||||
|
}
|
||||||
|
|
||||||
|
// newScanCommand builds the scan subcommand. Its help names the
|
||||||
|
// database, and lists the default excludes of the system it runs on and
|
||||||
|
// the filesystem types it never records anything on.
|
||||||
func newScanCommand() *cobra.Command {
|
func newScanCommand() *cobra.Command {
|
||||||
var (
|
var opts scanOptions
|
||||||
scanWorkers int
|
|
||||||
scanOneFS bool
|
|
||||||
scanNoDefaultExcludes bool
|
|
||||||
)
|
|
||||||
|
|
||||||
short := "Walk trees and synchronize the scan database"
|
short := "Walk trees and synchronize the scan database"
|
||||||
|
|
||||||
scanCmd := &cobra.Command{
|
scanCmd := &cobra.Command{
|
||||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
Use: cmdScan + " " + scanFlagsUsage + " PATH...",
|
||||||
// The Use line already names every flag, so cobra must not
|
// The Use line already names every flag, so cobra must not
|
||||||
// append its generic "[flags]" to it.
|
// append its generic "[flags]" to it.
|
||||||
DisableFlagsInUseLine: true,
|
DisableFlagsInUseLine: true,
|
||||||
Short: short,
|
Short: short,
|
||||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
Long: short + ".\n\n" + databaseHelp + "\n" +
|
||||||
Args: cobra.MinimumNArgs(1),
|
defaultExcludesHelp(runtime.GOOS),
|
||||||
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
Args: cobra.MinimumNArgs(1),
|
||||||
return checkScanWorkers(cmd, scanWorkers)
|
PreRunE: opts.check,
|
||||||
},
|
|
||||||
RunE: runE(func(ctx context.Context, args []string) error {
|
RunE: runE(func(ctx context.Context, args []string) error {
|
||||||
ctx, stop := interruptContext(ctx)
|
ctx, stop := interruptContext(ctx)
|
||||||
defer stop()
|
defer stop()
|
||||||
|
|
||||||
return runScan(ctx, args, scanWorkers, scanOneFS,
|
return runScan(ctx, args, opts)
|
||||||
scanNoDefaultExcludes)
|
|
||||||
}),
|
}),
|
||||||
}
|
}
|
||||||
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
|
addScanFlags(scanCmd, &opts)
|
||||||
"concurrent workers for the walk and hash phases")
|
|
||||||
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
|
|
||||||
"do not cross filesystem boundaries")
|
|
||||||
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
|
|
||||||
false, "enter the default excludes, the paths listed above")
|
|
||||||
|
|
||||||
return scanCmd
|
return scanCmd
|
||||||
}
|
}
|
||||||
|
|
||||||
// checkScanWorkers rejects a scan --workers value below 1. That is a
|
// newDwimCommand builds the dwim subcommand, which takes the scan flags
|
||||||
// usage error reported in one line: cobra prints the returned message
|
// with the meaning they have for scan. Its help says what dwim runs,
|
||||||
// without the usage text, and run exits 2.
|
// then lists the default excludes and the filesystem types the scan
|
||||||
|
// never records anything on.
|
||||||
|
func newDwimCommand(stdout io.Writer) *cobra.Command {
|
||||||
|
var opts scanOptions
|
||||||
|
|
||||||
|
short := "Scan /, then print the duplicate-tree report"
|
||||||
|
|
||||||
|
dwimCmd := &cobra.Command{
|
||||||
|
Use: cmdDwim + " " + scanFlagsUsage,
|
||||||
|
DisableFlagsInUseLine: true,
|
||||||
|
Short: short,
|
||||||
|
Long: short + ".\n\n" + dwimHelp + "\n" +
|
||||||
|
defaultExcludesHelp(runtime.GOOS),
|
||||||
|
Args: cobra.NoArgs,
|
||||||
|
PreRunE: opts.check,
|
||||||
|
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||||
|
return runDwim(ctx, "/", opts, stdout)
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
addScanFlags(dwimCmd, &opts)
|
||||||
|
|
||||||
|
return dwimCmd
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkScanWorkers rejects a --workers value below 1. That is a usage
|
||||||
|
// error reported in one line: cobra prints the returned message without
|
||||||
|
// the usage text, and run exits 2.
|
||||||
func checkScanWorkers(cmd *cobra.Command, workers int) error {
|
func checkScanWorkers(cmd *cobra.Command, workers int) error {
|
||||||
if workers >= 1 {
|
if workers >= 1 {
|
||||||
return nil
|
return nil
|
||||||
@@ -256,6 +375,18 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
|
|||||||
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
|
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// rejectEmptyExcludeValue rejects an empty --exclude value. That
|
||||||
|
// is a usage error reported in one line, as for --workers below 1.
|
||||||
|
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
|
||||||
|
if !slices.Contains(excludePaths, "") {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd.SilenceUsage = true
|
||||||
|
|
||||||
|
return errEmptyExcludeValue
|
||||||
|
}
|
||||||
|
|
||||||
// runE adapts a subcommand implementation, or the version print, to
|
// runE adapts a subcommand implementation, or the version print, to
|
||||||
// cobra's RunE. Cobra prints the error and the command's usage text for
|
// cobra's RunE. Cobra prints the error and the command's usage text for
|
||||||
// every error RunE returns, but a subcommand that ran and failed has no
|
// every error RunE returns, but a subcommand that ran and failed has no
|
||||||
|
|||||||
@@ -22,6 +22,12 @@ import (
|
|||||||
// error prints and a runtime failure must not.
|
// error prints and a runtime failure must not.
|
||||||
const usageMarker = "Usage:"
|
const usageMarker = "Usage:"
|
||||||
|
|
||||||
|
// helpFlag asks for the help text.
|
||||||
|
const helpFlag = "--help"
|
||||||
|
|
||||||
|
// workersFlag is the scan and dwim flag that sets the worker count.
|
||||||
|
const workersFlag = "--workers"
|
||||||
|
|
||||||
// TestMain stops the tests at once, saying why, when their temporary
|
// TestMain stops the tests at once, saying why, when their temporary
|
||||||
// directory is not on a persistent filesystem: scan records nothing
|
// directory is not on a persistent filesystem: scan records nothing
|
||||||
// there (README "Rules for the walk"), so most scan tests would fail
|
// there (README "Rules for the walk"), so most scan tests would fail
|
||||||
@@ -570,7 +576,8 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
cmdScan: {
|
cmdScan: {
|
||||||
run: func() error {
|
run: func() error {
|
||||||
return runScan(t.Context(), []string{missing}, 1, false, false)
|
return runScan(t.Context(), []string{missing},
|
||||||
|
scanOptions{workers: 1})
|
||||||
},
|
},
|
||||||
want: scanWant,
|
want: scanWant,
|
||||||
},
|
},
|
||||||
@@ -661,7 +668,7 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
|
|||||||
|
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
args := []string{cmdScan, "--workers", workers, t.TempDir()}
|
args := []string{cmdScan, workersFlag, workers, t.TempDir()}
|
||||||
|
|
||||||
code := run(args, &stdout, &stderr)
|
code := run(args, &stdout, &stderr)
|
||||||
if code != exitUsage {
|
if code != exitUsage {
|
||||||
@@ -692,7 +699,7 @@ func TestRunHelp(t *testing.T) {
|
|||||||
|
|
||||||
// README §Settings: help goes to stderr, exits 0, and leaves stdout
|
// README §Settings: help goes to stderr, exits 0, and leaves stdout
|
||||||
// empty.
|
// empty.
|
||||||
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
|
cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
|
||||||
|
|
||||||
for _, args := range cases {
|
for _, args := range cases {
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
@@ -712,6 +719,16 @@ func TestRunHelp(t *testing.T) {
|
|||||||
args, got)
|
args, got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The scan help lists --exclude with its argument.
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
|
run([]string{cmdScan, "-h"}, &stdout, &stderr)
|
||||||
|
|
||||||
|
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
|
||||||
|
t.Errorf("scan -h stderr = %q, want %s PATH", stderr.String(),
|
||||||
|
excludeFlag)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunScanUsageLine(t *testing.T) {
|
func TestRunScanUsageLine(t *testing.T) {
|
||||||
@@ -719,7 +736,8 @@ func TestRunScanUsageLine(t *testing.T) {
|
|||||||
|
|
||||||
// README §Subcommands shows this usage line for scan, and the help
|
// README §Subcommands shows this usage line for scan, and the help
|
||||||
// and the usage error both print it.
|
// and the usage error both print it.
|
||||||
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...\n"
|
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] " +
|
||||||
|
"[--exclude PATH]... PATH...\n"
|
||||||
|
|
||||||
cases := map[string][]string{
|
cases := map[string][]string{
|
||||||
"help": {cmdScan, "-h"},
|
"help": {cmdScan, "-h"},
|
||||||
@@ -742,6 +760,73 @@ func TestRunScanUsageLine(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// README §Settings: the help names the database, and says what
|
||||||
|
// report, trees, info and dwim print.
|
||||||
|
database := []string{
|
||||||
|
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
|
||||||
|
}
|
||||||
|
escapes := []string{`\\`, `\t`, `\n`, `\r`}
|
||||||
|
|
||||||
|
// The subcommand whose help is printed; none for the root.
|
||||||
|
cases := []struct {
|
||||||
|
subcommand []string
|
||||||
|
want []string
|
||||||
|
}{
|
||||||
|
{nil, database},
|
||||||
|
{[]string{cmdScan}, database},
|
||||||
|
{
|
||||||
|
[]string{cmdReport},
|
||||||
|
append([]string{
|
||||||
|
"tab-separated", "\n first ", "\n dupe ", "\n size ",
|
||||||
|
"largest first",
|
||||||
|
}, escapes...),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
[]string{cmdTrees},
|
||||||
|
append([]string{
|
||||||
|
"tab-separated", "\n first ", "\n dupe ", "\n files ",
|
||||||
|
"\n size ", "\n disk_size ", "largest first",
|
||||||
|
}, escapes...),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
[]string{cmdInfo},
|
||||||
|
[]string{
|
||||||
|
"\n database ", "\n database file size ",
|
||||||
|
"\n -wal file size ", "\n modified ",
|
||||||
|
"\n schema version ", "\n records ",
|
||||||
|
"\n size of recorded files ",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
[]string{cmdDwim},
|
||||||
|
[]string{
|
||||||
|
"sfdupes scan /", "as root", "sfdupes trees", "database",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
|
args := slices.Concat(tc.subcommand, []string{helpFlag})
|
||||||
|
|
||||||
|
code := run(args, &stdout, &stderr)
|
||||||
|
if code != exitOK {
|
||||||
|
t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, want := range tc.want {
|
||||||
|
if !strings.Contains(stderr.String(), want) {
|
||||||
|
t.Errorf("run(%v) help lacks %q:\n%s",
|
||||||
|
args, want, stderr.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunVersion(t *testing.T) {
|
func TestRunVersion(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -5,19 +5,20 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"os"
|
"os"
|
||||||
"runtime"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
|
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
|
||||||
// root" runs with no flags, then what "sfdupes trees" runs, and ends
|
// root" runs with the scan flags in opts, then what "sfdupes trees"
|
||||||
// with a line on stderr saying how to print the report again. The dwim
|
// runs, and ends with a line on stderr saying how to print the report
|
||||||
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
|
// again. The dwim command passes "/" as root. It catches SIGINT and
|
||||||
// does, but only during the scan. An error from the scan, an interrupt
|
// SIGTERM as scan does, but only during the scan. An error from the
|
||||||
// included, ends it before the report. During the report, a signal
|
// scan, an interrupt included, ends it before the report. During the
|
||||||
// ends it at once, as it ends trees.
|
// report, a signal ends it at once, as it ends trees.
|
||||||
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
func runDwim(
|
||||||
|
ctx context.Context, root string, opts scanOptions, stdout io.Writer,
|
||||||
|
) error {
|
||||||
scanCtx, stop := interruptContext(ctx)
|
scanCtx, stop := interruptContext(ctx)
|
||||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
|
err := runScan(scanCtx, []string{root}, opts)
|
||||||
|
|
||||||
stop()
|
stop()
|
||||||
|
|
||||||
|
|||||||
@@ -2,7 +2,10 @@ package sfdupes //nolint:testpackage // calls the unexported runDwim
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"errors"
|
||||||
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
@@ -28,7 +31,7 @@ func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
|
|||||||
|
|
||||||
stderr := captureStderr(t)
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
err := runDwim(t.Context(), root, &stdout)
|
err := runDwim(t.Context(), root, scanOptions{workers: 1}, &stdout)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
||||||
}
|
}
|
||||||
@@ -67,7 +70,7 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
|||||||
|
|
||||||
stderr := captureStderr(t)
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
err := runDwim(t.Context(), missing, &stdout)
|
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, &stdout)
|
||||||
if err == nil {
|
if err == nil {
|
||||||
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
|
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
|
||||||
}
|
}
|
||||||
@@ -81,3 +84,127 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
|||||||
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
|
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
|
||||||
|
// dwim passes its --exclude paths to the scan: an excluded copy is
|
||||||
|
// not recorded, so its pair is not reported.
|
||||||
|
t.Setenv(databaseEnv, testDBPath(t))
|
||||||
|
|
||||||
|
root := realTempDir(t)
|
||||||
|
|
||||||
|
kept := []string{
|
||||||
|
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
|
||||||
|
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
|
||||||
|
}
|
||||||
|
writeFile(t, root, "b1/f.bin", pattern(2, 5000))
|
||||||
|
excluded := filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000)))
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
|
opts := scanOptions{workers: 1, excludePaths: []string{excluded}}
|
||||||
|
|
||||||
|
err := runDwim(t.Context(), root, opts, &stdout)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
||||||
|
}
|
||||||
|
|
||||||
|
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
|
||||||
|
if len(lines) != 2 {
|
||||||
|
t.Fatalf("stdout = %q, want a header and one group",
|
||||||
|
stdout.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
want := kept[0] + "\t" + kept[1] + "\t1\t300\t"
|
||||||
|
if !strings.HasPrefix(lines[1], want) {
|
||||||
|
t.Errorf("group = %q, want it to start %q", lines[1], want)
|
||||||
|
}
|
||||||
|
|
||||||
|
skipped := "scan: skipped " + excluded + " (--exclude)\n"
|
||||||
|
if got := stderr(); !strings.Contains(got, skipped) {
|
||||||
|
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunDwimTakesScanFlags(t *testing.T) {
|
||||||
|
// The dwim command parses and checks the scan flags as scan does.
|
||||||
|
// Each case stops before the walk of / begins, and before the
|
||||||
|
// database is created. "dwim --exclude /" excludes its own operand.
|
||||||
|
cases := map[string]struct {
|
||||||
|
args []string
|
||||||
|
code int
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
"--exclude /": {
|
||||||
|
[]string{cmdDwim, excludeFlag, "/"},
|
||||||
|
exitFatal,
|
||||||
|
"sfdupes: scan /: operand excluded by --exclude /\n",
|
||||||
|
},
|
||||||
|
"--exclude empty": {
|
||||||
|
[]string{cmdDwim, excludeFlag, ""},
|
||||||
|
exitUsage,
|
||||||
|
"Error: --exclude must not be empty\n",
|
||||||
|
},
|
||||||
|
"--workers 0": {
|
||||||
|
[]string{cmdDwim, workersFlag, "0", "-x", "--no-default-excludes"},
|
||||||
|
exitUsage,
|
||||||
|
"Error: --workers must be at least 1, got 0\n",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, tc := range cases {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
stdout := captureStdout(t)
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
|
code := run(tc.args, os.Stdout, os.Stderr)
|
||||||
|
if code != tc.code {
|
||||||
|
t.Errorf("run(%q) = %d, want %d", tc.args, code, tc.code)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.HasSuffix(stderr(), tc.want) || stdout() != "" {
|
||||||
|
t.Errorf("stdout, stderr = %q, %q, want \"\" and stderr "+
|
||||||
|
"ending %q", stdout(), stderr(), tc.want)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := os.Lstat(path)
|
||||||
|
if !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunDwimHelpListsScanFlags(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// README "Subcommands" shows this usage line for dwim.
|
||||||
|
const want = "\n sfdupes dwim [--workers N] [-x] [--no-default-excludes] " +
|
||||||
|
"[--exclude PATH]...\n"
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
|
run([]string{cmdDwim, "-h"}, &stdout, &stderr)
|
||||||
|
|
||||||
|
if !strings.Contains(stderr.String(), want) {
|
||||||
|
t.Errorf("dwim -h stderr = %q, want the usage line %q",
|
||||||
|
stderr.String(), want)
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
|
||||||
|
t.Errorf("dwim -h stderr = %q, want %s PATH", stderr.String(),
|
||||||
|
excludeFlag)
|
||||||
|
}
|
||||||
|
|
||||||
|
// README "Settings": the help lists the default excludes and the
|
||||||
|
// filesystem types the walk never records anything on.
|
||||||
|
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
|
||||||
|
if !strings.Contains(stderr.String(), help) {
|
||||||
|
t.Errorf("dwim -h stderr = %q, want it to hold %q", stderr.String(),
|
||||||
|
help)
|
||||||
|
}
|
||||||
|
}
|
||||||
+104
-12
@@ -1,8 +1,9 @@
|
|||||||
package sfdupes
|
package sfdupes
|
||||||
|
|
||||||
// This file holds two rules for the walk (README "Rules for the walk"):
|
// This file holds three rules for the walk (README "Rules for the
|
||||||
|
// walk"): the paths given with --exclude, which the scan never enters;
|
||||||
// the default excludes, the paths a scan's walk does not enter below an
|
// the default excludes, the paths a scan's walk does not enter below an
|
||||||
// operand unless scan is given --no-default-excludes, and the
|
// operand unless scan is given --no-default-excludes; and the
|
||||||
// filesystems that are not persistent, on which the scan never records
|
// filesystems that are not persistent, on which the scan never records
|
||||||
// or reads anything.
|
// or reads anything.
|
||||||
|
|
||||||
@@ -12,6 +13,7 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -43,17 +45,25 @@ const typesPerHelpLine = 6
|
|||||||
// readMountInfo reads.
|
// readMountInfo reads.
|
||||||
var errMountInfoLine = errors.New("unexpected line")
|
var errMountInfoLine = errors.New("unexpected line")
|
||||||
|
|
||||||
|
// errExcludedOperand reports an operand at or below a path given with
|
||||||
|
// --exclude.
|
||||||
|
var errExcludedOperand = errors.New("operand excluded by --exclude")
|
||||||
|
|
||||||
// walkRules holds the rules for the walk that a scan sets up as it
|
// walkRules holds the rules for the walk that a scan sets up as it
|
||||||
// starts. excludes are the default excludes, or none with
|
// starts. excludePaths are the paths given with --exclude, each with
|
||||||
// --no-default-excludes. nonPersistent maps the device ID of every
|
// the device ID and inode number of the file it names (see
|
||||||
// mounted filesystem that is not persistent to its type; the scan
|
// excludePathInodes), which the walk neither enters nor records,
|
||||||
// records and reads nothing on one of those, whatever the flags.
|
// whatever path leads to them. defaultExcludes are the default
|
||||||
// databaseFiles are the scan's own database files, which the walk
|
// excludes, or none with --no-default-excludes. nonPersistent maps the
|
||||||
// passes over (see databaseFiles in db.go).
|
// device ID of every mounted filesystem that is not persistent to its
|
||||||
|
// type; the scan records and reads nothing on one of those, whatever
|
||||||
|
// the flags. databaseFiles are the scan's own database files, which the
|
||||||
|
// walk passes over (see databaseFiles in db.go).
|
||||||
type walkRules struct {
|
type walkRules struct {
|
||||||
excludes []string
|
excludePaths []fileRec
|
||||||
nonPersistent map[uint64]string
|
defaultExcludes []string
|
||||||
databaseFiles []fileRec
|
nonPersistent map[uint64]string
|
||||||
|
databaseFiles []fileRec
|
||||||
}
|
}
|
||||||
|
|
||||||
// defaultExcludes returns the default excludes of the system goos
|
// defaultExcludes returns the default excludes of the system goos
|
||||||
@@ -97,6 +107,88 @@ func isDefaultExclude(path string, excludes []string) bool {
|
|||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// excludePathInodes returns each of paths, the --exclude values, as its
|
||||||
|
// real path (see realPaths) with the device ID and inode number of the
|
||||||
|
// file there. The walk compares what it reaches with those, as it does
|
||||||
|
// with the database files, so it also leaves out an --exclude path
|
||||||
|
// spelled another way, such as in another letter case on macOS.
|
||||||
|
func excludePathInodes(paths []string) ([]fileRec, error) {
|
||||||
|
resolved, err := realPaths(paths)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
files := make([]fileRec, 0, len(resolved))
|
||||||
|
|
||||||
|
for _, path := range resolved {
|
||||||
|
fi, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
dev, ino := inodeOfInfo(fi)
|
||||||
|
files = append(files, fileRec{path: path, dev: dev, ino: ino})
|
||||||
|
}
|
||||||
|
|
||||||
|
return files, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// matchingExcludePath returns the path of the one of excludePaths with
|
||||||
|
// the device ID and inode number of fi, an lstat result, or "" when
|
||||||
|
// there is none.
|
||||||
|
func matchingExcludePath(fi fs.FileInfo, excludePaths []fileRec) string {
|
||||||
|
dev, ino := inodeOfInfo(fi)
|
||||||
|
f := fileRec{dev: dev, ino: ino}
|
||||||
|
|
||||||
|
for _, exclude := range excludePaths {
|
||||||
|
if sameInode(exclude, f) {
|
||||||
|
return exclude.path
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
// checkOperandsNotExcluded returns an error when one of roots, the
|
||||||
|
// operands, or a directory above it is one of excludePaths, the
|
||||||
|
// --exclude paths, by device ID and inode number. The error names the
|
||||||
|
// operand and the --exclude path. The roots are absolute, so the
|
||||||
|
// directories above each one end at /. A path here that lstat cannot
|
||||||
|
// read is an error too.
|
||||||
|
func checkOperandsNotExcluded(roots []string, excludePaths []fileRec) error {
|
||||||
|
for _, root := range roots {
|
||||||
|
for path := root; ; path = filepath.Dir(path) {
|
||||||
|
fi, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("scan %s: %w", root, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
exclude := matchingExcludePath(fi, excludePaths)
|
||||||
|
if exclude != "" {
|
||||||
|
return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand,
|
||||||
|
exclude)
|
||||||
|
}
|
||||||
|
|
||||||
|
if path == "/" {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// excludePathLine returns the line the scan prints for path when fi,
|
||||||
|
// its lstat, makes it one of rules.excludePaths, and "" when it does
|
||||||
|
// not.
|
||||||
|
func (rules walkRules) excludePathLine(path string, fi fs.FileInfo) string {
|
||||||
|
if matchingExcludePath(fi, rules.excludePaths) == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
|
||||||
|
return fmt.Sprintf("scan: skipped %s (--exclude)", path)
|
||||||
|
}
|
||||||
|
|
||||||
// nonPersistentFilesystemTypes returns the filesystem types that are
|
// nonPersistentFilesystemTypes returns the filesystem types that are
|
||||||
// not persistent: filesystems that the kernel makes up or that are kept
|
// not persistent: filesystems that the kernel makes up or that are kept
|
||||||
// in memory. devfs is the macOS filesystem of /dev; the others are
|
// in memory. devfs is the macOS filesystem of /dev; the others are
|
||||||
@@ -162,7 +254,7 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
|
|||||||
return devices, lines.Err()
|
return devices, lines.Err()
|
||||||
}
|
}
|
||||||
|
|
||||||
// defaultExcludesHelp describes, for the scan command's help, the
|
// defaultExcludesHelp describes, for the scan and dwim help, the
|
||||||
// default excludes of the system goos names and, where the scan lists
|
// default excludes of the system goos names and, where the scan lists
|
||||||
// the mounted filesystems, the types it never records anything on.
|
// the mounted filesystems, the types it never records anything on.
|
||||||
func defaultExcludesHelp(goos string) string {
|
func defaultExcludesHelp(goos string) string {
|
||||||
|
|||||||
@@ -14,6 +14,9 @@ import (
|
|||||||
"golang.org/x/sys/unix"
|
"golang.org/x/sys/unix"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// excludeFlag is the scan flag that names a path the walk never enters.
|
||||||
|
const excludeFlag = "--exclude"
|
||||||
|
|
||||||
func TestDefaultExcludes(t *testing.T) {
|
func TestDefaultExcludes(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -188,7 +191,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
|
|||||||
unseen := writeFile(t, excluded, "new", pattern(2, 10))
|
unseen := writeFile(t, excluded, "new", pattern(2, 10))
|
||||||
|
|
||||||
st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
|
st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
|
||||||
walkRules{excludes: []string{excluded}})
|
walkRules{defaultExcludes: []string{excluded}})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -234,7 +237,7 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
|
|||||||
db := openTestDB(t)
|
db := openTestDB(t)
|
||||||
|
|
||||||
st, err := syncScan(t.Context(), db, roots, 1, false,
|
st, err := syncScan(t.Context(), db, roots, 1, false,
|
||||||
walkRules{excludes: []string{excluded}})
|
walkRules{defaultExcludes: []string{excluded}})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -262,7 +265,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
|
|||||||
f := writeFile(t, excluded, "f", pattern(1, 10))
|
f := writeFile(t, excluded, "f", pattern(1, 10))
|
||||||
|
|
||||||
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
|
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
|
||||||
walkRules{excludes: []string{excluded}})
|
walkRules{defaultExcludes: []string{excluded}})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -356,6 +359,217 @@ func TestRunScanNoDefaultExcludes(t *testing.T) {
|
|||||||
assertRecorded(t, path, f, true)
|
assertRecorded(t, path, f, true)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRunScanExcludeNested scans a tree through the command with an
|
||||||
|
// --exclude path two levels below the operand. The walk records nothing
|
||||||
|
// at or below it and says so. The record an earlier scan left there is
|
||||||
|
// kept although its file is gone.
|
||||||
|
func TestRunScanExcludeNested(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
excluded := filepath.Join(dir, "a", "b")
|
||||||
|
kept := writeFile(t, dir, "a/f", pattern(1, 10))
|
||||||
|
gone := writeFile(t, excluded, "gone", pattern(2, 10))
|
||||||
|
|
||||||
|
scanOK(t, dir)
|
||||||
|
|
||||||
|
err := os.Remove(gone)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
unseen := writeFile(t, excluded, "c/new", pattern(3, 10))
|
||||||
|
|
||||||
|
stderr := scanOK(t, excludeFlag, excluded, dir)
|
||||||
|
|
||||||
|
line := "scan: skipped " + excluded + " (--exclude)\n"
|
||||||
|
if !strings.Contains(stderr, line) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertRecorded(t, path, kept, true)
|
||||||
|
assertRecorded(t, path, gone, true)
|
||||||
|
assertRecorded(t, path, unseen, false)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanExcludeRepeated gives --exclude twice, once for a
|
||||||
|
// directory and once for a file, the file through a symlink to it. The
|
||||||
|
// walk records neither, and records the file beside them.
|
||||||
|
func TestRunScanExcludeRepeated(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
inDir := writeFile(t, dir, "skip/f", pattern(1, 10))
|
||||||
|
file := writeFile(t, dir, "skip-file", pattern(2, 10))
|
||||||
|
kept := writeFile(t, dir, "keep", pattern(3, 10))
|
||||||
|
|
||||||
|
link := filepath.Join(t.TempDir(), "link")
|
||||||
|
symlink(t, file, link)
|
||||||
|
|
||||||
|
scanOK(t, excludeFlag, filepath.Dir(inDir), excludeFlag, link, dir)
|
||||||
|
|
||||||
|
assertRecorded(t, path, inDir, false)
|
||||||
|
assertRecorded(t, path, file, false)
|
||||||
|
assertRecorded(t, path, kept, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanExcludeReachedByAnotherPath gives --exclude a directory
|
||||||
|
// through a symlink and a file through "..", so the walk reaches
|
||||||
|
// neither by the path given. The walk also reaches the file through a
|
||||||
|
// hard link. It records none of these, prints a line naming each, and
|
||||||
|
// records the file beside them.
|
||||||
|
func TestRunScanExcludeReachedByAnotherPath(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
inDir := writeFile(t, dir, "a/cache/f", pattern(1, 10))
|
||||||
|
file := writeFile(t, dir, "a/big", pattern(2, 10))
|
||||||
|
kept := writeFile(t, dir, "keep", pattern(3, 10))
|
||||||
|
|
||||||
|
hardLink := filepath.Join(dir, "big-link")
|
||||||
|
|
||||||
|
err := os.Link(file, hardLink)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
link := filepath.Join(t.TempDir(), "link")
|
||||||
|
symlink(t, filepath.Join(dir, "a"), link)
|
||||||
|
|
||||||
|
// Joined as text, since filepath.Join would drop the "..".
|
||||||
|
dotDot := dir + "/a/cache/../big"
|
||||||
|
|
||||||
|
stderr := scanOK(t, excludeFlag, filepath.Join(link, "cache"),
|
||||||
|
excludeFlag, dotDot, dir)
|
||||||
|
|
||||||
|
for _, skipped := range []string{filepath.Dir(inDir), file, hardLink} {
|
||||||
|
line := "scan: skipped " + skipped + " (--exclude)\n"
|
||||||
|
if !strings.Contains(stderr, line) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
assertRecorded(t, path, inDir, false)
|
||||||
|
assertRecorded(t, path, file, false)
|
||||||
|
assertRecorded(t, path, hardLink, false)
|
||||||
|
assertRecorded(t, path, kept, true)
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanExcludedOperand checks that an operand at or below an
|
||||||
|
// --exclude path is a fatal error that names both, and so is an
|
||||||
|
// --exclude path that does not exist. An operand that is a hard link to
|
||||||
|
// an --exclude file is at it. Neither scan writes a database.
|
||||||
|
func TestRunScanExcludedOperand(t *testing.T) {
|
||||||
|
dir := realTempDir(t)
|
||||||
|
sub := filepath.Join(dir, "sub")
|
||||||
|
f := writeFile(t, sub, "f", pattern(1, 10))
|
||||||
|
|
||||||
|
hardLink := filepath.Join(dir, "f-link")
|
||||||
|
|
||||||
|
err := os.Link(f, hardLink)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
cases := map[string]struct {
|
||||||
|
args []string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
"at it": {
|
||||||
|
[]string{excludeFlag, sub, sub},
|
||||||
|
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
|
||||||
|
},
|
||||||
|
"below it": {
|
||||||
|
[]string{excludeFlag, dir, sub},
|
||||||
|
"scan " + sub + ": operand excluded by --exclude " + dir + "\n",
|
||||||
|
},
|
||||||
|
"after an operand it does not exclude": {
|
||||||
|
[]string{dir, excludeFlag, sub, sub},
|
||||||
|
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
|
||||||
|
},
|
||||||
|
"a hard link to it": {
|
||||||
|
[]string{excludeFlag, f, hardLink},
|
||||||
|
"scan " + hardLink + ": operand excluded by --exclude " + f + "\n",
|
||||||
|
},
|
||||||
|
"exclude missing": {
|
||||||
|
[]string{excludeFlag, filepath.Join(dir, "nope"), dir},
|
||||||
|
"--exclude: resolve " + filepath.Join(dir, "nope") + ": ",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, tc := range cases {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
stdout := captureStdout(t)
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
|
code := run(append([]string{cmdScan}, tc.args...), os.Stdout,
|
||||||
|
os.Stderr)
|
||||||
|
if code != exitFatal {
|
||||||
|
t.Errorf("run(scan %q) = %d, want %d", tc.args, code,
|
||||||
|
exitFatal)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertFatalOutput(t, stderr(), stdout())
|
||||||
|
|
||||||
|
if !strings.Contains(stderr(), "sfdupes: "+tc.want) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr(), tc.want)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := os.Lstat(path)
|
||||||
|
if !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanRejectsEmptyExclude checks that an empty --exclude value,
|
||||||
|
// given as its own argument, after "=" or after another --exclude, is
|
||||||
|
// a usage error reported in one line before anything is scanned
|
||||||
|
// (README "Settings").
|
||||||
|
func TestRunScanRejectsEmptyExclude(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
|
||||||
|
cases := map[string][]string{
|
||||||
|
"separate": {excludeFlag, "", dir},
|
||||||
|
"joined by =": {excludeFlag + "=", dir},
|
||||||
|
"after a path": {excludeFlag, t.TempDir(), excludeFlag, "", dir},
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, args := range cases {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
|
code := run(append([]string{cmdScan}, args...), &stdout,
|
||||||
|
&stderr)
|
||||||
|
if code != exitUsage {
|
||||||
|
t.Errorf("run(scan %q) = %d, want %d", args, code,
|
||||||
|
exitUsage)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := "Error: --exclude must not be empty\n"
|
||||||
|
if stdout.String() != "" || stderr.String() != want {
|
||||||
|
t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
|
||||||
|
stdout.String(), stderr.String(), want)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := os.Lstat(path)
|
||||||
|
if !errors.Is(err, os.ErrNotExist) {
|
||||||
|
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestRunScanSkipsNonPersistentOperand scans, through the command, a
|
// TestRunScanSkipsNonPersistentOperand scans, through the command, a
|
||||||
// directory on a filesystem that is not persistent, with and without
|
// directory on a filesystem that is not persistent, with and without
|
||||||
// --no-default-excludes: either way it is skipped, with a line saying
|
// --no-default-excludes: either way it is skipped, with a line saying
|
||||||
|
|||||||
+84
-55
@@ -98,44 +98,47 @@ func scanPhase(name string) string {
|
|||||||
// runs, and the lock is released after it. When ctx is cancelled, as by
|
// runs, and the lock is released after it. When ctx is cancelled, as by
|
||||||
// the SIGINT or SIGTERM that interruptContext catches, the scan keeps
|
// the SIGINT or SIGTERM that interruptContext catches, the scan keeps
|
||||||
// what it has hashed (see syncScan), prints how many files its walk
|
// what it has hashed (see syncScan), prints how many files its walk
|
||||||
// reached, and returns errInterrupted. workers must be at least 1; the
|
// reached, and returns errInterrupted. opts.workers must be at least 1;
|
||||||
// scan command rejects anything less. Before the lock is taken, the
|
// the scan and dwim commands reject anything less. Before the lock is
|
||||||
// scan lists the filesystems that are not persistent, on which the walk
|
// taken, the scan lists the filesystems that are not persistent, on
|
||||||
// records nothing; unless noDefaultExcludes is set, the walk does not
|
// which the walk records nothing; unless opts.noDefaultExcludes is set,
|
||||||
// enter the default excludes either. Once the database is open, the
|
// the walk does not enter the default excludes either. The walk neither
|
||||||
// scan notes its files, which the walk passes over.
|
// enters nor records the paths in opts.excludePaths, the --exclude
|
||||||
func runScan(ctx context.Context, roots []string, workers int,
|
// values; an operand at or below one is a fatal error. Once the
|
||||||
oneFS, noDefaultExcludes bool,
|
// database is open, the scan notes its files, which the walk passes
|
||||||
) error {
|
// over.
|
||||||
|
func runScan(ctx context.Context, roots []string, opts scanOptions) error {
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
// A database file missing now is created by lockScanDatabase below.
|
printDatabasePath(dbPath, source)
|
||||||
what := "database"
|
|
||||||
|
|
||||||
_, err = os.Stat(dbPath)
|
roots, err = realPaths(roots)
|
||||||
if errors.Is(err, fs.ErrNotExist) {
|
|
||||||
what = "new database"
|
|
||||||
}
|
|
||||||
|
|
||||||
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
|
|
||||||
source)
|
|
||||||
|
|
||||||
roots, err = resolveRoots(roots)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
nonPersistent, err := nonPersistentFilesystems()
|
var rules walkRules
|
||||||
|
|
||||||
|
rules.excludePaths, err = excludePathInodes(opts.excludePaths)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("--exclude: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = checkOperandsNotExcluded(roots, rules.excludePaths)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
rules.nonPersistent, err = nonPersistentFilesystems()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("list the mounted filesystems: %w", err)
|
return fmt.Errorf("list the mounted filesystems: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
rules := walkRules{nonPersistent: nonPersistent}
|
if !opts.noDefaultExcludes {
|
||||||
if !noDefaultExcludes {
|
rules.defaultExcludes = defaultExcludes(runtime.GOOS)
|
||||||
rules.excludes = defaultExcludes(runtime.GOOS)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
lock, err := lockScanDatabase(dbPath)
|
lock, err := lockScanDatabase(dbPath)
|
||||||
@@ -163,7 +166,7 @@ func runScan(ctx context.Context, roots []string, workers int,
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
|
st, err := syncScan(ctx, db, roots, opts.workers, opts.oneFS, rules)
|
||||||
if errors.Is(err, context.Canceled) {
|
if errors.Is(err, context.Canceled) {
|
||||||
return interrupted(st.walked)
|
return interrupted(st.walked)
|
||||||
}
|
}
|
||||||
@@ -181,6 +184,21 @@ func runScan(ctx context.Context, roots []string, workers int,
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// printDatabasePath prints on stderr the line that names the database
|
||||||
|
// at dbPath, with source, saying whether it is new. A database file
|
||||||
|
// missing now is new: lockScanDatabase creates it.
|
||||||
|
func printDatabasePath(dbPath, source string) {
|
||||||
|
what := "database"
|
||||||
|
|
||||||
|
_, err := os.Stat(dbPath)
|
||||||
|
if errors.Is(err, fs.ErrNotExist) {
|
||||||
|
what = "new database"
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
|
||||||
|
source)
|
||||||
|
}
|
||||||
|
|
||||||
// interruptContext returns a copy of ctx that the first SIGINT or
|
// interruptContext returns a copy of ctx that the first SIGINT or
|
||||||
// SIGTERM cancels; the scan command runs the scan under it. stop
|
// SIGTERM cancels; the scan command runs the scan under it. stop
|
||||||
// releases the signals.
|
// releases the signals.
|
||||||
@@ -209,24 +227,25 @@ func interrupted(walked int) error {
|
|||||||
return errInterrupted
|
return errInterrupted
|
||||||
}
|
}
|
||||||
|
|
||||||
// resolveRoots converts each PATH operand to its real path: absolute,
|
// realPaths returns the real path of each of paths, the PATH operands
|
||||||
// with every symlink in it followed. Database records are keyed by
|
// or the --exclude values: absolute, with every symlink in it followed.
|
||||||
// that path, so scan results depend neither on the working directory
|
// Database records are keyed by that path, so scan results depend
|
||||||
// nor on the symlinks an operand was given through. An operand that
|
// neither on the working directory nor on the symlinks an operand was
|
||||||
// cannot be resolved, because it or a symlink's target does not exist
|
// given through. A path that cannot be resolved, because it or a
|
||||||
// or cannot be read, is a fatal error before any scanning.
|
// symlink's target does not exist or cannot be read, is a fatal error
|
||||||
func resolveRoots(roots []string) ([]string, error) {
|
// before any scanning.
|
||||||
resolved := make([]string, 0, len(roots))
|
func realPaths(paths []string) ([]string, error) {
|
||||||
|
resolved := make([]string, 0, len(paths))
|
||||||
|
|
||||||
for _, root := range roots {
|
for _, path := range paths {
|
||||||
a, err := absolutePath(root)
|
a, err := absolutePath(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
return nil, fmt.Errorf("resolve %s: %w", path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
r, err := filepath.EvalSymlinks(a)
|
r, err := filepath.EvalSymlinks(a)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
return nil, fmt.Errorf("resolve %s: %w", path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
resolved = append(resolved, r)
|
resolved = append(resolved, r)
|
||||||
@@ -295,7 +314,7 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
|
|||||||
}
|
}
|
||||||
|
|
||||||
for ; dir != above; dir = filepath.Dir(dir) {
|
for ; dir != above; dir = filepath.Dir(dir) {
|
||||||
if isDefaultExclude(dir, rules.excludes) {
|
if isDefaultExclude(dir, rules.defaultExcludes) {
|
||||||
return false
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -418,7 +437,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
|||||||
|
|
||||||
// walkableRoots returns the operands the walk can start from: regular
|
// walkableRoots returns the operands the walk can start from: regular
|
||||||
// files, and directories not named .zfs, none of them inside a
|
// files, and directories not named .zfs, none of them inside a
|
||||||
// directory named .zfs. The operands are real paths from resolveRoots,
|
// directory named .zfs. The operands are real paths from realPaths,
|
||||||
// so these checks apply to what a symlink operand points at. Every
|
// so these checks apply to what a symlink operand points at. Every
|
||||||
// other operand is warned about, counted as skipped, and dropped. A
|
// other operand is warned about, counted as skipped, and dropped. A
|
||||||
// dropped operand is no longer a root, so the records stored beneath it
|
// dropped operand is no longer a root, so the records stored beneath it
|
||||||
@@ -493,10 +512,11 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
|||||||
// as verified. When its disk_size differs from the record's, it is
|
// as verified. When its disk_size differs from the record's, it is
|
||||||
// queued for the update phase, which writes the new disk_size. A record
|
// queued for the update phase, which writes the new disk_size. A record
|
||||||
// at or below a path the walk left out, because -x stopped at it, it is
|
// at or below a path the walk left out, because -x stopped at it, it is
|
||||||
// a default exclude, or it is not on a persistent filesystem, counts as
|
// an --exclude path or a default exclude, or it is not on a persistent
|
||||||
// outside the roots: it is not deleted. That does not hold below a root
|
// filesystem, counts as outside the roots: it is not deleted. That does
|
||||||
// at or inside that path, since that root's own walk verifies the
|
// not hold below a root at or inside that path, since that root's own
|
||||||
// records below it; nor is a line printed for such a root.
|
// walk verifies the records below it; nor is a line printed for such a
|
||||||
|
// root.
|
||||||
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||||
// The records the last scan left under the roots estimate how many
|
// The records the last scan left under the roots estimate how many
|
||||||
// files the walk will find. A first scan has none, and no estimate.
|
// files the walk will find. A first scan has none, and no estimate.
|
||||||
@@ -834,9 +854,9 @@ type dirJob struct {
|
|||||||
// file's statted record, a warning when fail is set, or a path the walk
|
// file's statted record, a warning when fail is set, or a path the walk
|
||||||
// left out: when otherFSDir is set, a directory -x did not enter
|
// left out: when otherFSDir is set, a directory -x did not enter
|
||||||
// because it is on another filesystem than its operand, and when
|
// because it is on another filesystem than its operand, and when
|
||||||
// leftOut is set, a default exclude or a directory or file on a
|
// leftOut is set, an --exclude path, a default exclude or a directory or
|
||||||
// filesystem that is not persistent, with warn holding the line that
|
// file on a filesystem that is not persistent, with warn holding the
|
||||||
// says so.
|
// line that says so.
|
||||||
type walkEvent struct {
|
type walkEvent struct {
|
||||||
rec fileRec
|
rec fileRec
|
||||||
warn string
|
warn string
|
||||||
@@ -887,7 +907,7 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
|
|||||||
// operandWarning returns the one-line warning for an operand the walk
|
// operandWarning returns the one-line warning for an operand the walk
|
||||||
// does not start from, naming the path and what it is, or "" for one it
|
// does not start from, naming the path and what it is, or "" for one it
|
||||||
// does: a regular file, or a directory not named .zfs, that is not
|
// does: a regular file, or a directory not named .zfs, that is not
|
||||||
// inside a directory named .zfs. resolveRoots has already followed the
|
// inside a directory named .zfs. realPaths has already followed the
|
||||||
// operand's symlinks, so a symlink here is an operand replaced by one
|
// operand's symlinks, so a symlink here is an operand replaced by one
|
||||||
// after it was resolved, and is not followed.
|
// after it was resolved, and is not followed.
|
||||||
func operandWarning(root string, fi fs.FileInfo) string {
|
func operandWarning(root string, fi fs.FileInfo) string {
|
||||||
@@ -1118,8 +1138,9 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
|||||||
// no longer exists, and one that has stopped being a regular file is
|
// no longer exists, and one that has stopped being a regular file is
|
||||||
// skipped. A file on a filesystem in rules that is not persistent, as
|
// skipped. A file on a filesystem in rules that is not persistent, as
|
||||||
// through a bind mount of one file, is not recorded: it gets an event
|
// through a bind mount of one file, is not recorded: it gets an event
|
||||||
// naming it instead. One of the scan's own database files in rules gets
|
// naming it instead, and so does a file given with --exclude, reached
|
||||||
// no event at all.
|
// by any path, a hard link to it included. One of the scan's own
|
||||||
|
// database files in rules gets no event at all.
|
||||||
func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||||
rules walkRules, events chan<- walkEvent,
|
rules walkRules, events chan<- walkEvent,
|
||||||
) {
|
) {
|
||||||
@@ -1143,6 +1164,10 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
}
|
}
|
||||||
|
|
||||||
line := rules.nonPersistentLine(p, info)
|
line := rules.nonPersistentLine(p, info)
|
||||||
|
if line == "" {
|
||||||
|
line = rules.excludePathLine(p, info)
|
||||||
|
}
|
||||||
|
|
||||||
if line != "" {
|
if line != "" {
|
||||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
||||||
|
|
||||||
@@ -1167,11 +1192,11 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
// subdirJob applies the descent rules to directory p: never enter
|
// subdirJob applies the descent rules to directory p: never enter
|
||||||
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
|
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
|
||||||
// snapshot), never enter a directory on a filesystem in rules that is
|
// snapshot), never enter a directory on a filesystem in rules that is
|
||||||
// not persistent, nor a default exclude in rules, and with -x never
|
// not persistent, nor an --exclude path or a default exclude in rules,
|
||||||
// enter a directory on a different filesystem than its operand. For the
|
// and with -x never enter a directory on a different filesystem than
|
||||||
// last three it sends an event that names the directory, so the scan
|
// its operand. For all but .zfs it sends an event that names the
|
||||||
// keeps the records below it, except those below an operand at or
|
// directory, so the scan keeps the records below it, except those below
|
||||||
// inside it.
|
// an operand at or inside it.
|
||||||
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||||
parent dirJob, oneFS bool, rules walkRules,
|
parent dirJob, oneFS bool, rules walkRules,
|
||||||
events chan<- walkEvent,
|
events chan<- walkEvent,
|
||||||
@@ -1196,13 +1221,17 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
}
|
}
|
||||||
|
|
||||||
line := rules.nonPersistentLine(p, info)
|
line := rules.nonPersistentLine(p, info)
|
||||||
|
if line == "" {
|
||||||
|
line = rules.excludePathLine(p, info)
|
||||||
|
}
|
||||||
|
|
||||||
if line != "" {
|
if line != "" {
|
||||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
||||||
|
|
||||||
return dirJob{}, false
|
return dirJob{}, false
|
||||||
}
|
}
|
||||||
|
|
||||||
if isDefaultExclude(p, rules.excludes) {
|
if isDefaultExclude(p, rules.defaultExcludes) {
|
||||||
sendEvent(ctx, events, walkEvent{
|
sendEvent(ctx, events, walkEvent{
|
||||||
leftOut: p,
|
leftOut: p,
|
||||||
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
|
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
|
||||||
|
|||||||
@@ -2196,7 +2196,7 @@ func TestScanHashWriteFailureUnwindsPool(t *testing.T) {
|
|||||||
var stderr bytes.Buffer
|
var stderr bytes.Buffer
|
||||||
|
|
||||||
code := run([]string{
|
code := run([]string{
|
||||||
cmdScan, "--workers", strconv.Itoa(hashLeakWorkers), dir,
|
cmdScan, workersFlag, strconv.Itoa(hashLeakWorkers), dir,
|
||||||
}, io.Discard, &stderr)
|
}, io.Discard, &stderr)
|
||||||
if code != exitFatal {
|
if code != exitFatal {
|
||||||
t.Fatalf("run(scan) = %d, want %d; stderr: %s",
|
t.Fatalf("run(scan) = %d, want %d; stderr: %s",
|
||||||
@@ -2289,7 +2289,7 @@ func TestPruneRoots(t *testing.T) {
|
|||||||
|
|
||||||
// The walk of /p does not enter /p/q when it is a default exclude, so
|
// The walk of /p does not enter /p/q when it is a default exclude, so
|
||||||
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
|
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
|
||||||
got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
|
got = pruneRoots(nested, false, walkRules{defaultExcludes: []string{pqRoot}},
|
||||||
device)
|
device)
|
||||||
if !slices.Equal(got, want) {
|
if !slices.Equal(got, want) {
|
||||||
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
|
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
|
||||||
|
|||||||
Reference in new issue
Block a user