The walk neither enters nor records an --exclude path or anything below it, and prints "scan: skipped PATH (--exclude)" for each one it reaches. The flag repeats. Each value is resolved to its real path, as an operand is. The records below an excluded path are kept, as for a default exclude. An empty value, which would name the working directory, is a usage error. An --exclude path whose real path scan cannot find, or an operand at or below one, is a fatal error before the scan starts. walkRules names its two lists excludePaths and defaultExcludes. The line naming the database moved out of runScan into printDatabasePath, which keeps runScan under the lint length limit. Model: opus-5-5
This commit is contained in:
1 parent
a27d651532
commit
8b92b9b4dd
10 files changed
+369
-77
No files matched your search
@@ -252,7 +252,7 @@ In order:
|
||||
report.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
@@ -297,6 +297,13 @@ The reference below is also the program's specification.
|
||||
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
|
||||
Those are the system and temporary directories the walk otherwise leaves out,
|
||||
such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||
- `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
|
||||
it. `PATH` may be a directory or a file. The flag may be given more than once.
|
||||
`PATH` is resolved to its real path, as an operand is. An empty `PATH`, as in
|
||||
`--exclude ""`, is a usage error, reported in one line on stderr with exit 2
|
||||
before anything is scanned. A `PATH` that does not exist, or whose real path
|
||||
`scan` cannot find, is a fatal error. So is an operand at or below one (see
|
||||
"Operands").
|
||||
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
|
||||
exits 0. It writes nothing to stderr.
|
||||
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
|
||||
@@ -355,6 +362,8 @@ default. An operand may be a directory or a regular file.
|
||||
- An operand that does not exist is a fatal error, and so is a symlink whose
|
||||
target does not exist. `scan` then stops before it walks anything, and exits
|
||||
with code 1.
|
||||
- An operand at or below an `--exclude` path is a fatal error too. The message
|
||||
names the operand and the `--exclude` path. Both are compared as real paths.
|
||||
- Each operand is resolved to its real path before the walk, as `realpath`
|
||||
prints it. A relative operand is joined to the working directory as `pwd -P`
|
||||
prints it. Every symlink in the path is then followed. A `..` after a symlink
|
||||
@@ -430,15 +439,16 @@ operands:
|
||||
scanned on different schedules into one database.
|
||||
- The scan keeps every record at or below a path that the walk leaves out for
|
||||
one of these reasons:
|
||||
- it is an `--exclude` path;
|
||||
- it is a default exclude;
|
||||
- it is not on a persistent filesystem;
|
||||
- with `-x`, it is a directory on another filesystem than its operand.
|
||||
|
||||
Such records count as outside the scanned operands. So with `-x`, a
|
||||
filesystem mounted under an operand can be scanned on its own schedule into
|
||||
the same database. An operand may itself lie at or inside such a path, as
|
||||
"Operands" describes. Its walk verifies the records below it, so they are
|
||||
deleted when their files are gone.
|
||||
the same database. An operand may itself lie at or inside such a path, other
|
||||
than an `--exclude` path (see "Operands"). Its walk verifies the records
|
||||
below it, so they are deleted when their files are gone.
|
||||
|
||||
#### Phases
|
||||
|
||||
@@ -537,6 +547,17 @@ walk practical on large or busy storage pools.
|
||||
The path is not counted as skipped. The records below it are kept, as
|
||||
described under "What a scan does to the database".
|
||||
|
||||
- The walk never enters or records an `--exclude` path, nor anything below it.
|
||||
For each one it reaches, `scan` prints a line on stderr naming it:
|
||||
|
||||
```
|
||||
scan: skipped /srv/cache (--exclude)
|
||||
```
|
||||
|
||||
The path is not counted as skipped. The scan keeps the records at and below
|
||||
it, as described under "What a scan does to the database".
|
||||
`--no-default-excludes` does not change this rule.
|
||||
|
||||
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
||||
default excludes hold temporary files and the kernel's view of the running
|
||||
system. They depend on the system, and `sfdupes scan --help` lists those of
|
||||
@@ -611,7 +632,8 @@ from a thread of its own and never blocks the workers.
|
||||
it repeats another or lies under another;
|
||||
- the progress display of each phase (see "Progress"). `scan` prints each
|
||||
warning for a skipped path as it happens. It prints a line the same way for
|
||||
each path left out as a default exclude or as not on a persistent filesystem;
|
||||
each path left out as an `--exclude` path, as a default exclude or as not on a
|
||||
persistent filesystem;
|
||||
- the summary line;
|
||||
- a warning when a report still has the database open as the scan ends (see
|
||||
"Concurrent reads").
|
||||
@@ -1178,6 +1200,9 @@ showing the count it stopped at.
|
||||
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
|
||||
stopped by either during its scan. A fatal error is one of these:
|
||||
- an operand that does not exist, or whose real path `scan` cannot find;
|
||||
- an `--exclude` path that does not exist, or whose real path `scan` cannot
|
||||
find;
|
||||
- an operand at or below an `--exclude` path;
|
||||
- a list of mounted filesystems that `scan` cannot read;
|
||||
- another `scan` holding the lock (see "The scan lock");
|
||||
- a database, its directory or its lock file that cannot be created or
|
||||
@@ -1197,6 +1222,7 @@ showing the count it stopped at.
|
||||
`--workers abc`;
|
||||
- `scan` with no operand;
|
||||
- `--workers` below 1;
|
||||
- an empty `--exclude` value, as in `--exclude ""`;
|
||||
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
||||
|
||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||
|
||||
@@ -29,6 +29,11 @@
|
||||
Newest first. An entry ends with its date and its issue or branch, where those
|
||||
are known.
|
||||
|
||||
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
|
||||
enters nor records an excluded path, and keeps the records below it. An
|
||||
operand at or below one is a fatal error (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/163)
|
||||
|
||||
- The usage line of `scan` no longer ends in `[flags]`. It is the line
|
||||
`README.md` shows under "Subcommands" (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/169)
|
||||
|
||||
@@ -359,7 +359,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
dir := buildSmokeTree(t)
|
||||
|
||||
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
||||
false)
|
||||
false, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -383,7 +383,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
defer close(done)
|
||||
|
||||
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
|
||||
false, false)
|
||||
false, false, nil)
|
||||
}()
|
||||
|
||||
awaitReturn(t, done, "runScan")
|
||||
@@ -411,7 +411,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
}
|
||||
|
||||
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
||||
false)
|
||||
false, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -437,7 +437,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
|
||||
dir := buildWalkCancelTree(t)
|
||||
|
||||
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
|
||||
walkCancelWorkers, false, false)
|
||||
walkCancelWorkers, false, false, nil)
|
||||
if !errors.Is(err, errInterrupted) {
|
||||
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
|
||||
err, errInterrupted)
|
||||
|
||||
+36
-7
@@ -10,6 +10,7 @@ import (
|
||||
"os"
|
||||
"runtime"
|
||||
"runtime/debug"
|
||||
"slices"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
@@ -40,6 +41,11 @@ var errNoSubcommand = errors.New("no subcommand")
|
||||
// below 1.
|
||||
var errWorkersBelowOne = errors.New("--workers must be at least 1")
|
||||
|
||||
// errEmptyExcludeValue is the usage error for an empty scan --exclude
|
||||
// value, as from --exclude "$DIR" with DIR unset. Taken as a path, it
|
||||
// would name the working directory.
|
||||
var errEmptyExcludeValue = errors.New("--exclude must not be empty")
|
||||
|
||||
// Version is the build version, injected at link time via -ldflags
|
||||
// (see the Makefile); empty when the linker set none, as for a binary
|
||||
// built by go install. buildVersion turns it into what --version prints.
|
||||
@@ -201,21 +207,23 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
return root
|
||||
}
|
||||
|
||||
// newScanCommand builds the scan subcommand, with its --workers, -x and
|
||||
// --no-default-excludes flags. Its help lists the default excludes of
|
||||
// the system it runs on, and the filesystem types it never records
|
||||
// anything on.
|
||||
// newScanCommand builds the scan subcommand, with its --workers, -x,
|
||||
// --no-default-excludes and --exclude flags. Its help lists the default
|
||||
// excludes of the system it runs on, and the filesystem types it never
|
||||
// records anything on.
|
||||
func newScanCommand() *cobra.Command {
|
||||
var (
|
||||
scanWorkers int
|
||||
scanOneFS bool
|
||||
scanNoDefaultExcludes bool
|
||||
scanExclude []string
|
||||
)
|
||||
|
||||
short := "Walk trees and synchronize the scan database"
|
||||
|
||||
scanCmd := &cobra.Command{
|
||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]... PATH...",
|
||||
// The Use line already names every flag, so cobra must not
|
||||
// append its generic "[flags]" to it.
|
||||
DisableFlagsInUseLine: true,
|
||||
@@ -223,14 +231,19 @@ func newScanCommand() *cobra.Command {
|
||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return checkScanWorkers(cmd, scanWorkers)
|
||||
err := checkScanWorkers(cmd, scanWorkers)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return rejectEmptyExcludeValue(cmd, scanExclude)
|
||||
},
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
ctx, stop := interruptContext(ctx)
|
||||
defer stop()
|
||||
|
||||
return runScan(ctx, args, scanWorkers, scanOneFS,
|
||||
scanNoDefaultExcludes)
|
||||
scanNoDefaultExcludes, scanExclude)
|
||||
}),
|
||||
}
|
||||
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
|
||||
@@ -239,6 +252,10 @@ func newScanCommand() *cobra.Command {
|
||||
"do not cross filesystem boundaries")
|
||||
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
|
||||
false, "enter the default excludes, the paths listed above")
|
||||
// A string array, not a slice, so a comma in a path is not a
|
||||
// separator.
|
||||
scanCmd.Flags().StringArrayVar(&scanExclude, "exclude", nil,
|
||||
"never enter or record `PATH` or anything below it; repeatable")
|
||||
|
||||
return scanCmd
|
||||
}
|
||||
@@ -256,6 +273,18 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
|
||||
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
|
||||
}
|
||||
|
||||
// rejectEmptyExcludeValue rejects an empty scan --exclude value. That
|
||||
// is a usage error reported in one line, as for --workers below 1.
|
||||
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
|
||||
if !slices.Contains(excludePaths, "") {
|
||||
return nil
|
||||
}
|
||||
|
||||
cmd.SilenceUsage = true
|
||||
|
||||
return errEmptyExcludeValue
|
||||
}
|
||||
|
||||
// runE adapts a subcommand implementation, or the version print, to
|
||||
// cobra's RunE. Cobra prints the error and the command's usage text for
|
||||
// every error RunE returns, but a subcommand that ran and failed has no
|
||||
|
||||
@@ -570,7 +570,8 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
}{
|
||||
cmdScan: {
|
||||
run: func() error {
|
||||
return runScan(t.Context(), []string{missing}, 1, false, false)
|
||||
return runScan(t.Context(), []string{missing}, 1, false, false,
|
||||
nil)
|
||||
},
|
||||
want: scanWant,
|
||||
},
|
||||
@@ -712,6 +713,16 @@ func TestRunHelp(t *testing.T) {
|
||||
args, got)
|
||||
}
|
||||
}
|
||||
|
||||
// The scan help lists --exclude with its argument.
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
run([]string{cmdScan, "-h"}, &stdout, &stderr)
|
||||
|
||||
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
|
||||
t.Errorf("scan -h stderr = %q, want %s PATH", stderr.String(),
|
||||
excludeFlag)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunScanUsageLine(t *testing.T) {
|
||||
@@ -719,7 +730,8 @@ func TestRunScanUsageLine(t *testing.T) {
|
||||
|
||||
// README §Subcommands shows this usage line for scan, and the help
|
||||
// and the usage error both print it.
|
||||
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...\n"
|
||||
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]... PATH...\n"
|
||||
|
||||
cases := map[string][]string{
|
||||
"help": {cmdScan, "-h"},
|
||||
|
||||
@@ -17,7 +17,8 @@ import (
|
||||
// ends it at once, as it ends trees.
|
||||
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
||||
scanCtx, stop := interruptContext(ctx)
|
||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
|
||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false,
|
||||
nil)
|
||||
|
||||
stop()
|
||||
|
||||
|
||||
+43
-11
@@ -1,8 +1,9 @@
|
||||
package sfdupes
|
||||
|
||||
// This file holds two rules for the walk (README "Rules for the walk"):
|
||||
// This file holds three rules for the walk (README "Rules for the
|
||||
// walk"): the paths given with --exclude, which the scan never enters;
|
||||
// the default excludes, the paths a scan's walk does not enter below an
|
||||
// operand unless scan is given --no-default-excludes, and the
|
||||
// operand unless scan is given --no-default-excludes; and the
|
||||
// filesystems that are not persistent, on which the scan never records
|
||||
// or reads anything.
|
||||
|
||||
@@ -43,17 +44,23 @@ const typesPerHelpLine = 6
|
||||
// readMountInfo reads.
|
||||
var errMountInfoLine = errors.New("unexpected line")
|
||||
|
||||
// errExcludedOperand reports an operand at or below a path given with
|
||||
// --exclude.
|
||||
var errExcludedOperand = errors.New("operand excluded by --exclude")
|
||||
|
||||
// walkRules holds the rules for the walk that a scan sets up as it
|
||||
// starts. excludes are the default excludes, or none with
|
||||
// --no-default-excludes. nonPersistent maps the device ID of every
|
||||
// mounted filesystem that is not persistent to its type; the scan
|
||||
// records and reads nothing on one of those, whatever the flags.
|
||||
// databaseFiles are the scan's own database files, which the walk
|
||||
// passes over (see databaseFiles in db.go).
|
||||
// starts. excludePaths are the real paths given with --exclude, which
|
||||
// the walk neither enters nor records. defaultExcludes are the default
|
||||
// excludes, or none with --no-default-excludes. nonPersistent maps the
|
||||
// device ID of every mounted filesystem that is not persistent to its
|
||||
// type; the scan records and reads nothing on one of those, whatever
|
||||
// the flags. databaseFiles are the scan's own database files, which the
|
||||
// walk passes over (see databaseFiles in db.go).
|
||||
type walkRules struct {
|
||||
excludes []string
|
||||
nonPersistent map[uint64]string
|
||||
databaseFiles []fileRec
|
||||
excludePaths []string
|
||||
defaultExcludes []string
|
||||
nonPersistent map[uint64]string
|
||||
databaseFiles []fileRec
|
||||
}
|
||||
|
||||
// defaultExcludes returns the default excludes of the system goos
|
||||
@@ -97,6 +104,31 @@ func isDefaultExclude(path string, excludes []string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// checkOperandsNotExcluded returns an error when one of roots, the
|
||||
// operands, is at or below one of excludePaths, the --exclude values.
|
||||
// The error names both. Both lists hold real paths.
|
||||
func checkOperandsNotExcluded(roots, excludePaths []string) error {
|
||||
for _, root := range roots {
|
||||
exclude := deepestRoot(root, excludePaths)
|
||||
if exclude != "" {
|
||||
return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand,
|
||||
exclude)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// excludePathLine returns the line the scan prints for path when it is
|
||||
// one of rules.excludePaths, and "" when it is not.
|
||||
func (rules walkRules) excludePathLine(path string) string {
|
||||
if !slices.Contains(rules.excludePaths, path) {
|
||||
return ""
|
||||
}
|
||||
|
||||
return fmt.Sprintf("scan: skipped %s (--exclude)", path)
|
||||
}
|
||||
|
||||
// nonPersistentFilesystemTypes returns the filesystem types that are
|
||||
// not persistent: filesystems that the kernel makes up or that are kept
|
||||
// in memory. devfs is the macOS filesystem of /dev; the others are
|
||||
|
||||
@@ -14,6 +14,9 @@ import (
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// excludeFlag is the scan flag that names a path the walk never enters.
|
||||
const excludeFlag = "--exclude"
|
||||
|
||||
func TestDefaultExcludes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -188,7 +191,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
|
||||
unseen := writeFile(t, excluded, "new", pattern(2, 10))
|
||||
|
||||
st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
|
||||
walkRules{excludes: []string{excluded}})
|
||||
walkRules{defaultExcludes: []string{excluded}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -234,7 +237,7 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
|
||||
st, err := syncScan(t.Context(), db, roots, 1, false,
|
||||
walkRules{excludes: []string{excluded}})
|
||||
walkRules{defaultExcludes: []string{excluded}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -262,7 +265,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
|
||||
f := writeFile(t, excluded, "f", pattern(1, 10))
|
||||
|
||||
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
|
||||
walkRules{excludes: []string{excluded}})
|
||||
walkRules{defaultExcludes: []string{excluded}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -356,6 +359,162 @@ func TestRunScanNoDefaultExcludes(t *testing.T) {
|
||||
assertRecorded(t, path, f, true)
|
||||
}
|
||||
|
||||
// TestRunScanExcludeNested scans a tree through the command with an
|
||||
// --exclude path two levels below the operand. The walk records nothing
|
||||
// at or below it and says so. The record an earlier scan left there is
|
||||
// kept although its file is gone.
|
||||
func TestRunScanExcludeNested(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
excluded := filepath.Join(dir, "a", "b")
|
||||
kept := writeFile(t, dir, "a/f", pattern(1, 10))
|
||||
gone := writeFile(t, excluded, "gone", pattern(2, 10))
|
||||
|
||||
scanOK(t, dir)
|
||||
|
||||
err := os.Remove(gone)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
unseen := writeFile(t, excluded, "c/new", pattern(3, 10))
|
||||
|
||||
stderr := scanOK(t, excludeFlag, excluded, dir)
|
||||
|
||||
line := "scan: skipped " + excluded + " (--exclude)\n"
|
||||
if !strings.Contains(stderr, line) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||
}
|
||||
|
||||
assertRecorded(t, path, kept, true)
|
||||
assertRecorded(t, path, gone, true)
|
||||
assertRecorded(t, path, unseen, false)
|
||||
}
|
||||
|
||||
// TestRunScanExcludeRepeated gives --exclude twice, once for a
|
||||
// directory and once for a file, the file through a symlink to it. The
|
||||
// walk records neither, and records the file beside them.
|
||||
func TestRunScanExcludeRepeated(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
inDir := writeFile(t, dir, "skip/f", pattern(1, 10))
|
||||
file := writeFile(t, dir, "skip-file", pattern(2, 10))
|
||||
kept := writeFile(t, dir, "keep", pattern(3, 10))
|
||||
|
||||
link := filepath.Join(t.TempDir(), "link")
|
||||
symlink(t, file, link)
|
||||
|
||||
scanOK(t, excludeFlag, filepath.Dir(inDir), excludeFlag, link, dir)
|
||||
|
||||
assertRecorded(t, path, inDir, false)
|
||||
assertRecorded(t, path, file, false)
|
||||
assertRecorded(t, path, kept, true)
|
||||
}
|
||||
|
||||
// TestRunScanExcludedOperand checks that an operand at or below an
|
||||
// --exclude path is a fatal error that names both, and so is an
|
||||
// --exclude path that does not exist. Neither scan writes a database.
|
||||
func TestRunScanExcludedOperand(t *testing.T) {
|
||||
dir := realTempDir(t)
|
||||
sub := filepath.Join(dir, "sub")
|
||||
writeFile(t, sub, "f", pattern(1, 10))
|
||||
|
||||
cases := map[string]struct {
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
"at it": {
|
||||
[]string{excludeFlag, sub, sub},
|
||||
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
|
||||
},
|
||||
"below it": {
|
||||
[]string{excludeFlag, dir, sub},
|
||||
"scan " + sub + ": operand excluded by --exclude " + dir + "\n",
|
||||
},
|
||||
"after an operand it does not exclude": {
|
||||
[]string{dir, excludeFlag, sub, sub},
|
||||
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
|
||||
},
|
||||
"exclude missing": {
|
||||
[]string{excludeFlag, filepath.Join(dir, "nope"), dir},
|
||||
"--exclude: resolve " + filepath.Join(dir, "nope") + ": ",
|
||||
},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
stdout := captureStdout(t)
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run(append([]string{cmdScan}, tc.args...), os.Stdout,
|
||||
os.Stderr)
|
||||
if code != exitFatal {
|
||||
t.Errorf("run(scan %q) = %d, want %d", tc.args, code,
|
||||
exitFatal)
|
||||
}
|
||||
|
||||
assertFatalOutput(t, stderr(), stdout())
|
||||
|
||||
if !strings.Contains(stderr(), "sfdupes: "+tc.want) {
|
||||
t.Errorf("stderr = %q, want %q", stderr(), tc.want)
|
||||
}
|
||||
|
||||
_, err := os.Lstat(path)
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanRejectsEmptyExclude checks that an empty --exclude value,
|
||||
// given as its own argument, after "=" or after another --exclude, is
|
||||
// a usage error reported in one line before anything is scanned
|
||||
// (README "Settings").
|
||||
func TestRunScanRejectsEmptyExclude(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
cases := map[string][]string{
|
||||
"separate": {excludeFlag, "", dir},
|
||||
"joined by =": {excludeFlag + "=", dir},
|
||||
"after a path": {excludeFlag, t.TempDir(), excludeFlag, "", dir},
|
||||
}
|
||||
|
||||
for name, args := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := run(append([]string{cmdScan}, args...), &stdout,
|
||||
&stderr)
|
||||
if code != exitUsage {
|
||||
t.Errorf("run(scan %q) = %d, want %d", args, code,
|
||||
exitUsage)
|
||||
}
|
||||
|
||||
want := "Error: --exclude must not be empty\n"
|
||||
if stdout.String() != "" || stderr.String() != want {
|
||||
t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
|
||||
stdout.String(), stderr.String(), want)
|
||||
}
|
||||
|
||||
_, err := os.Lstat(path)
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanSkipsNonPersistentOperand scans, through the command, a
|
||||
// directory on a filesystem that is not persistent, with and without
|
||||
// --no-default-excludes: either way it is skipped, with a line saying
|
||||
|
||||
+71
-43
@@ -102,28 +102,31 @@ func scanPhase(name string) string {
|
||||
// scan command rejects anything less. Before the lock is taken, the
|
||||
// scan lists the filesystems that are not persistent, on which the walk
|
||||
// records nothing; unless noDefaultExcludes is set, the walk does not
|
||||
// enter the default excludes either. Once the database is open, the
|
||||
// scan notes its files, which the walk passes over.
|
||||
// enter the default excludes either. The walk neither enters nor
|
||||
// records the paths in excludePaths, the --exclude values; an operand
|
||||
// at or below one is a fatal error. Once the database is open, the scan
|
||||
// notes its files, which the walk passes over.
|
||||
func runScan(ctx context.Context, roots []string, workers int,
|
||||
oneFS, noDefaultExcludes bool,
|
||||
oneFS, noDefaultExcludes bool, excludePaths []string,
|
||||
) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// A database file missing now is created by lockScanDatabase below.
|
||||
what := "database"
|
||||
printDatabasePath(dbPath, source)
|
||||
|
||||
_, err = os.Stat(dbPath)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
what = "new database"
|
||||
roots, err = realPaths(roots)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
|
||||
source)
|
||||
excludePaths, err = realPaths(excludePaths)
|
||||
if err != nil {
|
||||
return fmt.Errorf("--exclude: %w", err)
|
||||
}
|
||||
|
||||
roots, err = resolveRoots(roots)
|
||||
err = checkOperandsNotExcluded(roots, excludePaths)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -133,9 +136,9 @@ func runScan(ctx context.Context, roots []string, workers int,
|
||||
return fmt.Errorf("list the mounted filesystems: %w", err)
|
||||
}
|
||||
|
||||
rules := walkRules{nonPersistent: nonPersistent}
|
||||
rules := walkRules{excludePaths: excludePaths, nonPersistent: nonPersistent}
|
||||
if !noDefaultExcludes {
|
||||
rules.excludes = defaultExcludes(runtime.GOOS)
|
||||
rules.defaultExcludes = defaultExcludes(runtime.GOOS)
|
||||
}
|
||||
|
||||
lock, err := lockScanDatabase(dbPath)
|
||||
@@ -181,6 +184,21 @@ func runScan(ctx context.Context, roots []string, workers int,
|
||||
return nil
|
||||
}
|
||||
|
||||
// printDatabasePath prints on stderr the line that names the database
|
||||
// at dbPath, with source, saying whether it is new. A database file
|
||||
// missing now is new: lockScanDatabase creates it.
|
||||
func printDatabasePath(dbPath, source string) {
|
||||
what := "database"
|
||||
|
||||
_, err := os.Stat(dbPath)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
what = "new database"
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
|
||||
source)
|
||||
}
|
||||
|
||||
// interruptContext returns a copy of ctx that the first SIGINT or
|
||||
// SIGTERM cancels; the scan command runs the scan under it. stop
|
||||
// releases the signals.
|
||||
@@ -209,24 +227,25 @@ func interrupted(walked int) error {
|
||||
return errInterrupted
|
||||
}
|
||||
|
||||
// resolveRoots converts each PATH operand to its real path: absolute,
|
||||
// with every symlink in it followed. Database records are keyed by
|
||||
// that path, so scan results depend neither on the working directory
|
||||
// nor on the symlinks an operand was given through. An operand that
|
||||
// cannot be resolved, because it or a symlink's target does not exist
|
||||
// or cannot be read, is a fatal error before any scanning.
|
||||
func resolveRoots(roots []string) ([]string, error) {
|
||||
resolved := make([]string, 0, len(roots))
|
||||
// realPaths returns the real path of each of paths, the PATH operands
|
||||
// or the --exclude values: absolute, with every symlink in it followed.
|
||||
// Database records are keyed by that path, so scan results depend
|
||||
// neither on the working directory nor on the symlinks an operand was
|
||||
// given through. A path that cannot be resolved, because it or a
|
||||
// symlink's target does not exist or cannot be read, is a fatal error
|
||||
// before any scanning.
|
||||
func realPaths(paths []string) ([]string, error) {
|
||||
resolved := make([]string, 0, len(paths))
|
||||
|
||||
for _, root := range roots {
|
||||
a, err := absolutePath(root)
|
||||
for _, path := range paths {
|
||||
a, err := absolutePath(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
return nil, fmt.Errorf("resolve %s: %w", path, err)
|
||||
}
|
||||
|
||||
r, err := filepath.EvalSymlinks(a)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
return nil, fmt.Errorf("resolve %s: %w", path, err)
|
||||
}
|
||||
|
||||
resolved = append(resolved, r)
|
||||
@@ -295,7 +314,7 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
|
||||
}
|
||||
|
||||
for ; dir != above; dir = filepath.Dir(dir) {
|
||||
if isDefaultExclude(dir, rules.excludes) {
|
||||
if isDefaultExclude(dir, rules.defaultExcludes) {
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -418,7 +437,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
||||
|
||||
// walkableRoots returns the operands the walk can start from: regular
|
||||
// files, and directories not named .zfs, none of them inside a
|
||||
// directory named .zfs. The operands are real paths from resolveRoots,
|
||||
// directory named .zfs. The operands are real paths from realPaths,
|
||||
// so these checks apply to what a symlink operand points at. Every
|
||||
// other operand is warned about, counted as skipped, and dropped. A
|
||||
// dropped operand is no longer a root, so the records stored beneath it
|
||||
@@ -493,10 +512,11 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||
// as verified. When its disk_size differs from the record's, it is
|
||||
// queued for the update phase, which writes the new disk_size. A record
|
||||
// at or below a path the walk left out, because -x stopped at it, it is
|
||||
// a default exclude, or it is not on a persistent filesystem, counts as
|
||||
// outside the roots: it is not deleted. That does not hold below a root
|
||||
// at or inside that path, since that root's own walk verifies the
|
||||
// records below it; nor is a line printed for such a root.
|
||||
// an --exclude path or a default exclude, or it is not on a persistent
|
||||
// filesystem, counts as outside the roots: it is not deleted. That does
|
||||
// not hold below a root at or inside that path, since that root's own
|
||||
// walk verifies the records below it; nor is a line printed for such a
|
||||
// root.
|
||||
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
// The records the last scan left under the roots estimate how many
|
||||
// files the walk will find. A first scan has none, and no estimate.
|
||||
@@ -834,9 +854,9 @@ type dirJob struct {
|
||||
// file's statted record, a warning when fail is set, or a path the walk
|
||||
// left out: when otherFSDir is set, a directory -x did not enter
|
||||
// because it is on another filesystem than its operand, and when
|
||||
// leftOut is set, a default exclude or a directory or file on a
|
||||
// filesystem that is not persistent, with warn holding the line that
|
||||
// says so.
|
||||
// leftOut is set, an --exclude path, a default exclude or a directory or
|
||||
// file on a filesystem that is not persistent, with warn holding the
|
||||
// line that says so.
|
||||
type walkEvent struct {
|
||||
rec fileRec
|
||||
warn string
|
||||
@@ -887,7 +907,7 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
|
||||
// operandWarning returns the one-line warning for an operand the walk
|
||||
// does not start from, naming the path and what it is, or "" for one it
|
||||
// does: a regular file, or a directory not named .zfs, that is not
|
||||
// inside a directory named .zfs. resolveRoots has already followed the
|
||||
// inside a directory named .zfs. realPaths has already followed the
|
||||
// operand's symlinks, so a symlink here is an operand replaced by one
|
||||
// after it was resolved, and is not followed.
|
||||
func operandWarning(root string, fi fs.FileInfo) string {
|
||||
@@ -1118,8 +1138,8 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
||||
// no longer exists, and one that has stopped being a regular file is
|
||||
// skipped. A file on a filesystem in rules that is not persistent, as
|
||||
// through a bind mount of one file, is not recorded: it gets an event
|
||||
// naming it instead. One of the scan's own database files in rules gets
|
||||
// no event at all.
|
||||
// naming it instead, and so does a file given with --exclude. One of
|
||||
// the scan's own database files in rules gets no event at all.
|
||||
func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
rules walkRules, events chan<- walkEvent,
|
||||
) {
|
||||
@@ -1143,6 +1163,10 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
}
|
||||
|
||||
line := rules.nonPersistentLine(p, info)
|
||||
if line == "" {
|
||||
line = rules.excludePathLine(p)
|
||||
}
|
||||
|
||||
if line != "" {
|
||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
||||
|
||||
@@ -1167,11 +1191,11 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
// subdirJob applies the descent rules to directory p: never enter
|
||||
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
|
||||
// snapshot), never enter a directory on a filesystem in rules that is
|
||||
// not persistent, nor a default exclude in rules, and with -x never
|
||||
// enter a directory on a different filesystem than its operand. For the
|
||||
// last three it sends an event that names the directory, so the scan
|
||||
// keeps the records below it, except those below an operand at or
|
||||
// inside it.
|
||||
// not persistent, nor an --exclude path or a default exclude in rules,
|
||||
// and with -x never enter a directory on a different filesystem than
|
||||
// its operand. For all but .zfs it sends an event that names the
|
||||
// directory, so the scan keeps the records below it, except those below
|
||||
// an operand at or inside it.
|
||||
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||
parent dirJob, oneFS bool, rules walkRules,
|
||||
events chan<- walkEvent,
|
||||
@@ -1196,13 +1220,17 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||
}
|
||||
|
||||
line := rules.nonPersistentLine(p, info)
|
||||
if line == "" {
|
||||
line = rules.excludePathLine(p)
|
||||
}
|
||||
|
||||
if line != "" {
|
||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
||||
|
||||
return dirJob{}, false
|
||||
}
|
||||
|
||||
if isDefaultExclude(p, rules.excludes) {
|
||||
if isDefaultExclude(p, rules.defaultExcludes) {
|
||||
sendEvent(ctx, events, walkEvent{
|
||||
leftOut: p,
|
||||
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
|
||||
|
||||
@@ -2289,7 +2289,7 @@ func TestPruneRoots(t *testing.T) {
|
||||
|
||||
// The walk of /p does not enter /p/q when it is a default exclude, so
|
||||
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
|
||||
got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
|
||||
got = pruneRoots(nested, false, walkRules{defaultExcludes: []string{pqRoot}},
|
||||
device)
|
||||
if !slices.Equal(got, want) {
|
||||
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
|
||||
|
||||
Reference in new issue
Block a user