Compare commits
4
Commits
v0.0.4
...
8b92b9b4dd
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8b92b9b4dd | ||
|
|
a27d651532 | ||
|
|
a1ec1e0ca4 | ||
|
|
6c529fcc6d |
No files matched your search
@@ -208,8 +208,8 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
|
||||
of reads, at the price of a comparison that is not exact.
|
||||
|
||||
The hashes live in a database that is kept between runs. A rescan reads only new
|
||||
files, and files whose size changed or whose modification time (mtime) is newer
|
||||
than recorded. All analysis then works from the database alone.
|
||||
files, and files whose size or modification time (mtime) differs from their
|
||||
record. All analysis then works from the database alone.
|
||||
|
||||
The goal is whole duplicated trees, not only single files. A download extracted
|
||||
twice, or a project copied once too often, is such a tree. It can be removed as
|
||||
@@ -252,7 +252,7 @@ In order:
|
||||
report.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
@@ -297,6 +297,13 @@ The reference below is also the program's specification.
|
||||
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
|
||||
Those are the system and temporary directories the walk otherwise leaves out,
|
||||
such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||
- `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
|
||||
it. `PATH` may be a directory or a file. The flag may be given more than once.
|
||||
`PATH` is resolved to its real path, as an operand is. An empty `PATH`, as in
|
||||
`--exclude ""`, is a usage error, reported in one line on stderr with exit 2
|
||||
before anything is scanned. A `PATH` that does not exist, or whose real path
|
||||
`scan` cannot find, is a fatal error. So is an operand at or below one (see
|
||||
"Operands").
|
||||
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
|
||||
exits 0. It writes nothing to stderr.
|
||||
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
|
||||
@@ -355,6 +362,8 @@ default. An operand may be a directory or a regular file.
|
||||
- An operand that does not exist is a fatal error, and so is a symlink whose
|
||||
target does not exist. `scan` then stops before it walks anything, and exits
|
||||
with code 1.
|
||||
- An operand at or below an `--exclude` path is a fatal error too. The message
|
||||
names the operand and the `--exclude` path. Both are compared as real paths.
|
||||
- Each operand is resolved to its real path before the walk, as `realpath`
|
||||
prints it. A relative operand is joined to the working directory as `pwd -P`
|
||||
prints it. Every symlink in the path is then followed. A `..` after a symlink
|
||||
@@ -410,17 +419,16 @@ operands:
|
||||
|
||||
- A file not yet in the database is hashed and inserted.
|
||||
- A file already in the database is not read again when its size and mtime match
|
||||
its record. The mtime matches when it is not newer than the recorded one. This
|
||||
is what makes a daily rescan cheap.
|
||||
its record. This is what makes a daily rescan cheap.
|
||||
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
|
||||
updates `disk_size` alone. It still does not read the file, and counts it as
|
||||
unchanged. ZFS, for example, counts a new file's blocks up to seconds after
|
||||
the write. A block is the unit in which a filesystem allocates disk space. A
|
||||
file scanned within those seconds gets its real `disk_size` at the next scan.
|
||||
- `scan` hashes a file again when its size differs or its mtime is newer. The
|
||||
mtime is compared to the nanosecond, as finely as the filesystem records it. A
|
||||
same-size rewrite counts as a change whenever the filesystem gives it a later
|
||||
mtime, even within the same second.
|
||||
- `scan` hashes a file again when its size or its mtime differs from its record.
|
||||
An older mtime counts as much as a newer one. A file put back by `cp -p`,
|
||||
`rsync -a` or a restore from backup often has an older mtime. The mtime is
|
||||
compared to the nanosecond, as finely as the filesystem records it.
|
||||
- The scan deletes each record under a scanned operand whose file it could not
|
||||
verify. That covers deleted files, and paths the scan could not stat or read
|
||||
this run. To stat a file is to read its details, such as its size, without
|
||||
@@ -431,15 +439,16 @@ operands:
|
||||
scanned on different schedules into one database.
|
||||
- The scan keeps every record at or below a path that the walk leaves out for
|
||||
one of these reasons:
|
||||
- it is an `--exclude` path;
|
||||
- it is a default exclude;
|
||||
- it is not on a persistent filesystem;
|
||||
- with `-x`, it is a directory on another filesystem than its operand.
|
||||
|
||||
Such records count as outside the scanned operands. So with `-x`, a
|
||||
filesystem mounted under an operand can be scanned on its own schedule into
|
||||
the same database. An operand may itself lie at or inside such a path, as
|
||||
"Operands" describes. Its walk verifies the records below it, so they are
|
||||
deleted when their files are gone.
|
||||
the same database. An operand may itself lie at or inside such a path, other
|
||||
than an `--exclude` path (see "Operands"). Its walk verifies the records
|
||||
below it, so they are deleted when their files are gone.
|
||||
|
||||
#### Phases
|
||||
|
||||
@@ -538,6 +547,17 @@ walk practical on large or busy storage pools.
|
||||
The path is not counted as skipped. The records below it are kept, as
|
||||
described under "What a scan does to the database".
|
||||
|
||||
- The walk never enters or records an `--exclude` path, nor anything below it.
|
||||
For each one it reaches, `scan` prints a line on stderr naming it:
|
||||
|
||||
```
|
||||
scan: skipped /srv/cache (--exclude)
|
||||
```
|
||||
|
||||
The path is not counted as skipped. The scan keeps the records at and below
|
||||
it, as described under "What a scan does to the database".
|
||||
`--no-default-excludes` does not change this rule.
|
||||
|
||||
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
||||
default excludes hold temporary files and the kernel's view of the running
|
||||
system. They depend on the system, and `sfdupes scan --help` lists those of
|
||||
@@ -612,7 +632,8 @@ from a thread of its own and never blocks the workers.
|
||||
it repeats another or lies under another;
|
||||
- the progress display of each phase (see "Progress"). `scan` prints each
|
||||
warning for a skipped path as it happens. It prints a line the same way for
|
||||
each path left out as a default exclude or as not on a persistent filesystem;
|
||||
each path left out as an `--exclude` path, as a default exclude or as not on a
|
||||
persistent filesystem;
|
||||
- the summary line;
|
||||
- a warning when a report still has the database open as the scan ends (see
|
||||
"Concurrent reads").
|
||||
@@ -665,6 +686,7 @@ Processing:
|
||||
- Records are grouped by signature, the four values
|
||||
`(size, head, tail, content)`.
|
||||
- Every group with two or more paths is a duplicate group.
|
||||
- Empty files form no group, because removing one frees nothing.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
every other path is a `dupe`.
|
||||
- Groups are ordered by size, largest first, then by `first` path. The output is
|
||||
@@ -713,7 +735,7 @@ The summary line on stderr gives:
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
|
||||
```
|
||||
report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable
|
||||
report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
||||
```
|
||||
|
||||
### `trees` mode
|
||||
@@ -741,6 +763,9 @@ Processing:
|
||||
- `trees` builds the directory tree and computes every directory's digest. It
|
||||
then groups the directories by digest. Every group with two or more
|
||||
directories is a duplicate-tree group.
|
||||
- `trees` leaves out a group whose trees hold 0 bytes, because removing one
|
||||
frees nothing. An empty file still counts toward the digest of a tree that
|
||||
holds other files.
|
||||
- `trees` leaves out a group that a group of its members' parents already
|
||||
implies. A group is left out when its members' parents are distinct
|
||||
directories that all have the same digest. The parents' group, or an
|
||||
@@ -1175,6 +1200,9 @@ showing the count it stopped at.
|
||||
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
|
||||
stopped by either during its scan. A fatal error is one of these:
|
||||
- an operand that does not exist, or whose real path `scan` cannot find;
|
||||
- an `--exclude` path that does not exist, or whose real path `scan` cannot
|
||||
find;
|
||||
- an operand at or below an `--exclude` path;
|
||||
- a list of mounted filesystems that `scan` cannot read;
|
||||
- another `scan` holding the lock (see "The scan lock");
|
||||
- a database, its directory or its lock file that cannot be created or
|
||||
@@ -1194,6 +1222,7 @@ showing the count it stopped at.
|
||||
`--workers abc`;
|
||||
- `scan` with no operand;
|
||||
- `--workers` below 1;
|
||||
- an empty `--exclude` value, as in `--exclude ""`;
|
||||
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
||||
|
||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||
@@ -1382,7 +1411,7 @@ All of the following, run in this directory, must pass:
|
||||
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
||||
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
||||
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
||||
: > "$d/empty1"; : > "$d/empty2" # empty duplicates
|
||||
: > "$d/empty1"; : > "$d/empty2" # empty files
|
||||
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
||||
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
||||
head -c 3000 /dev/urandom > "$d/t1/f1"
|
||||
@@ -1407,8 +1436,8 @@ All of the following, run in this directory, must pass:
|
||||
is the path that sorts first.
|
||||
- `t1/f1`, `t2/f1` and `t3/f1` form one group.
|
||||
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
|
||||
- `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`.
|
||||
- `unique.bin` and `tiny3` appear nowhere.
|
||||
- `tiny1` and `tiny2` form a pair.
|
||||
- `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
|
||||
- The groups come in order of size, largest first.
|
||||
|
||||
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
|
||||
|
||||
@@ -29,6 +29,18 @@
|
||||
Newest first. An entry ends with its date and its issue or branch, where those
|
||||
are known.
|
||||
|
||||
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
|
||||
enters nor records an excluded path, and keeps the records below it. An
|
||||
operand at or below one is a fatal error (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/163)
|
||||
|
||||
- The usage line of `scan` no longer ends in `[flags]`. It is the line
|
||||
`README.md` shows under "Subcommands" (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/169)
|
||||
- `report` and `trees` leave out empty files. `report` writes no group of size
|
||||
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
|
||||
counts toward a tree's digest (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/178)
|
||||
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
|
||||
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
|
||||
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
|
||||
|
||||
@@ -359,7 +359,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
dir := buildSmokeTree(t)
|
||||
|
||||
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
||||
false)
|
||||
false, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -383,7 +383,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
defer close(done)
|
||||
|
||||
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
|
||||
false, false)
|
||||
false, false, nil)
|
||||
}()
|
||||
|
||||
awaitReturn(t, done, "runScan")
|
||||
@@ -411,7 +411,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
}
|
||||
|
||||
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
||||
false)
|
||||
false, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -437,7 +437,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
|
||||
dir := buildWalkCancelTree(t)
|
||||
|
||||
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
|
||||
walkCancelWorkers, false, false)
|
||||
walkCancelWorkers, false, false, nil)
|
||||
if !errors.Is(err, errInterrupted) {
|
||||
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
|
||||
err, errInterrupted)
|
||||
|
||||
+42
-10
@@ -10,6 +10,7 @@ import (
|
||||
"os"
|
||||
"runtime"
|
||||
"runtime/debug"
|
||||
"slices"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
@@ -40,6 +41,11 @@ var errNoSubcommand = errors.New("no subcommand")
|
||||
// below 1.
|
||||
var errWorkersBelowOne = errors.New("--workers must be at least 1")
|
||||
|
||||
// errEmptyExcludeValue is the usage error for an empty scan --exclude
|
||||
// value, as from --exclude "$DIR" with DIR unset. Taken as a path, it
|
||||
// would name the working directory.
|
||||
var errEmptyExcludeValue = errors.New("--exclude must not be empty")
|
||||
|
||||
// Version is the build version, injected at link time via -ldflags
|
||||
// (see the Makefile); empty when the linker set none, as for a binary
|
||||
// built by go install. buildVersion turns it into what --version prints.
|
||||
@@ -201,33 +207,43 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
return root
|
||||
}
|
||||
|
||||
// newScanCommand builds the scan subcommand, with its --workers, -x and
|
||||
// --no-default-excludes flags. Its help lists the default excludes of
|
||||
// the system it runs on, and the filesystem types it never records
|
||||
// anything on.
|
||||
// newScanCommand builds the scan subcommand, with its --workers, -x,
|
||||
// --no-default-excludes and --exclude flags. Its help lists the default
|
||||
// excludes of the system it runs on, and the filesystem types it never
|
||||
// records anything on.
|
||||
func newScanCommand() *cobra.Command {
|
||||
var (
|
||||
scanWorkers int
|
||||
scanOneFS bool
|
||||
scanNoDefaultExcludes bool
|
||||
scanExclude []string
|
||||
)
|
||||
|
||||
short := "Walk trees and synchronize the scan database"
|
||||
|
||||
scanCmd := &cobra.Command{
|
||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
||||
Short: short,
|
||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]... PATH...",
|
||||
// The Use line already names every flag, so cobra must not
|
||||
// append its generic "[flags]" to it.
|
||||
DisableFlagsInUseLine: true,
|
||||
Short: short,
|
||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return checkScanWorkers(cmd, scanWorkers)
|
||||
err := checkScanWorkers(cmd, scanWorkers)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return rejectEmptyExcludeValue(cmd, scanExclude)
|
||||
},
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
ctx, stop := interruptContext(ctx)
|
||||
defer stop()
|
||||
|
||||
return runScan(ctx, args, scanWorkers, scanOneFS,
|
||||
scanNoDefaultExcludes)
|
||||
scanNoDefaultExcludes, scanExclude)
|
||||
}),
|
||||
}
|
||||
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
|
||||
@@ -236,6 +252,10 @@ func newScanCommand() *cobra.Command {
|
||||
"do not cross filesystem boundaries")
|
||||
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
|
||||
false, "enter the default excludes, the paths listed above")
|
||||
// A string array, not a slice, so a comma in a path is not a
|
||||
// separator.
|
||||
scanCmd.Flags().StringArrayVar(&scanExclude, "exclude", nil,
|
||||
"never enter or record `PATH` or anything below it; repeatable")
|
||||
|
||||
return scanCmd
|
||||
}
|
||||
@@ -253,6 +273,18 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
|
||||
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
|
||||
}
|
||||
|
||||
// rejectEmptyExcludeValue rejects an empty scan --exclude value. That
|
||||
// is a usage error reported in one line, as for --workers below 1.
|
||||
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
|
||||
if !slices.Contains(excludePaths, "") {
|
||||
return nil
|
||||
}
|
||||
|
||||
cmd.SilenceUsage = true
|
||||
|
||||
return errEmptyExcludeValue
|
||||
}
|
||||
|
||||
// runE adapts a subcommand implementation, or the version print, to
|
||||
// cobra's RunE. Cobra prints the error and the command's usage text for
|
||||
// every error RunE returns, but a subcommand that ran and failed has no
|
||||
|
||||
@@ -570,7 +570,8 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
}{
|
||||
cmdScan: {
|
||||
run: func() error {
|
||||
return runScan(t.Context(), []string{missing}, 1, false, false)
|
||||
return runScan(t.Context(), []string{missing}, 1, false, false,
|
||||
nil)
|
||||
},
|
||||
want: scanWant,
|
||||
},
|
||||
@@ -712,6 +713,45 @@ func TestRunHelp(t *testing.T) {
|
||||
args, got)
|
||||
}
|
||||
}
|
||||
|
||||
// The scan help lists --exclude with its argument.
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
run([]string{cmdScan, "-h"}, &stdout, &stderr)
|
||||
|
||||
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
|
||||
t.Errorf("scan -h stderr = %q, want %s PATH", stderr.String(),
|
||||
excludeFlag)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunScanUsageLine(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// README §Subcommands shows this usage line for scan, and the help
|
||||
// and the usage error both print it.
|
||||
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]... PATH...\n"
|
||||
|
||||
cases := map[string][]string{
|
||||
"help": {cmdScan, "-h"},
|
||||
"missing paths": {cmdScan},
|
||||
}
|
||||
|
||||
for name, args := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
run(args, &stdout, &stderr)
|
||||
|
||||
if !strings.Contains(stderr.String(), want) {
|
||||
t.Errorf("run(%v) stderr = %q, want the usage line %q",
|
||||
args, stderr.String(), want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunVersion(t *testing.T) {
|
||||
|
||||
@@ -17,7 +17,8 @@ import (
|
||||
// ends it at once, as it ends trees.
|
||||
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
||||
scanCtx, stop := interruptContext(ctx)
|
||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
|
||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false,
|
||||
nil)
|
||||
|
||||
stop()
|
||||
|
||||
|
||||
+43
-11
@@ -1,8 +1,9 @@
|
||||
package sfdupes
|
||||
|
||||
// This file holds two rules for the walk (README "Rules for the walk"):
|
||||
// This file holds three rules for the walk (README "Rules for the
|
||||
// walk"): the paths given with --exclude, which the scan never enters;
|
||||
// the default excludes, the paths a scan's walk does not enter below an
|
||||
// operand unless scan is given --no-default-excludes, and the
|
||||
// operand unless scan is given --no-default-excludes; and the
|
||||
// filesystems that are not persistent, on which the scan never records
|
||||
// or reads anything.
|
||||
|
||||
@@ -43,17 +44,23 @@ const typesPerHelpLine = 6
|
||||
// readMountInfo reads.
|
||||
var errMountInfoLine = errors.New("unexpected line")
|
||||
|
||||
// errExcludedOperand reports an operand at or below a path given with
|
||||
// --exclude.
|
||||
var errExcludedOperand = errors.New("operand excluded by --exclude")
|
||||
|
||||
// walkRules holds the rules for the walk that a scan sets up as it
|
||||
// starts. excludes are the default excludes, or none with
|
||||
// --no-default-excludes. nonPersistent maps the device ID of every
|
||||
// mounted filesystem that is not persistent to its type; the scan
|
||||
// records and reads nothing on one of those, whatever the flags.
|
||||
// databaseFiles are the scan's own database files, which the walk
|
||||
// passes over (see databaseFiles in db.go).
|
||||
// starts. excludePaths are the real paths given with --exclude, which
|
||||
// the walk neither enters nor records. defaultExcludes are the default
|
||||
// excludes, or none with --no-default-excludes. nonPersistent maps the
|
||||
// device ID of every mounted filesystem that is not persistent to its
|
||||
// type; the scan records and reads nothing on one of those, whatever
|
||||
// the flags. databaseFiles are the scan's own database files, which the
|
||||
// walk passes over (see databaseFiles in db.go).
|
||||
type walkRules struct {
|
||||
excludes []string
|
||||
nonPersistent map[uint64]string
|
||||
databaseFiles []fileRec
|
||||
excludePaths []string
|
||||
defaultExcludes []string
|
||||
nonPersistent map[uint64]string
|
||||
databaseFiles []fileRec
|
||||
}
|
||||
|
||||
// defaultExcludes returns the default excludes of the system goos
|
||||
@@ -97,6 +104,31 @@ func isDefaultExclude(path string, excludes []string) bool {
|
||||
return false
|
||||
}
|
||||
|
||||
// checkOperandsNotExcluded returns an error when one of roots, the
|
||||
// operands, is at or below one of excludePaths, the --exclude values.
|
||||
// The error names both. Both lists hold real paths.
|
||||
func checkOperandsNotExcluded(roots, excludePaths []string) error {
|
||||
for _, root := range roots {
|
||||
exclude := deepestRoot(root, excludePaths)
|
||||
if exclude != "" {
|
||||
return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand,
|
||||
exclude)
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// excludePathLine returns the line the scan prints for path when it is
|
||||
// one of rules.excludePaths, and "" when it is not.
|
||||
func (rules walkRules) excludePathLine(path string) string {
|
||||
if !slices.Contains(rules.excludePaths, path) {
|
||||
return ""
|
||||
}
|
||||
|
||||
return fmt.Sprintf("scan: skipped %s (--exclude)", path)
|
||||
}
|
||||
|
||||
// nonPersistentFilesystemTypes returns the filesystem types that are
|
||||
// not persistent: filesystems that the kernel makes up or that are kept
|
||||
// in memory. devfs is the macOS filesystem of /dev; the others are
|
||||
|
||||
@@ -14,6 +14,9 @@ import (
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// excludeFlag is the scan flag that names a path the walk never enters.
|
||||
const excludeFlag = "--exclude"
|
||||
|
||||
func TestDefaultExcludes(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -188,7 +191,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
|
||||
unseen := writeFile(t, excluded, "new", pattern(2, 10))
|
||||
|
||||
st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
|
||||
walkRules{excludes: []string{excluded}})
|
||||
walkRules{defaultExcludes: []string{excluded}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -234,7 +237,7 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
|
||||
db := openTestDB(t)
|
||||
|
||||
st, err := syncScan(t.Context(), db, roots, 1, false,
|
||||
walkRules{excludes: []string{excluded}})
|
||||
walkRules{defaultExcludes: []string{excluded}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -262,7 +265,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
|
||||
f := writeFile(t, excluded, "f", pattern(1, 10))
|
||||
|
||||
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
|
||||
walkRules{excludes: []string{excluded}})
|
||||
walkRules{defaultExcludes: []string{excluded}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -356,6 +359,162 @@ func TestRunScanNoDefaultExcludes(t *testing.T) {
|
||||
assertRecorded(t, path, f, true)
|
||||
}
|
||||
|
||||
// TestRunScanExcludeNested scans a tree through the command with an
|
||||
// --exclude path two levels below the operand. The walk records nothing
|
||||
// at or below it and says so. The record an earlier scan left there is
|
||||
// kept although its file is gone.
|
||||
func TestRunScanExcludeNested(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
excluded := filepath.Join(dir, "a", "b")
|
||||
kept := writeFile(t, dir, "a/f", pattern(1, 10))
|
||||
gone := writeFile(t, excluded, "gone", pattern(2, 10))
|
||||
|
||||
scanOK(t, dir)
|
||||
|
||||
err := os.Remove(gone)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
unseen := writeFile(t, excluded, "c/new", pattern(3, 10))
|
||||
|
||||
stderr := scanOK(t, excludeFlag, excluded, dir)
|
||||
|
||||
line := "scan: skipped " + excluded + " (--exclude)\n"
|
||||
if !strings.Contains(stderr, line) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||
}
|
||||
|
||||
assertRecorded(t, path, kept, true)
|
||||
assertRecorded(t, path, gone, true)
|
||||
assertRecorded(t, path, unseen, false)
|
||||
}
|
||||
|
||||
// TestRunScanExcludeRepeated gives --exclude twice, once for a
|
||||
// directory and once for a file, the file through a symlink to it. The
|
||||
// walk records neither, and records the file beside them.
|
||||
func TestRunScanExcludeRepeated(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
inDir := writeFile(t, dir, "skip/f", pattern(1, 10))
|
||||
file := writeFile(t, dir, "skip-file", pattern(2, 10))
|
||||
kept := writeFile(t, dir, "keep", pattern(3, 10))
|
||||
|
||||
link := filepath.Join(t.TempDir(), "link")
|
||||
symlink(t, file, link)
|
||||
|
||||
scanOK(t, excludeFlag, filepath.Dir(inDir), excludeFlag, link, dir)
|
||||
|
||||
assertRecorded(t, path, inDir, false)
|
||||
assertRecorded(t, path, file, false)
|
||||
assertRecorded(t, path, kept, true)
|
||||
}
|
||||
|
||||
// TestRunScanExcludedOperand checks that an operand at or below an
|
||||
// --exclude path is a fatal error that names both, and so is an
|
||||
// --exclude path that does not exist. Neither scan writes a database.
|
||||
func TestRunScanExcludedOperand(t *testing.T) {
|
||||
dir := realTempDir(t)
|
||||
sub := filepath.Join(dir, "sub")
|
||||
writeFile(t, sub, "f", pattern(1, 10))
|
||||
|
||||
cases := map[string]struct {
|
||||
args []string
|
||||
want string
|
||||
}{
|
||||
"at it": {
|
||||
[]string{excludeFlag, sub, sub},
|
||||
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
|
||||
},
|
||||
"below it": {
|
||||
[]string{excludeFlag, dir, sub},
|
||||
"scan " + sub + ": operand excluded by --exclude " + dir + "\n",
|
||||
},
|
||||
"after an operand it does not exclude": {
|
||||
[]string{dir, excludeFlag, sub, sub},
|
||||
"scan " + sub + ": operand excluded by --exclude " + sub + "\n",
|
||||
},
|
||||
"exclude missing": {
|
||||
[]string{excludeFlag, filepath.Join(dir, "nope"), dir},
|
||||
"--exclude: resolve " + filepath.Join(dir, "nope") + ": ",
|
||||
},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
stdout := captureStdout(t)
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run(append([]string{cmdScan}, tc.args...), os.Stdout,
|
||||
os.Stderr)
|
||||
if code != exitFatal {
|
||||
t.Errorf("run(scan %q) = %d, want %d", tc.args, code,
|
||||
exitFatal)
|
||||
}
|
||||
|
||||
assertFatalOutput(t, stderr(), stdout())
|
||||
|
||||
if !strings.Contains(stderr(), "sfdupes: "+tc.want) {
|
||||
t.Errorf("stderr = %q, want %q", stderr(), tc.want)
|
||||
}
|
||||
|
||||
_, err := os.Lstat(path)
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanRejectsEmptyExclude checks that an empty --exclude value,
|
||||
// given as its own argument, after "=" or after another --exclude, is
|
||||
// a usage error reported in one line before anything is scanned
|
||||
// (README "Settings").
|
||||
func TestRunScanRejectsEmptyExclude(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
|
||||
cases := map[string][]string{
|
||||
"separate": {excludeFlag, "", dir},
|
||||
"joined by =": {excludeFlag + "=", dir},
|
||||
"after a path": {excludeFlag, t.TempDir(), excludeFlag, "", dir},
|
||||
}
|
||||
|
||||
for name, args := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := run(append([]string{cmdScan}, args...), &stdout,
|
||||
&stderr)
|
||||
if code != exitUsage {
|
||||
t.Errorf("run(scan %q) = %d, want %d", args, code,
|
||||
exitUsage)
|
||||
}
|
||||
|
||||
want := "Error: --exclude must not be empty\n"
|
||||
if stdout.String() != "" || stderr.String() != want {
|
||||
t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
|
||||
stdout.String(), stderr.String(), want)
|
||||
}
|
||||
|
||||
_, err := os.Lstat(path)
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanSkipsNonPersistentOperand scans, through the command, a
|
||||
// directory on a filesystem that is not persistent, with and without
|
||||
// --no-default-excludes: either way it is skipped, with a line saying
|
||||
|
||||
@@ -195,10 +195,11 @@ func (w *reportWriter) add(r scanRec) error {
|
||||
}
|
||||
|
||||
// endGroup ends the paths read with w.sig: two or more are a duplicate
|
||||
// group, held with its paths sorted until writeGroups writes it.
|
||||
// group, held with its paths sorted until writeGroups writes it. Empty
|
||||
// files form no group, since removing one frees nothing.
|
||||
func (w *reportWriter) endGroup() {
|
||||
if len(w.paths) < minGroupSize {
|
||||
// A path with no duplicate is dropped, and its slice reused.
|
||||
if len(w.paths) < minGroupSize || w.sig.size == 0 {
|
||||
// The paths are dropped, and their slice reused.
|
||||
w.paths = w.paths[:0]
|
||||
|
||||
return
|
||||
|
||||
@@ -434,6 +434,27 @@ func TestDupeGroups(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
|
||||
// duplicate group, while 1-byte files still do.
|
||||
func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e, x := sha256.Sum256(nil), testHash("x")
|
||||
recs := []scanRec{
|
||||
{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
|
||||
{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
|
||||
{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
|
||||
}
|
||||
|
||||
groups := dupeGroupsOf(t, recs)
|
||||
if len(groups) != 1 || groups[0].size != 1 ||
|
||||
!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
|
||||
t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDupeGroupsContentSeparates(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
+72
-54
@@ -102,28 +102,31 @@ func scanPhase(name string) string {
|
||||
// scan command rejects anything less. Before the lock is taken, the
|
||||
// scan lists the filesystems that are not persistent, on which the walk
|
||||
// records nothing; unless noDefaultExcludes is set, the walk does not
|
||||
// enter the default excludes either. Once the database is open, the
|
||||
// scan notes its files, which the walk passes over.
|
||||
// enter the default excludes either. The walk neither enters nor
|
||||
// records the paths in excludePaths, the --exclude values; an operand
|
||||
// at or below one is a fatal error. Once the database is open, the scan
|
||||
// notes its files, which the walk passes over.
|
||||
func runScan(ctx context.Context, roots []string, workers int,
|
||||
oneFS, noDefaultExcludes bool,
|
||||
oneFS, noDefaultExcludes bool, excludePaths []string,
|
||||
) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// A database file missing now is created by lockScanDatabase below.
|
||||
what := "database"
|
||||
printDatabasePath(dbPath, source)
|
||||
|
||||
_, err = os.Stat(dbPath)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
what = "new database"
|
||||
roots, err = realPaths(roots)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
|
||||
source)
|
||||
excludePaths, err = realPaths(excludePaths)
|
||||
if err != nil {
|
||||
return fmt.Errorf("--exclude: %w", err)
|
||||
}
|
||||
|
||||
roots, err = resolveRoots(roots)
|
||||
err = checkOperandsNotExcluded(roots, excludePaths)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -133,9 +136,9 @@ func runScan(ctx context.Context, roots []string, workers int,
|
||||
return fmt.Errorf("list the mounted filesystems: %w", err)
|
||||
}
|
||||
|
||||
rules := walkRules{nonPersistent: nonPersistent}
|
||||
rules := walkRules{excludePaths: excludePaths, nonPersistent: nonPersistent}
|
||||
if !noDefaultExcludes {
|
||||
rules.excludes = defaultExcludes(runtime.GOOS)
|
||||
rules.defaultExcludes = defaultExcludes(runtime.GOOS)
|
||||
}
|
||||
|
||||
lock, err := lockScanDatabase(dbPath)
|
||||
@@ -181,6 +184,21 @@ func runScan(ctx context.Context, roots []string, workers int,
|
||||
return nil
|
||||
}
|
||||
|
||||
// printDatabasePath prints on stderr the line that names the database
|
||||
// at dbPath, with source, saying whether it is new. A database file
|
||||
// missing now is new: lockScanDatabase creates it.
|
||||
func printDatabasePath(dbPath, source string) {
|
||||
what := "database"
|
||||
|
||||
_, err := os.Stat(dbPath)
|
||||
if errors.Is(err, fs.ErrNotExist) {
|
||||
what = "new database"
|
||||
}
|
||||
|
||||
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
|
||||
source)
|
||||
}
|
||||
|
||||
// interruptContext returns a copy of ctx that the first SIGINT or
|
||||
// SIGTERM cancels; the scan command runs the scan under it. stop
|
||||
// releases the signals.
|
||||
@@ -209,24 +227,25 @@ func interrupted(walked int) error {
|
||||
return errInterrupted
|
||||
}
|
||||
|
||||
// resolveRoots converts each PATH operand to its real path: absolute,
|
||||
// with every symlink in it followed. Database records are keyed by
|
||||
// that path, so scan results depend neither on the working directory
|
||||
// nor on the symlinks an operand was given through. An operand that
|
||||
// cannot be resolved, because it or a symlink's target does not exist
|
||||
// or cannot be read, is a fatal error before any scanning.
|
||||
func resolveRoots(roots []string) ([]string, error) {
|
||||
resolved := make([]string, 0, len(roots))
|
||||
// realPaths returns the real path of each of paths, the PATH operands
|
||||
// or the --exclude values: absolute, with every symlink in it followed.
|
||||
// Database records are keyed by that path, so scan results depend
|
||||
// neither on the working directory nor on the symlinks an operand was
|
||||
// given through. A path that cannot be resolved, because it or a
|
||||
// symlink's target does not exist or cannot be read, is a fatal error
|
||||
// before any scanning.
|
||||
func realPaths(paths []string) ([]string, error) {
|
||||
resolved := make([]string, 0, len(paths))
|
||||
|
||||
for _, root := range roots {
|
||||
a, err := absolutePath(root)
|
||||
for _, path := range paths {
|
||||
a, err := absolutePath(path)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
return nil, fmt.Errorf("resolve %s: %w", path, err)
|
||||
}
|
||||
|
||||
r, err := filepath.EvalSymlinks(a)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
return nil, fmt.Errorf("resolve %s: %w", path, err)
|
||||
}
|
||||
|
||||
resolved = append(resolved, r)
|
||||
@@ -295,7 +314,7 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
|
||||
}
|
||||
|
||||
for ; dir != above; dir = filepath.Dir(dir) {
|
||||
if isDefaultExclude(dir, rules.excludes) {
|
||||
if isDefaultExclude(dir, rules.defaultExcludes) {
|
||||
return false
|
||||
}
|
||||
|
||||
@@ -418,7 +437,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
||||
|
||||
// walkableRoots returns the operands the walk can start from: regular
|
||||
// files, and directories not named .zfs, none of them inside a
|
||||
// directory named .zfs. The operands are real paths from resolveRoots,
|
||||
// directory named .zfs. The operands are real paths from realPaths,
|
||||
// so these checks apply to what a symlink operand points at. Every
|
||||
// other operand is warned about, counted as skipped, and dropped. A
|
||||
// dropped operand is no longer a root, so the records stored beneath it
|
||||
@@ -493,10 +512,11 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||
// as verified. When its disk_size differs from the record's, it is
|
||||
// queued for the update phase, which writes the new disk_size. A record
|
||||
// at or below a path the walk left out, because -x stopped at it, it is
|
||||
// a default exclude, or it is not on a persistent filesystem, counts as
|
||||
// outside the roots: it is not deleted. That does not hold below a root
|
||||
// at or inside that path, since that root's own walk verifies the
|
||||
// records below it; nor is a line printed for such a root.
|
||||
// an --exclude path or a default exclude, or it is not on a persistent
|
||||
// filesystem, counts as outside the roots: it is not deleted. That does
|
||||
// not hold below a root at or inside that path, since that root's own
|
||||
// walk verifies the records below it; nor is a line printed for such a
|
||||
// root.
|
||||
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
// The records the last scan left under the roots estimate how many
|
||||
// files the walk will find. A first scan has none, and no estimate.
|
||||
@@ -545,7 +565,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
prog.increment()
|
||||
|
||||
old, ok := s.existing[ev.rec.path]
|
||||
if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) {
|
||||
if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) {
|
||||
s.toHash = append(s.toHash, ev.rec)
|
||||
|
||||
continue
|
||||
@@ -780,16 +800,6 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
||||
return applyChanges(ctx, s.db, nil, deletes, prog)
|
||||
}
|
||||
|
||||
// mtimeAfter reports whether mtime a is later than mtime b.
|
||||
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
|
||||
func mtimeAfter(a, b time.Time) bool {
|
||||
if a.Unix() != b.Unix() {
|
||||
return a.Unix() > b.Unix()
|
||||
}
|
||||
|
||||
return a.Nanosecond() > b.Nanosecond()
|
||||
}
|
||||
|
||||
// underAnyRoot reports whether path is any of the roots or lies under
|
||||
// one of them.
|
||||
func underAnyRoot(path string, roots []string) bool {
|
||||
@@ -844,9 +854,9 @@ type dirJob struct {
|
||||
// file's statted record, a warning when fail is set, or a path the walk
|
||||
// left out: when otherFSDir is set, a directory -x did not enter
|
||||
// because it is on another filesystem than its operand, and when
|
||||
// leftOut is set, a default exclude or a directory or file on a
|
||||
// filesystem that is not persistent, with warn holding the line that
|
||||
// says so.
|
||||
// leftOut is set, an --exclude path, a default exclude or a directory or
|
||||
// file on a filesystem that is not persistent, with warn holding the
|
||||
// line that says so.
|
||||
type walkEvent struct {
|
||||
rec fileRec
|
||||
warn string
|
||||
@@ -897,7 +907,7 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
|
||||
// operandWarning returns the one-line warning for an operand the walk
|
||||
// does not start from, naming the path and what it is, or "" for one it
|
||||
// does: a regular file, or a directory not named .zfs, that is not
|
||||
// inside a directory named .zfs. resolveRoots has already followed the
|
||||
// inside a directory named .zfs. realPaths has already followed the
|
||||
// operand's symlinks, so a symlink here is an operand replaced by one
|
||||
// after it was resolved, and is not followed.
|
||||
func operandWarning(root string, fi fs.FileInfo) string {
|
||||
@@ -1128,8 +1138,8 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
||||
// no longer exists, and one that has stopped being a regular file is
|
||||
// skipped. A file on a filesystem in rules that is not persistent, as
|
||||
// through a bind mount of one file, is not recorded: it gets an event
|
||||
// naming it instead. One of the scan's own database files in rules gets
|
||||
// no event at all.
|
||||
// naming it instead, and so does a file given with --exclude. One of
|
||||
// the scan's own database files in rules gets no event at all.
|
||||
func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
rules walkRules, events chan<- walkEvent,
|
||||
) {
|
||||
@@ -1153,6 +1163,10 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
}
|
||||
|
||||
line := rules.nonPersistentLine(p, info)
|
||||
if line == "" {
|
||||
line = rules.excludePathLine(p)
|
||||
}
|
||||
|
||||
if line != "" {
|
||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
||||
|
||||
@@ -1177,11 +1191,11 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
// subdirJob applies the descent rules to directory p: never enter
|
||||
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
|
||||
// snapshot), never enter a directory on a filesystem in rules that is
|
||||
// not persistent, nor a default exclude in rules, and with -x never
|
||||
// enter a directory on a different filesystem than its operand. For the
|
||||
// last three it sends an event that names the directory, so the scan
|
||||
// keeps the records below it, except those below an operand at or
|
||||
// inside it.
|
||||
// not persistent, nor an --exclude path or a default exclude in rules,
|
||||
// and with -x never enter a directory on a different filesystem than
|
||||
// its operand. For all but .zfs it sends an event that names the
|
||||
// directory, so the scan keeps the records below it, except those below
|
||||
// an operand at or inside it.
|
||||
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||
parent dirJob, oneFS bool, rules walkRules,
|
||||
events chan<- walkEvent,
|
||||
@@ -1206,13 +1220,17 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
||||
}
|
||||
|
||||
line := rules.nonPersistentLine(p, info)
|
||||
if line == "" {
|
||||
line = rules.excludePathLine(p)
|
||||
}
|
||||
|
||||
if line != "" {
|
||||
sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
|
||||
|
||||
return dirJob{}, false
|
||||
}
|
||||
|
||||
if isDefaultExclude(p, rules.excludes) {
|
||||
if isDefaultExclude(p, rules.defaultExcludes) {
|
||||
sendEvent(ctx, events, walkEvent{
|
||||
leftOut: p,
|
||||
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
|
||||
|
||||
@@ -1185,11 +1185,12 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
groups := dupeGroups(t, db)
|
||||
if len(groups) != 5 {
|
||||
t.Fatalf("len(groups) = %d, want 5", len(groups))
|
||||
if len(groups) != 4 {
|
||||
t.Fatalf("len(groups) = %d, want 4", len(groups))
|
||||
}
|
||||
|
||||
wantSizes := []int64{3000, 2000, 100, 1, 0}
|
||||
// The empty files empty1 and empty2 form no group.
|
||||
wantSizes := []int64{3000, 2000, 100, 1}
|
||||
for i, g := range groups {
|
||||
if g.size != wantSizes[i] {
|
||||
t.Errorf("groups[%d].size = %d, want %d",
|
||||
@@ -1383,6 +1384,47 @@ func TestSyncScanMtimeBump(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestSyncScanMtimeMovedBack replaces a file with other content of the
|
||||
// same size and an older mtime, as cp -p or a restore from backup does.
|
||||
// The next scan must hash the file again.
|
||||
func TestSyncScanMtimeMovedBack(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t)
|
||||
a := writeFile(t, dir, "a.bin", pattern(1, 500))
|
||||
|
||||
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
|
||||
err := os.Chtimes(a, recorded, recorded)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
syncTree(t, db, dir)
|
||||
|
||||
replaced := pattern(3, 500)
|
||||
writeFile(t, dir, "a.bin", replaced)
|
||||
|
||||
older := recorded.Add(-time.Hour)
|
||||
|
||||
err = os.Chtimes(a, older, older)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
st := syncTree(t, db, dir)
|
||||
if st != (scanStats{walked: 1, updated: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
||||
}
|
||||
|
||||
assertWholeFileHashed(t, db, a, replaced)
|
||||
|
||||
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
|
||||
t.Fatalf("mtime = %v, want %v", r.mtime, older)
|
||||
}
|
||||
}
|
||||
|
||||
// assertWholeFileHashed fails unless the record for path holds the
|
||||
// whole-file hash of data as its head, tail, and content.
|
||||
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
|
||||
@@ -2247,7 +2289,7 @@ func TestPruneRoots(t *testing.T) {
|
||||
|
||||
// The walk of /p does not enter /p/q when it is a default exclude, so
|
||||
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
|
||||
got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
|
||||
got = pruneRoots(nested, false, walkRules{defaultExcludes: []string{pqRoot}},
|
||||
device)
|
||||
if !slices.Equal(got, want) {
|
||||
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
|
||||
@@ -2367,8 +2409,8 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||
report := runStdout(t, cmdReport)
|
||||
if strings.Contains(report, "db.sqlite") ||
|
||||
strings.Contains(report, hardLink) ||
|
||||
!strings.Contains(report, filepath.Join(tree, "empty2")) {
|
||||
t.Errorf("report = %q, want the empty files and none of "+
|
||||
!strings.Contains(report, filepath.Join(tree, "b")) {
|
||||
t.Errorf("report = %q, want a and b and none of "+
|
||||
"the database's files", report)
|
||||
}
|
||||
})
|
||||
|
||||
@@ -301,7 +301,9 @@ func fileEntry(name string, r scanRec) string {
|
||||
// collectTreeGroups groups directories by digest, counting each one on
|
||||
// prog as it is grouped, and returns every maximal group with two or
|
||||
// more members, each group's members sorted by path, groups ordered by
|
||||
// tree size descending then by first path ascending.
|
||||
// tree size descending then by first path ascending. A group whose
|
||||
// trees hold 0 bytes is left out, since removing one frees nothing. The
|
||||
// groups it implies hold 0 bytes too, so leaving it out hides nothing.
|
||||
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||
prog *progress,
|
||||
) [][]*treeNode {
|
||||
@@ -315,7 +317,9 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||
var dupes [][]*treeNode
|
||||
|
||||
for _, g := range groups {
|
||||
if len(g) < minGroupSize || suppressed(g, super) {
|
||||
// The members share a digest, so they share a total size.
|
||||
if len(g) < minGroupSize || g[0].totalSize == 0 ||
|
||||
suppressed(g, super) {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
@@ -2,6 +2,7 @@ package sfdupes //nolint:testpackage // checks the unexported treeNode results
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"os"
|
||||
"slices"
|
||||
@@ -435,6 +436,35 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
|
||||
// holding only empty files form no group, and that an empty file still
|
||||
// counts toward the digest of a tree that holds other files.
|
||||
func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
|
||||
sha256.Sum256(nil)
|
||||
recs := []scanRec{
|
||||
{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
|
||||
// Without the empty file, /p/x3 is no copy of /p/x1.
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
|
||||
}
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||
|
||||
want := [][]string{{"/p/x1", "/p/x2"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
t.Fatalf("groups = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
Reference in new issue
Block a user