Compare commits

..
1 Commits
Author SHA1 Message Date
sneak 8e4e23f0af Add scan --exclude PATH (closes #163)
check / check (push) Canceled after 0s
The walk neither enters nor records an --exclude path or anything
below it, and prints "scan: skipped PATH (--exclude)" for each one it
reaches. The flag repeats. Each value is resolved to its real path, as
an operand is. The records below an excluded path are kept, as for a
default exclude. An empty value, which would name the working
directory, is a usage error. An --exclude path whose real path scan
cannot find, or an operand at or below one, is a fatal error before
the scan starts. The line naming the database moved out of runScan
into printDatabasePath, which keeps runScan under the lint length
limit.

Model: opus-5-5
2026-11-11 03:08:30 +00:00
8 changed files with 54 additions and 119 deletions

No files matched your search

+8 -7
View File
@@ -208,8 +208,8 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
of reads, at the price of a comparison that is not exact. of reads, at the price of a comparison that is not exact.
The hashes live in a database that is kept between runs. A rescan reads only new The hashes live in a database that is kept between runs. A rescan reads only new
files, and files whose size or modification time (mtime) differs from their files, and files whose size changed or whose modification time (mtime) is newer
record. All analysis then works from the database alone. than recorded. All analysis then works from the database alone.
The goal is whole duplicated trees, not only single files. A download extracted The goal is whole duplicated trees, not only single files. A download extracted
twice, or a project copied once too often, is such a tree. It can be removed as twice, or a project copied once too often, is such a tree. It can be removed as
@@ -419,16 +419,17 @@ operands:
- A file not yet in the database is hashed and inserted. - A file not yet in the database is hashed and inserted.
- A file already in the database is not read again when its size and mtime match - A file already in the database is not read again when its size and mtime match
its record. This is what makes a daily rescan cheap. its record. The mtime matches when it is not newer than the recorded one. This
is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan` - When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks up to seconds after unchanged. ZFS, for example, counts a new file's blocks up to seconds after
the write. A block is the unit in which a filesystem allocates disk space. A the write. A block is the unit in which a filesystem allocates disk space. A
file scanned within those seconds gets its real `disk_size` at the next scan. file scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size or its mtime differs from its record. - `scan` hashes a file again when its size differs or its mtime is newer. The
An older mtime counts as much as a newer one. A file put back by `cp -p`, mtime is compared to the nanosecond, as finely as the filesystem records it. A
`rsync -a` or a restore from backup often has an older mtime. The mtime is same-size rewrite counts as a change whenever the filesystem gives it a later
compared to the nanosecond, as finely as the filesystem records it. mtime, even within the same second.
- The scan deletes each record under a scanned operand whose file it could not - The scan deletes each record under a scanned operand whose file it could not
verify. That covers deleted files, and paths the scan could not stat or read verify. That covers deleted files, and paths the scan could not stat or read
this run. To stat a file is to read its details, such as its size, without this run. To stat a file is to read its details, such as its size, without
-3
View File
@@ -34,9 +34,6 @@ are known.
operand at or below one is a fatal error (2026-11-11, operand at or below one is a fatal error (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/163) https://git.eeqj.de/sneak/sfdupes/issues/163)
- The usage line of `scan` no longer ends in `[flags]`. It is the line
`README.md` shows under "Subcommands" (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/169)
- `report` and `trees` leave out empty files. `report` writes no group of size - `report` and `trees` leave out empty files. `report` writes no group of size
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still 0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
counts toward a tree's digest (2026-11-11, counts toward a tree's digest (2026-11-11,
+2 -5
View File
@@ -224,9 +224,6 @@ func newScanCommand() *cobra.Command {
scanCmd := &cobra.Command{ scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " + Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]... PATH...", "[--exclude PATH]... PATH...",
// The Use line already names every flag, so cobra must not
// append its generic "[flags]" to it.
DisableFlagsInUseLine: true,
Short: short, Short: short,
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS), Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1), Args: cobra.MinimumNArgs(1),
@@ -275,8 +272,8 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
// rejectEmptyExcludeValue rejects an empty scan --exclude value. That // rejectEmptyExcludeValue rejects an empty scan --exclude value. That
// is a usage error reported in one line, as for --workers below 1. // is a usage error reported in one line, as for --workers below 1.
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error { func rejectEmptyExcludeValue(cmd *cobra.Command, excluded []string) error {
if !slices.Contains(excludePaths, "") { if !slices.Contains(excluded, "") {
return nil return nil
} }
-29
View File
@@ -725,35 +725,6 @@ func TestRunHelp(t *testing.T) {
} }
} }
func TestRunScanUsageLine(t *testing.T) {
t.Parallel()
// README §Subcommands shows this usage line for scan, and the help
// and the usage error both print it.
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]... PATH...\n"
cases := map[string][]string{
"help": {cmdScan, "-h"},
"missing paths": {cmdScan},
}
for name, args := range cases {
t.Run(name, func(t *testing.T) {
t.Parallel()
var stdout, stderr bytes.Buffer
run(args, &stdout, &stderr)
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) stderr = %q, want the usage line %q",
args, stderr.String(), want)
}
})
}
}
func TestRunVersion(t *testing.T) { func TestRunVersion(t *testing.T) {
t.Parallel() t.Parallel()
+17 -17
View File
@@ -49,16 +49,16 @@ var errMountInfoLine = errors.New("unexpected line")
var errExcludedOperand = errors.New("operand excluded by --exclude") var errExcludedOperand = errors.New("operand excluded by --exclude")
// walkRules holds the rules for the walk that a scan sets up as it // walkRules holds the rules for the walk that a scan sets up as it
// starts. excludePaths are the real paths given with --exclude, which // starts. excluded are the real paths given with --exclude, which the
// the walk neither enters nor records. defaultExcludes are the default // walk neither enters nor records. excludes are the default excludes,
// excludes, or none with --no-default-excludes. nonPersistent maps the // or none with --no-default-excludes. nonPersistent maps the device ID
// device ID of every mounted filesystem that is not persistent to its // of every mounted filesystem that is not persistent to its type; the
// type; the scan records and reads nothing on one of those, whatever // scan records and reads nothing on one of those, whatever the flags.
// the flags. databaseFiles are the scan's own database files, which the // databaseFiles are the scan's own database files, which the walk
// walk passes over (see databaseFiles in db.go). // passes over (see databaseFiles in db.go).
type walkRules struct { type walkRules struct {
excludePaths []string excluded []string
defaultExcludes []string excludes []string
nonPersistent map[uint64]string nonPersistent map[uint64]string
databaseFiles []fileRec databaseFiles []fileRec
} }
@@ -105,11 +105,11 @@ func isDefaultExclude(path string, excludes []string) bool {
} }
// checkOperandsNotExcluded returns an error when one of roots, the // checkOperandsNotExcluded returns an error when one of roots, the
// operands, is at or below one of excludePaths, the --exclude values. // operands, is at or below one of excluded, the --exclude values. The
// The error names both. Both lists hold real paths. // error names both. Both lists hold real paths.
func checkOperandsNotExcluded(roots, excludePaths []string) error { func checkOperandsNotExcluded(roots, excluded []string) error {
for _, root := range roots { for _, root := range roots {
exclude := deepestRoot(root, excludePaths) exclude := deepestRoot(root, excluded)
if exclude != "" { if exclude != "" {
return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand, return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand,
exclude) exclude)
@@ -119,10 +119,10 @@ func checkOperandsNotExcluded(roots, excludePaths []string) error {
return nil return nil
} }
// excludePathLine returns the line the scan prints for path when it is // excludedLine returns the line the scan prints for path when it is one
// one of rules.excludePaths, and "" when it is not. // of rules.excluded, and "" when it is not.
func (rules walkRules) excludePathLine(path string) string { func (rules walkRules) excludedLine(path string) string {
if !slices.Contains(rules.excludePaths, path) { if !slices.Contains(rules.excluded, path) {
return "" return ""
} }
+3 -3
View File
@@ -191,7 +191,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
unseen := writeFile(t, excluded, "new", pattern(2, 10)) unseen := writeFile(t, excluded, "new", pattern(2, 10))
st, err := syncScan(t.Context(), db, []string{dir}, 1, false, st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
walkRules{defaultExcludes: []string{excluded}}) walkRules{excludes: []string{excluded}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -237,7 +237,7 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
db := openTestDB(t) db := openTestDB(t)
st, err := syncScan(t.Context(), db, roots, 1, false, st, err := syncScan(t.Context(), db, roots, 1, false,
walkRules{defaultExcludes: []string{excluded}}) walkRules{excludes: []string{excluded}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -265,7 +265,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
f := writeFile(t, excluded, "f", pattern(1, 10)) f := writeFile(t, excluded, "f", pattern(1, 10))
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false, st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
walkRules{defaultExcludes: []string{excluded}}) walkRules{excludes: []string{excluded}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+23 -13
View File
@@ -103,11 +103,11 @@ func scanPhase(name string) string {
// scan lists the filesystems that are not persistent, on which the walk // scan lists the filesystems that are not persistent, on which the walk
// records nothing; unless noDefaultExcludes is set, the walk does not // records nothing; unless noDefaultExcludes is set, the walk does not
// enter the default excludes either. The walk neither enters nor // enter the default excludes either. The walk neither enters nor
// records the paths in excludePaths, the --exclude values; an operand // records the paths in excluded, the --exclude values; an operand at or
// at or below one is a fatal error. Once the database is open, the scan // below one is a fatal error. Once the database is open, the scan notes
// notes its files, which the walk passes over. // its files, which the walk passes over.
func runScan(ctx context.Context, roots []string, workers int, func runScan(ctx context.Context, roots []string, workers int,
oneFS, noDefaultExcludes bool, excludePaths []string, oneFS, noDefaultExcludes bool, excluded []string,
) error { ) error {
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
@@ -121,12 +121,12 @@ func runScan(ctx context.Context, roots []string, workers int,
return err return err
} }
excludePaths, err = realPaths(excludePaths) excluded, err = realPaths(excluded)
if err != nil { if err != nil {
return fmt.Errorf("--exclude: %w", err) return fmt.Errorf("--exclude: %w", err)
} }
err = checkOperandsNotExcluded(roots, excludePaths) err = checkOperandsNotExcluded(roots, excluded)
if err != nil { if err != nil {
return err return err
} }
@@ -136,9 +136,9 @@ func runScan(ctx context.Context, roots []string, workers int,
return fmt.Errorf("list the mounted filesystems: %w", err) return fmt.Errorf("list the mounted filesystems: %w", err)
} }
rules := walkRules{excludePaths: excludePaths, nonPersistent: nonPersistent} rules := walkRules{excluded: excluded, nonPersistent: nonPersistent}
if !noDefaultExcludes { if !noDefaultExcludes {
rules.defaultExcludes = defaultExcludes(runtime.GOOS) rules.excludes = defaultExcludes(runtime.GOOS)
} }
lock, err := lockScanDatabase(dbPath) lock, err := lockScanDatabase(dbPath)
@@ -314,7 +314,7 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
} }
for ; dir != above; dir = filepath.Dir(dir) { for ; dir != above; dir = filepath.Dir(dir) {
if isDefaultExclude(dir, rules.defaultExcludes) { if isDefaultExclude(dir, rules.excludes) {
return false return false
} }
@@ -565,7 +565,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
prog.increment() prog.increment()
old, ok := s.existing[ev.rec.path] old, ok := s.existing[ev.rec.path]
if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) { if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) {
s.toHash = append(s.toHash, ev.rec) s.toHash = append(s.toHash, ev.rec)
continue continue
@@ -800,6 +800,16 @@ func (s *scanState) updatePhase(ctx context.Context) error {
return applyChanges(ctx, s.db, nil, deletes, prog) return applyChanges(ctx, s.db, nil, deletes, prog)
} }
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under // underAnyRoot reports whether path is any of the roots or lies under
// one of them. // one of them.
func underAnyRoot(path string, roots []string) bool { func underAnyRoot(path string, roots []string) bool {
@@ -1164,7 +1174,7 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
line := rules.nonPersistentLine(p, info) line := rules.nonPersistentLine(p, info)
if line == "" { if line == "" {
line = rules.excludePathLine(p) line = rules.excludedLine(p)
} }
if line != "" { if line != "" {
@@ -1221,7 +1231,7 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
line := rules.nonPersistentLine(p, info) line := rules.nonPersistentLine(p, info)
if line == "" { if line == "" {
line = rules.excludePathLine(p) line = rules.excludedLine(p)
} }
if line != "" { if line != "" {
@@ -1230,7 +1240,7 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
return dirJob{}, false return dirJob{}, false
} }
if isDefaultExclude(p, rules.defaultExcludes) { if isDefaultExclude(p, rules.excludes) {
sendEvent(ctx, events, walkEvent{ sendEvent(ctx, events, walkEvent{
leftOut: p, leftOut: p,
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p), warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
+1 -42
View File
@@ -1384,47 +1384,6 @@ func TestSyncScanMtimeBump(t *testing.T) {
} }
} }
// TestSyncScanMtimeMovedBack replaces a file with other content of the
// same size and an older mtime, as cp -p or a restore from backup does.
// The next scan must hash the file again.
func TestSyncScanMtimeMovedBack(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err := os.Chtimes(a, recorded, recorded)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
replaced := pattern(3, 500)
writeFile(t, dir, "a.bin", replaced)
older := recorded.Add(-time.Hour)
err = os.Chtimes(a, older, older)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
assertWholeFileHashed(t, db, a, replaced)
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
t.Fatalf("mtime = %v, want %v", r.mtime, older)
}
}
// assertWholeFileHashed fails unless the record for path holds the // assertWholeFileHashed fails unless the record for path holds the
// whole-file hash of data as its head, tail, and content. // whole-file hash of data as its head, tail, and content.
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string, func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
@@ -2289,7 +2248,7 @@ func TestPruneRoots(t *testing.T) {
// The walk of /p does not enter /p/q when it is a default exclude, so // The walk of /p does not enter /p/q when it is a default exclude, so
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped. // /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
got = pruneRoots(nested, false, walkRules{defaultExcludes: []string{pqRoot}}, got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
device) device)
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want) t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)