Compare commits

..
3 Commits
Author SHA1 Message Date
sneak 8b92b9b4dd Add scan --exclude PATH (closes #163)
check / check (push) Canceled after 0s
The walk neither enters nor records an --exclude path or anything
below it, and prints "scan: skipped PATH (--exclude)" for each one it
reaches. The flag repeats. Each value is resolved to its real path, as
an operand is. The records below an excluded path are kept, as for a
default exclude. An empty value, which would name the working
directory, is a usage error. An --exclude path whose real path scan
cannot find, or an operand at or below one, is a fatal error before
the scan starts. walkRules names its two lists excludePaths and
defaultExcludes. The line naming the database moved out of runScan
into printDatabasePath, which keeps runScan under the lint length
limit.

Model: opus-5-5
2026-11-11 04:42:02 +00:00
clawbot a27d651532 scan: drop the stray [flags] from the usage line (closes #169) (#187)
check / check (push) Canceled after 0s
2026-11-11 05:37:23 +01:00
clawbot a1ec1e0ca4 Hash a file again when its mtime moved back, not only forward (closes #176) (#182)
check / check (push) Canceled after 0s
2026-11-11 05:22:54 +01:00
8 changed files with 119 additions and 54 deletions

No files matched your search

+7 -8
View File
@@ -208,8 +208,8 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
of reads, at the price of a comparison that is not exact. of reads, at the price of a comparison that is not exact.
The hashes live in a database that is kept between runs. A rescan reads only new The hashes live in a database that is kept between runs. A rescan reads only new
files, and files whose size changed or whose modification time (mtime) is newer files, and files whose size or modification time (mtime) differs from their
than recorded. All analysis then works from the database alone. record. All analysis then works from the database alone.
The goal is whole duplicated trees, not only single files. A download extracted The goal is whole duplicated trees, not only single files. A download extracted
twice, or a project copied once too often, is such a tree. It can be removed as twice, or a project copied once too often, is such a tree. It can be removed as
@@ -419,17 +419,16 @@ operands:
- A file not yet in the database is hashed and inserted. - A file not yet in the database is hashed and inserted.
- A file already in the database is not read again when its size and mtime match - A file already in the database is not read again when its size and mtime match
its record. The mtime matches when it is not newer than the recorded one. This its record. This is what makes a daily rescan cheap.
is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan` - When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks up to seconds after unchanged. ZFS, for example, counts a new file's blocks up to seconds after
the write. A block is the unit in which a filesystem allocates disk space. A the write. A block is the unit in which a filesystem allocates disk space. A
file scanned within those seconds gets its real `disk_size` at the next scan. file scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The - `scan` hashes a file again when its size or its mtime differs from its record.
mtime is compared to the nanosecond, as finely as the filesystem records it. A An older mtime counts as much as a newer one. A file put back by `cp -p`,
same-size rewrite counts as a change whenever the filesystem gives it a later `rsync -a` or a restore from backup often has an older mtime. The mtime is
mtime, even within the same second. compared to the nanosecond, as finely as the filesystem records it.
- The scan deletes each record under a scanned operand whose file it could not - The scan deletes each record under a scanned operand whose file it could not
verify. That covers deleted files, and paths the scan could not stat or read verify. That covers deleted files, and paths the scan could not stat or read
this run. To stat a file is to read its details, such as its size, without this run. To stat a file is to read its details, such as its size, without
+3
View File
@@ -34,6 +34,9 @@ are known.
operand at or below one is a fatal error (2026-11-11, operand at or below one is a fatal error (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/163) https://git.eeqj.de/sneak/sfdupes/issues/163)
- The usage line of `scan` no longer ends in `[flags]`. It is the line
`README.md` shows under "Subcommands" (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/169)
- `report` and `trees` leave out empty files. `report` writes no group of size - `report` and `trees` leave out empty files. `report` writes no group of size
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still 0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
counts toward a tree's digest (2026-11-11, counts toward a tree's digest (2026-11-11,
+5 -2
View File
@@ -224,6 +224,9 @@ func newScanCommand() *cobra.Command {
scanCmd := &cobra.Command{ scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " + Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]... PATH...", "[--exclude PATH]... PATH...",
// The Use line already names every flag, so cobra must not
// append its generic "[flags]" to it.
DisableFlagsInUseLine: true,
Short: short, Short: short,
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS), Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1), Args: cobra.MinimumNArgs(1),
@@ -272,8 +275,8 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
// rejectEmptyExcludeValue rejects an empty scan --exclude value. That // rejectEmptyExcludeValue rejects an empty scan --exclude value. That
// is a usage error reported in one line, as for --workers below 1. // is a usage error reported in one line, as for --workers below 1.
func rejectEmptyExcludeValue(cmd *cobra.Command, excluded []string) error { func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
if !slices.Contains(excluded, "") { if !slices.Contains(excludePaths, "") {
return nil return nil
} }
+29
View File
@@ -725,6 +725,35 @@ func TestRunHelp(t *testing.T) {
} }
} }
func TestRunScanUsageLine(t *testing.T) {
t.Parallel()
// README §Subcommands shows this usage line for scan, and the help
// and the usage error both print it.
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]... PATH...\n"
cases := map[string][]string{
"help": {cmdScan, "-h"},
"missing paths": {cmdScan},
}
for name, args := range cases {
t.Run(name, func(t *testing.T) {
t.Parallel()
var stdout, stderr bytes.Buffer
run(args, &stdout, &stderr)
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) stderr = %q, want the usage line %q",
args, stderr.String(), want)
}
})
}
}
func TestRunVersion(t *testing.T) { func TestRunVersion(t *testing.T) {
t.Parallel() t.Parallel()
+17 -17
View File
@@ -49,16 +49,16 @@ var errMountInfoLine = errors.New("unexpected line")
var errExcludedOperand = errors.New("operand excluded by --exclude") var errExcludedOperand = errors.New("operand excluded by --exclude")
// walkRules holds the rules for the walk that a scan sets up as it // walkRules holds the rules for the walk that a scan sets up as it
// starts. excluded are the real paths given with --exclude, which the // starts. excludePaths are the real paths given with --exclude, which
// walk neither enters nor records. excludes are the default excludes, // the walk neither enters nor records. defaultExcludes are the default
// or none with --no-default-excludes. nonPersistent maps the device ID // excludes, or none with --no-default-excludes. nonPersistent maps the
// of every mounted filesystem that is not persistent to its type; the // device ID of every mounted filesystem that is not persistent to its
// scan records and reads nothing on one of those, whatever the flags. // type; the scan records and reads nothing on one of those, whatever
// databaseFiles are the scan's own database files, which the walk // the flags. databaseFiles are the scan's own database files, which the
// passes over (see databaseFiles in db.go). // walk passes over (see databaseFiles in db.go).
type walkRules struct { type walkRules struct {
excluded []string excludePaths []string
excludes []string defaultExcludes []string
nonPersistent map[uint64]string nonPersistent map[uint64]string
databaseFiles []fileRec databaseFiles []fileRec
} }
@@ -105,11 +105,11 @@ func isDefaultExclude(path string, excludes []string) bool {
} }
// checkOperandsNotExcluded returns an error when one of roots, the // checkOperandsNotExcluded returns an error when one of roots, the
// operands, is at or below one of excluded, the --exclude values. The // operands, is at or below one of excludePaths, the --exclude values.
// error names both. Both lists hold real paths. // The error names both. Both lists hold real paths.
func checkOperandsNotExcluded(roots, excluded []string) error { func checkOperandsNotExcluded(roots, excludePaths []string) error {
for _, root := range roots { for _, root := range roots {
exclude := deepestRoot(root, excluded) exclude := deepestRoot(root, excludePaths)
if exclude != "" { if exclude != "" {
return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand, return fmt.Errorf("scan %s: %w %s", root, errExcludedOperand,
exclude) exclude)
@@ -119,10 +119,10 @@ func checkOperandsNotExcluded(roots, excluded []string) error {
return nil return nil
} }
// excludedLine returns the line the scan prints for path when it is one // excludePathLine returns the line the scan prints for path when it is
// of rules.excluded, and "" when it is not. // one of rules.excludePaths, and "" when it is not.
func (rules walkRules) excludedLine(path string) string { func (rules walkRules) excludePathLine(path string) string {
if !slices.Contains(rules.excluded, path) { if !slices.Contains(rules.excludePaths, path) {
return "" return ""
} }
+3 -3
View File
@@ -191,7 +191,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
unseen := writeFile(t, excluded, "new", pattern(2, 10)) unseen := writeFile(t, excluded, "new", pattern(2, 10))
st, err := syncScan(t.Context(), db, []string{dir}, 1, false, st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
walkRules{excludes: []string{excluded}}) walkRules{defaultExcludes: []string{excluded}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -237,7 +237,7 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
db := openTestDB(t) db := openTestDB(t)
st, err := syncScan(t.Context(), db, roots, 1, false, st, err := syncScan(t.Context(), db, roots, 1, false,
walkRules{excludes: []string{excluded}}) walkRules{defaultExcludes: []string{excluded}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -265,7 +265,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
f := writeFile(t, excluded, "f", pattern(1, 10)) f := writeFile(t, excluded, "f", pattern(1, 10))
st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false, st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
walkRules{excludes: []string{excluded}}) walkRules{defaultExcludes: []string{excluded}})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+13 -23
View File
@@ -103,11 +103,11 @@ func scanPhase(name string) string {
// scan lists the filesystems that are not persistent, on which the walk // scan lists the filesystems that are not persistent, on which the walk
// records nothing; unless noDefaultExcludes is set, the walk does not // records nothing; unless noDefaultExcludes is set, the walk does not
// enter the default excludes either. The walk neither enters nor // enter the default excludes either. The walk neither enters nor
// records the paths in excluded, the --exclude values; an operand at or // records the paths in excludePaths, the --exclude values; an operand
// below one is a fatal error. Once the database is open, the scan notes // at or below one is a fatal error. Once the database is open, the scan
// its files, which the walk passes over. // notes its files, which the walk passes over.
func runScan(ctx context.Context, roots []string, workers int, func runScan(ctx context.Context, roots []string, workers int,
oneFS, noDefaultExcludes bool, excluded []string, oneFS, noDefaultExcludes bool, excludePaths []string,
) error { ) error {
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
@@ -121,12 +121,12 @@ func runScan(ctx context.Context, roots []string, workers int,
return err return err
} }
excluded, err = realPaths(excluded) excludePaths, err = realPaths(excludePaths)
if err != nil { if err != nil {
return fmt.Errorf("--exclude: %w", err) return fmt.Errorf("--exclude: %w", err)
} }
err = checkOperandsNotExcluded(roots, excluded) err = checkOperandsNotExcluded(roots, excludePaths)
if err != nil { if err != nil {
return err return err
} }
@@ -136,9 +136,9 @@ func runScan(ctx context.Context, roots []string, workers int,
return fmt.Errorf("list the mounted filesystems: %w", err) return fmt.Errorf("list the mounted filesystems: %w", err)
} }
rules := walkRules{excluded: excluded, nonPersistent: nonPersistent} rules := walkRules{excludePaths: excludePaths, nonPersistent: nonPersistent}
if !noDefaultExcludes { if !noDefaultExcludes {
rules.excludes = defaultExcludes(runtime.GOOS) rules.defaultExcludes = defaultExcludes(runtime.GOOS)
} }
lock, err := lockScanDatabase(dbPath) lock, err := lockScanDatabase(dbPath)
@@ -314,7 +314,7 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
} }
for ; dir != above; dir = filepath.Dir(dir) { for ; dir != above; dir = filepath.Dir(dir) {
if isDefaultExclude(dir, rules.excludes) { if isDefaultExclude(dir, rules.defaultExcludes) {
return false return false
} }
@@ -565,7 +565,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
prog.increment() prog.increment()
old, ok := s.existing[ev.rec.path] old, ok := s.existing[ev.rec.path]
if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) { if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) {
s.toHash = append(s.toHash, ev.rec) s.toHash = append(s.toHash, ev.rec)
continue continue
@@ -800,16 +800,6 @@ func (s *scanState) updatePhase(ctx context.Context) error {
return applyChanges(ctx, s.db, nil, deletes, prog) return applyChanges(ctx, s.db, nil, deletes, prog)
} }
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under // underAnyRoot reports whether path is any of the roots or lies under
// one of them. // one of them.
func underAnyRoot(path string, roots []string) bool { func underAnyRoot(path string, roots []string) bool {
@@ -1174,7 +1164,7 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
line := rules.nonPersistentLine(p, info) line := rules.nonPersistentLine(p, info)
if line == "" { if line == "" {
line = rules.excludedLine(p) line = rules.excludePathLine(p)
} }
if line != "" { if line != "" {
@@ -1231,7 +1221,7 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
line := rules.nonPersistentLine(p, info) line := rules.nonPersistentLine(p, info)
if line == "" { if line == "" {
line = rules.excludedLine(p) line = rules.excludePathLine(p)
} }
if line != "" { if line != "" {
@@ -1240,7 +1230,7 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
return dirJob{}, false return dirJob{}, false
} }
if isDefaultExclude(p, rules.excludes) { if isDefaultExclude(p, rules.defaultExcludes) {
sendEvent(ctx, events, walkEvent{ sendEvent(ctx, events, walkEvent{
leftOut: p, leftOut: p,
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p), warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
+42 -1
View File
@@ -1384,6 +1384,47 @@ func TestSyncScanMtimeBump(t *testing.T) {
} }
} }
// TestSyncScanMtimeMovedBack replaces a file with other content of the
// same size and an older mtime, as cp -p or a restore from backup does.
// The next scan must hash the file again.
func TestSyncScanMtimeMovedBack(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err := os.Chtimes(a, recorded, recorded)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
replaced := pattern(3, 500)
writeFile(t, dir, "a.bin", replaced)
older := recorded.Add(-time.Hour)
err = os.Chtimes(a, older, older)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
assertWholeFileHashed(t, db, a, replaced)
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
t.Fatalf("mtime = %v, want %v", r.mtime, older)
}
}
// assertWholeFileHashed fails unless the record for path holds the // assertWholeFileHashed fails unless the record for path holds the
// whole-file hash of data as its head, tail, and content. // whole-file hash of data as its head, tail, and content.
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string, func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
@@ -2248,7 +2289,7 @@ func TestPruneRoots(t *testing.T) {
// The walk of /p does not enter /p/q when it is a default exclude, so // The walk of /p does not enter /p/q when it is a default exclude, so
// /p/q is kept, without -x too; /p/q/r lies under it and is dropped. // /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}}, got = pruneRoots(nested, false, walkRules{defaultExcludes: []string{pqRoot}},
device) device)
if !slices.Equal(got, want) { if !slices.Equal(got, want) {
t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want) t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)