Add scan --exclude PATH (closes #163) #165

Merged
clawbot merged 1 commits from issue-163-scan-exclude into next 2026-11-11 06:37:46 +01:00
Collaborator

Adds --exclude PATH to sfdupes scan. The flag repeats. The walk neither enters nor records the path or anything below it, and prints scan: skipped PATH (--exclude) for each one it reaches. README (Subcommands, Settings, Operands, "What a scan does to the database", "Rules for the walk", exit codes) and scan --help describe it.

What the diff does not show:

  • The walk and the operand check know each --exclude path by device ID and inode number, as the walk knows the database files. Another spelling, such as another letter case on macOS, and a hard link to an excluded file are left out too. The skipped line names the path the walk reached. A symlink given to --exclude excludes its target.
  • An operand at or below an excluded path exits 1 with sfdupes: scan OPERAND: operand excluded by --exclude PATH. It is a fatal error, not a usage error, because it is known only after both are resolved.
  • dwim passes no excludes; #162 adds the scan options there.

Judgement calls:

  • Records at and below an excluded path are kept, as for a default exclude. Deleting them instead means not sending the leftOut event for an excluded path in subdirJob and emitFile.
  • An --exclude path that does not exist is a fatal error, like a missing operand. It catches typos; the cost is that a cron line excluding a path that may be absent fails.

Model: opus-5-5

Adds `--exclude PATH` to `sfdupes scan`. The flag repeats. The walk neither enters nor records the path or anything below it, and prints `scan: skipped PATH (--exclude)` for each one it reaches. README (Subcommands, Settings, Operands, "What a scan does to the database", "Rules for the walk", exit codes) and `scan --help` describe it. What the diff does not show: - The walk and the operand check know each `--exclude` path by device ID and inode number, as the walk knows the database files. Another spelling, such as another letter case on macOS, and a hard link to an excluded file are left out too. The `skipped` line names the path the walk reached. A symlink given to `--exclude` excludes its target. - An operand at or below an excluded path exits 1 with `sfdupes: scan OPERAND: operand excluded by --exclude PATH`. It is a fatal error, not a usage error, because it is known only after both are resolved. - `dwim` passes no excludes; https://git.eeqj.de/sneak/sfdupes/issues/162 adds the scan options there. Judgement calls: - Records at and below an excluded path are kept, as for a default exclude. Deleting them instead means not sending the `leftOut` event for an excluded path in `subdirJob` and `emitFile`. - An `--exclude` path that does not exist is a fatal error, like a missing operand. It catches typos; the cost is that a `cron` line excluding a path that may be absent fails. Model: opus-5-5
clawbot added the needs-review label 2026-11-11 00:45:48 +01:00
clawbot self-assigned this 2026-11-11 00:45:48 +01:00
Author
Collaborator

FAIL, needs-rework.

  1. internal/sfdupes/cli.go line 243, and resolveExcluded in internal/sfdupes/exclude.go: an empty value, --exclude "", is accepted and excludes the working directory. A cron line with --exclude "$CACHE" and the variable unset then leaves out the directory cron starts in. Make an empty --exclude value a usage error (exit 2, before anything is scanned), list it under exit code 2 in README.md, and add a test.
  2. README.md line 303 (Settings) and line 1197 (exit codes) say an --exclude path is fatal only when it does not exist. One whose real path scan cannot find, such as a path under a directory it may not search, is fatal too, and the exit-code list is meant to be complete. Use the operand wording: "an --exclude path that does not exist, or whose real path scan cannot find".
  3. internal/sfdupes/exclude.go line 110: resolveExcluded also fails on an operand at or below an excluded path, which its name does not say. Move that check into a function named for it. resolveRoots in internal/sfdupes/scan.go now resolves --exclude values as well; rename it for both uses, for example realPaths.

Model: opus-5-5

FAIL, `needs-rework`. 1. `internal/sfdupes/cli.go` line 243, and `resolveExcluded` in `internal/sfdupes/exclude.go`: an empty value, `--exclude ""`, is accepted and excludes the working directory. A `cron` line with `--exclude "$CACHE"` and the variable unset then leaves out the directory `cron` starts in. Make an empty `--exclude` value a usage error (exit 2, before anything is scanned), list it under exit code `2` in `README.md`, and add a test. 2. `README.md` line 303 (Settings) and line 1197 (exit codes) say an `--exclude` path is fatal only when it does not exist. One whose real path `scan` cannot find, such as a path under a directory it may not search, is fatal too, and the exit-code list is meant to be complete. Use the operand wording: "an `--exclude` path that does not exist, or whose real path `scan` cannot find". 3. `internal/sfdupes/exclude.go` line 110: `resolveExcluded` also fails on an operand at or below an excluded path, which its name does not say. Move that check into a function named for it. `resolveRoots` in `internal/sfdupes/scan.go` now resolves `--exclude` values as well; rename it for both uses, for example `realPaths`. Model: opus-5-5
clawbot force-pushed issue-163-scan-exclude from 20966b8ae2 to 8e4e23f0af 2026-11-11 04:08:53 +01:00 Compare
Author
Collaborator
  1. rejectEmptyExcludeValue in internal/sfdupes/cli.go makes it exit 2 before the scan; README Settings and exit code 2 list it; TestRunScanRejectsEmptyExclude covers --exclude "", --exclude= and an empty value after another --exclude.
  2. Both now use the operand wording; the exit-code entry is split into two bullets.
  3. The check is checkOperandsNotExcluded in internal/sfdupes/exclude.go; resolveRoots is now realPaths, which runScan calls for operands and --exclude values alike.

Deviation: runScan went one line over the lint length limit, so its database line moved, unchanged, into printDatabasePath.
Judgement call: TODO.md keeps resolveRoots in its 2026-08-09 entry, which records history.

Model: opus-5-5

1. `rejectEmptyExcludeValue` in `internal/sfdupes/cli.go` makes it exit 2 before the scan; README Settings and exit code `2` list it; `TestRunScanRejectsEmptyExclude` covers `--exclude ""`, `--exclude=` and an empty value after another `--exclude`. 2. Both now use the operand wording; the exit-code entry is split into two bullets. 3. The check is `checkOperandsNotExcluded` in `internal/sfdupes/exclude.go`; `resolveRoots` is now `realPaths`, which `runScan` calls for operands and `--exclude` values alike. Deviation: `runScan` went one line over the lint length limit, so its database line moved, unchanged, into `printDatabasePath`. Judgement call: `TODO.md` keeps `resolveRoots` in its 2026-08-09 entry, which records history. Model: opus-5-5
Author
Collaborator

FAIL, needs-rework.

  1. internal/sfdupes/exclude.go lines 60–61 (walkRules) and internal/sfdupes/scan.go lines 139–141: the new field excluded holds the --exclude paths, and the existing field excludes beside it holds the default excludes. The names differ by one letter, and neither says which list it holds. A reader of rules := walkRules{excluded: excluded, ...} followed by rules.excludes = defaultExcludes(runtime.GOOS) cannot tell them apart. Use the README's terms for the two names, for example defaultExcludes and excludePaths. Carry the new name through the runScan parameter, the rejectEmptyExcludeValue parameter and excludedLine.

Judgement call: keeping the records at and below an --exclude path, as for a default exclude, is not counted as a finding.

Model: opus-5-5

FAIL, `needs-rework`. 1. `internal/sfdupes/exclude.go` lines 60–61 (`walkRules`) and `internal/sfdupes/scan.go` lines 139–141: the new field `excluded` holds the `--exclude` paths, and the existing field `excludes` beside it holds the default excludes. The names differ by one letter, and neither says which list it holds. A reader of `rules := walkRules{excluded: excluded, ...}` followed by `rules.excludes = defaultExcludes(runtime.GOOS)` cannot tell them apart. Use the README's terms for the two names, for example `defaultExcludes` and `excludePaths`. Carry the new name through the `runScan` parameter, the `rejectEmptyExcludeValue` parameter and `excludedLine`. Judgement call: keeping the records at and below an `--exclude` path, as for a default exclude, is not counted as a finding. Model: opus-5-5
clawbot force-pushed issue-163-scan-exclude from 8e4e23f0af to 8b92b9b4dd 2026-11-11 05:42:06 +01:00 Compare
Author
Collaborator
  1. The fields are excludePaths and defaultExcludes; excludedLine is excludePathLine, and the runScan, rejectEmptyExcludeValue and checkOperandsNotExcluded parameters are excludePaths.

Rebased onto next: TestRunScanUsageLine, added by #187, now expects [--exclude PATH]... in the usage line.

Model: opus-5-5

1. The fields are `excludePaths` and `defaultExcludes`; `excludedLine` is `excludePathLine`, and the `runScan`, `rejectEmptyExcludeValue` and `checkOperandsNotExcluded` parameters are `excludePaths`. Rebased onto `next`: `TestRunScanUsageLine`, added by https://git.eeqj.de/sneak/sfdupes/pulls/187, now expects `[--exclude PATH]...` in the usage line. Model: opus-5-5
Author
Collaborator

FAIL, needs-rework.

  1. internal/sfdupes/exclude.go lines 110 and 124 (checkOperandsNotExcluded, excludePathLine), with realPaths in internal/sfdupes/scan.go line 237: an --exclude path is matched by how it is spelled, and realPaths keeps the spelling it was given. macOS filesystems ignore letter case by default. There, --exclude ~/library/caches is accepted, but it never matches the ~/Library/Caches the walk reaches. The scan then records everything in that directory, prints no skipped line and exits 0. An operand inside an --exclude path, spelled differently, also passes the check, and its files are recorded. README.md line 302 says the value is resolved as an operand is, and operands are resolved "as realpath prints it". That does not hold either: macOS realpath returns the name as stored on disk.

    Acceptable: the walk leaves the path out however the user spelled it, and an operand at or inside it is the fatal error under any spelling. A plain way: identify each --exclude path by device ID and inode number, as the walk already identifies the database files. Check each directory and file the walk reaches against those, and check each operand and its parent directories the same way. Update the README sentences that say the paths are compared as real paths (lines 302 and 366), and say that a hard link to an excluded file is left out too. Add a test in which the walk reaches an --exclude path through a second path, such as a hard link to an excluded file.

Model: opus-5-5

FAIL, `needs-rework`. 1. `internal/sfdupes/exclude.go` lines 110 and 124 (`checkOperandsNotExcluded`, `excludePathLine`), with `realPaths` in `internal/sfdupes/scan.go` line 237: an `--exclude` path is matched by how it is spelled, and `realPaths` keeps the spelling it was given. macOS filesystems ignore letter case by default. There, `--exclude ~/library/caches` is accepted, but it never matches the `~/Library/Caches` the walk reaches. The scan then records everything in that directory, prints no `skipped` line and exits 0. An operand inside an `--exclude` path, spelled differently, also passes the check, and its files are recorded. `README.md` line 302 says the value is resolved as an operand is, and operands are resolved "as `realpath` prints it". That does not hold either: macOS `realpath` returns the name as stored on disk. Acceptable: the walk leaves the path out however the user spelled it, and an operand at or inside it is the fatal error under any spelling. A plain way: identify each `--exclude` path by device ID and inode number, as the walk already identifies the database files. Check each directory and file the walk reaches against those, and check each operand and its parent directories the same way. Update the README sentences that say the paths are compared as real paths (lines 302 and 366), and say that a hard link to an excluded file is left out too. Add a test in which the walk reaches an `--exclude` path through a second path, such as a hard link to an excluded file. Model: opus-5-5
Author
Collaborator

For the reworker (this ends the review loop): implement exactly the "Acceptable" of the newest review: identify each --exclude path by device ID and inode number, as the walk already does for the database files; check each directory the walk reaches and each operand with its parents against those; update the two README sentences; add a test where the exclude is spelled differently from the walked path (a symlinked or .. spelling, since the test filesystem is case-sensitive). Nothing else. The next review checks only this change and anything it breaks.

Model: opus-5-5

For the reworker (this ends the review loop): implement exactly the "Acceptable" of the newest review: identify each `--exclude` path by device ID and inode number, as the walk already does for the database files; check each directory the walk reaches and each operand with its parents against those; update the two README sentences; add a test where the exclude is spelled differently from the walked path (a symlinked or `..` spelling, since the test filesystem is case-sensitive). Nothing else. The next review checks only this change and anything it breaks. Model: opus-5-5
clawbot added 1 commit 2026-11-11 06:19:20 +01:00
Add scan --exclude PATH (closes #163)
check / check (push) Waiting to run
bcd373da7a
The walk neither enters nor records an --exclude path or anything
below it, and prints "scan: skipped PATH (--exclude)" for each one it
reaches. The flag repeats. The walk knows each --exclude path by
device ID and inode number, as it knows the database files, so it also
leaves one out when it reaches it by another path, such as a hard
link. The records below an excluded path are kept, as for a default
exclude. An empty value is a usage error. An --exclude path whose real
path scan cannot find, or an operand at or below one, is a fatal error
before the scan starts. walkRules names its two lists excludePaths and
defaultExcludes. printDatabasePath, split out of runScan, keeps runScan
under the lint length limit.

Model: opus-5-5
clawbot force-pushed issue-163-scan-exclude from 8b92b9b4dd to bcd373da7a 2026-11-11 06:19:20 +01:00 Compare
Author
Collaborator
  1. walkRules.excludePaths holds each --exclude path with its device ID and inode number, from excludePathInodes in internal/sfdupes/exclude.go, matched with sameInode as the database files are.
  2. emitFile and subdirJob check every file and directory the walk reaches; checkOperandsNotExcluded checks each operand and every directory above it, up to /.
  3. Settings now says the walk knows PATH by device ID and inode number, a hard link to a file given as PATH included; Operands says the comparison is by device ID and inode number, not by name.
  4. TestRunScanExcludeReachedByAnotherPath excludes a directory through a symlink and a file through .., and the walk also reaches that file through a hard link; TestRunScanExcludedOperand adds an operand that is a hard link to an --exclude file.

Judgement call: in the operand check, a path that lstat cannot read is a fatal error.
Unverified: another letter case on macOS itself; the test filesystem is case-sensitive.
Rebased onto next: the TODO.md conflict with #184 keeps both entries.

Model: opus-5-5

1. `walkRules.excludePaths` holds each `--exclude` path with its device ID and inode number, from `excludePathInodes` in `internal/sfdupes/exclude.go`, matched with `sameInode` as the database files are. 2. `emitFile` and `subdirJob` check every file and directory the walk reaches; `checkOperandsNotExcluded` checks each operand and every directory above it, up to `/`. 3. Settings now says the walk knows `PATH` by device ID and inode number, a hard link to a file given as `PATH` included; Operands says the comparison is by device ID and inode number, not by name. 4. `TestRunScanExcludeReachedByAnotherPath` excludes a directory through a symlink and a file through `..`, and the walk also reaches that file through a hard link; `TestRunScanExcludedOperand` adds an operand that is a hard link to an `--exclude` file. Judgement call: in the operand check, a path that `lstat` cannot read is a fatal error. Unverified: another letter case on macOS itself; the test filesystem is case-sensitive. Rebased onto `next`: the `TODO.md` conflict with https://git.eeqj.de/sneak/sfdupes/pulls/184 keeps both entries. Model: opus-5-5
Author
Collaborator

PASS: scan --exclude PATH meets the definition of done in #163, and the walk and the operand check now know each --exclude path by device ID and inode number, as the last review asked.

Unverified: another letter case on macOS itself; only Linux filesystems, which are case-sensitive, were available.

Model: opus-5-5

PASS: `scan --exclude PATH` meets the definition of done in https://git.eeqj.de/sneak/sfdupes/issues/163, and the walk and the operand check now know each `--exclude` path by device ID and inode number, as the last review asked. Unverified: another letter case on macOS itself; only Linux filesystems, which are case-sensitive, were available. Model: opus-5-5
clawbot merged commit 6caaff935a into next 2026-11-11 06:37:46 +01:00
Sign in to join this conversation.