Make scan paths required operands; add -x/--one-file-system
All checks were successful
check / check (push) Successful in 4s

scan now takes one or more PATH operands (directories or regular
files) via cobra flags instead of the -root flag with its /srv
default; invoking scan with no operand is a usage error and a
nonexistent operand is fatal. Filesystem boundaries are crossed by
default; the new -x/--one-file-system flag (GNU du/rsync convention)
stops the walk at each operand's filesystem, implemented by comparing
lstat device IDs with build-tagged helpers for darwin's int32 Dev.
Verified against a real mounted disk image: default crosses, -x does
not, --one-file-system is identical to -x.
This commit is contained in:
2026-07-23 08:55:17 +07:00
parent 3391207f8d
commit 4b1c3cbf70
7 changed files with 262 additions and 84 deletions

View File

@@ -20,16 +20,17 @@ This README is the complete and authoritative specification.
```sh
make build
./sfdupes scan -root /srv > files.dat
./sfdupes scan /srv > files.dat
./sfdupes report files.dat > dupes.tsv
./sfdupes trees files.dat > dupetrees.tsv
```
`scan` walks a filesystem tree and emits one record per regular file
(path, size, mtime, head hash, tail hash). `report` ingests that stream
and prints the file-level duplicates report. `trees` ingests the same
stream and prints the duplicate-tree report. A missing/invalid
subcommand prints a usage message and exits 2.
`scan` walks one or more filesystem trees and emits one record per
regular file (path, size, mtime, head hash, tail hash). `report`
ingests that stream and prints the file-level duplicates report.
`trees` ingests the same stream and prints the duplicate-tree report. A
missing/invalid subcommand — or a `scan` invocation with no `PATH`
operand — prints a usage message and exits 2.
## Rationale
@@ -92,18 +93,26 @@ Three subcommands, all implemented:
directory, and report maximal groups of identical trees.
```
sfdupes scan [-root /srv] [-workers N] > files.dat
sfdupes scan [--workers N] [-x] PATH... > files.dat
sfdupes report [files.dat|-] > dupes.tsv
sfdupes trees [files.dat|-] > dupetrees.tsv
```
### `scan` mode
`scan` requires one or more `PATH` operands naming the trees to scan.
There is no default path; invoking `scan` with no operand is a usage
error (usage message on stderr, exit 2). An operand may be a directory
or a regular file; an operand that does not exist is a fatal error
(exit 1). Operands are walked in the order given; overlapping operands
(one containing another) emit their common files once per operand, so
callers should pass disjoint paths.
`scan` runs **three sequential passes**, in this order, so that every
expensive pass has an exact total for meaningful progress and ETA:
1. **walk** — recursively enumerate the tree under `-root` (default
`/srv`), collecting the list of regular-file paths. Total unknown
1. **walk** — recursively enumerate the tree under each `PATH` in
turn, collecting the list of regular-file paths. Total unknown
while running: show a live count, not a percentage.
2. **stat**`lstat` every collected path, recording size and mtime.
3. **hash** — for each file, read the first `min(1024, size)` bytes and
@@ -113,16 +122,21 @@ expensive pass has an exact total for meaningful progress and ETA:
Rules for the walk:
- Only regular files. Skip directories, symlinks (do not follow),
sockets, FIFOs, and device nodes.
- Only regular files. Skip directories, symlinks (do not follow,
including symlink operands), sockets, FIFOs, and device nodes.
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs;
walking them would list every file once per snapshot).
- Filesystem boundaries are crossed by default. With `-x`
(long form `--one-file-system`, following the GNU `du`/`rsync`
convention), never descend into a directory on a different
filesystem than its `PATH` operand; each operand is bounded by its
own filesystem.
- On any per-path error (permission denied, file vanished between
passes, unreadable): print a one-line warning to stderr, skip the
path, and continue. Per-file errors never abort the run; the final
summary reports how many were skipped.
Concurrency: the stat and hash passes use a worker pool (`-workers`,
Concurrency: the stat and hash passes use a worker pool (`--workers`,
default `runtime.NumCPU()`). The main goroutine owns stdout writing and
progress rendering; progress display must never block the workers.
@@ -278,9 +292,9 @@ Additional requirements:
### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., `-root` does not exist, cannot read the scan
input, stdout write failure).
- `2`: usage error.
- `1`: fatal error (e.g., a `PATH` operand does not exist, cannot
read the scan input, stdout write failure).
- `2`: usage error (including `scan` with no `PATH` operand).
## Build
@@ -325,7 +339,7 @@ All of the following, run in this directory, must pass:
cp "$d/t1/sub/f2" "$d/t2/sub/f2"
cp "$d/t1/f1" "$d/t3/f1"
cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
./sfdupes scan -root "$d" > files.dat
./sfdupes scan "$d" > files.dat
./sfdupes report files.dat
./sfdupes trees files.dat
```
@@ -336,7 +350,7 @@ All of the following, run in this directory, must pass:
`t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
`empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere;
groups ordered by size descending; piping scan directly into report
(`./sfdupes scan -root "$d" | ./sfdupes report`) gives the same
(`./sfdupes scan "$d" | ./sfdupes report`) gives the same
rows.
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`