Make scan paths required operands; add -x/--one-file-system
All checks were successful
check / check (push) Successful in 4s
All checks were successful
check / check (push) Successful in 4s
scan now takes one or more PATH operands (directories or regular files) via cobra flags instead of the -root flag with its /srv default; invoking scan with no operand is a usage error and a nonexistent operand is fatal. Filesystem boundaries are crossed by default; the new -x/--one-file-system flag (GNU du/rsync convention) stops the walk at each operand's filesystem, implemented by comparing lstat device IDs with build-tagged helpers for darwin's int32 Dev. Verified against a real mounted disk image: default crosses, -x does not, --one-file-system is identical to -x.
This commit is contained in:
48
README.md
48
README.md
@@ -20,16 +20,17 @@ This README is the complete and authoritative specification.
|
||||
|
||||
```sh
|
||||
make build
|
||||
./sfdupes scan -root /srv > files.dat
|
||||
./sfdupes scan /srv > files.dat
|
||||
./sfdupes report files.dat > dupes.tsv
|
||||
./sfdupes trees files.dat > dupetrees.tsv
|
||||
```
|
||||
|
||||
`scan` walks a filesystem tree and emits one record per regular file
|
||||
(path, size, mtime, head hash, tail hash). `report` ingests that stream
|
||||
and prints the file-level duplicates report. `trees` ingests the same
|
||||
stream and prints the duplicate-tree report. A missing/invalid
|
||||
subcommand prints a usage message and exits 2.
|
||||
`scan` walks one or more filesystem trees and emits one record per
|
||||
regular file (path, size, mtime, head hash, tail hash). `report`
|
||||
ingests that stream and prints the file-level duplicates report.
|
||||
`trees` ingests the same stream and prints the duplicate-tree report. A
|
||||
missing/invalid subcommand — or a `scan` invocation with no `PATH`
|
||||
operand — prints a usage message and exits 2.
|
||||
|
||||
## Rationale
|
||||
|
||||
@@ -92,18 +93,26 @@ Three subcommands, all implemented:
|
||||
directory, and report maximal groups of identical trees.
|
||||
|
||||
```
|
||||
sfdupes scan [-root /srv] [-workers N] > files.dat
|
||||
sfdupes scan [--workers N] [-x] PATH... > files.dat
|
||||
sfdupes report [files.dat|-] > dupes.tsv
|
||||
sfdupes trees [files.dat|-] > dupetrees.tsv
|
||||
```
|
||||
|
||||
### `scan` mode
|
||||
|
||||
`scan` requires one or more `PATH` operands naming the trees to scan.
|
||||
There is no default path; invoking `scan` with no operand is a usage
|
||||
error (usage message on stderr, exit 2). An operand may be a directory
|
||||
or a regular file; an operand that does not exist is a fatal error
|
||||
(exit 1). Operands are walked in the order given; overlapping operands
|
||||
(one containing another) emit their common files once per operand, so
|
||||
callers should pass disjoint paths.
|
||||
|
||||
`scan` runs **three sequential passes**, in this order, so that every
|
||||
expensive pass has an exact total for meaningful progress and ETA:
|
||||
|
||||
1. **walk** — recursively enumerate the tree under `-root` (default
|
||||
`/srv`), collecting the list of regular-file paths. Total unknown
|
||||
1. **walk** — recursively enumerate the tree under each `PATH` in
|
||||
turn, collecting the list of regular-file paths. Total unknown
|
||||
while running: show a live count, not a percentage.
|
||||
2. **stat** — `lstat` every collected path, recording size and mtime.
|
||||
3. **hash** — for each file, read the first `min(1024, size)` bytes and
|
||||
@@ -113,16 +122,21 @@ expensive pass has an exact total for meaningful progress and ETA:
|
||||
|
||||
Rules for the walk:
|
||||
|
||||
- Only regular files. Skip directories, symlinks (do not follow),
|
||||
sockets, FIFOs, and device nodes.
|
||||
- Only regular files. Skip directories, symlinks (do not follow,
|
||||
including symlink operands), sockets, FIFOs, and device nodes.
|
||||
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs;
|
||||
walking them would list every file once per snapshot).
|
||||
- Filesystem boundaries are crossed by default. With `-x`
|
||||
(long form `--one-file-system`, following the GNU `du`/`rsync`
|
||||
convention), never descend into a directory on a different
|
||||
filesystem than its `PATH` operand; each operand is bounded by its
|
||||
own filesystem.
|
||||
- On any per-path error (permission denied, file vanished between
|
||||
passes, unreadable): print a one-line warning to stderr, skip the
|
||||
path, and continue. Per-file errors never abort the run; the final
|
||||
summary reports how many were skipped.
|
||||
|
||||
Concurrency: the stat and hash passes use a worker pool (`-workers`,
|
||||
Concurrency: the stat and hash passes use a worker pool (`--workers`,
|
||||
default `runtime.NumCPU()`). The main goroutine owns stdout writing and
|
||||
progress rendering; progress display must never block the workers.
|
||||
|
||||
@@ -278,9 +292,9 @@ Additional requirements:
|
||||
### Error handling and exit codes
|
||||
|
||||
- `0`: success, even if individual files were skipped with warnings.
|
||||
- `1`: fatal error (e.g., `-root` does not exist, cannot read the scan
|
||||
input, stdout write failure).
|
||||
- `2`: usage error.
|
||||
- `1`: fatal error (e.g., a `PATH` operand does not exist, cannot
|
||||
read the scan input, stdout write failure).
|
||||
- `2`: usage error (including `scan` with no `PATH` operand).
|
||||
|
||||
## Build
|
||||
|
||||
@@ -325,7 +339,7 @@ All of the following, run in this directory, must pass:
|
||||
cp "$d/t1/sub/f2" "$d/t2/sub/f2"
|
||||
cp "$d/t1/f1" "$d/t3/f1"
|
||||
cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
|
||||
./sfdupes scan -root "$d" > files.dat
|
||||
./sfdupes scan "$d" > files.dat
|
||||
./sfdupes report files.dat
|
||||
./sfdupes trees files.dat
|
||||
```
|
||||
@@ -336,7 +350,7 @@ All of the following, run in this directory, must pass:
|
||||
`t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
|
||||
`empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere;
|
||||
groups ordered by size descending; piping scan directly into report
|
||||
(`./sfdupes scan -root "$d" | ./sfdupes report`) gives the same
|
||||
(`./sfdupes scan "$d" | ./sfdupes report`) gives the same
|
||||
rows.
|
||||
|
||||
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`
|
||||
|
||||
Reference in New Issue
Block a user