Files

sneak dbc7f5a786 Add README: complete sfdupes specification

2026-07-22 22:06:22 +07:00

14 KiB

Raw Blame History

sfdupes

sfdupes quickly identifies candidate duplicate files — and, ultimately, entire duplicate directory trees — across very large filesystems without reading full file contents. Files are considered duplicates when they have identical size, identical SHA-256 of their first 1024 bytes, and identical SHA-256 of their last 1024 bytes. This is a strong candidate signal, not proof of identical content (the middle of the file is never read); the intended use is finding duplicate downloads and duplicated directory trees on multi-terabyte ZFS servers where reading every byte is prohibitively expensive.

This README is the complete and authoritative specification.

Design goals

Find whole duplicate trees, not just files. The end goal is to identify places where the exact same set of files and directories exists at two or more paths (duplicate extractions, duplicate downloads, copied project trees), so the operator can consider removing an entire subtree at once. File-level duplicate detection is the foundation; tree-level detection is built on top of it.
Never read full file contents. At most 2 KiB is read per file (first and last 1024 bytes). Scale target: ~10 million files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool under resilver). Holding the full file list in memory is acceptable; reading file contents beyond 2 KiB per file is not.
Scan once, analyze offline. The expensive filesystem scan produces a self-contained stream; all analysis (report, trees) works from that stream alone and must never touch the scanned filesystem again.
Clean stream separation. Everything on stdout is machine-readable data. All progress, warnings, and summaries go to stderr. Never mix them.

Constraints

Language: Go (module sneak.berlin/go/sfdupes, already initialized in go.mod). Binary name: sfdupes.
Dependencies: standard library, github.com/spf13/cobra for the CLI, and one progress-bar library (github.com/schollz/progressbar/v3). github.com/spf13/viper is permitted if configuration-file support is ever needed, but is not currently used. No other third-party deps.
Cross-compilation is not a concern. Builds run with cgo disabled (the Makefile exports CGO_ENABLED=0); the code must remain pure Go.
Analysis modes (report, trees) must be deterministic: identical input stream, identical output, regardless of record order.

Plan

Three subcommands, built in this order:

scan — walk the filesystem and emit one signature record per regular file (implemented).
report — file-level duplicate report from the scan stream (implemented).
trees — tree-level duplicate report: reconstruct the directory hierarchy from the scan stream, compute a Merkle-style digest per directory, and report maximal groups of identical trees (implemented).

Possible later work, explicitly out of scope for now: full-content verification of candidates, and helpers that emit removal scripts.

Usage

sfdupes scan [-root /srv] [-workers N] > files.dat
sfdupes report [files.dat|-] > dupes.tsv
sfdupes trees [files.dat|-] > dupetrees.tsv

scan walks a filesystem tree and emits one record per regular file (path, size, mtime, head hash, tail hash). report ingests that stream and prints the file-level duplicates report. trees ingests the same stream and prints the duplicate-tree report. A missing/invalid subcommand prints a usage message and exits 2.

`scan` mode

scan runs three sequential passes, in this order, so that every expensive pass has an exact total for meaningful progress and ETA:

walk — recursively enumerate the tree under -root (default /srv), collecting the list of regular-file paths. Total unknown while running: show a live count, not a percentage.
stat — lstat every collected path, recording size and mtime.
hash — for each file, read the first min(1024, size) bytes and the last min(1024, size) bytes (the two reads overlap when size < 2048; for size == 0 hash the empty input) and compute the SHA-256 of each. Emit the output record.

Rules for the walk:

Only regular files. Skip directories, symlinks (do not follow), sockets, FIFOs, and device nodes.
Never descend into a directory named .zfs (ZFS snapshot pseudo-dirs; walking them would list every file once per snapshot).
On any per-path error (permission denied, file vanished between passes, unreadable): print a one-line warning to stderr, skip the path, and continue. Per-file errors never abort the run; the final summary reports how many were skipped.

Concurrency: the stat and hash passes use a worker pool (-workers, default runtime.NumCPU()). The main goroutine owns stdout writing and progress rendering; progress display must never block the workers.

Output record format

One record per file on stdout, NUL-terminated (\x00), with tab-separated fields, path last so tabs or newlines embedded in paths cannot corrupt the record structure:

<size>\t<mtime_unix>\t<sha256_first1k_hex>\t<sha256_last1k_hex>\t<path>\x00

size: decimal bytes, from the stat pass.
mtime_unix: decimal Unix seconds. Informational only; not part of the duplicate key.
Hashes: lowercase hex, 64 chars each.
Record order is unspecified (workers complete out of order); the analysis modes must not depend on ordering.

`report` mode

report reads the scan stream from the file named in its first positional argument, or from stdin if the argument is absent or -.

report must never touch the filesystem being analyzed. It does not stat, open, or otherwise access any path that appears in the records; its only I/O is reading the scan file/stdin and writing stdout/stderr. It must produce identical output whether or not the scanned filesystem is still mounted.

Processing:

Parse records; a record that does not have exactly 5 fields or whose size is non-numeric is counted as malformed and skipped (warn once with the total malformed count in the summary, not per record).
Group records by the key (size, head_hash, tail_hash).
Every group with two or more paths is a duplicate group.
Within each group, sort paths lexicographically (byte order). The first path is the group's first; every other path is a dupe.
Order groups by size descending (biggest reclaimable space first), tie-broken by first path ascending. Output must be fully deterministic for a given input.

Report output format

TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a group of N):

first	dupe	size
/srv/a/big.iso	/srv/b/big-copy.iso	4294967296
/srv/a/big.iso	/srv/c/big-copy2.iso	4294967296

Summary to stderr: records read, malformed count (if any), number of duplicate groups, number of dupe files, and total reclaimable bytes (sum of size over all dupe rows) in human units.

`trees` mode

trees reads the same scan stream as report (same argument handling, same parsing and malformed-record rules) and reports entire duplicate directory trees: directories under which the exact same set of relative paths exists with the exact same file signatures.

trees must never touch the filesystem being analyzed — the same rule as report. The directory hierarchy is reconstructed purely from the paths in the records, split on /.

Definitions:

A file's signature is (size, head_hash, tail_hash) — mtime is informational and excluded.
A directory's digest is a SHA-256 Merkle digest computed bottom-up: serialize the directory's child entries — for a file child, its name and signature; for a subdirectory child, its name and that subdirectory's digest — sort the serialized entries byte-lexicographically, and hash the concatenation. Names are part of the digest: two trees whose files differ only in name are not duplicates.
Two directories are duplicate trees when their digests are equal. Equal digests imply equal recursive file count and equal total byte size.

Known limitation (accepted): only regular files that appear in the scan stream define a tree. Empty directories are invisible, and a file skipped during the scan (e.g. permission error) in one copy but not the other will make otherwise-identical trees compare as different.

Processing:

Build the hierarchy, compute every directory's digest, and group directories by digest. Every group with two or more directories is a duplicate-tree group.
Report only maximal trees. A group is suppressed when its members' parents are pairwise distinct directories that all share a single digest — such a group is wholly implied by its parents' (or a further ancestor's) group. Groups containing sibling directories, or members whose parents differ, are always reported.
Within each group, sort paths lexicographically (byte order); the first path is first, every other path is a dupe.
Order groups by total tree size descending, tie-broken by first path ascending. Output must be fully deterministic for a given input.

Trees output format

TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a group of N). files is the recursive regular-file count of one copy of the tree; size is the recursive total byte size of one copy:

first	dupe	files	size
/srv/a/project	/srv/backup/project	3417	104857600

Summary to stderr: records read, malformed count (if any), number of duplicate-tree groups, number of dupe trees, and total reclaimable bytes (sum of size over all dupe rows) in human units.

Progress

Use the progress-bar library for all scan-pass progress; rendering in the style of pv is the model. All progress goes to stderr.

Each scan pass gets its own bar. Required elements for the stat and hash passes (known totals):

elapsed time
estimated time remaining
a [m/n] x% display (files processed / total files, percent)
current rate (files/s)

Example shape (exact layout is flexible, content is not):

hash: [1234567/9876543] 12% |████      | 8123 files/s elapsed 2:32 eta 17:54

The walk pass has no known total: show a live file count and elapsed time (spinner-style, no percentage or ETA).

Additional requirements:

When stderr is not a TTY, do not emit ANSI redraws: print a plain one-line progress update no more often than every 5 seconds instead.
Progress updates are driven from the main goroutine and must be non-blocking with respect to the worker pool.
report and trees modes need no progress display, only their stderr summaries.

Error handling and exit codes

0: success, even if individual files were skipped with warnings.
1: fatal error (e.g., -root does not exist, cannot read the scan input, stdout write failure).
2: usage error.

Build

make (or make build) builds the binary with cgo disabled. make check builds and runs gofmt and go vet. make clean removes the binary and any local files.dat.

Definition of done

All of the following, run in this directory, must pass:

gofmt -l . prints nothing.
go vet ./... is clean.
go build succeeds (equivalently, make check passes).

Smoke test — create a throwaway tree in a temp dir (never test against real data):

d=$(mktemp -d)
mkdir -p "$d/a" "$d/b"
head -c 2000 /dev/urandom > "$d/a/one.bin"
cp "$d/a/one.bin" "$d/b/copy.bin"
cp "$d/a/one.bin" "$d/b/copy2.bin"
head -c 2000 /dev/urandom > "$d/a/unique.bin"     # same size, different content
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2"  # 1-byte duplicates
printf 'y' > "$d/tiny3"                           # 1-byte non-duplicate
: > "$d/empty1"; : > "$d/empty2"                  # empty duplicates
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
head -c 3000 /dev/urandom > "$d/t1/f1"
head -c 100  /dev/urandom > "$d/t1/sub/f2"
cp "$d/t1/f1" "$d/t2/f1"
cp "$d/t1/sub/f2" "$d/t2/sub/f2"
cp "$d/t1/f1" "$d/t3/f1"
cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
./sfdupes scan -root "$d" > files.dat
./sfdupes report files.dat
./sfdupes trees files.dat

Expected from report: one.bin/copy.bin/copy2.bin form one group (two dupe rows, first is the lexicographically smallest path); t1/f1/t2/f1/t3/f1 form one group; t1/sub/f2/ t2/sub/f2/t3/sub/f2renamed form one group; tiny1/tiny2 pair; empty1/empty2 pair; unique.bin and tiny3 appear nowhere; groups ordered by size descending; piping scan directly into report (./sfdupes scan -root "$d" | ./sfdupes report) gives the same rows.

Expected from trees: exactly one row — first $d/t1, dupe $d/t2, 2 files, 3100 bytes. $d/t1/sub vs $d/t2/sub is suppressed as non-maximal (implied by the t1/t2 group), and t3 appears nowhere (its file set differs by name).

A negative check: run report and trees after deleting the temp tree — output must be unchanged (proves the analysis modes never touch the filesystem).

Non-goals

No full-content verification, no byte-for-byte compare, no deletion or linking of duplicates. The reports are advisory; acting on them is the user's job.
No persistence formats beyond the scan stream described above.
No git repository setup and no CI — code, go.mod/go.sum, the Makefile, and this README only. Do not write anything outside this directory (module cache aside).

14 KiB Raw Blame History