sneak/sfdupes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restructure README with required policy sections

Browse Source 
Adds Description (name/purpose/category/license/author), Getting
Started, Rationale, TODO, License, and Author sections; the full
normative specification is preserved under Design. The stale non-goal
about having no git repository or CI is removed, and the Build section
now documents the Makefile targets.
This commit is contained in:
Jeffrey Paul
2026-07-23 06:44:32 +07:00
parent b4d013cb34
commit 375233fff4
1 changed files with 90 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

142
README.md
View File

@@ -1,18 +1,51 @@
# sfdupes # sfdupes
`sfdupes` quickly identifies *candidate* duplicate files — and, ultimately, ## Description
entire duplicate directory trees — across very large filesystems without
reading full file contents. Files are considered duplicates when they have `sfdupes` is an MIT-licensed Go CLI tool by
identical size, identical SHA-256 of their first 1024 bytes, and identical [@sneak](https://sneak.berlin) that quickly identifies *candidate*
SHA-256 of their last 1024 bytes. This is a strong candidate signal, not duplicate files — and, ultimately, entire duplicate directory trees —
proof of identical content (the middle of the file is never read); the across very large filesystems without reading full file contents. Files
intended use is finding duplicate downloads and duplicated directory trees are considered duplicates when they have identical size, identical
on multi-terabyte ZFS servers where reading every byte is prohibitively SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
1024 bytes. This is a strong candidate signal, not proof of identical
content (the middle of the file is never read); the intended use is
finding duplicate downloads and duplicated directory trees on
multi-terabyte ZFS servers where reading every byte is prohibitively
expensive. expensive.
This README is the complete and authoritative specification. This README is the complete and authoritative specification.
## Design goals ## Getting Started
```sh
make build
./sfdupes scan -root /srv > files.dat
./sfdupes report files.dat > dupes.tsv
./sfdupes trees files.dat > dupetrees.tsv
```
`scan` walks a filesystem tree and emits one record per regular file
(path, size, mtime, head hash, tail hash). `report` ingests that stream
and prints the file-level duplicates report. `trees` ingests the same
stream and prints the duplicate-tree report. A missing/invalid
subcommand prints a usage message and exits 2.
## Rationale
Duplicate finders that hash entire files do not scale to the target
environment: ~10 million files and ~150 TB on possibly slow or busy
disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes
a full-filesystem sweep tractable, and the resulting scan stream is
self-contained, so the expensive filesystem pass runs exactly once and
all analysis happens offline. The end goal is not individual files but
whole duplicated trees — duplicate extractions, duplicate downloads,
copied project trees — which an operator can consider removing as a
unit.
## Design
Goals, in order:
1. **Find whole duplicate trees, not just files.** The end goal is to 1. **Find whole duplicate trees, not just files.** The end goal is to
identify places where the exact same set of files and directories identify places where the exact same set of files and directories
@@ -33,10 +66,10 @@ This README is the complete and authoritative specification.
data. All progress, warnings, and summaries go to stderr. Never mix data. All progress, warnings, and summaries go to stderr. Never mix
them. them.
## Constraints ### Constraints
- Language: Go (module `sneak.berlin/go/sfdupes`, already initialized - Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
in `go.mod`). Binary name: `sfdupes`. `sfdupes`.
- Dependencies: standard library, `github.com/spf13/cobra` for the CLI, - Dependencies: standard library, `github.com/spf13/cobra` for the CLI,
and **one progress-bar library** and **one progress-bar library**
(`github.com/schollz/progressbar/v3`). `github.com/spf13/viper` is (`github.com/schollz/progressbar/v3`). `github.com/spf13/viper` is
@@ -47,23 +80,16 @@ This README is the complete and authoritative specification.
- Analysis modes (`report`, `trees`) must be deterministic: identical - Analysis modes (`report`, `trees`) must be deterministic: identical
input stream, identical output, regardless of record order. input stream, identical output, regardless of record order.
## Plan ### Subcommands
Three subcommands, built in this order: Three subcommands, all implemented:
1. `scan` — walk the filesystem and emit one signature record per 1. `scan` — walk the filesystem and emit one signature record per
regular file (implemented). regular file.
2. `report` — file-level duplicate report from the scan stream 2. `report` — file-level duplicate report from the scan stream.
(implemented).
3. `trees` — tree-level duplicate report: reconstruct the directory 3. `trees` — tree-level duplicate report: reconstruct the directory
hierarchy from the scan stream, compute a Merkle-style digest per hierarchy from the scan stream, compute a Merkle-style digest per
directory, and report maximal groups of identical trees directory, and report maximal groups of identical trees.
(implemented).
Possible later work, explicitly out of scope for now: full-content
verification of candidates, and helpers that emit removal scripts.
## Usage
``` ```
sfdupes scan [-root /srv] [-workers N] > files.dat sfdupes scan [-root /srv] [-workers N] > files.dat
@@ -71,13 +97,7 @@ sfdupes report [files.dat|-] > dupes.tsv
sfdupes trees [files.dat|-] > dupetrees.tsv sfdupes trees [files.dat|-] > dupetrees.tsv
``` ```
`scan` walks a filesystem tree and emits one record per regular file ### `scan` mode
(path, size, mtime, head hash, tail hash). `report` ingests that stream
and prints the file-level duplicates report. `trees` ingests the same
stream and prints the duplicate-tree report. A missing/invalid subcommand
prints a usage message and exits 2.
## `scan` mode
`scan` runs **three sequential passes**, in this order, so that every `scan` runs **three sequential passes**, in this order, so that every
expensive pass has an exact total for meaningful progress and ETA: expensive pass has an exact total for meaningful progress and ETA:
@@ -106,7 +126,7 @@ Concurrency: the stat and hash passes use a worker pool (`-workers`,
default `runtime.NumCPU()`). The main goroutine owns stdout writing and default `runtime.NumCPU()`). The main goroutine owns stdout writing and
progress rendering; progress display must never block the workers. progress rendering; progress display must never block the workers.
### Output record format #### Output record format
One record per file on stdout, NUL-terminated (`\x00`), with One record per file on stdout, NUL-terminated (`\x00`), with
tab-separated fields, **path last** so tabs or newlines embedded in tab-separated fields, **path last** so tabs or newlines embedded in
@@ -123,7 +143,7 @@ paths cannot corrupt the record structure:
- Record order is unspecified (workers complete out of order); the - Record order is unspecified (workers complete out of order); the
analysis modes must not depend on ordering. analysis modes must not depend on ordering.
## `report` mode ### `report` mode
`report` reads the scan stream from the file named in its first `report` reads the scan stream from the file named in its first
positional argument, or from stdin if the argument is absent or `-`. positional argument, or from stdin if the argument is absent or `-`.
@@ -147,7 +167,7 @@ Processing:
tie-broken by `first` path ascending. Output must be fully tie-broken by `first` path ascending. Output must be fully
deterministic for a given input. deterministic for a given input.
### Report output format #### Report output format
TSV on stdout: a header line, then one row per duplicate file (N-1 rows TSV on stdout: a header line, then one row per duplicate file (N-1 rows
for a group of N): for a group of N):
@@ -162,7 +182,7 @@ Summary to stderr: records read, malformed count (if any), number of
duplicate groups, number of dupe files, and total reclaimable bytes duplicate groups, number of dupe files, and total reclaimable bytes
(sum of `size` over all dupe rows) in human units. (sum of `size` over all dupe rows) in human units.
## `trees` mode ### `trees` mode
`trees` reads the same scan stream as `report` (same argument handling, `trees` reads the same scan stream as `report` (same argument handling,
same parsing and malformed-record rules) and reports **entire duplicate same parsing and malformed-record rules) and reports **entire duplicate
@@ -209,7 +229,7 @@ Processing:
path ascending. Output must be fully deterministic for a given path ascending. Output must be fully deterministic for a given
input. input.
### Trees output format #### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows TSV on stdout: a header line, then one row per duplicate tree (N-1 rows
for a group of N). `files` is the recursive regular-file count of one for a group of N). `files` is the recursive regular-file count of one
@@ -224,7 +244,7 @@ Summary to stderr: records read, malformed count (if any), number of
duplicate-tree groups, number of dupe trees, and total reclaimable bytes duplicate-tree groups, number of dupe trees, and total reclaimable bytes
(sum of `size` over all dupe rows) in human units. (sum of `size` over all dupe rows) in human units.
## Progress ### Progress
Use the progress-bar library for all scan-pass progress; rendering in the Use the progress-bar library for all scan-pass progress; rendering in the
style of `pv` is the model. All progress goes to stderr. style of `pv` is the model. All progress goes to stderr.
@@ -255,7 +275,7 @@ Additional requirements:
- `report` and `trees` modes need no progress display, only their - `report` and `trees` modes need no progress display, only their
stderr summaries. stderr summaries.
## Error handling and exit codes ### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings. - `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., `-root` does not exist, cannot read the scan - `1`: fatal error (e.g., `-root` does not exist, cannot read the scan
@@ -264,18 +284,27 @@ Additional requirements:
## Build ## Build
`make` (or `make build`) builds the binary with cgo disabled. `make The `Makefile` is the single source of truth for all operations:
check` builds and runs `gofmt` and `go vet`. `make clean` removes the
binary and any local `files.dat`.
## Definition of done - `make build` — build the `sfdupes` binary (cgo disabled).
- `make test` — run the test suite (30-second timeout; reruns with
`-v` on failure).
- `make lint` — run `golangci-lint` with the repo config.
- `make fmt` / `make fmt-check` — format Go sources / verify
formatting without writing.
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing.
- `make docker` — build the Docker image, which runs `make check` as
a build stage.
- `make hooks` — install the pre-commit hook.
- `make clean` — remove the binary and any local `files.dat`.
### Definition of done
All of the following, run in this directory, must pass: All of the following, run in this directory, must pass:
1. `gofmt -l .` prints nothing. 1. `make check` passes (tests, lint, `gofmt`).
2. `go vet ./...` is clean. 2. `make docker` succeeds.
3. `go build` succeeds (equivalently, `make check` passes). 3. Smoke test — create a throwaway tree in a temp dir (never test
4. Smoke test — create a throwaway tree in a temp dir (never test
against real data): against real data):
```sh ```sh
@@ -315,9 +344,13 @@ All of the following, run in this directory, must pass:
suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3` suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
appears nowhere (its file set differs by name). appears nowhere (its file set differs by name).
5. A negative check: run `report` and `trees` after deleting the temp The test suite automates this scenario (see `scan_test.go`), plus a
tree — output must be unchanged (proves the analysis modes never negative check: `report` and `trees` operate on the captured stream
touch the filesystem). alone and never touch the scanned filesystem.
## TODO
Tracked in [TODO.md](TODO.md).
## Non-goals ## Non-goals
@@ -325,6 +358,11 @@ All of the following, run in this directory, must pass:
or linking of duplicates. The reports are advisory; acting on them is or linking of duplicates. The reports are advisory; acting on them is
the user's job. the user's job.
- No persistence formats beyond the scan stream described above. - No persistence formats beyond the scan stream described above.
- No git repository setup and no CI — code, `go.mod`/`go.sum`, the
`Makefile`, and this README only. Do not write anything outside this ## License
directory (module cache aside).
MIT. See [LICENSE](LICENSE).
## Author
[@sneak](https://sneak.berlin)