sneak/sfdupes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Restructure README with required policy sections

Browse Source 
Adds Description (name/purpose/category/license/author), Getting
Started, Rationale, TODO, License, and Author sections; the full
normative specification is preserved under Design. The stale non-goal
about having no git repository or CI is removed, and the Build section
now documents the Makefile targets.
This commit is contained in:
Jeffrey Paul
2026-07-23 06:44:32 +07:00
parent b4d013cb34
commit 375233fff4
1 changed files with 90 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

142
README.md
View File

@@ -1,18 +1,51 @@
# sfdupes
`sfdupes` quickly identifies *candidate* duplicate files — and, ultimately,
entire duplicate directory trees — across very large filesystems without
reading full file contents. Files are considered duplicates when they have
identical size, identical SHA-256 of their first 1024 bytes, and identical
SHA-256 of their last 1024 bytes. This is a strong candidate signal, not
proof of identical content (the middle of the file is never read); the
intended use is finding duplicate downloads and duplicated directory trees
on multi-terabyte ZFS servers where reading every byte is prohibitively
## Description
`sfdupes` is an MIT-licensed Go CLI tool by
[@sneak](https://sneak.berlin) that quickly identifies *candidate*
duplicate files — and, ultimately, entire duplicate directory trees —
across very large filesystems without reading full file contents. Files
are considered duplicates when they have identical size, identical
SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
1024 bytes. This is a strong candidate signal, not proof of identical
content (the middle of the file is never read); the intended use is
finding duplicate downloads and duplicated directory trees on
multi-terabyte ZFS servers where reading every byte is prohibitively
expensive.
This README is the complete and authoritative specification.
## Design goals
## Getting Started
```sh
make build
./sfdupes scan -root /srv > files.dat
./sfdupes report files.dat > dupes.tsv
./sfdupes trees files.dat > dupetrees.tsv
```
`scan` walks a filesystem tree and emits one record per regular file
(path, size, mtime, head hash, tail hash). `report` ingests that stream
and prints the file-level duplicates report. `trees` ingests the same
stream and prints the duplicate-tree report. A missing/invalid
subcommand prints a usage message and exits 2.
## Rationale
Duplicate finders that hash entire files do not scale to the target
environment: ~10 million files and ~150 TB on possibly slow or busy
disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes
a full-filesystem sweep tractable, and the resulting scan stream is
self-contained, so the expensive filesystem pass runs exactly once and
all analysis happens offline. The end goal is not individual files but
whole duplicated trees — duplicate extractions, duplicate downloads,
copied project trees — which an operator can consider removing as a
unit.
## Design
Goals, in order:
1. **Find whole duplicate trees, not just files.** The end goal is to
identify places where the exact same set of files and directories
@@ -33,10 +66,10 @@ This README is the complete and authoritative specification.
data. All progress, warnings, and summaries go to stderr. Never mix
them.
## Constraints
### Constraints
- Language: Go (module `sneak.berlin/go/sfdupes`, already initialized
in `go.mod`). Binary name: `sfdupes`.
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
`sfdupes`.
- Dependencies: standard library, `github.com/spf13/cobra` for the CLI,
and **one progress-bar library**
(`github.com/schollz/progressbar/v3`). `github.com/spf13/viper` is
@@ -47,23 +80,16 @@ This README is the complete and authoritative specification.
- Analysis modes (`report`, `trees`) must be deterministic: identical
input stream, identical output, regardless of record order.
## Plan
### Subcommands
Three subcommands, built in this order:
Three subcommands, all implemented:
1. `scan` — walk the filesystem and emit one signature record per
regular file (implemented).
2. `report` — file-level duplicate report from the scan stream
(implemented).
regular file.
2. `report` — file-level duplicate report from the scan stream.
3. `trees` — tree-level duplicate report: reconstruct the directory
hierarchy from the scan stream, compute a Merkle-style digest per
directory, and report maximal groups of identical trees
(implemented).
Possible later work, explicitly out of scope for now: full-content
verification of candidates, and helpers that emit removal scripts.
## Usage
directory, and report maximal groups of identical trees.
```
sfdupes scan [-root /srv] [-workers N] > files.dat
@@ -71,13 +97,7 @@ sfdupes report [files.dat|-] > dupes.tsv
sfdupes trees [files.dat|-] > dupetrees.tsv
```
`scan` walks a filesystem tree and emits one record per regular file
(path, size, mtime, head hash, tail hash). `report` ingests that stream
and prints the file-level duplicates report. `trees` ingests the same
stream and prints the duplicate-tree report. A missing/invalid subcommand
prints a usage message and exits 2.
## `scan` mode
### `scan` mode
`scan` runs **three sequential passes**, in this order, so that every
expensive pass has an exact total for meaningful progress and ETA:
@@ -106,7 +126,7 @@ Concurrency: the stat and hash passes use a worker pool (`-workers`,
default `runtime.NumCPU()`). The main goroutine owns stdout writing and
progress rendering; progress display must never block the workers.
### Output record format
#### Output record format
One record per file on stdout, NUL-terminated (`\x00`), with
tab-separated fields, **path last** so tabs or newlines embedded in
@@ -123,7 +143,7 @@ paths cannot corrupt the record structure:
- Record order is unspecified (workers complete out of order); the
analysis modes must not depend on ordering.
## `report` mode
### `report` mode
`report` reads the scan stream from the file named in its first
positional argument, or from stdin if the argument is absent or `-`.
@@ -147,7 +167,7 @@ Processing:
tie-broken by `first` path ascending. Output must be fully
deterministic for a given input.
### Report output format
#### Report output format
TSV on stdout: a header line, then one row per duplicate file (N-1 rows
for a group of N):
@@ -162,7 +182,7 @@ Summary to stderr: records read, malformed count (if any), number of
duplicate groups, number of dupe files, and total reclaimable bytes
(sum of `size` over all dupe rows) in human units.
## `trees` mode
### `trees` mode
`trees` reads the same scan stream as `report` (same argument handling,
same parsing and malformed-record rules) and reports **entire duplicate
@@ -209,7 +229,7 @@ Processing:
path ascending. Output must be fully deterministic for a given
input.
### Trees output format
#### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows
for a group of N). `files` is the recursive regular-file count of one
@@ -224,7 +244,7 @@ Summary to stderr: records read, malformed count (if any), number of
duplicate-tree groups, number of dupe trees, and total reclaimable bytes
(sum of `size` over all dupe rows) in human units.
## Progress
### Progress
Use the progress-bar library for all scan-pass progress; rendering in the
style of `pv` is the model. All progress goes to stderr.
@@ -255,7 +275,7 @@ Additional requirements:
- `report` and `trees` modes need no progress display, only their
stderr summaries.
## Error handling and exit codes
### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., `-root` does not exist, cannot read the scan
@@ -264,18 +284,27 @@ Additional requirements:
## Build
`make` (or `make build`) builds the binary with cgo disabled. `make
check` builds and runs `gofmt` and `go vet`. `make clean` removes the
binary and any local `files.dat`.
The `Makefile` is the single source of truth for all operations:
## Definition of done
- `make build` — build the `sfdupes` binary (cgo disabled).
- `make test` — run the test suite (30-second timeout; reruns with
`-v` on failure).
- `make lint` — run `golangci-lint` with the repo config.
- `make fmt` / `make fmt-check` — format Go sources / verify
formatting without writing.
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing.
- `make docker` — build the Docker image, which runs `make check` as
a build stage.
- `make hooks` — install the pre-commit hook.
- `make clean` — remove the binary and any local `files.dat`.
### Definition of done
All of the following, run in this directory, must pass:
1. `gofmt -l .` prints nothing.
2. `go vet ./...` is clean.
3. `go build` succeeds (equivalently, `make check` passes).
4. Smoke test — create a throwaway tree in a temp dir (never test
1. `make check` passes (tests, lint, `gofmt`).
2. `make docker` succeeds.
3. Smoke test — create a throwaway tree in a temp dir (never test
against real data):
```sh
@@ -315,9 +344,13 @@ All of the following, run in this directory, must pass:
suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
appears nowhere (its file set differs by name).
5. A negative check: run `report` and `trees` after deleting the temp
tree — output must be unchanged (proves the analysis modes never
touch the filesystem).
The test suite automates this scenario (see `scan_test.go`), plus a
negative check: `report` and `trees` operate on the captured stream
alone and never touch the scanned filesystem.
## TODO
Tracked in [TODO.md](TODO.md).
## Non-goals
@@ -325,6 +358,11 @@ All of the following, run in this directory, must pass:
or linking of duplicates. The reports are advisory; acting on them is
the user's job.
- No persistence formats beyond the scan stream described above.
- No git repository setup and no CI — code, `go.mod`/`go.sum`, the
`Makefile`, and this README only. Do not write anything outside this
directory (module cache aside).
## License
MIT. See [LICENSE](LICENSE).
## Author
[@sneak](https://sneak.berlin)