From 375233fff4a4f8d2562a12d3383d1fe9e5942e52 Mon Sep 17 00:00:00 2001 From: sneak Date: Thu, 23 Jul 2026 06:44:32 +0700 Subject: [PATCH] Restructure README with required policy sections Adds Description (name/purpose/category/license/author), Getting Started, Rationale, TODO, License, and Author sections; the full normative specification is preserved under Design. The stale non-goal about having no git repository or CI is removed, and the Build section now documents the Makefile targets. --- README.md | 142 ++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 90 insertions(+), 52 deletions(-) diff --git a/README.md b/README.md index 2d8984b..98deca3 100644 --- a/README.md +++ b/README.md @@ -1,18 +1,51 @@ # sfdupes -`sfdupes` quickly identifies *candidate* duplicate files — and, ultimately, -entire duplicate directory trees — across very large filesystems without -reading full file contents. Files are considered duplicates when they have -identical size, identical SHA-256 of their first 1024 bytes, and identical -SHA-256 of their last 1024 bytes. This is a strong candidate signal, not -proof of identical content (the middle of the file is never read); the -intended use is finding duplicate downloads and duplicated directory trees -on multi-terabyte ZFS servers where reading every byte is prohibitively +## Description + +`sfdupes` is an MIT-licensed Go CLI tool by +[@sneak](https://sneak.berlin) that quickly identifies *candidate* +duplicate files — and, ultimately, entire duplicate directory trees — +across very large filesystems without reading full file contents. Files +are considered duplicates when they have identical size, identical +SHA-256 of their first 1024 bytes, and identical SHA-256 of their last +1024 bytes. This is a strong candidate signal, not proof of identical +content (the middle of the file is never read); the intended use is +finding duplicate downloads and duplicated directory trees on +multi-terabyte ZFS servers where reading every byte is prohibitively expensive. This README is the complete and authoritative specification. -## Design goals +## Getting Started + +```sh +make build +./sfdupes scan -root /srv > files.dat +./sfdupes report files.dat > dupes.tsv +./sfdupes trees files.dat > dupetrees.tsv +``` + +`scan` walks a filesystem tree and emits one record per regular file +(path, size, mtime, head hash, tail hash). `report` ingests that stream +and prints the file-level duplicates report. `trees` ingests the same +stream and prints the duplicate-tree report. A missing/invalid +subcommand prints a usage message and exits 2. + +## Rationale + +Duplicate finders that hash entire files do not scale to the target +environment: ~10 million files and ~150 TB on possibly slow or busy +disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes +a full-filesystem sweep tractable, and the resulting scan stream is +self-contained, so the expensive filesystem pass runs exactly once and +all analysis happens offline. The end goal is not individual files but +whole duplicated trees — duplicate extractions, duplicate downloads, +copied project trees — which an operator can consider removing as a +unit. + +## Design + +Goals, in order: 1. **Find whole duplicate trees, not just files.** The end goal is to identify places where the exact same set of files and directories @@ -33,10 +66,10 @@ This README is the complete and authoritative specification. data. All progress, warnings, and summaries go to stderr. Never mix them. -## Constraints +### Constraints -- Language: Go (module `sneak.berlin/go/sfdupes`, already initialized - in `go.mod`). Binary name: `sfdupes`. +- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: + `sfdupes`. - Dependencies: standard library, `github.com/spf13/cobra` for the CLI, and **one progress-bar library** (`github.com/schollz/progressbar/v3`). `github.com/spf13/viper` is @@ -47,23 +80,16 @@ This README is the complete and authoritative specification. - Analysis modes (`report`, `trees`) must be deterministic: identical input stream, identical output, regardless of record order. -## Plan +### Subcommands -Three subcommands, built in this order: +Three subcommands, all implemented: 1. `scan` — walk the filesystem and emit one signature record per - regular file (implemented). -2. `report` — file-level duplicate report from the scan stream - (implemented). + regular file. +2. `report` — file-level duplicate report from the scan stream. 3. `trees` — tree-level duplicate report: reconstruct the directory hierarchy from the scan stream, compute a Merkle-style digest per - directory, and report maximal groups of identical trees - (implemented). - -Possible later work, explicitly out of scope for now: full-content -verification of candidates, and helpers that emit removal scripts. - -## Usage + directory, and report maximal groups of identical trees. ``` sfdupes scan [-root /srv] [-workers N] > files.dat @@ -71,13 +97,7 @@ sfdupes report [files.dat|-] > dupes.tsv sfdupes trees [files.dat|-] > dupetrees.tsv ``` -`scan` walks a filesystem tree and emits one record per regular file -(path, size, mtime, head hash, tail hash). `report` ingests that stream -and prints the file-level duplicates report. `trees` ingests the same -stream and prints the duplicate-tree report. A missing/invalid subcommand -prints a usage message and exits 2. - -## `scan` mode +### `scan` mode `scan` runs **three sequential passes**, in this order, so that every expensive pass has an exact total for meaningful progress and ETA: @@ -106,7 +126,7 @@ Concurrency: the stat and hash passes use a worker pool (`-workers`, default `runtime.NumCPU()`). The main goroutine owns stdout writing and progress rendering; progress display must never block the workers. -### Output record format +#### Output record format One record per file on stdout, NUL-terminated (`\x00`), with tab-separated fields, **path last** so tabs or newlines embedded in @@ -123,7 +143,7 @@ paths cannot corrupt the record structure: - Record order is unspecified (workers complete out of order); the analysis modes must not depend on ordering. -## `report` mode +### `report` mode `report` reads the scan stream from the file named in its first positional argument, or from stdin if the argument is absent or `-`. @@ -147,7 +167,7 @@ Processing: tie-broken by `first` path ascending. Output must be fully deterministic for a given input. -### Report output format +#### Report output format TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a group of N): @@ -162,7 +182,7 @@ Summary to stderr: records read, malformed count (if any), number of duplicate groups, number of dupe files, and total reclaimable bytes (sum of `size` over all dupe rows) in human units. -## `trees` mode +### `trees` mode `trees` reads the same scan stream as `report` (same argument handling, same parsing and malformed-record rules) and reports **entire duplicate @@ -209,7 +229,7 @@ Processing: path ascending. Output must be fully deterministic for a given input. -### Trees output format +#### Trees output format TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a group of N). `files` is the recursive regular-file count of one @@ -224,7 +244,7 @@ Summary to stderr: records read, malformed count (if any), number of duplicate-tree groups, number of dupe trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human units. -## Progress +### Progress Use the progress-bar library for all scan-pass progress; rendering in the style of `pv` is the model. All progress goes to stderr. @@ -255,7 +275,7 @@ Additional requirements: - `report` and `trees` modes need no progress display, only their stderr summaries. -## Error handling and exit codes +### Error handling and exit codes - `0`: success, even if individual files were skipped with warnings. - `1`: fatal error (e.g., `-root` does not exist, cannot read the scan @@ -264,18 +284,27 @@ Additional requirements: ## Build -`make` (or `make build`) builds the binary with cgo disabled. `make -check` builds and runs `gofmt` and `go vet`. `make clean` removes the -binary and any local `files.dat`. +The `Makefile` is the single source of truth for all operations: -## Definition of done +- `make build` — build the `sfdupes` binary (cgo disabled). +- `make test` — run the test suite (30-second timeout; reruns with + `-v` on failure). +- `make lint` — run `golangci-lint` with the repo config. +- `make fmt` / `make fmt-check` — format Go sources / verify + formatting without writing. +- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. +- `make docker` — build the Docker image, which runs `make check` as + a build stage. +- `make hooks` — install the pre-commit hook. +- `make clean` — remove the binary and any local `files.dat`. + +### Definition of done All of the following, run in this directory, must pass: -1. `gofmt -l .` prints nothing. -2. `go vet ./...` is clean. -3. `go build` succeeds (equivalently, `make check` passes). -4. Smoke test — create a throwaway tree in a temp dir (never test +1. `make check` passes (tests, lint, `gofmt`). +2. `make docker` succeeds. +3. Smoke test — create a throwaway tree in a temp dir (never test against real data): ```sh @@ -315,9 +344,13 @@ All of the following, run in this directory, must pass: suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set differs by name). -5. A negative check: run `report` and `trees` after deleting the temp - tree — output must be unchanged (proves the analysis modes never - touch the filesystem). + The test suite automates this scenario (see `scan_test.go`), plus a + negative check: `report` and `trees` operate on the captured stream + alone and never touch the scanned filesystem. + +## TODO + +Tracked in [TODO.md](TODO.md). ## Non-goals @@ -325,6 +358,11 @@ All of the following, run in this directory, must pass: or linking of duplicates. The reports are advisory; acting on them is the user's job. - No persistence formats beyond the scan stream described above. -- No git repository setup and no CI — code, `go.mod`/`go.sum`, the - `Makefile`, and this README only. Do not write anything outside this - directory (module cache aside). + +## License + +MIT. See [LICENSE](LICENSE). + +## Author + +[@sneak](https://sneak.berlin)