sfdupes/README.md

# sfdupes

## Description

`sfdupes` is an MIT-licensed Go CLI tool by
[@sneak](https://sneak.berlin) that quickly identifies *candidate*
duplicate files — and, ultimately, entire duplicate directory trees —
across very large filesystems without reading full file contents. Files
are considered duplicates when they have identical size, identical
SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
1024 bytes. This is a strong candidate signal, not proof of identical
content (the middle of the file is never read); the intended use is
finding duplicate downloads and duplicated directory trees on
multi-terabyte ZFS servers where reading every byte is prohibitively
expensive. `scan` maintains a persistent SQLite database of file
signatures that survives between runs, so it can be run from cron and
the reports can be generated at any time from the most recent scan.

This tool was created by [@sneak](https://sneak.berlin) to scratch an itch,
using Claude Code/Fable.

This README is the complete and authoritative specification.

## Getting Started

```sh
make build
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes scan /srv
./sfdupes report > dupes.tsv
./sfdupes trees > dupetrees.tsv
```

`scan` walks one or more filesystem trees and maintains one database
record per regular file (path, size, mtime, head hash, tail hash). The
database persists between runs; a rescan only hashes files that are new
or changed, and removes records for files that no longer exist.
`report` reads the database and prints the file-level duplicates
report. `trees` reads the same database and prints the duplicate-tree
report. A missing/invalid subcommand — or a `scan` invocation with no
`PATH` operand — prints a usage message and exits 2.

The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed
anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a
daily `sfdupes scan` cron job, with the reporting commands run
interactively whenever needed; their results are as fresh as the last
completed scan.

## Rationale

Duplicate finders that hash entire files do not scale to the target
environment: ~10 million files and ~150 TB on possibly slow or busy
disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes
a full-filesystem sweep tractable, and the signatures are kept in a
persistent database, so the expensive filesystem pass is incremental: a
rescan re-hashes only files whose recorded mtime or size changed, and
all analysis happens offline from the database alone. The end goal is
not individual files but whole duplicated trees — duplicate
extractions, duplicate downloads, copied project trees — which an
operator can consider removing as a unit.

## Design

Goals, in order:

1.  **Find whole duplicate trees, not just files.** The end goal is to
    identify places where the exact same set of files and directories
    exists at two or more paths (duplicate extractions, duplicate
    downloads, copied project trees), so the operator can consider
    removing an entire subtree at once. File-level duplicate detection is
    the foundation; tree-level detection is built on top of it.
2.  **Never read full file contents.** At most 2 KiB is read per file
    (first and last 1024 bytes). Scale target: ~10 million files, ~150 TB
    filesystem, possibly slow or busy disks (ZFS pool under resilver).
    Holding the full file list in memory is acceptable; reading file
    contents beyond 2 KiB per file is not.
3.  **Scan incrementally, analyze offline.** The expensive filesystem
    scan maintains a persistent database; an unchanged file is never
    read again on a rescan. All analysis (`report`, `trees`) works from
    the database alone and must never touch the scanned filesystem
    again. `scan` is designed to be cronned; the reports run at any
    time against the last completed scan.
4.  **Clean stream separation.** Everything on stdout is machine-readable
    data. All progress, warnings, and summaries go to stderr. Never mix
    them.

### Constraints

-   Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
    `sfdupes`.
-   Dependencies: standard library, `github.com/spf13/cobra` for the
    CLI, **one progress-bar library**
    (`github.com/schollz/progressbar/v3`), and **one SQLite driver**
    (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
    `github.com/spf13/viper` is permitted if configuration-file support
    is ever needed, but is not currently used. No other third-party
    deps.
-   Cross-compilation is not a concern. Builds run with cgo disabled (the
    `Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
-   Analysis modes (`report`, `trees`) must be deterministic: identical
    database contents, identical output, regardless of the order in
    which records were inserted.

### Subcommands

Three subcommands, all implemented:

1.  `scan` — walk the filesystem and synchronize the database: one
    signature record per regular file.
2.  `report` — file-level duplicate report from the database.
3.  `trees` — tree-level duplicate report: reconstruct the directory
    hierarchy from the database records, compute a Merkle-style digest
    per directory, and report maximal groups of identical trees.

```
sfdupes scan [--workers N] [-x] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
```

### Database

All three subcommands operate on a single SQLite database file:

-   Location: the value of the `SFDUPES_DATABASE` environment variable
    when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`.
    There is no command-line flag.
-   `scan` creates the database (and its parent directory) on first
    use. `report` and `trees` require an existing database; a missing
    database file is a fatal error (exit 1) telling the user to run
    `scan` first.
-   The database uses WAL journal mode and a busy timeout, so running a
    report while a cron `scan` is in progress is safe; the reports see
    the last committed state. Each `PATH` operand's changes are
    committed in a single transaction when that operand completes, so
    a report never observes a half-finished operand, and a scan that
    dies partway keeps every operand completed so far (the interrupted
    operand's changes are lost, not corrupted).
-   Schema (`PRAGMA user_version` is the schema version, currently 1; a
    database with any other version is a fatal error):

    ```sql
    CREATE TABLE files (
        path  BLOB PRIMARY KEY,  -- absolute path, raw bytes
        size  INTEGER NOT NULL,  -- bytes, from lstat
        mtime INTEGER NOT NULL,  -- Unix seconds, from lstat
        head  TEXT NOT NULL,     -- lowercase-hex SHA-256, first 1 KiB
        tail  TEXT NOT NULL      -- lowercase-hex SHA-256, last 1 KiB
    ) WITHOUT ROWID;
    ```

    Paths are stored as BLOBs because Unix paths are raw bytes, not
    guaranteed UTF-8. `mtime` is used only for change detection; it is
    not part of the duplicate key.

### `scan` mode

`scan` requires one or more `PATH` operands naming the trees to scan.
There is no default path; invoking `scan` with no operand is a usage
error (usage message on stderr, exit 2). An operand may be a directory
or a regular file; an operand that does not exist is a fatal error
(exit 1). Because database records persist between runs and are keyed
by absolute path, each operand is resolved to an absolute, lexically
cleaned path (symlinks are not resolved) before walking, so results do
not depend on the working directory. Operands are processed in the
order given, each one walked and committed independently; overlapping
operands (one containing another) are harmless — a file reached via
multiple operands produces one database record (later operands see the
records committed by earlier ones and reuse them unchanged).

`scan` synchronizes the database with the filesystem state under the
scanned operands:

-   A file not yet in the database is hashed and inserted.
-   A file already in the database is **skipped without reading its
    contents** when its lstat size equals the recorded size and its
    lstat mtime is not newer than the recorded mtime. This is what
    makes a daily rescan cheap.
-   A file whose mtime is newer than recorded, or whose size differs,
    is re-hashed and its record updated.
-   A database record whose path lies under one of the scanned operands
    but was not successfully processed this run is deleted. This
    removes records for deleted files. It also removes records for
    paths that failed to stat or hash this run: the database only ever
    contains signatures verified by the most recent scan that covered
    them (a subsequent successful scan re-adds such files).
-   Database records outside the scanned operands are untouched, so
    disjoint trees can be scanned on different schedules into the same
    database.

`scan` runs **three sequential passes per operand**, committing each
operand before starting the next:

1.  **walk** — enumerate the tree with the worker pool: each worker
    reads one directory at a time, records size and mtime for every
    regular-file entry (`lstat` while the directory is fresh in
    cache), and hands discovered subdirectories back to the shared
    queue. Sequential directory enumeration is metadata-latency-bound
    and takes hours at tens of millions of files; per-directory
    parallelism is what makes the walk tractable on large or busy
    pools. Total unknown while running: show a live count, not a
    percentage.
2.  **hash** — for each new or changed file (per the rules above), read
    the first `min(1024, size)` bytes and the last `min(1024, size)`
    bytes (the two reads overlap when `size < 2048`; for `size == 0`
    hash the empty input) and compute the SHA-256 of each. Unchanged
    files are not read and do not appear in this pass's total, which
    is therefore exact for meaningful progress and ETA.
3.  **update** — apply the operand's insertions, updates, and
    deletions to the database in a single transaction.

Because every operand runs its own three passes with its own totals,
`scan` announces each operand on stderr before starting it, so a
multi-operand invocation (e.g. a shell glob) is not mistaken for a
nearly-finished run:

```
scan: /srv/media (operand 3 of 14)
```

Rules for the walk:

-   Only regular files. Skip directories, symlinks (do not follow,
    including symlink operands), sockets, FIFOs, and device nodes.
-   Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs;
    walking them would list every file once per snapshot).
-   Filesystem boundaries are crossed by default. With `-x`
    (long form `--one-file-system`, following the GNU `du`/`rsync`
    convention), never descend into a directory on a different
    filesystem than its `PATH` operand; each operand is bounded by its
    own filesystem.
-   On any per-path error (permission denied, file vanished between
    passes, unreadable): print a one-line warning to stderr, skip the
    path, and continue. Per-file errors never abort the run; the final
    summary reports how many were skipped. As specified above, a
    skipped path that has a database record from an earlier scan loses
    that record; an unreadable directory subtree likewise loses its
    records (accepted: the database mirrors what the latest scan could
    actually verify).

Concurrency: the walk and hash passes use a worker pool (`--workers`,
default `runtime.NumCPU()`); the walk parallelizes across directories,
so raising `--workers` can speed up metadata-bound walks on busy
pools. The main goroutine owns database writes and progress rendering;
progress display must never block the workers.

`scan` writes nothing to stdout. The summary line on stderr reports the
files seen this run broken down by disposition, plus skips:

```
scan: 123456 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
```

(`removed` counts deleted database records, which are not part of the
files-seen total.)

### `report` mode

`report` reads every record from the database and takes no positional
arguments.

**`report` must never touch the filesystem being analyzed.** It does not
stat, open, or otherwise access any path that appears in the records; its
only I/O is reading the database and writing stdout/stderr. It must
produce identical output whether or not the scanned filesystem is still
mounted.

Processing:

-   Group records by the key `(size, head_hash, tail_hash)`.
-   Every group with two or more paths is a duplicate group.
-   Within each group, sort paths lexicographically (byte order). The
    first path is the group's `first`; every other path is a `dupe`.
-   Order groups by size descending (biggest reclaimable space first),
    tie-broken by `first` path ascending. Output must be fully
    deterministic for a given database state.

#### Report output format

TSV on stdout: a header line, then one row per duplicate file (N-1 rows
for a group of N):

```
first	dupe	size
/srv/a/big.iso	/srv/b/big-copy.iso	4294967296
/srv/a/big.iso	/srv/c/big-copy2.iso	4294967296
```

Summary to stderr: records read, number of duplicate groups, number of
dupe files, and total reclaimable bytes (sum of `size` over all dupe
rows) in human units.

### `trees` mode

`trees` reads the same database as `report` (no positional arguments)
and reports **entire duplicate directory trees**: directories under
which the exact same set of relative paths exists with the exact same
file signatures.

**`trees` must never touch the filesystem being analyzed** — the same
rule as `report`. The directory hierarchy is reconstructed purely from
the paths in the records, split on `/`.

Definitions:

-   A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
    informational and excluded.
-   A directory's **digest** is a SHA-256 Merkle digest computed
    bottom-up: serialize the directory's child entries — for a file
    child, its name and signature; for a subdirectory child, its name
    and that subdirectory's digest — sort the serialized entries
    byte-lexicographically, and hash the concatenation. Names are part
    of the digest: two trees whose files differ only in name are *not*
    duplicates.
-   Two directories are **duplicate trees** when their digests are
    equal. Equal digests imply equal recursive file count and equal
    total byte size.

Known limitation (accepted): only regular files that appear in the
database define a tree. Empty directories are invisible, and a file
skipped during the scan (e.g. permission error) in one copy but not the
other will make otherwise-identical trees compare as different.

Processing:

-   Build the hierarchy, compute every directory's digest, and group
    directories by digest. Every group with two or more directories is a
    duplicate-tree group.
-   **Report only maximal trees.** A group is suppressed when its
    members' parents are pairwise distinct directories that all share a
    single digest — such a group is wholly implied by its parents' (or a
    further ancestor's) group. Groups containing sibling directories, or
    members whose parents differ, are always reported.
-   Within each group, sort paths lexicographically (byte order); the
    first path is `first`, every other path is a `dupe`.
-   Order groups by total tree size descending, tie-broken by `first`
    path ascending. Output must be fully deterministic for a given
    input.

#### Trees output format

TSV on stdout: a header line, then one row per duplicate tree (N-1 rows
for a group of N). `files` is the recursive regular-file count of one
copy of the tree; `size` is the recursive total byte size of one copy:

```
first	dupe	files	size
/srv/a/project	/srv/backup/project	3417	104857600
```

Summary to stderr: records read, number of duplicate-tree groups,
number of dupe trees, and total reclaimable bytes (sum of `size` over
all dupe rows) in human units.

### Progress

Use the progress-bar library for all scan-pass progress; rendering in the
style of `pv` is the model. All progress goes to stderr.

Each scan pass gets its own bar, repeated per operand. Required
elements for the hash and update passes (known totals):

-   elapsed time
-   estimated time remaining
-   a `[m/n] x%` display (items processed / total items, percent)
-   current rate (items/s)

Example shape (exact layout is flexible, content is not):

```
hash: [1234567/9876543] 12% |████      | 8123 files/s elapsed 2:32 eta 17:54
```

The walk pass has no known total: show a live file count and elapsed time
(spinner-style, no percentage or ETA).

Additional requirements:

-   When stderr is not a TTY, do not emit ANSI redraws: print a plain
    one-line progress update no more often than every 5 seconds instead.
-   Progress updates are driven from the main goroutine and must be
    non-blocking with respect to the worker pool.
-   `report` and `trees` modes need no progress display, only their
    stderr summaries.

### Error handling and exit codes

-   `0`: success, even if individual files were skipped with warnings.
-   `1`: fatal error (e.g., a `PATH` operand does not exist, the
    database cannot be created/opened/read/written, a missing database
    for `report`/`trees`, stdout write failure).
-   `2`: usage error (including `scan` with no `PATH` operand and
    `report`/`trees` with any positional argument).

## Build

The `Makefile` is the single source of truth for all operations:

-   `make build` — build the `sfdupes` binary (cgo disabled).
-   `make test` — run the test suite (30-second timeout; reruns with
    `-v` on failure).
-   `make lint` — run `golangci-lint` with the repo config.
-   `make fmt` / `make fmt-check` — format Go sources / verify
    formatting without writing.
-   `make check` — `test`, `lint`, and `fmt-check`; modifies nothing.
-   `make docker` — build the Docker image, which runs `make check` as
    a build stage.
-   `make hooks` — install the pre-commit hook.
-   `make clean` — remove the binary and any legacy local `files.dat`.

### Definition of done

All of the following, run in this directory, must pass:

1.  `make check` passes (tests, lint, `gofmt`).
2.  `make docker` succeeds.
3.  Smoke test — create a throwaway tree in a temp dir (never test
    against real data):

    ```sh
    d=$(mktemp -d)
    export SFDUPES_DATABASE="$d/db.sqlite"
    mkdir -p "$d/a" "$d/b"
    head -c 2000 /dev/urandom > "$d/a/one.bin"
    cp "$d/a/one.bin" "$d/b/copy.bin"
    cp "$d/a/one.bin" "$d/b/copy2.bin"
    head -c 2000 /dev/urandom > "$d/a/unique.bin"     # same size, different content
    printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2"  # 1-byte duplicates
    printf 'y' > "$d/tiny3"                           # 1-byte non-duplicate
    : > "$d/empty1"; : > "$d/empty2"                  # empty duplicates
    # duplicate trees: t1 and t2 are identical; t3 differs by one filename
    mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
    head -c 3000 /dev/urandom > "$d/t1/f1"
    head -c 100  /dev/urandom > "$d/t1/sub/f2"
    cp "$d/t1/f1" "$d/t2/f1"
    cp "$d/t1/sub/f2" "$d/t2/sub/f2"
    cp "$d/t1/f1" "$d/t3/f1"
    cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
    ./sfdupes scan "$d"
    ./sfdupes report
    ./sfdupes trees
    # incremental behavior (scan a subtree; records elsewhere persist):
    ./sfdupes scan "$d/a"                  # everything unchanged, nothing hashed
    printf 'z' >> "$d/a/one.bin"           # modify: next scan re-hashes it
    rm "$d/a/unique.bin"                   # delete: next scan removes its record
    ./sfdupes scan "$d/a"                  # 1 updated, 1 removed
    ./sfdupes report
    ```

    (The scan database lives inside `$d` here purely for test hygiene;
    scanning `$d` therefore also records the SQLite file itself, which
    is harmless.)

    Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin`
    form one group (two dupe rows, `first` is the lexicographically
    smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group;
    `t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group;
    `tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and
    `tiny3` appear nowhere; groups ordered by size descending.

    Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`
    `$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is
    suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
    appears nowhere (its file set differs by name).

    Expected from the second `report` (after the modify/delete rescan):
    `one.bin` has left its group (its content changed), so
    `copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is
    gone from the database.

    The test suite automates this scenario (see `scan_test.go`), plus a
    negative check: `report` and `trees` operate on the database alone
    and never touch the scanned filesystem.

## TODO

Tracked in [TODO.md](TODO.md).

## Non-goals

-   No full-content verification, no byte-for-byte compare, no deletion
    or linking of duplicates. The reports are advisory; acting on them is
    the user's job.
-   No persistence beyond the SQLite database described above; no
    export/import formats.
-   No daemon or filesystem watcher; scheduling rescans is cron's job.

## License

MIT. See [LICENSE](LICENSE).

## Author

[@sneak](https://sneak.berlin)