sfdupes/README.md

# sfdupes

## Description

`sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that
quickly identifies _candidate_ duplicate files — and, ultimately, entire
duplicate directory trees — across very large filesystems without reading full
file contents. Files are considered duplicates when they have identical size,
identical SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
1024 bytes. This is a strong candidate signal, not proof of identical content
(the middle of the file is never read); the intended use is finding duplicate
downloads and duplicated directory trees on multi-terabyte ZFS servers where
reading every byte is prohibitively expensive. `scan` maintains a persistent
SQLite database of file signatures that survives between runs, so it can be run
from cron and the reports can be generated at any time from the most recent
scan.

This README is the complete and authoritative specification.

## Getting Started

```sh
make build
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes scan /srv
./sfdupes report > dupes.tsv
./sfdupes trees > dupetrees.tsv
```

`scan` walks one or more filesystem trees and maintains one database record per
regular file (path, size, mtime, head hash, tail hash). The database persists
between runs; a rescan only hashes files that are new or changed, and removes
records for files that no longer exist. `report` reads the database and prints
the file-level duplicates report. `trees` reads the same database and prints the
duplicate-tree report. A missing/invalid subcommand — or a `scan` invocation
with no `PATH` operand — prints a usage message and exits 2.

The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
cron job, with the reporting commands run interactively whenever needed; their
results are as fresh as the last completed scan.

## Rationale

Duplicate finders that hash entire files do not scale to the target environment:
~10 million files and ~150 TB on possibly slow or busy disks (a ZFS pool under
resilver). Reading at most 2 KiB per file — and only from files whose size at
least one other file shares, since a size-unique file cannot be a duplicate —
makes a full-filesystem sweep tractable, and the signatures are kept in a
persistent database, so the expensive filesystem pass is incremental: a rescan
re-hashes only files whose recorded mtime or size changed, and all analysis
happens offline from the database alone. The end goal is not individual files
but whole duplicated trees — duplicate extractions, duplicate downloads, copied
project trees — which an operator can consider removing as a unit.

## Design

Goals, in order:

1.  **Find whole duplicate trees, not just files.** The end goal is to identify
    places where the exact same set of files and directories exists at two or
    more paths (duplicate extractions, duplicate downloads, copied project
    trees), so the operator can consider removing an entire subtree at once.
    File-level duplicate detection is the foundation; tree-level detection is
    built on top of it.
2.  **Never read full file contents.** At most 2 KiB is read per file (first and
    last 1024 bytes), and only files whose size at least one other file shares
    are read at all — a size-unique file cannot be a duplicate. Scale target:
    tens of millions of files, ~150 TB filesystem, possibly slow or busy disks
    (ZFS pool under resilver). Holding one small record (path, size, mtime) per
    file in memory during a scan is acceptable; holding every file's hashes is
    not (they stay in the database).
3.  **Scan incrementally, analyze offline.** The expensive filesystem scan
    maintains a persistent database; an unchanged file is never read again on a
    rescan. All analysis (`report`, `trees`) works from the database alone and
    must never touch the scanned filesystem again. `scan` is designed to be
    cronned; the reports run at any time against the last completed scan.
4.  **Clean stream separation.** Everything on stdout is machine-readable data.
    All progress, warnings, and summaries go to stderr. Never mix them.

### Constraints

- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: `sfdupes`.
- Dependencies: standard library, `github.com/spf13/cobra` for the CLI, **one
  progress-bar library** (`github.com/schollz/progressbar/v3`), and **one SQLite
  driver** (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
  `github.com/spf13/viper` is permitted if configuration-file support is ever
  needed, but is not currently used. No other third-party deps.
- Cross-compilation is not a concern. Builds run with cgo disabled (the
  `Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
- Analysis modes (`report`, `trees`) must be deterministic: identical database
  contents, identical output, regardless of the order in which records were
  inserted.

### Subcommands

Three subcommands, all implemented:

1.  `scan` — walk the filesystem and synchronize the database: one signature
    record per regular file.
2.  `report` — file-level duplicate report from the database.
3.  `trees` — tree-level duplicate report: reconstruct the directory hierarchy
    from the database records, compute a Merkle-style digest per directory, and
    report maximal groups of identical trees.

```
sfdupes scan [--workers N] [-x] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
```

### Database

All three subcommands operate on a single SQLite database file:

- Location: the value of the `SFDUPES_DATABASE` environment variable when set
  and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. There is no
  command-line flag.
- `scan` creates the database (and its parent directory) on first use. `report`
  and `trees` require an existing database; a missing database file is a fatal
  error (exit 1) telling the user to run `scan` first.
- The database uses WAL journal mode and a busy timeout, so running a report
  while a cron `scan` is in progress is safe. The filesystem is authoritative;
  the database is an eventually-consistent reflection of it. Hashed records are
  committed in batched transactions while the scan is still running (keeping the
  WAL small and letting concurrent reports observe progress), so a report may
  see a scan's changes partially applied, and a scan that dies partway leaves a
  valid database holding everything hashed so far; the next scan skips those
  records and converges toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a database
  with any other version is a fatal error):

    ```sql
    CREATE TABLE files (
        path  BLOB PRIMARY KEY,  -- absolute path, raw bytes
        size  INTEGER NOT NULL,  -- bytes, from lstat
        mtime INTEGER NOT NULL,  -- Unix seconds, from lstat
        head  TEXT NOT NULL,     -- lowercase-hex SHA-256, first 1 KiB
        tail  TEXT NOT NULL      -- lowercase-hex SHA-256, last 1 KiB
    ) WITHOUT ROWID;
    ```

    Paths are stored as BLOBs because Unix paths are raw bytes, not guaranteed
    UTF-8. `mtime` is used only for change detection; it is not part of the
    duplicate key. `head` and `tail` are empty strings when the file has never
    been hashed because its size was unique as of the last scan that covered it;
    such records still define the file for tree reconstruction but never
    participate in duplicate groups.

### `scan` mode

`scan` requires one or more `PATH` operands naming the trees to scan. There is
no default path; invoking `scan` with no operand is a usage error (usage message
on stderr, exit 2). An operand may be a directory or a regular file; an operand
that does not exist is a fatal error (exit 1). Because database records persist
between runs and are keyed by absolute path, each operand is resolved to an
absolute, lexically cleaned path (symlinks are not resolved) before walking, so
results do not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared walk worker
pool. Overlapping operands are harmless — an operand that duplicates another or
lies under another is dropped before walking, so every file is reached exactly
once and produces one database record.

`scan` synchronizes the database with the filesystem state under the scanned
operands:

- Only a file whose size at least one other file shares is ever read: a
  size-unique file cannot be a duplicate, so it is recorded without hashes
  (`head` and `tail` empty). The size census covers every file walked this scan
  plus every database record outside the scanned operands, so a possible
  duplicate of a separately scanned tree is still recognized.
- A file not yet in the database is inserted: hashed when its size is shared,
  without hashes otherwise.
- A file already in the database is **skipped without reading its contents**
  when its lstat size equals the recorded size and its lstat mtime is not newer
  than the recorded mtime. This is what makes a daily rescan cheap. Exception:
  an unchanged file whose record lacks hashes is hashed — and its record updated
  — once its size becomes shared, so hashing deferred by size-uniqueness happens
  as soon as it could matter.
- A file whose mtime is newer than recorded, or whose size differs, is processed
  as if new: re-hashed, or recorded without hashes, per the shared-size rule.
- A database record whose path lies under one of the scanned operands but was
  not successfully processed this run is deleted. This removes records for
  deleted files. It also removes records for paths that failed to stat or hash
  this run: the database only ever contains signatures verified by the most
  recent scan that covered them (a subsequent successful scan re-adds such
  files).
- Database records outside the scanned operands are untouched, so disjoint trees
  can be scanned on different schedules into the same database.

`scan` runs **three sequential phases over the whole scan**. Parallelism lives
inside each phase; batched database writes begin during the hash phase:

1.  **walk + stat** — enumerate the trees under all `PATH` operands concurrently
    with the walk worker pool: every operand seeds the shared queue, and each
    worker reads one directory at a time, handing discovered subdirectories back
    to the queue and running `lstat` on each regular file as it is discovered
    (while the directory's metadata is still hot). Sequential directory
    enumeration is metadata-latency-bound and takes hours at tens of millions of
    files; per-directory parallelism is what makes the walk tractable on large
    or busy pools. The walk builds the size census and resolves unchanged
    already-hashed files on the fly; every other file is carried to the hash
    phase as a (path, size, mtime) record.
2.  **hash** — with the census complete, each carried file's size decides its
    fate. Size-unique files are never read: new or changed ones are recorded
    without hashes in the update phase, unchanged unhashed ones simply keep
    their records. Every file with a shared size is hashed by the worker pool:
    read the first `min(1024, size)` bytes and the last `min(1024, size)` bytes
    (one read when `size <= 1024`, since the two windows coincide) and compute
    the SHA-256 of each. Zero-length files have constant hashes and are never
    opened. Files are hashed in **inode order** (minimizing seeks on spinning
    disks), and paths that are hard links to the same inode are **read once**,
    all sharing the one result — a hard-link backup farm costs one read per
    inode, not per path. The phase total counts actual reads, so progress and
    ETA are meaningful. Completed records are committed in batched transactions
    **while hashing runs**, so a scan interrupted after hours keeps everything
    hashed so far and the next scan resumes cheaply, skipping records already
    written.
3.  **update** — commit the final partial batch, the hash-less records for
    size-unique new and changed files, and the deletions for records the scan
    did not verify (vanished files, plus paths that failed to stat or hash).

Rules for the walk:

- Only regular files. Skip directories, symlinks (do not follow, including
  symlink operands), sockets, FIFOs, and device nodes.
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs; walking
  them would list every file once per snapshot).
- Filesystem boundaries are crossed by default. With `-x` (long form
  `--one-file-system`, following the GNU `du`/`rsync` convention), never descend
  into a directory on a different filesystem than its `PATH` operand; each
  operand is bounded by its own filesystem.
- On any per-path error (permission denied, file vanished between passes,
  unreadable): print a one-line warning to stderr, skip the path, and continue.
  Per-file errors never abort the run; the final summary reports how many were
  skipped. As specified above, a skipped path that has a database record from an
  earlier scan loses that record; an unreadable directory subtree likewise loses
  its records (accepted: the database mirrors what the latest scan could
  actually verify).

Concurrency: the walk phase (which also stats files) and the hash phase each use
a worker pool of `--workers` workers (default `runtime.NumCPU()`); the walk
parallelizes across directories, hashing across files. Both phases are
seek-bound on spinning disks, so raising `--workers` well past the core count
can help on pools with many spindles. The main goroutine owns partitioning,
database writes, and progress rendering; progress display must never block the
workers.

`scan` writes nothing to stdout. The summary line on stderr reports the files
seen this run broken down by disposition, plus skips:

```
scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
```

(`removed` counts deleted database records, which are not part of the files-seen
total.)

### `report` mode

`report` reads every record from the database and takes no positional arguments.

**`report` must never touch the filesystem being analyzed.** It does not stat,
open, or otherwise access any path that appears in the records; its only I/O is
reading the database and writing stdout/stderr. It must produce identical output
whether or not the scanned filesystem is still mounted.

Processing:

- Records without hashes (size-unique when last scanned) are excluded: their
  content is unknown, so they are never reported as duplicates.
- Group the remaining records by the key `(size, head_hash, tail_hash)`.
- Every group with two or more paths is a duplicate group.
- Within each group, sort paths lexicographically (byte order). The first path
  is the group's `first`; every other path is a `dupe`.
- Order groups by size descending (biggest reclaimable space first), tie-broken
  by `first` path ascending. Output must be fully deterministic for a given
  database state.

#### Report output format

TSV on stdout: a header line, then one row per duplicate file (N-1 rows for a
group of N):

```
first	dupe	size
/srv/a/big.iso	/srv/b/big-copy.iso	4294967296
/srv/a/big.iso	/srv/c/big-copy2.iso	4294967296
```

Summary to stderr: records read, number of duplicate groups, number of dupe
files, and total reclaimable bytes (sum of `size` over all dupe rows) in human
units.

### `trees` mode

`trees` reads the same database as `report` (no positional arguments) and
reports **entire duplicate directory trees**: directories under which the exact
same set of relative paths exists with the exact same file signatures.

**`trees` must never touch the filesystem being analyzed** — the same rule as
`report`. The directory hierarchy is reconstructed purely from the paths in the
records, split on `/`.

Definitions:

- A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
  informational and excluded. An unhashed record (empty hashes) has unknown
  content: its signature is treated as unique to that file, so a tree containing
  an unhashed file never compares equal to any other tree.
- A directory's **digest** is a SHA-256 Merkle digest computed bottom-up:
  serialize the directory's child entries — for a file child, its name and
  signature; for a subdirectory child, its name and that subdirectory's digest —
  sort the serialized entries byte-lexicographically, and hash the
  concatenation. Names are part of the digest: two trees whose files differ only
  in name are _not_ duplicates.
- Two directories are **duplicate trees** when their digests are equal. Equal
  digests imply equal recursive file count and equal total byte size.

Known limitation (accepted): hard-linked paths are reported as duplicates by
`report` and count toward duplicate trees — their content is genuinely identical
— even though they share storage, so removing one reclaims no space. Inode
identity is used during the scan to avoid redundant reads but is not persisted
in the database.

Known limitation (accepted): only regular files that appear in the database
define a tree. Empty directories are invisible, and a file skipped during the
scan (e.g. permission error) in one copy but not the other will make
otherwise-identical trees compare as different.

Processing:

- Build the hierarchy, compute every directory's digest, and group directories
  by digest. Every group with two or more directories is a duplicate-tree group.
- **Report only maximal trees.** A group is suppressed when its members' parents
  are pairwise distinct directories that all share a single digest — such a
  group is wholly implied by its parents' (or a further ancestor's) group.
  Groups containing sibling directories, or members whose parents differ, are
  always reported.
- Within each group, sort paths lexicographically (byte order); the first path
  is `first`, every other path is a `dupe`.
- Order groups by total tree size descending, tie-broken by `first` path
  ascending. Output must be fully deterministic for a given input.

#### Trees output format

TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
group of N). `files` is the recursive regular-file count of one copy of the
tree; `size` is the recursive total byte size of one copy:

```
first	dupe	files	size
/srv/a/project	/srv/backup/project	3417	104857600
```

Summary to stderr: records read, number of duplicate-tree groups, number of dupe
trees, and total reclaimable bytes (sum of `size` over all dupe rows) in human
units.

### Progress

Use the progress-bar library for all scan progress; rendering in the style of
`pv` is the model. All progress goes to stderr.

Each phase gets its own display, rendered the moment the phase starts — a scan
must never look hung. Loading the existing-record index (`load`) and the walk
have no known totals while running: show a live count, rate, and elapsed time
(spinner-style, no percentage or ETA). The hash and update phases have exact
totals — only files that actually need hashing appear in the hash total, so its
ETA is meaningful. Required elements for the bars with known totals:

- elapsed time
- estimated time remaining
- a `[m/n] x%` display (items processed / total items, percent)
- current rate (items/s)

Example shape (exact layout is flexible, content is not):

```
hash: [12345/98765] 12% |████      | 92 files/s elapsed 2:32 eta 17:54
```

Additional requirements:

- When stderr is not a TTY, do not emit ANSI redraws: print a plain one-line
  progress update no more often than every 5 seconds instead.
- Progress updates are driven from the main goroutine and must be non-blocking
  with respect to the worker pool.
- `report` and `trees` modes need no progress display, only their stderr
  summaries.

### Error handling and exit codes

- `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., a `PATH` operand does not exist, the database cannot
  be created/opened/read/written, a missing database for `report`/`trees`,
  stdout write failure).
- `2`: usage error (including `scan` with no `PATH` operand and `report`/`trees`
  with any positional argument).

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: the normalized executables in `script/` are the entrypoints for the
development workflow, and the `Makefile` targets are thin shims that call them.
Every script is POSIX `sh`, resolves the repository root itself so it can be run
from any working directory, and may be invoked directly. The provided
entrypoints are:

- `script/bootstrap` — install everything needed to build and develop this
  repository, idempotently, assuming nothing is present. `git`, `make`, `go`,
  and `node` come from the first of nix, apt, brew, or apk found on the host,
  and are presence-checked only; `node` is an unpinned host runtime like the
  rest, because nvm's prebuilt node is glibc-linked and does not run on this
  repo's musl/Alpine build image. The Markdown formatter itself — `prettier` —
  is pinned by `yarn.lock`'s integrity hash and installed with
  `yarn install --frozen-lockfile`. `golangci-lint` is deliberately **not**
  installed: it runs from a digest-pinned image via `script/lint` and never from
  a host install, so there is no host copy to drift from the pin. A missing
  `docker` is warned about rather than installed or treated as fatal —
  everything except linting works without it. Ends with `go mod download` and
  the `yarn` install.
- `script/setup` — make a fresh clone ready for development: runs
  `script/bootstrap`, then `script/install-precommit`.
- `script/projectname` — print this project's name (`sfdupes`). Scripts that
  need the name call it, so they stay identical across repositories.
- `script/test` — run the test suite with a 30-second timeout and coverage
  enabled, rerunning verbosely on failure so the logs show which test failed.
- `script/lint` — run the linter. It builds `Dockerfile.lint`, which copies the
  repository into the digest-pinned `golangci/golangci-lint` image and runs
  `golangci-lint config verify` and `golangci-lint run` as build steps, so a
  successful build is a clean lint. The linter is never run on the host, which
  makes a working `docker` the one prerequisite for linting — and therefore for
  `make check` and the pre-commit hook. Offline machines: the gate steps
  themselves make no network calls. `golangci-lint run` does not, and neither
  does `golangci-lint config verify` — it validates against a schema the pinned
  binary embeds, measured under `--network none` to both pass a valid config and
  reject an invalid one. The build around them does. `Dockerfile.lint` runs
  `go mod download` before the gates and this module has external dependencies,
  so a first lint on a machine with a cold BuildKit cache reaches the network
  there (as well as pulling the pinned image); under `--network none` it fails
  at that step, before any gate. That layer sits above the gates and stays
  cached, so once it is warm `script/lint` — and with it `make check` — runs
  entirely offline, until `go.mod` or `go.sum` changes and the download layer
  goes cold again. Because the daemon only ever sees a build context, this works
  when the docker daemon is remote and bind mounts are impossible.
- `script/fmt` — format in place: `gofmt -s -w` for Go sources and `prettier`
  for Markdown (`--tab-width 4 --prose-wrap always`, the house settings, also
  carried in `.prettierrc`). prettier is the pinned devDependency in
  `package.json`/`yarn.lock`, installed by `script/bootstrap`.
- `script/fmt-check` — the read-only counterpart of `script/fmt`: runs both
  checks, reports each independently so it is clear which failed, and exits
  non-zero if either found unformatted files instead of writing.
- `script/check` — run `script/test`, `script/lint`, and `script/fmt-check`, in
  that order. Modifies nothing. Needs `docker`, because `script/lint` does.
- `script/docker` — build the Docker image, tagged with the name from
  `script/projectname`. The `Dockerfile` runs the gates as build steps, so this
  is also the check a developer or reviewer runs by hand.
- `script/cibuild` — build the Docker image untagged. This is what the Gitea
  workflow runs on push; because the gates run as build steps, a successful
  build implies the repository is green.
- `script/precommit` — run by the git pre-commit hook: `go mod tidy` must be a
  no-op (a resulting change to `go.mod` or `go.sum` fails the commit), then
  `script/check`.
- `script/install-precommit` — install the git pre-commit hook that runs
  `script/precommit`. The hook is written to the common git directory, so the
  main checkout and every worktree share it.
- `script/verify-lint-image-pin` — fail unless the `golangci/golangci-lint`
  reference in `Dockerfile.lint` and the one in the `Dockerfile` lint stage are
  the same image at the same digest, naming both if not. The linter is pinned in
  those two files and nothing else keeps them in sync, so a bump applied to one
  alone would leave `make lint` and the `Dockerfile`'s fail-fast lint stage
  checking the same tree against different rulesets, both green. The guard
  restates neither pin — a third copy would be the same drift one file further
  out — and runs as a gate in both files, so `make lint`, `make check` and
  `make docker` all catch it.

`script/verify-linter-pin` used to live here. It compared a linter binary
against a version pin in `script/bootstrap`, and both of its subjects are gone:
no linter binary is copied between build stages any more, and bootstrap pins no
version because it installs no linter. The drift it existed to catch has moved
from binary-versus-pin to pin-versus-pin, which is what
`script/verify-lint-image-pin` above checks.

`script/lint`, `script/docker` and `script/cibuild` all pass a freshly computed
`CHECK_EPOCH` build argument, and the gate steps in `Dockerfile.lint` and
`Dockerfile` reference it. Without that, an unchanged tree lets Docker serve the
gate layers from cache and the build exits 0 having executed no tests and no
lint — a green it never earned, and one this repository has produced twice.
`CHECK_EPOCH` invalidates the gate layers on every run while leaving the pinned
base images and the dependency layers cached. `script/lint`'s value carries the
process id as well as the epoch, because two lint runs land inside the same
second easily and a bare epoch would cache the second one.

## Build

The `script/` entrypoints above are where the implementations live; the
`Makefile` targets are shims onto them, except `build`, which carries the
compile recipe:

- `make` / `make build` — build the `sfdupes` binary (cgo disabled); building is
  the default target.
- `make bootstrap` — install the build and development dependencies.
- `make setup` — prepare a fresh clone: `bootstrap` plus the pre-commit hook.
- `make test` — run the test suite (30-second timeout; reruns with `-v` on
  failure).
- `make lint` — run `golangci-lint` with the repo config, in Docker (see
  `script/lint`); requires `docker`.
- `make fmt` / `make fmt-check` — format Go and Markdown sources / verify both
  without writing.
- `make check` — `test`, `lint`, and `fmt-check`; modifies nothing. Requires
  `docker`, via `lint`.
- `make docker` — build the Docker image, which runs the gates as build stages.
- `make hooks` — install the pre-commit hook.
- `make clean` — remove the binary.

### Definition of done

All of the following, run in this directory, must pass:

1.  `make check` passes (tests, lint, `gofmt`).
2.  `make docker` succeeds.
3.  Smoke test — create a throwaway tree in a temp dir (never test against real
    data):

    ```sh
    d=$(mktemp -d)
    export SFDUPES_DATABASE="$d/db.sqlite"
    mkdir -p "$d/a" "$d/b"
    head -c 2000 /dev/urandom > "$d/a/one.bin"
    cp "$d/a/one.bin" "$d/b/copy.bin"
    cp "$d/a/one.bin" "$d/b/copy2.bin"
    head -c 2000 /dev/urandom > "$d/a/unique.bin"     # same size, different content
    printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2"  # 1-byte duplicates
    printf 'y' > "$d/tiny3"                           # 1-byte non-duplicate
    : > "$d/empty1"; : > "$d/empty2"                  # empty duplicates
    # duplicate trees: t1 and t2 are identical; t3 differs by one filename
    mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
    head -c 3000 /dev/urandom > "$d/t1/f1"
    head -c 100  /dev/urandom > "$d/t1/sub/f2"
    cp "$d/t1/f1" "$d/t2/f1"
    cp "$d/t1/sub/f2" "$d/t2/sub/f2"
    cp "$d/t1/f1" "$d/t3/f1"
    cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
    ./sfdupes scan "$d"
    ./sfdupes report
    ./sfdupes trees
    # incremental behavior (scan a subtree; records elsewhere persist):
    ./sfdupes scan "$d/a"                  # everything unchanged, nothing hashed
    printf 'z' >> "$d/a/one.bin"           # modify: next scan re-hashes it
    rm "$d/a/unique.bin"                   # delete: next scan removes its record
    ./sfdupes scan "$d/a"                  # 1 updated, 1 removed
    ./sfdupes report
    ```

    (The scan database lives inside `$d` here purely for test hygiene; scanning
    `$d` therefore also records the SQLite file itself, which is harmless.)

    Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
    group (two dupe rows, `first` is the lexicographically smallest path);
    `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
    `t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
    `empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere; groups
    ordered by size descending.

    Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2
    files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal
    (implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set
    differs by name).

    Expected from the second `report` (after the modify/delete rescan):
    `one.bin` has left its group (its content changed), so
    `copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is gone from the
    database.

    The test suite automates this scenario (see `scan_test.go`), plus a negative
    check: `report` and `trees` operate on the database alone and never touch
    the scanned filesystem.

## TODO

Tracked in [TODO.md](TODO.md).

## Non-goals

- No full-content verification, no byte-for-byte compare, no deletion or linking
  of duplicates. The reports are advisory; acting on them is the user's job.
- No persistence beyond the SQLite database described above; no export/import
  formats.
- No daemon or filesystem watcher; scheduling rescans is cron's job.

## License

MIT. See [LICENSE](LICENSE).

## Author

[@sneak](https://sneak.berlin)