sfdupes/README.md

# sfdupes

## Description

`sfdupes` is an MIT-licensed Go CLI tool by
[@sneak](https://sneak.berlin) that quickly identifies *candidate*
duplicate files — and, ultimately, entire duplicate directory trees —
across very large filesystems without reading every byte of every file.
Files are considered duplicates when their sizes are equal and they
agree on a short ladder of hashes. A file under 10 MiB is hashed in full
and compared directly. A larger file is gated first on the SHA-256 of
its first 64 KiB and of its last 64 KiB, and then compared on a content
hash — the SHA-256 of the whole file when it is under 50 MiB, or of
gigabyte-spaced 1 MiB samples when it is 50 MiB or larger. Below 50 MiB
the content hash is proof of identical content; at or above 50 MiB it is
a strong candidate signal rather than proof, because the gaps between
samples are never read. The intended use is
finding duplicate downloads and duplicated directory trees on
multi-terabyte ZFS servers where reading every byte of every file is
prohibitively expensive. `scan` maintains a persistent SQLite database of file
signatures that survives between runs, so it can be run from cron and
the reports can be generated at any time from the most recent scan.

This README is the complete and authoritative specification.

## Getting Started

```sh
make build
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes scan /srv
./sfdupes report > dupes.tsv
./sfdupes trees > dupetrees.tsv
```

`scan` walks one or more filesystem trees and maintains one database
record per regular file (path, size, mtime, head hash, tail hash,
content hash). The
database persists between runs; a rescan only hashes files that are new
or changed, and removes records for files that no longer exist.
`report` reads the database and prints the file-level duplicates
report. `trees` reads the same database and prints the duplicate-tree
report. A missing/invalid subcommand — or a `scan` invocation with no
`PATH` operand — prints a usage message and exits 2.

The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed
anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a
daily `sfdupes scan` cron job, with the reporting commands run
interactively whenever needed; their results are as fresh as the last
completed scan.

## Rationale

Duplicate finders that hash entire files do not scale to the target
environment: ~10 million files and ~150 TB on possibly slow or busy
disks (a ZFS pool under resilver). sfdupes spends disk I/O only on files
whose size at least one other file shares, since a size-unique file
cannot be a duplicate; for those it reads the cheap end windows first
and a content hash second — the whole file below 50 MiB, but only
gigabyte-spaced samples at or above 50 MiB, so the largest files are
never read in full. This keeps a full-filesystem sweep tractable, and
the signatures are kept in a persistent database, so
the expensive filesystem pass is incremental: a rescan re-hashes only
files whose recorded mtime or size changed, and all analysis happens
offline from the database alone. The end goal is
not individual files but whole duplicated trees — duplicate
extractions, duplicate downloads, copied project trees — which an
operator can consider removing as a unit.

## Design

Goals, in order:

1.  **Find whole duplicate trees, not just files.** The end goal is to
    identify places where the exact same set of files and directories
    exists at two or more paths (duplicate extractions, duplicate
    downloads, copied project trees), so the operator can consider
    removing an entire subtree at once. File-level duplicate detection is
    the foundation; tree-level detection is built on top of it.
2.  **Spend I/O in proportion to duplicate likelihood.** Only files
    whose size at least one other file shares are read at all — a
    size-unique file cannot be a duplicate. Those are compared by the
    ladder in "Duplicate detection" below: a file under 10 MiB is hashed
    in full, while a larger file is gated on cheap 64 KiB end windows
    first, then a content hash that reads the whole file below 50 MiB
    but only gigabyte-spaced 1 MiB samples at or above it, so the very
    largest files are still never read in full. Scale target: tens of
    millions of files, ~150 TB filesystem, possibly slow or busy disks
    (ZFS pool under resilver). Holding one small record (path, size,
    mtime) per file in memory during a scan is acceptable; holding
    every file's hashes is not (they stay in the database).
3.  **Scan incrementally, analyze offline.** The expensive filesystem
    scan maintains a persistent database; an unchanged file is never
    read again on a rescan. All analysis (`report`, `trees`) works from
    the database alone and must never touch the scanned filesystem
    again. `scan` is designed to be cronned; the reports run at any
    time against the last completed scan.
4.  **Clean stream separation.** Everything on stdout is machine-readable
    data. All progress, warnings, and summaries go to stderr. Never mix
    them.

### Constraints

-   Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
    `sfdupes`.
-   Dependencies: standard library, `github.com/spf13/cobra` for the
    CLI, **one progress-bar library**
    (`github.com/schollz/progressbar/v3`), and **one SQLite driver**
    (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
    `github.com/spf13/viper` is permitted if configuration-file support
    is ever needed, but is not currently used. No other third-party
    deps.
-   Cross-compilation is not a concern. Builds run with cgo disabled (the
    `Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
-   Analysis modes (`report`, `trees`) must be deterministic: identical
    database contents, identical output, regardless of the order in
    which records were inserted.

### Subcommands

Three subcommands, all implemented:

1.  `scan` — walk the filesystem and synchronize the database: one
    signature record per regular file.
2.  `report` — file-level duplicate report from the database.
3.  `trees` — tree-level duplicate report: reconstruct the directory
    hierarchy from the database records, compute a Merkle-style digest
    per directory, and report maximal groups of identical trees.

```
sfdupes scan [--workers N] [-x] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
```

### Database

All three subcommands operate on a single SQLite database file:

-   Location: the value of the `SFDUPES_DATABASE` environment variable
    when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`.
    There is no command-line flag.
-   `scan` creates the database (and its parent directory) on first
    use. `report` and `trees` require an existing database; a missing
    database file is a fatal error (exit 1) telling the user to run
    `scan` first.
-   The database uses WAL journal mode and a busy timeout, so running a
    report while a cron `scan` is in progress is safe. The filesystem
    is authoritative; the database is an eventually-consistent
    reflection of it. Hashed records are committed in batched
    transactions while the scan is still running (keeping the WAL
    small and letting concurrent reports observe progress), so a
    report may see a scan's changes partially applied, and a scan
    that dies partway leaves a valid database holding everything
    hashed so far; the next scan skips those records and converges
    toward the filesystem.
-   Schema (`PRAGMA user_version` is the schema version, currently 2; a
    database with any other version is a fatal error). Version 2 added
    the `content` column and the 64 KiB head/tail signature (replacing
    the version 1 1 KiB end windows), so a version 1 database cannot be
    reused: it is rejected and the tree must be rescanned from scratch.

    ```sql
    CREATE TABLE files (
        path    BLOB PRIMARY KEY,  -- absolute path, raw bytes
        size    INTEGER NOT NULL,  -- bytes, from lstat
        mtime   INTEGER NOT NULL,  -- Unix seconds, from lstat
        head    TEXT NOT NULL,     -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB
        tail    TEXT NOT NULL,     -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB
        content TEXT NOT NULL      -- lowercase-hex SHA-256, whole file or samples
    ) WITHOUT ROWID;
    ```

    Paths are stored as BLOBs because Unix paths are raw bytes, not
    guaranteed UTF-8. `mtime` is used only for change detection; it is
    not part of the duplicate key. For a file under 10 MiB `head`, `tail`,
    and `content` all hold the whole-file hash (that range is hashed in
    full, with no end windows); for a larger file `head` and `tail` hold
    the first- and last-64 KiB hashes and `content` the whole-file or
    sampled hash. All three are empty strings when the file has never
    been hashed because its size was unique as of the last scan that
    covered it; such records still define the file for tree
    reconstruction but never participate in duplicate groups.

### Duplicate detection

Two files are duplicates only when they agree on every rung of this
ladder; a mismatch at any rung means they are not duplicates. `scan`
stores each file's hashes once, and `report` and `trees` group files by
the whole signature — size, `head`, `tail`, and `content` — so the
grouping is exactly this ladder applied across everything scanned into
the database, even across separate scans.

1.  **Size.** Files of different sizes are never compared. Only files
    whose size at least one other file shares are hashed at all.
2.  **Under 10 MiB: whole file.** A file smaller than 10 MiB is hashed
    in full and compared directly, with no separate end-window step —
    small files are cheap to read to the last byte, and doing so makes
    the comparison exact. `head`, `tail`, and `content` all hold this
    whole-file SHA-256, so such a file's signature is decided entirely
    by its size and its content.
3.  **10 MiB and above: head and tail.** For a larger file, the SHA-256
    of the first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a
    cheap gate that eliminates most same-size pairs before any bulk
    reading. At 10 MiB and above the two windows never overlap.
4.  **10 MiB and above, content below 50 MiB.** The SHA-256 of the
    entire file. Agreement here is proof of identical content (barring a
    SHA-256 collision).
5.  **10 MiB and above, content 50 MiB and above.** A sampled SHA-256:
    the 1 MiB window at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, …
    while inside the file, the final window truncated at end of file) is
    fed, in order, into one hash. This is **deliberately probabilistic**
    — the gaps between samples are never read, so two large files that
    agree on every sample are reported as duplicates without being read
    in full. It is the price of never reading a 150 GB file end to end.
    Because size is already part of the signature, only equal-size files
    reach this rung, so their sample boundaries always align.

`head`, `tail`, and `content` are one column each. A file below 10 MiB
and one at or above it never share a size, and neither do a file below
50 MiB and one at or above it, so a stored value is never ambiguous
between the whole-file, end-window, and sampled forms.

### `scan` mode

`scan` requires one or more `PATH` operands naming the trees to scan.
There is no default path; invoking `scan` with no operand is a usage
error (usage message on stderr, exit 2). An operand may be a directory
or a regular file; an operand that does not exist is a fatal error
(exit 1). Because database records persist between runs and are keyed
by absolute path, each operand is resolved to an absolute, lexically
cleaned path (symlinks are not resolved) before walking, so results do
not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared
walk worker pool. Overlapping operands are harmless — an operand that
duplicates another or lies under another is dropped before walking,
so every file is reached exactly once and produces one database
record.

`scan` synchronizes the database with the filesystem state under the
scanned operands:

-   Only a file whose size at least one other file shares is ever
    read: a size-unique file cannot be a duplicate, so it is recorded
    without hashes (`head` and `tail` empty). The size census covers
    every file walked this scan plus every database record outside
    the scanned operands, so a possible duplicate of a separately
    scanned tree is still recognized.
-   A file not yet in the database is inserted: hashed when its size
    is shared, without hashes otherwise.
-   A file already in the database is **skipped without reading its
    contents** when its lstat size equals the recorded size and its
    lstat mtime is not newer than the recorded mtime. This is what
    makes a daily rescan cheap. Exception: an unchanged file whose
    record lacks hashes is hashed — and its record updated — once its
    size becomes shared, so hashing deferred by size-uniqueness
    happens as soon as it could matter.
-   A file whose mtime is newer than recorded, or whose size differs,
    is processed as if new: re-hashed, or recorded without hashes,
    per the shared-size rule.
-   A database record whose path lies under one of the scanned operands
    but was not successfully processed this run is deleted. This
    removes records for deleted files. It also removes records for
    paths that failed to stat or hash this run: the database only ever
    contains signatures verified by the most recent scan that covered
    them (a subsequent successful scan re-adds such files).
-   Database records outside the scanned operands are untouched, so
    disjoint trees can be scanned on different schedules into the same
    database.

`scan` runs **three sequential phases over the whole scan**.
Parallelism lives inside each phase; batched database writes begin
during the hash phase:

1.  **walk + stat** — enumerate the trees under all `PATH` operands
    concurrently with the walk worker pool: every operand seeds the
    shared queue, and each worker reads one directory at a time,
    handing discovered subdirectories back to the queue and running
    `lstat` on each regular file as it is discovered (while the
    directory's metadata is still hot). Sequential directory
    enumeration is metadata-latency-bound and takes hours at tens of
    millions of files; per-directory parallelism is what makes the
    walk tractable on large or busy pools. The walk builds the size
    census and resolves unchanged already-hashed files on the fly;
    every other file is carried to the hash phase as a (path, size,
    mtime) record.
2.  **hash** — with the census complete, each carried file's size
    decides its fate. Size-unique files are never read: new or
    changed ones are recorded without hashes in the update phase,
    unchanged unhashed ones simply keep their records. Every file
    with a shared size is hashed by the worker pool, computing the
    full signature — head, tail, and content — described in "Duplicate
    detection" below. Zero-length files have constant
    hashes and are never opened. Files are hashed in **inode order**
    (minimizing seeks on spinning disks), and paths that are hard
    links to the same inode are **read once**, all sharing the one
    result — a hard-link backup farm costs one read per inode, not
    per path. The phase total counts actual reads, so progress and
    ETA are meaningful. Completed records are committed in batched
    transactions **while hashing runs**, so a scan interrupted after
    hours keeps everything hashed so far and the next scan resumes
    cheaply, skipping records already written.
3.  **update** — commit the final partial batch, the hash-less
    records for size-unique new and changed files, and the deletions
    for records the scan did not verify (vanished files, plus paths
    that failed to stat or hash).

Rules for the walk:

-   Only regular files. Skip directories, symlinks (do not follow,
    including symlink operands), sockets, FIFOs, and device nodes.
-   Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs;
    walking them would list every file once per snapshot).
-   Filesystem boundaries are crossed by default. With `-x`
    (long form `--one-file-system`, following the GNU `du`/`rsync`
    convention), never descend into a directory on a different
    filesystem than its `PATH` operand; each operand is bounded by its
    own filesystem.
-   On any per-path error (permission denied, file vanished between
    passes, unreadable): print a one-line warning to stderr, skip the
    path, and continue. Per-file errors never abort the run; the final
    summary reports how many were skipped. As specified above, a
    skipped path that has a database record from an earlier scan loses
    that record; an unreadable directory subtree likewise loses its
    records (accepted: the database mirrors what the latest scan could
    actually verify).

Concurrency: the walk phase (which also stats files) and the hash
phase each use a worker pool of `--workers` workers (default
`runtime.NumCPU()`); the walk parallelizes across directories,
hashing across files. Both phases are seek-bound on spinning disks,
so raising `--workers` well past the core count can help on pools
with many spindles. The main goroutine owns partitioning, database
writes, and progress rendering; progress display must never block
the workers.

`scan` writes nothing to stdout. The summary line on stderr reports the
files seen this run broken down by disposition, plus skips:

```
scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
```

(`removed` counts deleted database records, which are not part of the
files-seen total.)

### `report` mode

`report` reads every record from the database and takes no positional
arguments.

**`report` must never touch the filesystem being analyzed.** It does not
stat, open, or otherwise access any path that appears in the records; its
only I/O is reading the database and writing stdout/stderr. It must
produce identical output whether or not the scanned filesystem is still
mounted.

Processing:

-   Records without hashes (size-unique when last scanned) are
    excluded: their content is unknown, so they are never reported as
    duplicates.
-   Group the remaining records by the key
    `(size, head_hash, tail_hash)`.
-   Every group with two or more paths is a duplicate group.
-   Within each group, sort paths lexicographically (byte order). The
    first path is the group's `first`; every other path is a `dupe`.
-   Order groups by size descending (biggest reclaimable space first),
    tie-broken by `first` path ascending. Output must be fully
    deterministic for a given database state.

#### Report output format

TSV on stdout: a header line, then one row per duplicate file (N-1 rows
for a group of N):

```
first	dupe	size
/srv/a/big.iso	/srv/b/big-copy.iso	4294967296
/srv/a/big.iso	/srv/c/big-copy2.iso	4294967296
```

Summary to stderr: records read, number of duplicate groups, number of
dupe files, and total reclaimable bytes (sum of `size` over all dupe
rows) in human units.

### `trees` mode

`trees` reads the same database as `report` (no positional arguments)
and reports **entire duplicate directory trees**: directories under
which the exact same set of relative paths exists with the exact same
file signatures.

**`trees` must never touch the filesystem being analyzed** — the same
rule as `report`. The directory hierarchy is reconstructed purely from
the paths in the records, split on `/`.

Definitions:

-   A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
    informational and excluded. An unhashed record (empty hashes) has
    unknown content: its signature is treated as unique to that file,
    so a tree containing an unhashed file never compares equal to any
    other tree.
-   A directory's **digest** is a SHA-256 Merkle digest computed
    bottom-up: serialize the directory's child entries — for a file
    child, its name and signature; for a subdirectory child, its name
    and that subdirectory's digest — sort the serialized entries
    byte-lexicographically, and hash the concatenation. Names are part
    of the digest: two trees whose files differ only in name are *not*
    duplicates.
-   Two directories are **duplicate trees** when their digests are
    equal. Equal digests imply equal recursive file count and equal
    total byte size.

Known limitation (accepted): hard-linked paths are reported as
duplicates by `report` and count toward duplicate trees — their
content is genuinely identical — even though they share storage, so
removing one reclaims no space. Inode identity is used during the
scan to avoid redundant reads but is not persisted in the database.

Known limitation (accepted): only regular files that appear in the
database define a tree. Empty directories are invisible, and a file
skipped during the scan (e.g. permission error) in one copy but not the
other will make otherwise-identical trees compare as different.

Processing:

-   Build the hierarchy, compute every directory's digest, and group
    directories by digest. Every group with two or more directories is a
    duplicate-tree group.
-   **Report only maximal trees.** A group is suppressed when its
    members' parents are pairwise distinct directories that all share a
    single digest — such a group is wholly implied by its parents' (or a
    further ancestor's) group. Groups containing sibling directories, or
    members whose parents differ, are always reported.
-   Within each group, sort paths lexicographically (byte order); the
    first path is `first`, every other path is a `dupe`.
-   Order groups by total tree size descending, tie-broken by `first`
    path ascending. Output must be fully deterministic for a given
    input.

#### Trees output format

TSV on stdout: a header line, then one row per duplicate tree (N-1 rows
for a group of N). `files` is the recursive regular-file count of one
copy of the tree; `size` is the recursive total byte size of one copy:

```
first	dupe	files	size
/srv/a/project	/srv/backup/project	3417	104857600
```

Summary to stderr: records read, number of duplicate-tree groups,
number of dupe trees, and total reclaimable bytes (sum of `size` over
all dupe rows) in human units.

### Progress

Use the progress-bar library for all scan progress; rendering in the
style of `pv` is the model. All progress goes to stderr.

Each phase gets its own display, rendered the moment the phase
starts — a scan must never look hung. Loading the existing-record
index (`load`) and the walk have no known totals while running: show
a live count, rate, and elapsed time (spinner-style, no percentage or
ETA). The hash and update phases
have exact totals — only files that actually need hashing appear in
the hash total, so its ETA is meaningful. Required elements for the
bars with known totals:

-   elapsed time
-   estimated time remaining
-   a `[m/n] x%` display (items processed / total items, percent)
-   current rate (items/s)

Example shape (exact layout is flexible, content is not):

```
hash: [12345/98765] 12% |████      | 92 files/s elapsed 2:32 eta 17:54
```

Additional requirements:

-   When stderr is not a TTY, do not emit ANSI redraws: print a plain
    one-line progress update no more often than every 5 seconds instead.
-   Progress updates are driven from the main goroutine and must be
    non-blocking with respect to the worker pool.
-   `report` and `trees` modes need no progress display, only their
    stderr summaries.

### Error handling and exit codes

-   `0`: success, even if individual files were skipped with warnings.
-   `1`: fatal error (e.g., a `PATH` operand does not exist, the
    database cannot be created/opened/read/written, a missing database
    for `report`/`trees`, stdout write failure).
-   `2`: usage error (including `scan` with no `PATH` operand and
    `report`/`trees` with any positional argument).

## Entrypoints

This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: the normalized executables in `script/` are the entrypoints
for the development workflow, and the `Makefile` targets are thin
shims that call them. Every script is POSIX `sh`, resolves the
repository root itself so it can be run from any working directory,
and may be invoked directly. The provided entrypoints are:

-   `script/bootstrap` — install everything needed to build and
    develop this repository, idempotently, assuming nothing is
    present. `git`, `make`, and `go` come from the first of nix, apt,
    brew, or apk found on the host, and are presence-checked only.
    `golangci-lint` is deliberately **not** installed: it runs from a
    digest-pinned image via `script/lint` and never from a host
    install, so there is no host copy to drift from the pin. A missing
    `docker` is warned about rather than installed or treated as
    fatal — everything except linting works without it. Ends with
    `go mod download`.
-   `script/setup` — make a fresh clone ready for development: runs
    `script/bootstrap`, then `script/install-precommit`.
-   `script/projectname` — print this project's name (`sfdupes`).
    Scripts that need the name call it, so they stay identical across
    repositories.
-   `script/test` — run the test suite with a 30-second timeout and
    coverage enabled, rerunning verbosely on failure so the logs show
    which test failed.
-   `script/lint` — run the linter. It builds `Dockerfile.lint`, which
    copies the repository into the digest-pinned
    `golangci/golangci-lint` image and runs
    `golangci-lint config verify` and `golangci-lint run` as build
    steps, so a successful build is a clean lint. The linter is never
    run on the host, which makes a working `docker` the one
    prerequisite for linting — and therefore for `make check` and the
    pre-commit hook. Offline machines: the gate steps themselves make
    no network calls. `golangci-lint run` does not, and neither does
    `golangci-lint config verify` — it validates against a schema the
    pinned binary embeds, measured under `--network none` to both
    pass a valid config and reject an invalid one. The build around
    them does. `Dockerfile.lint` runs `go mod download` before the
    gates and this module has external dependencies, so a first lint
    on a machine with a cold BuildKit cache reaches the network there
    (as well as pulling the pinned image); under `--network none` it
    fails at that step, before any gate. That layer sits above the
    gates and stays cached, so once it is warm `script/lint` — and
    with it `make check` — runs entirely offline, until `go.mod` or
    `go.sum` changes and the download layer goes cold again. Because
    the daemon only ever sees a build context, this works when the
    docker daemon is remote and bind mounts are impossible.
-   `script/fmt` — format the Go sources in place (`gofmt -s -w`).
    Markdown is not formatted.
-   `script/fmt-check` — the read-only counterpart of `script/fmt`:
    prints any unformatted file and exits non-zero instead of writing.
-   `script/check` — run `script/test`, `script/lint`, and
    `script/fmt-check`, in that order. Modifies nothing. Needs
    `docker`, because `script/lint` does.
-   `script/docker` — build the Docker image, tagged with the name
    from `script/projectname`. The `Dockerfile` runs the gates as
    build steps, so this is also the check a developer or reviewer
    runs by hand.
-   `script/cibuild` — build the Docker image untagged. This is what
    the Gitea workflow runs on push; because the gates run as build
    steps, a successful build implies the repository is green.
-   `script/precommit` — run by the git pre-commit hook: `go mod tidy`
    must be a no-op (a resulting change to `go.mod` or `go.sum` fails
    the commit), then `script/check`.
-   `script/install-precommit` — install the git pre-commit hook that
    runs `script/precommit`. The hook is written to the common git
    directory, so the main checkout and every worktree share it.
-   `script/verify-lint-image-pin` — fail unless the
    `golangci/golangci-lint` reference in `Dockerfile.lint` and the
    one in the `Dockerfile` lint stage are the same image at the same
    digest, naming both if not. The linter is pinned in those two
    files and nothing else keeps them in sync, so a bump applied to
    one alone would leave `make lint` and the `Dockerfile`'s
    fail-fast lint stage checking the same tree against different
    rulesets, both green. The guard restates neither pin — a third
    copy would be the same drift one file further out — and runs as a
    gate in both files, so `make lint`, `make check` and `make docker`
    all catch it.

`script/verify-linter-pin` used to live here. It compared a linter
binary against a version pin in `script/bootstrap`, and both of its
subjects are gone: no linter binary is copied between build stages any
more, and bootstrap pins no version because it installs no linter. The
drift it existed to catch has moved from binary-versus-pin to
pin-versus-pin, which is what `script/verify-lint-image-pin` above
checks.

`script/lint`, `script/docker` and `script/cibuild` all pass a freshly
computed `CHECK_EPOCH` build argument, and the gate steps in
`Dockerfile.lint` and `Dockerfile` reference it. Without that, an
unchanged tree lets Docker serve the gate layers from cache and the
build exits 0 having executed no tests and no lint — a green it never
earned, and one this repository has produced twice. `CHECK_EPOCH`
invalidates the gate layers on every run while leaving the pinned base
images and the dependency layers cached. `script/lint`'s value carries
the process id as well as the epoch, because two lint runs land inside
the same second easily and a bare epoch would cache the second one.

## Build

The `script/` entrypoints above are where the implementations live;
the `Makefile` targets are shims onto them, except `build`, which
carries the compile recipe:

-   `make` / `make build` — build the `sfdupes` binary (cgo
    disabled); building is the default target.
-   `make bootstrap` — install the build and development
    dependencies.
-   `make setup` — prepare a fresh clone: `bootstrap` plus the
    pre-commit hook.
-   `make test` — run the test suite (30-second timeout; reruns with
    `-v` on failure).
-   `make lint` — run `golangci-lint` with the repo config, in Docker
    (see `script/lint`); requires `docker`.
-   `make fmt` / `make fmt-check` — format Go sources / verify
    formatting without writing.
-   `make check` — `test`, `lint`, and `fmt-check`; modifies nothing.
    Requires `docker`, via `lint`.
-   `make docker` — build the Docker image, which runs the gates as
    build stages.
-   `make hooks` — install the pre-commit hook.
-   `make clean` — remove the binary.

### Definition of done

All of the following, run in this directory, must pass:

1.  `make check` passes (tests, lint, `gofmt`).
2.  `make docker` succeeds.
3.  Smoke test — create a throwaway tree in a temp dir (never test
    against real data):

    ```sh
    d=$(mktemp -d)
    export SFDUPES_DATABASE="$d/db.sqlite"
    mkdir -p "$d/a" "$d/b"
    head -c 2000 /dev/urandom > "$d/a/one.bin"
    cp "$d/a/one.bin" "$d/b/copy.bin"
    cp "$d/a/one.bin" "$d/b/copy2.bin"
    head -c 2000 /dev/urandom > "$d/a/unique.bin"     # same size, different content
    printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2"  # 1-byte duplicates
    printf 'y' > "$d/tiny3"                           # 1-byte non-duplicate
    : > "$d/empty1"; : > "$d/empty2"                  # empty duplicates
    # duplicate trees: t1 and t2 are identical; t3 differs by one filename
    mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
    head -c 3000 /dev/urandom > "$d/t1/f1"
    head -c 100  /dev/urandom > "$d/t1/sub/f2"
    cp "$d/t1/f1" "$d/t2/f1"
    cp "$d/t1/sub/f2" "$d/t2/sub/f2"
    cp "$d/t1/f1" "$d/t3/f1"
    cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
    ./sfdupes scan "$d"
    ./sfdupes report
    ./sfdupes trees
    # incremental behavior (scan a subtree; records elsewhere persist):
    ./sfdupes scan "$d/a"                  # everything unchanged, nothing hashed
    printf 'z' >> "$d/a/one.bin"           # modify: next scan re-hashes it
    rm "$d/a/unique.bin"                   # delete: next scan removes its record
    ./sfdupes scan "$d/a"                  # 1 updated, 1 removed
    ./sfdupes report
    ```

    (The scan database lives inside `$d` here purely for test hygiene;
    scanning `$d` therefore also records the SQLite file itself, which
    is harmless.)

    Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin`
    form one group (two dupe rows, `first` is the lexicographically
    smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group;
    `t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group;
    `tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and
    `tiny3` appear nowhere; groups ordered by size descending.

    Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`
    `$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is
    suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
    appears nowhere (its file set differs by name).

    Expected from the second `report` (after the modify/delete rescan):
    `one.bin` has left its group (its content changed), so
    `copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is
    gone from the database.

    The test suite automates this scenario (see `scan_test.go`), plus a
    negative check: `report` and `trees` operate on the database alone
    and never touch the scanned filesystem.

## TODO

Tracked in [TODO.md](TODO.md).

## Non-goals

-   No full-content verification, no byte-for-byte compare, no deletion
    or linking of duplicates. The reports are advisory; acting on them is
    the user's job.
-   No persistence beyond the SQLite database described above; no
    export/import formats.
-   No daemon or filesystem watcher; scheduling rescans is cron's job.

## License

MIT. See [LICENSE](LICENSE).

## Author

[@sneak](https://sneak.berlin)