diff --git a/README.md b/README.md index fcaed1a..02b1ee0 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,12 @@ SHA-256 of their first 1024 bytes, and identical SHA-256 of their last content (the middle of the file is never read); the intended use is finding duplicate downloads and duplicated directory trees on multi-terabyte ZFS servers where reading every byte is prohibitively -expensive. +expensive. `scan` maintains a persistent SQLite database of file +signatures that survives between runs, so it can be run from cron and +the reports can be generated at any time from the most recent scan. + +This tool was created by [@sneak](https://sneak.berlin) to scratch an itch, +using Claude Code/Fable. This README is the complete and authoritative specification. @@ -20,29 +25,39 @@ This README is the complete and authoritative specification. ```sh make build -./sfdupes scan /srv > files.dat -./sfdupes report files.dat > dupes.tsv -./sfdupes trees files.dat > dupetrees.tsv +export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite" +./sfdupes scan /srv +./sfdupes report > dupes.tsv +./sfdupes trees > dupetrees.tsv ``` -`scan` walks one or more filesystem trees and emits one record per -regular file (path, size, mtime, head hash, tail hash). `report` -ingests that stream and prints the file-level duplicates report. -`trees` ingests the same stream and prints the duplicate-tree report. A -missing/invalid subcommand — or a `scan` invocation with no `PATH` -operand — prints a usage message and exits 2. +`scan` walks one or more filesystem trees and maintains one database +record per regular file (path, size, mtime, head hash, tail hash). The +database persists between runs; a rescan only hashes files that are new +or changed, and removes records for files that no longer exist. +`report` reads the database and prints the file-level duplicates +report. `trees` reads the same database and prints the duplicate-tree +report. A missing/invalid subcommand — or a `scan` invocation with no +`PATH` operand — prints a usage message and exits 2. + +The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed +anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a +daily `sfdupes scan` cron job, with the reporting commands run +interactively whenever needed; their results are as fresh as the last +completed scan. ## Rationale Duplicate finders that hash entire files do not scale to the target environment: ~10 million files and ~150 TB on possibly slow or busy disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes -a full-filesystem sweep tractable, and the resulting scan stream is -self-contained, so the expensive filesystem pass runs exactly once and -all analysis happens offline. The end goal is not individual files but -whole duplicated trees — duplicate extractions, duplicate downloads, -copied project trees — which an operator can consider removing as a -unit. +a full-filesystem sweep tractable, and the signatures are kept in a +persistent database, so the expensive filesystem pass is incremental: a +rescan re-hashes only files whose recorded mtime or size changed, and +all analysis happens offline from the database alone. The end goal is +not individual files but whole duplicated trees — duplicate +extractions, duplicate downloads, copied project trees — which an +operator can consider removing as a unit. ## Design @@ -59,10 +74,12 @@ Goals, in order: filesystem, possibly slow or busy disks (ZFS pool under resilver). Holding the full file list in memory is acceptable; reading file contents beyond 2 KiB per file is not. -3. **Scan once, analyze offline.** The expensive filesystem scan - produces a self-contained stream; all analysis (`report`, `trees`) - works from that stream alone and must never touch the scanned - filesystem again. +3. **Scan incrementally, analyze offline.** The expensive filesystem + scan maintains a persistent database; an unchanged file is never + read again on a rescan. All analysis (`report`, `trees`) works from + the database alone and must never touch the scanned filesystem + again. `scan` is designed to be cronned; the reports run at any + time against the last completed scan. 4. **Clean stream separation.** Everything on stdout is machine-readable data. All progress, warnings, and summaries go to stderr. Never mix them. @@ -71,54 +88,116 @@ Goals, in order: - Language: Go (module `sneak.berlin/go/sfdupes`). Binary name: `sfdupes`. -- Dependencies: standard library, `github.com/spf13/cobra` for the CLI, - and **one progress-bar library** - (`github.com/schollz/progressbar/v3`). `github.com/spf13/viper` is - permitted if configuration-file support is ever needed, but is not - currently used. No other third-party deps. +- Dependencies: standard library, `github.com/spf13/cobra` for the + CLI, **one progress-bar library** + (`github.com/schollz/progressbar/v3`), and **one SQLite driver** + (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled). + `github.com/spf13/viper` is permitted if configuration-file support + is ever needed, but is not currently used. No other third-party + deps. - Cross-compilation is not a concern. Builds run with cgo disabled (the `Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go. - Analysis modes (`report`, `trees`) must be deterministic: identical - input stream, identical output, regardless of record order. + database contents, identical output, regardless of the order in + which records were inserted. ### Subcommands Three subcommands, all implemented: -1. `scan` — walk the filesystem and emit one signature record per - regular file. -2. `report` — file-level duplicate report from the scan stream. +1. `scan` — walk the filesystem and synchronize the database: one + signature record per regular file. +2. `report` — file-level duplicate report from the database. 3. `trees` — tree-level duplicate report: reconstruct the directory - hierarchy from the scan stream, compute a Merkle-style digest per - directory, and report maximal groups of identical trees. + hierarchy from the database records, compute a Merkle-style digest + per directory, and report maximal groups of identical trees. ``` -sfdupes scan [--workers N] [-x] PATH... > files.dat -sfdupes report [files.dat|-] > dupes.tsv -sfdupes trees [files.dat|-] > dupetrees.tsv +sfdupes scan [--workers N] [-x] PATH... +sfdupes report > dupes.tsv +sfdupes trees > dupetrees.tsv ``` +### Database + +All three subcommands operate on a single SQLite database file: + +- Location: the value of the `SFDUPES_DATABASE` environment variable + when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`. + There is no command-line flag. +- `scan` creates the database (and its parent directory) on first + use. `report` and `trees` require an existing database; a missing + database file is a fatal error (exit 1) telling the user to run + `scan` first. +- The database uses WAL journal mode and a busy timeout, so running a + report while a cron `scan` is in progress is safe; the reports see + the last committed state. Each scan commits its changes in a single + transaction, so a report never observes a half-finished scan and a + scan that dies partway leaves the previous state intact. +- Schema (`PRAGMA user_version` is the schema version, currently 1; a + database with any other version is a fatal error): + + ```sql + CREATE TABLE files ( + path BLOB PRIMARY KEY, -- absolute path, raw bytes + size INTEGER NOT NULL, -- bytes, from lstat + mtime INTEGER NOT NULL, -- Unix seconds, from lstat + head TEXT NOT NULL, -- lowercase-hex SHA-256, first 1 KiB + tail TEXT NOT NULL -- lowercase-hex SHA-256, last 1 KiB + ) WITHOUT ROWID; + ``` + + Paths are stored as BLOBs because Unix paths are raw bytes, not + guaranteed UTF-8. `mtime` is used only for change detection; it is + not part of the duplicate key. + ### `scan` mode `scan` requires one or more `PATH` operands naming the trees to scan. There is no default path; invoking `scan` with no operand is a usage error (usage message on stderr, exit 2). An operand may be a directory or a regular file; an operand that does not exist is a fatal error -(exit 1). Operands are walked in the order given; overlapping operands -(one containing another) emit their common files once per operand, so -callers should pass disjoint paths. +(exit 1). Because database records persist between runs and are keyed +by absolute path, each operand is resolved to an absolute, lexically +cleaned path (symlinks are not resolved) before walking, so results do +not depend on the working directory. Operands are walked in the order +given; overlapping operands (one containing another) are harmless — a +file reached via multiple operands produces one database record. -`scan` runs **three sequential passes**, in this order, so that every +`scan` synchronizes the database with the filesystem state under the +scanned operands: + +- A file not yet in the database is hashed and inserted. +- A file already in the database is **skipped without reading its + contents** when its lstat size equals the recorded size and its + lstat mtime is not newer than the recorded mtime. This is what + makes a daily rescan cheap. +- A file whose mtime is newer than recorded, or whose size differs, + is re-hashed and its record updated. +- A database record whose path lies under one of the scanned operands + but was not successfully processed this run is deleted. This + removes records for deleted files. It also removes records for + paths that failed to stat or hash this run: the database only ever + contains signatures verified by the most recent scan that covered + them (a subsequent successful scan re-adds such files). +- Database records outside the scanned operands are untouched, so + disjoint trees can be scanned on different schedules into the same + database. + +`scan` runs **four sequential passes**, in this order, so that every expensive pass has an exact total for meaningful progress and ETA: 1. **walk** — recursively enumerate the tree under each `PATH` in turn, collecting the list of regular-file paths. Total unknown while running: show a live count, not a percentage. 2. **stat** — `lstat` every collected path, recording size and mtime. -3. **hash** — for each file, read the first `min(1024, size)` bytes and - the last `min(1024, size)` bytes (the two reads overlap when - `size < 2048`; for `size == 0` hash the empty input) and compute the - SHA-256 of each. Emit the output record. +3. **hash** — for each new or changed file (per the rules above), read + the first `min(1024, size)` bytes and the last `min(1024, size)` + bytes (the two reads overlap when `size < 2048`; for `size == 0` + hash the empty input) and compute the SHA-256 of each. Unchanged + files are not read and do not appear in this pass's total. +4. **update** — apply all insertions, updates, and deletions to the + database in a single transaction. Rules for the walk: @@ -134,52 +213,46 @@ Rules for the walk: - On any per-path error (permission denied, file vanished between passes, unreadable): print a one-line warning to stderr, skip the path, and continue. Per-file errors never abort the run; the final - summary reports how many were skipped. + summary reports how many were skipped. As specified above, a + skipped path that has a database record from an earlier scan loses + that record; an unreadable directory subtree likewise loses its + records (accepted: the database mirrors what the latest scan could + actually verify). Concurrency: the stat and hash passes use a worker pool (`--workers`, -default `runtime.NumCPU()`). The main goroutine owns stdout writing and -progress rendering; progress display must never block the workers. +default `runtime.NumCPU()`). The main goroutine owns database writes +and progress rendering; progress display must never block the workers. -#### Output record format - -One record per file on stdout, NUL-terminated (`\x00`), with -tab-separated fields, **path last** so tabs or newlines embedded in -paths cannot corrupt the record structure: +`scan` writes nothing to stdout. The summary line on stderr reports the +files seen this run broken down by disposition, plus skips: ``` -\t\t\t\t\x00 +scan: 123456 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped ``` -- `size`: decimal bytes, from the stat pass. -- `mtime_unix`: decimal Unix seconds. Informational only; not part of - the duplicate key. -- Hashes: lowercase hex, 64 chars each. -- Record order is unspecified (workers complete out of order); the - analysis modes must not depend on ordering. +(`removed` counts deleted database records, which are not part of the +files-seen total.) ### `report` mode -`report` reads the scan stream from the file named in its first -positional argument, or from stdin if the argument is absent or `-`. +`report` reads every record from the database and takes no positional +arguments. **`report` must never touch the filesystem being analyzed.** It does not stat, open, or otherwise access any path that appears in the records; its -only I/O is reading the scan file/stdin and writing stdout/stderr. It must +only I/O is reading the database and writing stdout/stderr. It must produce identical output whether or not the scanned filesystem is still mounted. Processing: -- Parse records; a record that does not have exactly 5 fields or whose - size is non-numeric is counted as malformed and skipped (warn once - with the total malformed count in the summary, not per record). - Group records by the key `(size, head_hash, tail_hash)`. - Every group with two or more paths is a duplicate group. - Within each group, sort paths lexicographically (byte order). The first path is the group's `first`; every other path is a `dupe`. - Order groups by size descending (biggest reclaimable space first), tie-broken by `first` path ascending. Output must be fully - deterministic for a given input. + deterministic for a given database state. #### Report output format @@ -192,16 +265,16 @@ first dupe size /srv/a/big.iso /srv/c/big-copy2.iso 4294967296 ``` -Summary to stderr: records read, malformed count (if any), number of -duplicate groups, number of dupe files, and total reclaimable bytes -(sum of `size` over all dupe rows) in human units. +Summary to stderr: records read, number of duplicate groups, number of +dupe files, and total reclaimable bytes (sum of `size` over all dupe +rows) in human units. ### `trees` mode -`trees` reads the same scan stream as `report` (same argument handling, -same parsing and malformed-record rules) and reports **entire duplicate -directory trees**: directories under which the exact same set of relative -paths exists with the exact same file signatures. +`trees` reads the same database as `report` (no positional arguments) +and reports **entire duplicate directory trees**: directories under +which the exact same set of relative paths exists with the exact same +file signatures. **`trees` must never touch the filesystem being analyzed** — the same rule as `report`. The directory hierarchy is reconstructed purely from @@ -222,10 +295,10 @@ Definitions: equal. Equal digests imply equal recursive file count and equal total byte size. -Known limitation (accepted): only regular files that appear in the scan -stream define a tree. Empty directories are invisible, and a file skipped -during the scan (e.g. permission error) in one copy but not the other -will make otherwise-identical trees compare as different. +Known limitation (accepted): only regular files that appear in the +database define a tree. Empty directories are invisible, and a file +skipped during the scan (e.g. permission error) in one copy but not the +other will make otherwise-identical trees compare as different. Processing: @@ -254,22 +327,22 @@ first dupe files size /srv/a/project /srv/backup/project 3417 104857600 ``` -Summary to stderr: records read, malformed count (if any), number of -duplicate-tree groups, number of dupe trees, and total reclaimable bytes -(sum of `size` over all dupe rows) in human units. +Summary to stderr: records read, number of duplicate-tree groups, +number of dupe trees, and total reclaimable bytes (sum of `size` over +all dupe rows) in human units. ### Progress Use the progress-bar library for all scan-pass progress; rendering in the style of `pv` is the model. All progress goes to stderr. -Each scan pass gets its own bar. Required elements for the stat and hash -passes (known totals): +Each scan pass gets its own bar. Required elements for the stat, hash, +and update passes (known totals): - elapsed time - estimated time remaining -- a `[m/n] x%` display (files processed / total files, percent) -- current rate (files/s) +- a `[m/n] x%` display (items processed / total items, percent) +- current rate (items/s) Example shape (exact layout is flexible, content is not): @@ -292,9 +365,11 @@ Additional requirements: ### Error handling and exit codes - `0`: success, even if individual files were skipped with warnings. -- `1`: fatal error (e.g., a `PATH` operand does not exist, cannot - read the scan input, stdout write failure). -- `2`: usage error (including `scan` with no `PATH` operand). +- `1`: fatal error (e.g., a `PATH` operand does not exist, the + database cannot be created/opened/read/written, a missing database + for `report`/`trees`, stdout write failure). +- `2`: usage error (including `scan` with no `PATH` operand and + `report`/`trees` with any positional argument). ## Build @@ -310,7 +385,7 @@ The `Makefile` is the single source of truth for all operations: - `make docker` — build the Docker image, which runs `make check` as a build stage. - `make hooks` — install the pre-commit hook. -- `make clean` — remove the binary and any local `files.dat`. +- `make clean` — remove the binary and any legacy local `files.dat`. ### Definition of done @@ -323,6 +398,7 @@ All of the following, run in this directory, must pass: ```sh d=$(mktemp -d) + export SFDUPES_DATABASE="$d/db.sqlite" mkdir -p "$d/a" "$d/b" head -c 2000 /dev/urandom > "$d/a/one.bin" cp "$d/a/one.bin" "$d/b/copy.bin" @@ -339,28 +415,41 @@ All of the following, run in this directory, must pass: cp "$d/t1/sub/f2" "$d/t2/sub/f2" cp "$d/t1/f1" "$d/t3/f1" cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed" - ./sfdupes scan "$d" > files.dat - ./sfdupes report files.dat - ./sfdupes trees files.dat + ./sfdupes scan "$d" + ./sfdupes report + ./sfdupes trees + # incremental behavior (scan a subtree; records elsewhere persist): + ./sfdupes scan "$d/a" # everything unchanged, nothing hashed + printf 'z' >> "$d/a/one.bin" # modify: next scan re-hashes it + rm "$d/a/unique.bin" # delete: next scan removes its record + ./sfdupes scan "$d/a" # 1 updated, 1 removed + ./sfdupes report ``` - Expected from `report`: `one.bin`/`copy.bin`/`copy2.bin` form one - group (two dupe rows, `first` is the lexicographically smallest - path); `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/ - `t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair; - `empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere; - groups ordered by size descending; piping scan directly into report - (`./sfdupes scan "$d" | ./sfdupes report`) gives the same - rows. + (The scan database lives inside `$d` here purely for test hygiene; + scanning `$d` therefore also records the SQLite file itself, which + is harmless.) + + Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin` + form one group (two dupe rows, `first` is the lexicographically + smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group; + `t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group; + `tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and + `tiny3` appear nowhere; groups ordered by size descending. Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe` `$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3` appears nowhere (its file set differs by name). + Expected from the second `report` (after the modify/delete rescan): + `one.bin` has left its group (its content changed), so + `copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is + gone from the database. + The test suite automates this scenario (see `scan_test.go`), plus a - negative check: `report` and `trees` operate on the captured stream - alone and never touch the scanned filesystem. + negative check: `report` and `trees` operate on the database alone + and never touch the scanned filesystem. ## TODO @@ -371,7 +460,9 @@ Tracked in [TODO.md](TODO.md). - No full-content verification, no byte-for-byte compare, no deletion or linking of duplicates. The reports are advisory; acting on them is the user's job. -- No persistence formats beyond the scan stream described above. +- No persistence beyond the SQLite database described above; no + export/import formats. +- No daemon or filesystem watcher; scheduling rescans is cron's job. ## License diff --git a/TODO.md b/TODO.md index f9b1099..407e861 100644 --- a/TODO.md +++ b/TODO.md @@ -14,8 +14,41 @@ # Next Step -- convert Makefile targets to scripts-to-rule-them-all `script/` - entrypoints like the other managed repos +- persistent scan database (branch `persistent-database`): `scan` + maintains a SQLite database that survives between runs so it can be + cronned daily; `report` and `trees` read the database instead of a + scan stream. Database path defaults to `/var/lib/sfdupes/db.sqlite`, + overridable via `SFDUPES_DATABASE`. Plan: + - [x] update `README.md` (the authoritative spec) for the database: + new Database section, revised `scan`/`report`/`trees` modes, + dependency list (`modernc.org/sqlite`, pure Go, cgo stays + disabled), exit codes, smoke test with incremental steps + - [ ] add `modernc.org/sqlite` (hash-pinned via `go.sum`) + - [ ] `db.go`: path resolution (`SFDUPES_DATABASE` env, default + `/var/lib/sfdupes/db.sqlite`), open/create with WAL + + busy-timeout pragmas and `PRAGMA user_version` schema check, + `files` table (path BLOB primary key, size, mtime, head, + tail), load-all-rows, and single-transaction apply of + upserts/deletes + - [ ] `scan.go`: resolve operands to absolute paths; diff stat + results against loaded rows (reuse hashes when size matches + and mtime is not newer); hash only new/changed files; delete + rows under the scanned operands not successfully processed; + leave rows outside the operands untouched; new summary line + (added/updated/removed/unchanged/skipped); "update" progress + pass for the database write + - [ ] `report.go`/`trees.go`: read records from the database (no + positional args, no stream parsing); drop the NUL-stream + parser and malformed-record handling + - [ ] `main.go`: updated usage strings; `report`/`trees` take no + args (usage error otherwise) + - [ ] tests: db open/env-override/schema tests; incremental scan + tests (add, mtime update, delete, unchanged-hash-reuse, + out-of-scope rows untouched, error paths); rework pipeline + test to go through the database; keep walk/stat/hash/grouping + unit tests + - [ ] `.gitignore`: local `*.sqlite` artifacts + - [ ] `make check`, `make docker`, and the README smoke test pass # Completed Steps @@ -33,6 +66,8 @@ # Future Steps +- convert Makefile targets to scripts-to-rule-them-all `script/` + entrypoints like the other managed repos - possible later features (explicitly out of scope per README): full-content verification of candidates, removal-script helpers