Specify persistent SQLite scan database in README; plan in TODO.md

scan will maintain a persistent database of file signatures
(default /var/lib/sfdupes/db.sqlite, overridable via
SFDUPES_DATABASE) that survives between runs; rescans hash only new
or changed files (mtime/size) and remove records for vanished files,
so scan can be cronned daily. report and trees will read the
database instead of a scan stream.

README.md

@@ -12,7 +12,12 @@ SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
 content (the middle of the file is never read); the intended use is
 finding duplicate downloads and duplicated directory trees on
 multi-terabyte ZFS servers where reading every byte is prohibitively
-expensive.
+expensive. `scan` maintains a persistent SQLite database of file
+signatures that survives between runs, so it can be run from cron and
+the reports can be generated at any time from the most recent scan.
+
+This tool was created by [@sneak](https://sneak.berlin) to scratch an itch,
+using Claude Code/Fable.
 
 This README is the complete and authoritative specification.
 
@@ -20,29 +25,39 @@ This README is the complete and authoritative specification.
 
 ```sh
 make build
-./sfdupes scan /srv > files.dat
+export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
-./sfdupes report files.dat > dupes.tsv
+./sfdupes scan /srv
-./sfdupes trees files.dat > dupetrees.tsv
+./sfdupes report > dupes.tsv
+./sfdupes trees > dupetrees.tsv
 ```
 
-`scan` walks one or more filesystem trees and emits one record per
+`scan` walks one or more filesystem trees and maintains one database
-regular file (path, size, mtime, head hash, tail hash). `report`
+record per regular file (path, size, mtime, head hash, tail hash). The
-ingests that stream and prints the file-level duplicates report.
+database persists between runs; a rescan only hashes files that are new
-`trees` ingests the same stream and prints the duplicate-tree report. A
+or changed, and removes records for files that no longer exist.
-missing/invalid subcommand — or a `scan` invocation with no `PATH`
+`report` reads the database and prints the file-level duplicates
-operand — prints a usage message and exits 2.
+report. `trees` reads the same database and prints the duplicate-tree
+report. A missing/invalid subcommand — or a `scan` invocation with no
+`PATH` operand — prints a usage message and exits 2.
+
+The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed
+anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a
+daily `sfdupes scan` cron job, with the reporting commands run
+interactively whenever needed; their results are as fresh as the last
+completed scan.
 
 ## Rationale
 
 Duplicate finders that hash entire files do not scale to the target
 environment: ~10 million files and ~150 TB on possibly slow or busy
 disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes
-a full-filesystem sweep tractable, and the resulting scan stream is
+a full-filesystem sweep tractable, and the signatures are kept in a
-self-contained, so the expensive filesystem pass runs exactly once and
+persistent database, so the expensive filesystem pass is incremental: a
-all analysis happens offline. The end goal is not individual files but
+rescan re-hashes only files whose recorded mtime or size changed, and
-whole duplicated trees — duplicate extractions, duplicate downloads,
+all analysis happens offline from the database alone. The end goal is
-copied project trees — which an operator can consider removing as a
+not individual files but whole duplicated trees — duplicate
-unit.
+extractions, duplicate downloads, copied project trees — which an
+operator can consider removing as a unit.
 
 ## Design
 
@@ -59,10 +74,12 @@ Goals, in order:
     filesystem, possibly slow or busy disks (ZFS pool under resilver).
     Holding the full file list in memory is acceptable; reading file
     contents beyond 2 KiB per file is not.
-3.  **Scan once, analyze offline.** The expensive filesystem scan
+3.  **Scan incrementally, analyze offline.** The expensive filesystem
-    produces a self-contained stream; all analysis (`report`, `trees`)
+    scan maintains a persistent database; an unchanged file is never
-    works from that stream alone and must never touch the scanned
+    read again on a rescan. All analysis (`report`, `trees`) works from
-    filesystem again.
+    the database alone and must never touch the scanned filesystem
+    again. `scan` is designed to be cronned; the reports run at any
+    time against the last completed scan.
 4.  **Clean stream separation.** Everything on stdout is machine-readable
     data. All progress, warnings, and summaries go to stderr. Never mix
     them.
@@ -71,54 +88,116 @@ Goals, in order:
 
 -   Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
     `sfdupes`.
--   Dependencies: standard library, `github.com/spf13/cobra` for the CLI,
+-   Dependencies: standard library, `github.com/spf13/cobra` for the
-    and **one progress-bar library**
+    CLI, **one progress-bar library**
-    (`github.com/schollz/progressbar/v3`). `github.com/spf13/viper` is
+    (`github.com/schollz/progressbar/v3`), and **one SQLite driver**
-    permitted if configuration-file support is ever needed, but is not
+    (`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
-    currently used. No other third-party deps.
+    `github.com/spf13/viper` is permitted if configuration-file support
+    is ever needed, but is not currently used. No other third-party
+    deps.
 -   Cross-compilation is not a concern. Builds run with cgo disabled (the
     `Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
 -   Analysis modes (`report`, `trees`) must be deterministic: identical
-    input stream, identical output, regardless of record order.
+    database contents, identical output, regardless of the order in
+    which records were inserted.
 
 ### Subcommands
 
 Three subcommands, all implemented:
 
-1.  `scan` — walk the filesystem and emit one signature record per
+1.  `scan` — walk the filesystem and synchronize the database: one
-    regular file.
+    signature record per regular file.
-2.  `report` — file-level duplicate report from the scan stream.
+2.  `report` — file-level duplicate report from the database.
 3.  `trees` — tree-level duplicate report: reconstruct the directory
-    hierarchy from the scan stream, compute a Merkle-style digest per
+    hierarchy from the database records, compute a Merkle-style digest
-    directory, and report maximal groups of identical trees.
+    per directory, and report maximal groups of identical trees.
 
 ```
-sfdupes scan [--workers N] [-x] PATH... > files.dat
+sfdupes scan [--workers N] [-x] PATH...
-sfdupes report [files.dat|-] > dupes.tsv
+sfdupes report > dupes.tsv
-sfdupes trees [files.dat|-] > dupetrees.tsv
+sfdupes trees > dupetrees.tsv
 ```
 
+### Database
+
+All three subcommands operate on a single SQLite database file:
+
+-   Location: the value of the `SFDUPES_DATABASE` environment variable
+    when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`.
+    There is no command-line flag.
+-   `scan` creates the database (and its parent directory) on first
+    use. `report` and `trees` require an existing database; a missing
+    database file is a fatal error (exit 1) telling the user to run
+    `scan` first.
+-   The database uses WAL journal mode and a busy timeout, so running a
+    report while a cron `scan` is in progress is safe; the reports see
+    the last committed state. Each scan commits its changes in a single
+    transaction, so a report never observes a half-finished scan and a
+    scan that dies partway leaves the previous state intact.
+-   Schema (`PRAGMA user_version` is the schema version, currently 1; a
+    database with any other version is a fatal error):
+
+    ```sql
+    CREATE TABLE files (
+        path  BLOB PRIMARY KEY,  -- absolute path, raw bytes
+        size  INTEGER NOT NULL,  -- bytes, from lstat
+        mtime INTEGER NOT NULL,  -- Unix seconds, from lstat
+        head  TEXT NOT NULL,     -- lowercase-hex SHA-256, first 1 KiB
+        tail  TEXT NOT NULL      -- lowercase-hex SHA-256, last 1 KiB
+    ) WITHOUT ROWID;
+    ```
+
+    Paths are stored as BLOBs because Unix paths are raw bytes, not
+    guaranteed UTF-8. `mtime` is used only for change detection; it is
+    not part of the duplicate key.
+
 ### `scan` mode
 
 `scan` requires one or more `PATH` operands naming the trees to scan.
 There is no default path; invoking `scan` with no operand is a usage
 error (usage message on stderr, exit 2). An operand may be a directory
 or a regular file; an operand that does not exist is a fatal error
-(exit 1). Operands are walked in the order given; overlapping operands
+(exit 1). Because database records persist between runs and are keyed
-(one containing another) emit their common files once per operand, so
+by absolute path, each operand is resolved to an absolute, lexically
-callers should pass disjoint paths.
+cleaned path (symlinks are not resolved) before walking, so results do
+not depend on the working directory. Operands are walked in the order
+given; overlapping operands (one containing another) are harmless — a
+file reached via multiple operands produces one database record.
 
-`scan` runs **three sequential passes**, in this order, so that every
+`scan` synchronizes the database with the filesystem state under the
+scanned operands:
+
+-   A file not yet in the database is hashed and inserted.
+-   A file already in the database is **skipped without reading its
+    contents** when its lstat size equals the recorded size and its
+    lstat mtime is not newer than the recorded mtime. This is what
+    makes a daily rescan cheap.
+-   A file whose mtime is newer than recorded, or whose size differs,
+    is re-hashed and its record updated.
+-   A database record whose path lies under one of the scanned operands
+    but was not successfully processed this run is deleted. This
+    removes records for deleted files. It also removes records for
+    paths that failed to stat or hash this run: the database only ever
+    contains signatures verified by the most recent scan that covered
+    them (a subsequent successful scan re-adds such files).
+-   Database records outside the scanned operands are untouched, so
+    disjoint trees can be scanned on different schedules into the same
+    database.
+
+`scan` runs **four sequential passes**, in this order, so that every
 expensive pass has an exact total for meaningful progress and ETA:
 
 1.  **walk** — recursively enumerate the tree under each `PATH` in
     turn, collecting the list of regular-file paths. Total unknown
     while running: show a live count, not a percentage.
 2.  **stat** — `lstat` every collected path, recording size and mtime.
-3.  **hash** — for each file, read the first `min(1024, size)` bytes and
+3.  **hash** — for each new or changed file (per the rules above), read
-    the last `min(1024, size)` bytes (the two reads overlap when
+    the first `min(1024, size)` bytes and the last `min(1024, size)`
-    `size < 2048`; for `size == 0` hash the empty input) and compute the
+    bytes (the two reads overlap when `size < 2048`; for `size == 0`
-    SHA-256 of each. Emit the output record.
+    hash the empty input) and compute the SHA-256 of each. Unchanged
+    files are not read and do not appear in this pass's total.
+4.  **update** — apply all insertions, updates, and deletions to the
+    database in a single transaction.
 
 Rules for the walk:
 
@@ -134,52 +213,46 @@ Rules for the walk:
 -   On any per-path error (permission denied, file vanished between
     passes, unreadable): print a one-line warning to stderr, skip the
     path, and continue. Per-file errors never abort the run; the final
-    summary reports how many were skipped.
+    summary reports how many were skipped. As specified above, a
+    skipped path that has a database record from an earlier scan loses
+    that record; an unreadable directory subtree likewise loses its
+    records (accepted: the database mirrors what the latest scan could
+    actually verify).
 
 Concurrency: the stat and hash passes use a worker pool (`--workers`,
-default `runtime.NumCPU()`). The main goroutine owns stdout writing and
+default `runtime.NumCPU()`). The main goroutine owns database writes
-progress rendering; progress display must never block the workers.
+and progress rendering; progress display must never block the workers.
 
-#### Output record format
+`scan` writes nothing to stdout. The summary line on stderr reports the
-
+files seen this run broken down by disposition, plus skips:
-One record per file on stdout, NUL-terminated (`\x00`), with
-tab-separated fields, **path last** so tabs or newlines embedded in
-paths cannot corrupt the record structure:
 
 ```
-<size>\t<mtime_unix>\t<sha256_first1k_hex>\t<sha256_last1k_hex>\t<path>\x00
+scan: 123456 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
 ```
 
--   `size`: decimal bytes, from the stat pass.
+(`removed` counts deleted database records, which are not part of the
--   `mtime_unix`: decimal Unix seconds. Informational only; not part of
+files-seen total.)
-    the duplicate key.
--   Hashes: lowercase hex, 64 chars each.
--   Record order is unspecified (workers complete out of order); the
-    analysis modes must not depend on ordering.
 
 ### `report` mode
 
-`report` reads the scan stream from the file named in its first
+`report` reads every record from the database and takes no positional
-positional argument, or from stdin if the argument is absent or `-`.
+arguments.
 
 **`report` must never touch the filesystem being analyzed.** It does not
 stat, open, or otherwise access any path that appears in the records; its
-only I/O is reading the scan file/stdin and writing stdout/stderr. It must
+only I/O is reading the database and writing stdout/stderr. It must
 produce identical output whether or not the scanned filesystem is still
 mounted.
 
 Processing:
 
--   Parse records; a record that does not have exactly 5 fields or whose
-    size is non-numeric is counted as malformed and skipped (warn once
-    with the total malformed count in the summary, not per record).
 -   Group records by the key `(size, head_hash, tail_hash)`.
 -   Every group with two or more paths is a duplicate group.
 -   Within each group, sort paths lexicographically (byte order). The
     first path is the group's `first`; every other path is a `dupe`.
 -   Order groups by size descending (biggest reclaimable space first),
     tie-broken by `first` path ascending. Output must be fully
-    deterministic for a given input.
+    deterministic for a given database state.
 
 #### Report output format
 
@@ -192,16 +265,16 @@ first	dupe	size
 /srv/a/big.iso	/srv/c/big-copy2.iso	4294967296
 ```
 
-Summary to stderr: records read, malformed count (if any), number of
+Summary to stderr: records read, number of duplicate groups, number of
-duplicate groups, number of dupe files, and total reclaimable bytes
+dupe files, and total reclaimable bytes (sum of `size` over all dupe
-(sum of `size` over all dupe rows) in human units.
+rows) in human units.
 
 ### `trees` mode
 
-`trees` reads the same scan stream as `report` (same argument handling,
+`trees` reads the same database as `report` (no positional arguments)
-same parsing and malformed-record rules) and reports **entire duplicate
+and reports **entire duplicate directory trees**: directories under
-directory trees**: directories under which the exact same set of relative
+which the exact same set of relative paths exists with the exact same
-paths exists with the exact same file signatures.
+file signatures.
 
 **`trees` must never touch the filesystem being analyzed** — the same
 rule as `report`. The directory hierarchy is reconstructed purely from
@@ -222,10 +295,10 @@ Definitions:
     equal. Equal digests imply equal recursive file count and equal
     total byte size.
 
-Known limitation (accepted): only regular files that appear in the scan
+Known limitation (accepted): only regular files that appear in the
-stream define a tree. Empty directories are invisible, and a file skipped
+database define a tree. Empty directories are invisible, and a file
-during the scan (e.g. permission error) in one copy but not the other
+skipped during the scan (e.g. permission error) in one copy but not the
-will make otherwise-identical trees compare as different.
+other will make otherwise-identical trees compare as different.
 
 Processing:
 
@@ -254,22 +327,22 @@ first	dupe	files	size
 /srv/a/project	/srv/backup/project	3417	104857600
 ```
 
-Summary to stderr: records read, malformed count (if any), number of
+Summary to stderr: records read, number of duplicate-tree groups,
-duplicate-tree groups, number of dupe trees, and total reclaimable bytes
+number of dupe trees, and total reclaimable bytes (sum of `size` over
-(sum of `size` over all dupe rows) in human units.
+all dupe rows) in human units.
 
 ### Progress
 
 Use the progress-bar library for all scan-pass progress; rendering in the
 style of `pv` is the model. All progress goes to stderr.
 
-Each scan pass gets its own bar. Required elements for the stat and hash
+Each scan pass gets its own bar. Required elements for the stat, hash,
-passes (known totals):
+and update passes (known totals):
 
 -   elapsed time
 -   estimated time remaining
--   a `[m/n] x%` display (files processed / total files, percent)
+-   a `[m/n] x%` display (items processed / total items, percent)
--   current rate (files/s)
+-   current rate (items/s)
 
 Example shape (exact layout is flexible, content is not):
 
@@ -292,9 +365,11 @@ Additional requirements:
 ### Error handling and exit codes
 
 -   `0`: success, even if individual files were skipped with warnings.
--   `1`: fatal error (e.g., a `PATH` operand does not exist, cannot
+-   `1`: fatal error (e.g., a `PATH` operand does not exist, the
-    read the scan input, stdout write failure).
+    database cannot be created/opened/read/written, a missing database
--   `2`: usage error (including `scan` with no `PATH` operand).
+    for `report`/`trees`, stdout write failure).
+-   `2`: usage error (including `scan` with no `PATH` operand and
+    `report`/`trees` with any positional argument).
 
 ## Build
 
@@ -310,7 +385,7 @@ The `Makefile` is the single source of truth for all operations:
 -   `make docker` — build the Docker image, which runs `make check` as
     a build stage.
 -   `make hooks` — install the pre-commit hook.
--   `make clean` — remove the binary and any local `files.dat`.
+-   `make clean` — remove the binary and any legacy local `files.dat`.
 
 ### Definition of done
 
@@ -323,6 +398,7 @@ All of the following, run in this directory, must pass:
 
     ```sh
     d=$(mktemp -d)
+    export SFDUPES_DATABASE="$d/db.sqlite"
     mkdir -p "$d/a" "$d/b"
     head -c 2000 /dev/urandom > "$d/a/one.bin"
     cp "$d/a/one.bin" "$d/b/copy.bin"
@@ -339,28 +415,41 @@ All of the following, run in this directory, must pass:
     cp "$d/t1/sub/f2" "$d/t2/sub/f2"
     cp "$d/t1/f1" "$d/t3/f1"
     cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
-    ./sfdupes scan "$d" > files.dat
+    ./sfdupes scan "$d"
-    ./sfdupes report files.dat
+    ./sfdupes report
-    ./sfdupes trees files.dat
+    ./sfdupes trees
+    # incremental behavior (scan a subtree; records elsewhere persist):
+    ./sfdupes scan "$d/a"                  # everything unchanged, nothing hashed
+    printf 'z' >> "$d/a/one.bin"           # modify: next scan re-hashes it
+    rm "$d/a/unique.bin"                   # delete: next scan removes its record
+    ./sfdupes scan "$d/a"                  # 1 updated, 1 removed
+    ./sfdupes report
     ```
 
-    Expected from `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
+    (The scan database lives inside `$d` here purely for test hygiene;
-    group (two dupe rows, `first` is the lexicographically smallest
+    scanning `$d` therefore also records the SQLite file itself, which
-    path); `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
+    is harmless.)
-    `t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
+
-    `empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere;
+    Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin`
-    groups ordered by size descending; piping scan directly into report
+    form one group (two dupe rows, `first` is the lexicographically
-    (`./sfdupes scan "$d" | ./sfdupes report`) gives the same
+    smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group;
-    rows.
+    `t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group;
+    `tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and
+    `tiny3` appear nowhere; groups ordered by size descending.
 
     Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`
     `$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is
     suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
     appears nowhere (its file set differs by name).
 
+    Expected from the second `report` (after the modify/delete rescan):
+    `one.bin` has left its group (its content changed), so
+    `copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is
+    gone from the database.
+
     The test suite automates this scenario (see `scan_test.go`), plus a
-    negative check: `report` and `trees` operate on the captured stream
+    negative check: `report` and `trees` operate on the database alone
-    alone and never touch the scanned filesystem.
+    and never touch the scanned filesystem.
 
 ## TODO
 
@@ -371,7 +460,9 @@ Tracked in [TODO.md](TODO.md).
 -   No full-content verification, no byte-for-byte compare, no deletion
     or linking of duplicates. The reports are advisory; acting on them is
     the user's job.
--   No persistence formats beyond the scan stream described above.
+-   No persistence beyond the SQLite database described above; no
+    export/import formats.
+-   No daemon or filesystem watcher; scheduling rescans is cron's job.
 
 ## License
 

TODO.md

@@ -14,8 +14,41 @@
 
 # Next Step
 
-- convert Makefile targets to scripts-to-rule-them-all `script/`
+- persistent scan database (branch `persistent-database`): `scan`
-  entrypoints like the other managed repos
+  maintains a SQLite database that survives between runs so it can be
+  cronned daily; `report` and `trees` read the database instead of a
+  scan stream. Database path defaults to `/var/lib/sfdupes/db.sqlite`,
+  overridable via `SFDUPES_DATABASE`. Plan:
+    - [x] update `README.md` (the authoritative spec) for the database:
+          new Database section, revised `scan`/`report`/`trees` modes,
+          dependency list (`modernc.org/sqlite`, pure Go, cgo stays
+          disabled), exit codes, smoke test with incremental steps
+    - [ ] add `modernc.org/sqlite` (hash-pinned via `go.sum`)
+    - [ ] `db.go`: path resolution (`SFDUPES_DATABASE` env, default
+          `/var/lib/sfdupes/db.sqlite`), open/create with WAL +
+          busy-timeout pragmas and `PRAGMA user_version` schema check,
+          `files` table (path BLOB primary key, size, mtime, head,
+          tail), load-all-rows, and single-transaction apply of
+          upserts/deletes
+    - [ ] `scan.go`: resolve operands to absolute paths; diff stat
+          results against loaded rows (reuse hashes when size matches
+          and mtime is not newer); hash only new/changed files; delete
+          rows under the scanned operands not successfully processed;
+          leave rows outside the operands untouched; new summary line
+          (added/updated/removed/unchanged/skipped); "update" progress
+          pass for the database write
+    - [ ] `report.go`/`trees.go`: read records from the database (no
+          positional args, no stream parsing); drop the NUL-stream
+          parser and malformed-record handling
+    - [ ] `main.go`: updated usage strings; `report`/`trees` take no
+          args (usage error otherwise)
+    - [ ] tests: db open/env-override/schema tests; incremental scan
+          tests (add, mtime update, delete, unchanged-hash-reuse,
+          out-of-scope rows untouched, error paths); rework pipeline
+          test to go through the database; keep walk/stat/hash/grouping
+          unit tests
+    - [ ] `.gitignore`: local `*.sqlite` artifacts
+    - [ ] `make check`, `make docker`, and the README smoke test pass
 
 # Completed Steps
 
@@ -33,6 +66,8 @@
 
 # Future Steps
 
+- convert Makefile targets to scripts-to-rule-them-all `script/`
+  entrypoints like the other managed repos
 - possible later features (explicitly out of scope per README):
   full-content verification of candidates, removal-script helpers