Implement persistent SQLite scan database
All checks were successful
check / check (push) Successful in 3s

scan now synchronizes a database that survives between runs
(SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) instead of
emitting a stream: operands are resolved to absolute paths, unchanged
files (same size, mtime not newer than recorded) are never re-read, new
and changed files are hashed, and records under the scanned operands
that were not verified this run are deleted; records outside the
operands are untouched. All changes commit in a single transaction, and
WAL journaling with a busy timeout keeps a report run during a cron
scan safe.

report and trees read the database (no positional arguments); the
NUL-terminated stream format, its parser, and the malformed-record
handling are gone. The driver is modernc.org/sqlite (pure Go), so
builds keep cgo disabled.
This commit is contained in:
2026-07-24 03:08:49 +07:00
parent e0d578a707
commit d8fbcb32c2
12 changed files with 1229 additions and 379 deletions

50
TODO.md
View File

@@ -14,44 +14,22 @@
# Next Step
- persistent scan database (branch `persistent-database`): `scan`
maintains a SQLite database that survives between runs so it can be
cronned daily; `report` and `trees` read the database instead of a
scan stream. Database path defaults to `/var/lib/sfdupes/db.sqlite`,
overridable via `SFDUPES_DATABASE`. Plan:
- [x] update `README.md` (the authoritative spec) for the database:
new Database section, revised `scan`/`report`/`trees` modes,
dependency list (`modernc.org/sqlite`, pure Go, cgo stays
disabled), exit codes, smoke test with incremental steps
- [ ] add `modernc.org/sqlite` (hash-pinned via `go.sum`)
- [ ] `db.go`: path resolution (`SFDUPES_DATABASE` env, default
`/var/lib/sfdupes/db.sqlite`), open/create with WAL +
busy-timeout pragmas and `PRAGMA user_version` schema check,
`files` table (path BLOB primary key, size, mtime, head,
tail), load-all-rows, and single-transaction apply of
upserts/deletes
- [ ] `scan.go`: resolve operands to absolute paths; diff stat
results against loaded rows (reuse hashes when size matches
and mtime is not newer); hash only new/changed files; delete
rows under the scanned operands not successfully processed;
leave rows outside the operands untouched; new summary line
(added/updated/removed/unchanged/skipped); "update" progress
pass for the database write
- [ ] `report.go`/`trees.go`: read records from the database (no
positional args, no stream parsing); drop the NUL-stream
parser and malformed-record handling
- [ ] `main.go`: updated usage strings; `report`/`trees` take no
args (usage error otherwise)
- [ ] tests: db open/env-override/schema tests; incremental scan
tests (add, mtime update, delete, unchanged-hash-reuse,
out-of-scope rows untouched, error paths); rework pipeline
test to go through the database; keep walk/stat/hash/grouping
unit tests
- [ ] `.gitignore`: local `*.sqlite` artifacts
- [ ] `make check`, `make docker`, and the README smoke test pass
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos
# Completed Steps
- persistent scan database (2026-07-24, branch `persistent-database`):
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
Go, cgo stays disabled) keyed by absolute path that survives between
runs — a rescan hashes only new or changed files (by mtime/size),
deletes records for files vanished from under the scanned operands,
and leaves records outside them untouched, so `scan` can be cronned
daily; `report` and `trees` read the database (no positional
arguments) instead of a scan stream. Database at
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
WAL journaling plus a single-transaction update keep a report run
during a scan safe
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
`v0.0.1`, and push `main` plus tags (2026-07-23)
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
@@ -66,8 +44,6 @@
# Future Steps
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos
- possible later features (explicitly out of scope per README):
full-content verification of candidates, removal-script helpers