Implement persistent SQLite scan database
All checks were successful
check / check (push) Successful in 3s
All checks were successful
check / check (push) Successful in 3s
scan now synchronizes a database that survives between runs (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) instead of emitting a stream: operands are resolved to absolute paths, unchanged files (same size, mtime not newer than recorded) are never re-read, new and changed files are hashed, and records under the scanned operands that were not verified this run are deleted; records outside the operands are untouched. All changes commit in a single transaction, and WAL journaling with a busy timeout keeps a report run during a cron scan safe. report and trees read the database (no positional arguments); the NUL-terminated stream format, its parser, and the malformed-record handling are gone. The driver is modernc.org/sqlite (pure Go), so builds keep cgo disabled.
This commit is contained in:
50
TODO.md
50
TODO.md
@@ -14,44 +14,22 @@
|
||||
|
||||
# Next Step
|
||||
|
||||
- persistent scan database (branch `persistent-database`): `scan`
|
||||
maintains a SQLite database that survives between runs so it can be
|
||||
cronned daily; `report` and `trees` read the database instead of a
|
||||
scan stream. Database path defaults to `/var/lib/sfdupes/db.sqlite`,
|
||||
overridable via `SFDUPES_DATABASE`. Plan:
|
||||
- [x] update `README.md` (the authoritative spec) for the database:
|
||||
new Database section, revised `scan`/`report`/`trees` modes,
|
||||
dependency list (`modernc.org/sqlite`, pure Go, cgo stays
|
||||
disabled), exit codes, smoke test with incremental steps
|
||||
- [ ] add `modernc.org/sqlite` (hash-pinned via `go.sum`)
|
||||
- [ ] `db.go`: path resolution (`SFDUPES_DATABASE` env, default
|
||||
`/var/lib/sfdupes/db.sqlite`), open/create with WAL +
|
||||
busy-timeout pragmas and `PRAGMA user_version` schema check,
|
||||
`files` table (path BLOB primary key, size, mtime, head,
|
||||
tail), load-all-rows, and single-transaction apply of
|
||||
upserts/deletes
|
||||
- [ ] `scan.go`: resolve operands to absolute paths; diff stat
|
||||
results against loaded rows (reuse hashes when size matches
|
||||
and mtime is not newer); hash only new/changed files; delete
|
||||
rows under the scanned operands not successfully processed;
|
||||
leave rows outside the operands untouched; new summary line
|
||||
(added/updated/removed/unchanged/skipped); "update" progress
|
||||
pass for the database write
|
||||
- [ ] `report.go`/`trees.go`: read records from the database (no
|
||||
positional args, no stream parsing); drop the NUL-stream
|
||||
parser and malformed-record handling
|
||||
- [ ] `main.go`: updated usage strings; `report`/`trees` take no
|
||||
args (usage error otherwise)
|
||||
- [ ] tests: db open/env-override/schema tests; incremental scan
|
||||
tests (add, mtime update, delete, unchanged-hash-reuse,
|
||||
out-of-scope rows untouched, error paths); rework pipeline
|
||||
test to go through the database; keep walk/stat/hash/grouping
|
||||
unit tests
|
||||
- [ ] `.gitignore`: local `*.sqlite` artifacts
|
||||
- [ ] `make check`, `make docker`, and the README smoke test pass
|
||||
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
||||
entrypoints like the other managed repos
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- persistent scan database (2026-07-24, branch `persistent-database`):
|
||||
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
|
||||
Go, cgo stays disabled) keyed by absolute path that survives between
|
||||
runs — a rescan hashes only new or changed files (by mtime/size),
|
||||
deletes records for files vanished from under the scanned operands,
|
||||
and leaves records outside them untouched, so `scan` can be cronned
|
||||
daily; `report` and `trees` read the database (no positional
|
||||
arguments) instead of a scan stream. Database at
|
||||
`/var/lib/sfdupes/db.sqlite`, overridable via `SFDUPES_DATABASE`;
|
||||
WAL journaling plus a single-transaction update keep a report run
|
||||
during a scan safe
|
||||
- add the `origin` remote (`git@git.eeqj.de:sneak/sfdupes.git`), tag
|
||||
`v0.0.1`, and push `main` plus tags (2026-07-23)
|
||||
- `scan` CLI rework (2026-07-23, branch `scan-required-paths`): required
|
||||
@@ -66,8 +44,6 @@
|
||||
|
||||
# Future Steps
|
||||
|
||||
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
||||
entrypoints like the other managed repos
|
||||
- possible later features (explicitly out of scope per README):
|
||||
full-content verification of candidates, removal-script helpers
|
||||
|
||||
|
||||
Reference in New Issue
Block a user