sneak/sfdupes
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Specify persistent SQLite scan database in README; plan in TODO.md

Browse Source 
scan will maintain a persistent database of file signatures
(default /var/lib/sfdupes/db.sqlite, overridable via
SFDUPES_DATABASE) that survives between runs; rescans hash only new
or changed files (mtime/size) and remove records for vanished files,
so scan can be cronned daily. report and trees will read the
database instead of a scan stream.
This commit is contained in:
Jeffrey Paul
2026-07-24 02:54:08 +07:00
parent 90c9ef3546
commit e0d578a707
2 changed files with 228 additions and 102 deletions
Download Patch File Download Diff File Expand all files Collapse all files

291
README.md
View File

@@ -12,7 +12,12 @@ SHA-256 of their first 1024 bytes, and identical SHA-256 of their last
content (the middle of the file is never read); the intended use is
finding duplicate downloads and duplicated directory trees on
multi-terabyte ZFS servers where reading every byte is prohibitively
expensive.
expensive. `scan` maintains a persistent SQLite database of file
signatures that survives between runs, so it can be run from cron and
the reports can be generated at any time from the most recent scan.
This tool was created by [@sneak](https://sneak.berlin) to scratch an itch,
using Claude Code/Fable.
This README is the complete and authoritative specification.
@@ -20,29 +25,39 @@ This README is the complete and authoritative specification.
```sh
make build
./sfdupes scan /srv > files.dat
./sfdupes report files.dat > dupes.tsv
./sfdupes trees files.dat > dupetrees.tsv
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes scan /srv
./sfdupes report > dupes.tsv
./sfdupes trees > dupetrees.tsv
```
`scan` walks one or more filesystem trees and emits one record per
regular file (path, size, mtime, head hash, tail hash). `report`
ingests that stream and prints the file-level duplicates report.
`trees` ingests the same stream and prints the duplicate-tree report. A
missing/invalid subcommand — or a `scan` invocation with no `PATH`
operand — prints a usage message and exits 2.
`scan` walks one or more filesystem trees and maintains one database
record per regular file (path, size, mtime, head hash, tail hash). The
database persists between runs; a rescan only hashes files that are new
or changed, and removes records for files that no longer exist.
`report` reads the database and prints the file-level duplicates
report. `trees` reads the same database and prints the duplicate-tree
report. A missing/invalid subcommand — or a `scan` invocation with no
`PATH` operand — prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed
anywhere by setting `SFDUPES_DATABASE`. The intended deployment is a
daily `sfdupes scan` cron job, with the reporting commands run
interactively whenever needed; their results are as fresh as the last
completed scan.
## Rationale
Duplicate finders that hash entire files do not scale to the target
environment: ~10 million files and ~150 TB on possibly slow or busy
disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes
a full-filesystem sweep tractable, and the resulting scan stream is
self-contained, so the expensive filesystem pass runs exactly once and
all analysis happens offline. The end goal is not individual files but
whole duplicated trees — duplicate extractions, duplicate downloads,
copied project trees — which an operator can consider removing as a
unit.
a full-filesystem sweep tractable, and the signatures are kept in a
persistent database, so the expensive filesystem pass is incremental: a
rescan re-hashes only files whose recorded mtime or size changed, and
all analysis happens offline from the database alone. The end goal is
not individual files but whole duplicated trees — duplicate
extractions, duplicate downloads, copied project trees — which an
operator can consider removing as a unit.
## Design
@@ -59,10 +74,12 @@ Goals, in order:
filesystem, possibly slow or busy disks (ZFS pool under resilver).
Holding the full file list in memory is acceptable; reading file
contents beyond 2 KiB per file is not.
3. **Scan once, analyze offline.** The expensive filesystem scan
produces a self-contained stream; all analysis (`report`, `trees`)
works from that stream alone and must never touch the scanned
filesystem again.
3. **Scan incrementally, analyze offline.** The expensive filesystem
scan maintains a persistent database; an unchanged file is never
read again on a rescan. All analysis (`report`, `trees`) works from
the database alone and must never touch the scanned filesystem
again. `scan` is designed to be cronned; the reports run at any
time against the last completed scan.
4. **Clean stream separation.** Everything on stdout is machine-readable
data. All progress, warnings, and summaries go to stderr. Never mix
them.
@@ -71,54 +88,116 @@ Goals, in order:
- Language: Go (module `sneak.berlin/go/sfdupes`). Binary name:
`sfdupes`.
- Dependencies: standard library, `github.com/spf13/cobra` for the CLI,
and **one progress-bar library**
(`github.com/schollz/progressbar/v3`). `github.com/spf13/viper` is
permitted if configuration-file support is ever needed, but is not
currently used. No other third-party deps.
- Dependencies: standard library, `github.com/spf13/cobra` for the
CLI, **one progress-bar library**
(`github.com/schollz/progressbar/v3`), and **one SQLite driver**
(`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
`github.com/spf13/viper` is permitted if configuration-file support
is ever needed, but is not currently used. No other third-party
deps.
- Cross-compilation is not a concern. Builds run with cgo disabled (the
`Makefile` exports `CGO_ENABLED=0`); the code must remain pure Go.
- Analysis modes (`report`, `trees`) must be deterministic: identical
input stream, identical output, regardless of record order.
database contents, identical output, regardless of the order in
which records were inserted.
### Subcommands
Three subcommands, all implemented:
1. `scan` — walk the filesystem and emit one signature record per
regular file.
2. `report` — file-level duplicate report from the scan stream.
1. `scan` — walk the filesystem and synchronize the database: one
signature record per regular file.
2. `report` — file-level duplicate report from the database.
3. `trees` — tree-level duplicate report: reconstruct the directory
hierarchy from the scan stream, compute a Merkle-style digest per
directory, and report maximal groups of identical trees.
hierarchy from the database records, compute a Merkle-style digest
per directory, and report maximal groups of identical trees.
```
sfdupes scan [--workers N] [-x] PATH... > files.dat
sfdupes report [files.dat|-] > dupes.tsv
sfdupes trees [files.dat|-] > dupetrees.tsv
sfdupes scan [--workers N] [-x] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
```
### Database
All three subcommands operate on a single SQLite database file:
- Location: the value of the `SFDUPES_DATABASE` environment variable
when set and non-empty, otherwise `/var/lib/sfdupes/db.sqlite`.
There is no command-line flag.
- `scan` creates the database (and its parent directory) on first
use. `report` and `trees` require an existing database; a missing
database file is a fatal error (exit 1) telling the user to run
`scan` first.
- The database uses WAL journal mode and a busy timeout, so running a
report while a cron `scan` is in progress is safe; the reports see
the last committed state. Each scan commits its changes in a single
transaction, so a report never observes a half-finished scan and a
scan that dies partway leaves the previous state intact.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error):
```sql
CREATE TABLE files (
path BLOB PRIMARY KEY, -- absolute path, raw bytes
size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- Unix seconds, from lstat
head TEXT NOT NULL, -- lowercase-hex SHA-256, first 1 KiB
tail TEXT NOT NULL -- lowercase-hex SHA-256, last 1 KiB
) WITHOUT ROWID;
```
Paths are stored as BLOBs because Unix paths are raw bytes, not
guaranteed UTF-8. `mtime` is used only for change detection; it is
not part of the duplicate key.
### `scan` mode
`scan` requires one or more `PATH` operands naming the trees to scan.
There is no default path; invoking `scan` with no operand is a usage
error (usage message on stderr, exit 2). An operand may be a directory
or a regular file; an operand that does not exist is a fatal error
(exit 1). Operands are walked in the order given; overlapping operands
(one containing another) emit their common files once per operand, so
callers should pass disjoint paths.
(exit 1). Because database records persist between runs and are keyed
by absolute path, each operand is resolved to an absolute, lexically
cleaned path (symlinks are not resolved) before walking, so results do
not depend on the working directory. Operands are walked in the order
given; overlapping operands (one containing another) are harmless — a
file reached via multiple operands produces one database record.
`scan` runs **three sequential passes**, in this order, so that every
`scan` synchronizes the database with the filesystem state under the
scanned operands:
- A file not yet in the database is hashed and inserted.
- A file already in the database is **skipped without reading its
contents** when its lstat size equals the recorded size and its
lstat mtime is not newer than the recorded mtime. This is what
makes a daily rescan cheap.
- A file whose mtime is newer than recorded, or whose size differs,
is re-hashed and its record updated.
- A database record whose path lies under one of the scanned operands
but was not successfully processed this run is deleted. This
removes records for deleted files. It also removes records for
paths that failed to stat or hash this run: the database only ever
contains signatures verified by the most recent scan that covered
them (a subsequent successful scan re-adds such files).
- Database records outside the scanned operands are untouched, so
disjoint trees can be scanned on different schedules into the same
database.
`scan` runs **four sequential passes**, in this order, so that every
expensive pass has an exact total for meaningful progress and ETA:
1. **walk** — recursively enumerate the tree under each `PATH` in
turn, collecting the list of regular-file paths. Total unknown
while running: show a live count, not a percentage.
2. **stat** — `lstat` every collected path, recording size and mtime.
3. **hash** — for each file, read the first `min(1024, size)` bytes and
the last `min(1024, size)` bytes (the two reads overlap when
`size < 2048`; for `size == 0` hash the empty input) and compute the
SHA-256 of each. Emit the output record.
3. **hash** — for each new or changed file (per the rules above), read
the first `min(1024, size)` bytes and the last `min(1024, size)`
bytes (the two reads overlap when `size < 2048`; for `size == 0`
hash the empty input) and compute the SHA-256 of each. Unchanged
files are not read and do not appear in this pass's total.
4. **update** — apply all insertions, updates, and deletions to the
database in a single transaction.
Rules for the walk:
@@ -134,52 +213,46 @@ Rules for the walk:
- On any per-path error (permission denied, file vanished between
passes, unreadable): print a one-line warning to stderr, skip the
path, and continue. Per-file errors never abort the run; the final
summary reports how many were skipped.
summary reports how many were skipped. As specified above, a
skipped path that has a database record from an earlier scan loses
that record; an unreadable directory subtree likewise loses its
records (accepted: the database mirrors what the latest scan could
actually verify).
Concurrency: the stat and hash passes use a worker pool (`--workers`,
default `runtime.NumCPU()`). The main goroutine owns stdout writing and
progress rendering; progress display must never block the workers.
default `runtime.NumCPU()`). The main goroutine owns database writes
and progress rendering; progress display must never block the workers.
#### Output record format
One record per file on stdout, NUL-terminated (`\x00`), with
tab-separated fields, **path last** so tabs or newlines embedded in
paths cannot corrupt the record structure:
`scan` writes nothing to stdout. The summary line on stderr reports the
files seen this run broken down by disposition, plus skips:
```
<size>\t<mtime_unix>\t<sha256_first1k_hex>\t<sha256_last1k_hex>\t<path>\x00
scan: 123456 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged), 3 skipped
```
- `size`: decimal bytes, from the stat pass.
- `mtime_unix`: decimal Unix seconds. Informational only; not part of
the duplicate key.
- Hashes: lowercase hex, 64 chars each.
- Record order is unspecified (workers complete out of order); the
analysis modes must not depend on ordering.
(`removed` counts deleted database records, which are not part of the
files-seen total.)
### `report` mode
`report` reads the scan stream from the file named in its first
positional argument, or from stdin if the argument is absent or `-`.
`report` reads every record from the database and takes no positional
arguments.
**`report` must never touch the filesystem being analyzed.** It does not
stat, open, or otherwise access any path that appears in the records; its
only I/O is reading the scan file/stdin and writing stdout/stderr. It must
only I/O is reading the database and writing stdout/stderr. It must
produce identical output whether or not the scanned filesystem is still
mounted.
Processing:
- Parse records; a record that does not have exactly 5 fields or whose
size is non-numeric is counted as malformed and skipped (warn once
with the total malformed count in the summary, not per record).
- Group records by the key `(size, head_hash, tail_hash)`.
- Every group with two or more paths is a duplicate group.
- Within each group, sort paths lexicographically (byte order). The
first path is the group's `first`; every other path is a `dupe`.
- Order groups by size descending (biggest reclaimable space first),
tie-broken by `first` path ascending. Output must be fully
deterministic for a given input.
deterministic for a given database state.
#### Report output format
@@ -192,16 +265,16 @@ first dupe size
/srv/a/big.iso /srv/c/big-copy2.iso 4294967296
```
Summary to stderr: records read, malformed count (if any), number of
duplicate groups, number of dupe files, and total reclaimable bytes
(sum of `size` over all dupe rows) in human units.
Summary to stderr: records read, number of duplicate groups, number of
dupe files, and total reclaimable bytes (sum of `size` over all dupe
rows) in human units.
### `trees` mode
`trees` reads the same scan stream as `report` (same argument handling,
same parsing and malformed-record rules) and reports **entire duplicate
directory trees**: directories under which the exact same set of relative
paths exists with the exact same file signatures.
`trees` reads the same database as `report` (no positional arguments)
and reports **entire duplicate directory trees**: directories under
which the exact same set of relative paths exists with the exact same
file signatures.
**`trees` must never touch the filesystem being analyzed** — the same
rule as `report`. The directory hierarchy is reconstructed purely from
@@ -222,10 +295,10 @@ Definitions:
equal. Equal digests imply equal recursive file count and equal
total byte size.
Known limitation (accepted): only regular files that appear in the scan
stream define a tree. Empty directories are invisible, and a file skipped
during the scan (e.g. permission error) in one copy but not the other
will make otherwise-identical trees compare as different.
Known limitation (accepted): only regular files that appear in the
database define a tree. Empty directories are invisible, and a file
skipped during the scan (e.g. permission error) in one copy but not the
other will make otherwise-identical trees compare as different.
Processing:
@@ -254,22 +327,22 @@ first dupe files size
/srv/a/project /srv/backup/project 3417 104857600
```
Summary to stderr: records read, malformed count (if any), number of
duplicate-tree groups, number of dupe trees, and total reclaimable bytes
(sum of `size` over all dupe rows) in human units.
Summary to stderr: records read, number of duplicate-tree groups,
number of dupe trees, and total reclaimable bytes (sum of `size` over
all dupe rows) in human units.
### Progress
Use the progress-bar library for all scan-pass progress; rendering in the
style of `pv` is the model. All progress goes to stderr.
Each scan pass gets its own bar. Required elements for the stat and hash
passes (known totals):
Each scan pass gets its own bar. Required elements for the stat, hash,
and update passes (known totals):
- elapsed time
- estimated time remaining
- a `[m/n] x%` display (files processed / total files, percent)
- current rate (files/s)
- a `[m/n] x%` display (items processed / total items, percent)
- current rate (items/s)
Example shape (exact layout is flexible, content is not):
@@ -292,9 +365,11 @@ Additional requirements:
### Error handling and exit codes
- `0`: success, even if individual files were skipped with warnings.
- `1`: fatal error (e.g., a `PATH` operand does not exist, cannot
read the scan input, stdout write failure).
- `2`: usage error (including `scan` with no `PATH` operand).
- `1`: fatal error (e.g., a `PATH` operand does not exist, the
database cannot be created/opened/read/written, a missing database
for `report`/`trees`, stdout write failure).
- `2`: usage error (including `scan` with no `PATH` operand and
`report`/`trees` with any positional argument).
## Build
@@ -310,7 +385,7 @@ The `Makefile` is the single source of truth for all operations:
- `make docker` — build the Docker image, which runs `make check` as
a build stage.
- `make hooks` — install the pre-commit hook.
- `make clean` — remove the binary and any local `files.dat`.
- `make clean` — remove the binary and any legacy local `files.dat`.
### Definition of done
@@ -323,6 +398,7 @@ All of the following, run in this directory, must pass:
```sh
d=$(mktemp -d)
export SFDUPES_DATABASE="$d/db.sqlite"
mkdir -p "$d/a" "$d/b"
head -c 2000 /dev/urandom > "$d/a/one.bin"
cp "$d/a/one.bin" "$d/b/copy.bin"
@@ -339,28 +415,41 @@ All of the following, run in this directory, must pass:
cp "$d/t1/sub/f2" "$d/t2/sub/f2"
cp "$d/t1/f1" "$d/t3/f1"
cp "$d/t1/sub/f2" "$d/t3/sub/f2renamed"
./sfdupes scan "$d" > files.dat
./sfdupes report files.dat
./sfdupes trees files.dat
./sfdupes scan "$d"
./sfdupes report
./sfdupes trees
# incremental behavior (scan a subtree; records elsewhere persist):
./sfdupes scan "$d/a" # everything unchanged, nothing hashed
printf 'z' >> "$d/a/one.bin" # modify: next scan re-hashes it
rm "$d/a/unique.bin" # delete: next scan removes its record
./sfdupes scan "$d/a" # 1 updated, 1 removed
./sfdupes report
```
Expected from `report`: `one.bin`/`copy.bin`/`copy2.bin` form one
group (two dupe rows, `first` is the lexicographically smallest
path); `t1/f1`/`t2/f1`/`t3/f1` form one group; `t1/sub/f2`/
`t2/sub/f2`/`t3/sub/f2renamed` form one group; `tiny1`/`tiny2` pair;
`empty1`/`empty2` pair; `unique.bin` and `tiny3` appear nowhere;
groups ordered by size descending; piping scan directly into report
(`./sfdupes scan "$d" | ./sfdupes report`) gives the same
rows.
(The scan database lives inside `$d` here purely for test hygiene;
scanning `$d` therefore also records the SQLite file itself, which
is harmless.)
Expected from the first `report`: `one.bin`/`copy.bin`/`copy2.bin`
form one group (two dupe rows, `first` is the lexicographically
smallest path); `t1/f1`/`t2/f1`/`t3/f1` form one group;
`t1/sub/f2`/ `t2/sub/f2`/`t3/sub/f2renamed` form one group;
`tiny1`/`tiny2` pair; `empty1`/`empty2` pair; `unique.bin` and
`tiny3` appear nowhere; groups ordered by size descending.
Expected from `trees`: exactly one row — `first` `$d/t1`, `dupe`
`$d/t2`, 2 files, 3100 bytes. `$d/t1/sub` vs `$d/t2/sub` is
suppressed as non-maximal (implied by the `t1`/`t2` group), and `t3`
appears nowhere (its file set differs by name).
Expected from the second `report` (after the modify/delete rescan):
`one.bin` has left its group (its content changed), so
`copy.bin`/`copy2.bin` remain as one pair, and `unique.bin` is
gone from the database.
The test suite automates this scenario (see `scan_test.go`), plus a
negative check: `report` and `trees` operate on the captured stream
alone and never touch the scanned filesystem.
negative check: `report` and `trees` operate on the database alone
and never touch the scanned filesystem.
## TODO
@@ -371,7 +460,9 @@ Tracked in [TODO.md](TODO.md).
- No full-content verification, no byte-for-byte compare, no deletion
or linking of duplicates. The reports are advisory; acting on them is
the user's job.
- No persistence formats beyond the scan stream described above.
- No persistence beyond the SQLite database described above; no
export/import formats.
- No daemon or filesystem watcher; scheduling rescans is cron's job.
## License

39
TODO.md
View File

@@ -14,8 +14,41 @@
# Next Step
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos
- persistent scan database (branch `persistent-database`): `scan`
maintains a SQLite database that survives between runs so it can be
cronned daily; `report` and `trees` read the database instead of a
scan stream. Database path defaults to `/var/lib/sfdupes/db.sqlite`,
overridable via `SFDUPES_DATABASE`. Plan:
- [x] update `README.md` (the authoritative spec) for the database:
new Database section, revised `scan`/`report`/`trees` modes,
dependency list (`modernc.org/sqlite`, pure Go, cgo stays
disabled), exit codes, smoke test with incremental steps
- [ ] add `modernc.org/sqlite` (hash-pinned via `go.sum`)
- [ ] `db.go`: path resolution (`SFDUPES_DATABASE` env, default
`/var/lib/sfdupes/db.sqlite`), open/create with WAL +
busy-timeout pragmas and `PRAGMA user_version` schema check,
`files` table (path BLOB primary key, size, mtime, head,
tail), load-all-rows, and single-transaction apply of
upserts/deletes
- [ ] `scan.go`: resolve operands to absolute paths; diff stat
results against loaded rows (reuse hashes when size matches
and mtime is not newer); hash only new/changed files; delete
rows under the scanned operands not successfully processed;
leave rows outside the operands untouched; new summary line
(added/updated/removed/unchanged/skipped); "update" progress
pass for the database write
- [ ] `report.go`/`trees.go`: read records from the database (no
positional args, no stream parsing); drop the NUL-stream
parser and malformed-record handling
- [ ] `main.go`: updated usage strings; `report`/`trees` take no
args (usage error otherwise)
- [ ] tests: db open/env-override/schema tests; incremental scan
tests (add, mtime update, delete, unchanged-hash-reuse,
out-of-scope rows untouched, error paths); rework pipeline
test to go through the database; keep walk/stat/hash/grouping
unit tests
- [ ] `.gitignore`: local `*.sqlite` artifacts
- [ ] `make check`, `make docker`, and the README smoke test pass
# Completed Steps
@@ -33,6 +66,8 @@
# Future Steps
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos
- possible later features (explicitly out of scope per README):
full-content verification of candidates, removal-script helpers