sfdupes

65 Commits 6 Branches 1 Tag

3 Commits

Include renames

Author	SHA1	Message	Date
sneak	b80c7e805e	Add 64 KiB head/tail and content-hash duplicate ladder (closes #61 ) check / check (push) Successful in 1m0s Details Replace the 1 KiB end sampling with a ladder for same-size candidates: SHA-256 of the first and last 64 KiB, then a content hash that is the whole file below 50 MiB (proof of identity) and gigabyte-spaced 1 MiB samples at or above (deliberately probabilistic). Two files are duplicates only when size, head, tail, and content all agree. The signature gains a content column; schema bumps to version 2, so a version 1 database is rejected and must be rescanned (unavoidable — every stored hash changed). Because report and trees group stored signatures across separate scans, content is computed for every shared-size file, not only within-run head/tail collisions; size remains the sole read gate. README "Duplicate detection" documents each rung; tests cover the window boundaries, the 50 MiB boundary, and a multi-gigabyte sampled case with sparse temp files. Model: opus-4-8	2026-09-22 13:58:57 +00:00
sneak	d8fbcb32c2	Implement persistent SQLite scan database check / check (push) Successful in 3s Details scan now synchronizes a database that survives between runs (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) instead of emitting a stream: operands are resolved to absolute paths, unchanged files (same size, mtime not newer than recorded) are never re-read, new and changed files are hashed, and records under the scanned operands that were not verified this run are deleted; records outside the operands are untouched. All changes commit in a single transaction, and WAL journaling with a busy timeout keeps a report run during a cron scan safe. report and trees read the database (no positional arguments); the NUL-terminated stream format, its parser, and the malformed-record handling are gone. The driver is modernc.org/sqlite (pure Go), so builds keep cgo disabled.	2026-07-24 03:08:49 +07:00
sneak	7a6adae0d2	Add test suite covering parsing, grouping, digests, and scan passes Covers splitNUL framing, parseScanStream (malformed records, tabs and newlines in paths, unterminated trailing record), duplicate grouping with ordering/tie-break/determinism, humanBytes units, Merkle digest equality and name/content sensitivity, maximal-tree suppression (including sibling and differing-parent cases), hashHeadTail edge sizes and error paths, walkPass .zfs/symlink skipping, statPass vanished files, and an end-to-end walk/stat/hash pipeline test of the README smoke-test scenario. 64% statement coverage.	2026-07-23 05:42:38 +07:00

Author

SHA1

Message

Date

sneak

b80c7e805e

Add 64 KiB head/tail and content-hash duplicate ladder (closes #61 )

check / check (push) Successful in 1m0s

Details

Replace the 1 KiB end sampling with a ladder for same-size candidates:
SHA-256 of the first and last 64 KiB, then a content hash that is the
whole file below 50 MiB (proof of identity) and gigabyte-spaced 1 MiB
samples at or above (deliberately probabilistic). Two files are
duplicates only when size, head, tail, and content all agree.

The signature gains a content column; schema bumps to version 2, so a
version 1 database is rejected and must be rescanned (unavoidable — every
stored hash changed). Because report and trees group stored signatures
across separate scans, content is computed for every shared-size file,
not only within-run head/tail collisions; size remains the sole read
gate. README "Duplicate detection" documents each rung; tests cover the
window boundaries, the 50 MiB boundary, and a multi-gigabyte sampled
case with sparse temp files.

Model: opus-4-8

2026-09-22 13:58:57 +00:00

sneak

d8fbcb32c2

Implement persistent SQLite scan database

check / check (push) Successful in 3s

Details

scan now synchronizes a database that survives between runs
(SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite) instead of
emitting a stream: operands are resolved to absolute paths, unchanged
files (same size, mtime not newer than recorded) are never re-read, new
and changed files are hashed, and records under the scanned operands
that were not verified this run are deleted; records outside the
operands are untouched. All changes commit in a single transaction, and
WAL journaling with a busy timeout keeps a report run during a cron
scan safe.

report and trees read the database (no positional arguments); the
NUL-terminated stream format, its parser, and the malformed-record
handling are gone. The driver is modernc.org/sqlite (pure Go), so
builds keep cgo disabled.

2026-07-24 03:08:49 +07:00

sneak

7a6adae0d2

Add test suite covering parsing, grouping, digests, and scan passes

Covers splitNUL framing, parseScanStream (malformed records, tabs and
newlines in paths, unterminated trailing record), duplicate grouping
with ordering/tie-break/determinism, humanBytes units, Merkle digest
equality and name/content sensitivity, maximal-tree suppression
(including sibling and differing-parent cases), hashHeadTail edge
sizes and error paths, walkPass .zfs/symlink skipping, statPass
vanished files, and an end-to-end walk/stat/hash pipeline test of the
README smoke-test scenario. 64% statement coverage.

2026-07-23 05:42:38 +07:00