Add 64 KiB head/tail and content-hash duplicate ladder (closes #61)
check / check (push) Successful in 1m0s

Replace the 1 KiB end sampling with a ladder for same-size candidates:
SHA-256 of the first and last 64 KiB, then a content hash that is the
whole file below 50 MiB (proof of identity) and gigabyte-spaced 1 MiB
samples at or above (deliberately probabilistic). Two files are
duplicates only when size, head, tail, and content all agree.

The signature gains a content column; schema bumps to version 2, so a
version 1 database is rejected and must be rescanned (unavoidable — every
stored hash changed). Because report and trees group stored signatures
across separate scans, content is computed for every shared-size file,
not only within-run head/tail collisions; size remains the sole read
gate. README "Duplicate detection" documents each rung; tests cover the
window boundaries, the 50 MiB boundary, and a multi-gigabyte sampled
case with sparse temp files.

Model: opus-4-8
This commit is contained in:
2026-09-22 13:58:57 +00:00
parent 7ac4f6b723
commit b80c7e805e
8 changed files with 446 additions and 115 deletions
+12
View File
@@ -29,6 +29,18 @@
# Completed Steps
- replace the 1 KiB end-window sampling with the 64 KiB head/tail plus
content-hash ladder (2026-09-22, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/61): the duplicate signature
gains a `content` hash — the whole file below 50 MiB, gigabyte-spaced
1 MiB samples at or above — and the end windows widen from 1 KiB to
64 KiB. Schema bumps to version 2 (new `content` column); a version 1
database is rejected and must be rescanned, which is required anyway
since every stored hash changed. `report` and `trees` group by the
extended signature, so the ladder is applied across the whole
database. README "Duplicate detection" documents every rung including
the probabilistic large-file path.
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
and `.dockerignore` (2026-09-21, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/22)