Add 64 KiB head/tail and content-hash duplicate ladder (closes #61)
check / check (push) Successful in 1m0s
check / check (push) Successful in 1m0s
Replace the 1 KiB end sampling with a ladder for same-size candidates: SHA-256 of the first and last 64 KiB, then a content hash that is the whole file below 50 MiB (proof of identity) and gigabyte-spaced 1 MiB samples at or above (deliberately probabilistic). Two files are duplicates only when size, head, tail, and content all agree. The signature gains a content column; schema bumps to version 2, so a version 1 database is rejected and must be rescanned (unavoidable — every stored hash changed). Because report and trees group stored signatures across separate scans, content is computed for every shared-size file, not only within-run head/tail collisions; size remains the sole read gate. README "Duplicate detection" documents each rung; tests cover the window boundaries, the 50 MiB boundary, and a multi-gigabyte sampled case with sparse temp files. Model: opus-4-8
This commit is contained in:
@@ -29,6 +29,18 @@
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- replace the 1 KiB end-window sampling with the 64 KiB head/tail plus
|
||||
content-hash ladder (2026-09-22, branch `next`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/61): the duplicate signature
|
||||
gains a `content` hash — the whole file below 50 MiB, gigabyte-spaced
|
||||
1 MiB samples at or above — and the end windows widen from 1 KiB to
|
||||
64 KiB. Schema bumps to version 2 (new `content` column); a version 1
|
||||
database is rejected and must be rescanned, which is required anyway
|
||||
since every stored hash changed. `report` and `trees` group by the
|
||||
extended signature, so the ladder is applied across the whole
|
||||
database. README "Duplicate detection" documents every rung including
|
||||
the probabilistic large-file path.
|
||||
|
||||
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
|
||||
and `.dockerignore` (2026-09-21, branch `next`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/22)
|
||||
|
||||
Reference in New Issue
Block a user