Add 64 KiB head/tail and content-hash duplicate ladder (closes #61)

Conversation 4 Commits 2 Files Changed 8

+555 -128

2 Commits

Author	SHA1	Message	Date
sneak	dc30ad9f31	Hash files under 10 MiB in full, gate larger files on head/tail check / check (push) Successful in 59s Details Amends the issue 61 ladder per the owner's design change. A file under 10 MiB (new headTailMin) is now hashed in full and compared directly, with no end-window step: its head, tail, and content all carry the whole-file SHA-256, so its signature is decided by size and content alone. A file at 10 MiB or above keeps the 64 KiB head/tail gate, then the whole-file content hash below 50 MiB or gigabyte-spaced 1 MiB samples at or above. hashEnds drops its now-unreachable single-window branch, and hashWhole errors if the file shrank below its recorded size (the only read for the sub-10-MiB range). README, TODO, tests, and the schema-version note updated to match. Model: opus-4-8	2026-09-22 14:22:17 +00:00
sneak	b80c7e805e	Add 64 KiB head/tail and content-hash duplicate ladder (closes #61 ) check / check (push) Successful in 1m0s Details Replace the 1 KiB end sampling with a ladder for same-size candidates: SHA-256 of the first and last 64 KiB, then a content hash that is the whole file below 50 MiB (proof of identity) and gigabyte-spaced 1 MiB samples at or above (deliberately probabilistic). Two files are duplicates only when size, head, tail, and content all agree. The signature gains a content column; schema bumps to version 2, so a version 1 database is rejected and must be rescanned (unavoidable — every stored hash changed). Because report and trees group stored signatures across separate scans, content is computed for every shared-size file, not only within-run head/tail collisions; size remains the sole read gate. README "Duplicate detection" documents each rung; tests cover the window boundaries, the 50 MiB boundary, and a multi-gigabyte sampled case with sparse temp files. Model: opus-4-8	2026-09-22 13:58:57 +00:00

Author

SHA1

Message

Date

sneak

dc30ad9f31

Hash files under 10 MiB in full, gate larger files on head/tail

check / check (push) Successful in 59s

Details

Amends the issue 61 ladder per the owner's design change. A file under
10 MiB (new headTailMin) is now hashed in full and compared directly,
with no end-window step: its head, tail, and content all carry the
whole-file SHA-256, so its signature is decided by size and content
alone. A file at 10 MiB or above keeps the 64 KiB head/tail gate, then
the whole-file content hash below 50 MiB or gigabyte-spaced 1 MiB
samples at or above. hashEnds drops its now-unreachable single-window
branch, and hashWhole errors if the file shrank below its recorded
size (the only read for the sub-10-MiB range). README, TODO, tests,
and the schema-version note updated to match.

Model: opus-4-8

2026-09-22 14:22:17 +00:00

sneak

b80c7e805e

Add 64 KiB head/tail and content-hash duplicate ladder (closes #61 )

check / check (push) Successful in 1m0s

Details

Replace the 1 KiB end sampling with a ladder for same-size candidates:
SHA-256 of the first and last 64 KiB, then a content hash that is the
whole file below 50 MiB (proof of identity) and gigabyte-spaced 1 MiB
samples at or above (deliberately probabilistic). Two files are
duplicates only when size, head, tail, and content all agree.

The signature gains a content column; schema bumps to version 2, so a
version 1 database is rejected and must be rescanned (unavoidable — every
stored hash changed). Because report and trees group stored signatures
across separate scans, content is computed for every shared-size file,
not only within-run head/tail collisions; size remains the sole read
gate. README "Duplicate detection" documents each rung; tests cover the
window boundaries, the 50 MiB boundary, and a multi-gigabyte sampled
case with sparse temp files.

Model: opus-4-8

2026-09-22 13:58:57 +00:00

Add 64 KiB head/tail and content-hash duplicate ladder (closes #61) #62