Hash files under 10 MiB in full, gate larger files on head/tail
check / check (push) Successful in 59s

Amends the issue 61 ladder per the owner's design change. A file under
10 MiB (new headTailMin) is now hashed in full and compared directly,
with no end-window step: its head, tail, and content all carry the
whole-file SHA-256, so its signature is decided by size and content
alone. A file at 10 MiB or above keeps the 64 KiB head/tail gate, then
the whole-file content hash below 50 MiB or gigabyte-spaced 1 MiB
samples at or above. hashEnds drops its now-unreachable single-window
branch, and hashWhole errors if the file shrank below its recorded
size (the only read for the sub-10-MiB range). README, TODO, tests,
and the schema-version note updated to match.

Model: opus-4-8
This commit is contained in:
2026-09-22 14:22:17 +00:00
parent b80c7e805e
commit dc30ad9f31
5 changed files with 203 additions and 107 deletions
+12 -10
View File
@@ -29,17 +29,19 @@
# Completed Steps
- replace the 1 KiB end-window sampling with the 64 KiB head/tail plus
- replace the 1 KiB end-window sampling with the head/tail plus
content-hash ladder (2026-09-22, branch `next`, closes
https://git.eeqj.de/sneak/sfdupes/issues/61): the duplicate signature
gains a `content` hash — the whole file below 50 MiB, gigabyte-spaced
1 MiB samples at or above — and the end windows widen from 1 KiB to
64 KiB. Schema bumps to version 2 (new `content` column); a version 1
database is rejected and must be rescanned, which is required anyway
since every stored hash changed. `report` and `trees` group by the
extended signature, so the ladder is applied across the whole
database. README "Duplicate detection" documents every rung including
the probabilistic large-file path.
https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is
hashed in full and compared directly, with no end-window step — its
`head`, `tail`, and `content` all hold the whole-file hash. A file at
10 MiB or above is gated on the 64 KiB `head` and `tail`, then
compared on a `content` hash — the whole file below 50 MiB,
gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2
(new `content` column); a version 1 database is rejected and must be
rescanned, which is required anyway since every stored hash changed.
`report` and `trees` group by the extended signature, so the ladder is
applied across the whole database. README "Duplicate detection"
documents every rung including the probabilistic large-file path.
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
and `.dockerignore` (2026-09-21, branch `next`, closes