Hash files under 10 MiB in full, gate larger files on head/tail
check / check (push) Successful in 59s
check / check (push) Successful in 59s
Amends the issue 61 ladder per the owner's design change. A file under 10 MiB (new headTailMin) is now hashed in full and compared directly, with no end-window step: its head, tail, and content all carry the whole-file SHA-256, so its signature is decided by size and content alone. A file at 10 MiB or above keeps the 64 KiB head/tail gate, then the whole-file content hash below 50 MiB or gigabyte-spaced 1 MiB samples at or above. hashEnds drops its now-unreachable single-window branch, and hashWhole errors if the file shrank below its recorded size (the only read for the sub-10-MiB range). README, TODO, tests, and the schema-version note updated to match. Model: opus-4-8
This commit is contained in:
@@ -29,17 +29,19 @@
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- replace the 1 KiB end-window sampling with the 64 KiB head/tail plus
|
||||
- replace the 1 KiB end-window sampling with the head/tail plus
|
||||
content-hash ladder (2026-09-22, branch `next`, closes
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/61): the duplicate signature
|
||||
gains a `content` hash — the whole file below 50 MiB, gigabyte-spaced
|
||||
1 MiB samples at or above — and the end windows widen from 1 KiB to
|
||||
64 KiB. Schema bumps to version 2 (new `content` column); a version 1
|
||||
database is rejected and must be rescanned, which is required anyway
|
||||
since every stored hash changed. `report` and `trees` group by the
|
||||
extended signature, so the ladder is applied across the whole
|
||||
database. README "Duplicate detection" documents every rung including
|
||||
the probabilistic large-file path.
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is
|
||||
hashed in full and compared directly, with no end-window step — its
|
||||
`head`, `tail`, and `content` all hold the whole-file hash. A file at
|
||||
10 MiB or above is gated on the 64 KiB `head` and `tail`, then
|
||||
compared on a `content` hash — the whole file below 50 MiB,
|
||||
gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2
|
||||
(new `content` column); a version 1 database is rejected and must be
|
||||
rescanned, which is required anyway since every stored hash changed.
|
||||
`report` and `trees` group by the extended signature, so the ladder is
|
||||
applied across the whole database. README "Duplicate detection"
|
||||
documents every rung including the probabilistic large-file path.
|
||||
|
||||
- remove the dead `files.dat` references from `Makefile`, `.gitignore`
|
||||
and `.dockerignore` (2026-09-21, branch `next`, closes
|
||||
|
||||
Reference in New Issue
Block a user