Add 64 KiB head/tail and content-hash duplicate ladder (closes #61) (#62)

2026-09-22 16:40:43 +02:00
parent 7ac4f6b723
commit 29a65016d0
8 changed files with 555 additions and 128 deletions
@@ -29,6 +29,20 @@

 # Completed Steps

+- replace the 1 KiB end-window sampling with the head/tail plus
+  content-hash ladder (2026-09-22, branch `next`, closes
+  https://git.eeqj.de/sneak/sfdupes/issues/61): a file under 10 MiB is
+  hashed in full and compared directly, with no end-window step — its
+  `head`, `tail`, and `content` all hold the whole-file hash. A file at
+  10 MiB or above is gated on the 64 KiB `head` and `tail`, then
+  compared on a `content` hash — the whole file below 50 MiB,
+  gigabyte-spaced 1 MiB samples at or above. Schema bumps to version 2
+  (new `content` column); a version 1 database is rejected and must be
+  rescanned, which is required anyway since every stored hash changed.
+  `report` and `trees` group by the extended signature, so the ladder is
+  applied across the whole database. README "Duplicate detection"
+  documents every rung including the probabilistic large-file path.
+
 - remove the dead `files.dat` references from `Makefile`, `.gitignore`
  and `.dockerignore` (2026-09-21, branch `next`, closes
  https://git.eeqj.de/sneak/sfdupes/issues/22)