Today a file under 10 MiB is read in full; a larger one is first compared by its first and last 64 KiB, and if those match another file's it is read in full below 50 MiB, or only sampled (1 MiB per gigabyte) at 50 MiB and above. The two thresholds have no reason beyond saving reads.
Proposed change: one rule for every size. Files are grouped by size, then by their first and last 64 KiB (a small file's windows cover all of it), and every file that still matches another is read in full for its content hash. No 10 MiB or 50 MiB split, and no sampling, so a reported duplicate is always byte-identical. A large file is read in full only when its size and both ends already match another file, which is when it almost certainly is a duplicate and has to be read anyway before anyone deletes it.
Question: build this? Recommendation: yes.
Model: opus-5-5
Today a file under 10 MiB is read in full; a larger one is first compared by its first and last 64 KiB, and if those match another file's it is read in full below 50 MiB, or only sampled (1 MiB per gigabyte) at 50 MiB and above. The two thresholds have no reason beyond saving reads.
Proposed change: one rule for every size. Files are grouped by size, then by their first and last 64 KiB (a small file's windows cover all of it), and every file that still matches another is read in full for its content hash. No 10 MiB or 50 MiB split, and no sampling, so a reported duplicate is always byte-identical. A large file is read in full only when its size and both ends already match another file, which is when it almost certainly is a duplicate and has to be read anyway before anyone deletes it.
Question: build this? Recommendation: yes.
Model: opus-5-5
sneak
was assigned by clawbot2026-10-22 18:51:42 +02:00
no, you don't understand the requirements. large files cannot be read in full, it would take unreasonably long. this is why the per-gigabyte sampling exists. all files should have their head and tail 64k hashed and stored. files under 100MiB should have their full content hash stored. files >100MiB should use the sampling trick.
no, you don't understand the requirements. large files cannot be read in full, it would take unreasonably long. this is why the per-gigabyte sampling exists. all files should have their head and tail 64k hashed and stored. files under 100MiB should have their full content hash stored. files >100MiB should use the sampling trick.
Every scanned file has the SHA-256 of its first 64 KiB and of its last 64 KiB stored.
A file under 100 MiB also has the SHA-256 of its whole content stored.
A file of 100 MiB or more has the per-gigabyte sampled hash stored instead (the 1 MiB window at each gigabyte offset), so it is never read in full.
That replaces today's 10 MiB and 50 MiB thresholds. sfdupes' manager builds it on next, with the README's "Duplicate detection" and "Rationale" rewritten to match, when the fleet restarts at 23:00 UTC on 25 October.
Model: opus-5-5
Recorded, as the spec to build:
- Every scanned file has the SHA-256 of its first 64 KiB and of its last 64 KiB stored.
- A file under 100 MiB also has the SHA-256 of its whole content stored.
- A file of 100 MiB or more has the per-gigabyte sampled hash stored instead (the 1 MiB window at each gigabyte offset), so it is never read in full.
That replaces today's 10 MiB and 50 MiB thresholds. sfdupes' manager builds it on `next`, with the README's "Duplicate detection" and "Rationale" rewritten to match, when the fleet restarts at 23:00 UTC on 25 October.
Model: opus-5-5
sneak
was unassigned by clawbot2026-10-22 19:36:30 +02:00
clawbot
self-assigned this 2026-10-22 19:36:30 +02:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
what's with the 10mb/50mb split?
Today a file under 10 MiB is read in full; a larger one is first compared by its first and last 64 KiB, and if those match another file's it is read in full below 50 MiB, or only sampled (1 MiB per gigabyte) at 50 MiB and above. The two thresholds have no reason beyond saving reads.
Proposed change: one rule for every size. Files are grouped by size, then by their first and last 64 KiB (a small file's windows cover all of it), and every file that still matches another is read in full for its content hash. No 10 MiB or 50 MiB split, and no sampling, so a reported duplicate is always byte-identical. A large file is read in full only when its size and both ends already match another file, which is when it almost certainly is a duplicate and has to be read anyway before anyone deletes it.
Question: build this? Recommendation: yes.
Model: opus-5-5
no, you don't understand the requirements. large files cannot be read in full, it would take unreasonably long. this is why the per-gigabyte sampling exists. all files should have their head and tail 64k hashed and stored. files under 100MiB should have their full content hash stored. files >100MiB should use the sampling trick.
Recorded, as the spec to build:
That replaces today's 10 MiB and 50 MiB thresholds. sfdupes' manager builds it on
next, with the README's "Duplicate detection" and "Rationale" rewritten to match, when the fleet restarts at 23:00 UTC on 25 October.Model: opus-5-5