Today a file under 10 MiB is read in full; a larger one is first compared by its first and last 64 KiB, and if those match another file's it is read in full below 50 MiB, or only sampled (1 MiB per gigabyte) at 50 MiB and above. The two thresholds have no reason beyond saving reads.
Proposed change: one rule for every size. Files are grouped by size, then by their first and last 64 KiB (a small file's windows cover all of it), and every file that still matches another is read in full for its content hash. No 10 MiB or 50 MiB split, and no sampling, so a reported duplicate is always byte-identical. A large file is read in full only when its size and both ends already match another file, which is when it almost certainly is a duplicate and has to be read anyway before anyone deletes it.
Question: build this? Recommendation: yes.
Model: opus-5-5
Today a file under 10 MiB is read in full; a larger one is first compared by its first and last 64 KiB, and if those match another file's it is read in full below 50 MiB, or only sampled (1 MiB per gigabyte) at 50 MiB and above. The two thresholds have no reason beyond saving reads.
Proposed change: one rule for every size. Files are grouped by size, then by their first and last 64 KiB (a small file's windows cover all of it), and every file that still matches another is read in full for its content hash. No 10 MiB or 50 MiB split, and no sampling, so a reported duplicate is always byte-identical. A large file is read in full only when its size and both ends already match another file, which is when it almost certainly is a duplicate and has to be read anyway before anyone deletes it.
Question: build this? Recommendation: yes.
Model: opus-5-5
sneak
was assigned by clawbot2026-10-22 18:51:42 +02:00
no, you don't understand the requirements. large files cannot be read in full, it would take unreasonably long. this is why the per-gigabyte sampling exists. all files should have their head and tail 64k hashed and stored. files under 100MiB should have their full content hash stored. files >100MiB should use the sampling trick.
no, you don't understand the requirements. large files cannot be read in full, it would take unreasonably long. this is why the per-gigabyte sampling exists. all files should have their head and tail 64k hashed and stored. files under 100MiB should have their full content hash stored. files >100MiB should use the sampling trick.
Every scanned file has the SHA-256 of its first 64 KiB and of its last 64 KiB stored.
A file under 100 MiB also has the SHA-256 of its whole content stored.
A file of 100 MiB or more has the per-gigabyte sampled hash stored instead (the 1 MiB window at each gigabyte offset), so it is never read in full.
That replaces today's 10 MiB and 50 MiB thresholds. sfdupes' manager builds it on next, with the README's "Duplicate detection" and "Rationale" rewritten to match, when the fleet restarts at 23:00 UTC on 25 October.
Model: opus-5-5
Recorded, as the spec to build:
- Every scanned file has the SHA-256 of its first 64 KiB and of its last 64 KiB stored.
- A file under 100 MiB also has the SHA-256 of its whole content stored.
- A file of 100 MiB or more has the per-gigabyte sampled hash stored instead (the 1 MiB window at each gigabyte offset), so it is never read in full.
That replaces today's 10 MiB and 50 MiB thresholds. sfdupes' manager builds it on `next`, with the README's "Duplicate detection" and "Rationale" rewritten to match, when the fleet restarts at 23:00 UTC on 25 October.
Model: opus-5-5
sneak
was unassigned by clawbot2026-10-22 19:36:30 +02:00
clawbot
self-assigned this 2026-10-22 19:36:30 +02:00
In the hash phase every file the walk records gets three hashes: head and tail, the SHA-256 of its first and last 64 KiB, and content. Under 100 MiB, content is the SHA-256 of the whole file. At 100 MiB and above it is the sampled hash (1 MiB at each gigabyte offset), so such a file is never read in full.
The 10 MiB and 50 MiB thresholds go.
Judgement call: a file whose size no other file shares is hashed too, since the spec covers all files. In your trial that skip spared only about 4% of files (28.4 million of 29.5 million were queued for hashing anyway).
The separate content phase goes, because the hash phase computes every hash. A rescan hashes only new files and files whose size or mtime changed.
The README's "Duplicate detection" and "Rationale" are rewritten to match, and every other README sentence the change makes false is corrected.
Tests cover files just under and exactly at 100 MiB.
Plan, one PR into `next`:
- In the hash phase every file the walk records gets three hashes: `head` and `tail`, the SHA-256 of its first and last 64 KiB, and `content`. Under 100 MiB, `content` is the SHA-256 of the whole file. At 100 MiB and above it is the sampled hash (1 MiB at each gigabyte offset), so such a file is never read in full.
- The 10 MiB and 50 MiB thresholds go.
- Judgement call: a file whose size no other file shares is hashed too, since the spec covers all files. In your trial that skip spared only about 4% of files (28.4 million of 29.5 million were queued for hashing anyway).
- The separate content phase goes, because the hash phase computes every hash. A rescan hashes only new files and files whose size or mtime changed.
- The README's "Duplicate detection" and "Rationale" are rewritten to match, and every other README sentence the change makes false is corrected.
- Tests cover files just under and exactly at 100 MiB.
Raw hash storage (https://git.eeqj.de/sneak/sfdupes/issues/134) follows as its own PR.
Model: opus-5-5
Built in #136: every recorded file now gets its head, tail and content hash in the hash phase, with one 100 MiB boundary between the whole-file and the sampled content hash. The PR body has the details and disclosures.
Model: opus-5-5
Built in https://git.eeqj.de/sneak/sfdupes/pulls/136: every recorded file now gets its head, tail and content hash in the hash phase, with one 100 MiB boundary between the whole-file and the sampled content hash. The PR body has the details and disclosures.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
what's with the 10mb/50mb split?
Today a file under 10 MiB is read in full; a larger one is first compared by its first and last 64 KiB, and if those match another file's it is read in full below 50 MiB, or only sampled (1 MiB per gigabyte) at 50 MiB and above. The two thresholds have no reason beyond saving reads.
Proposed change: one rule for every size. Files are grouped by size, then by their first and last 64 KiB (a small file's windows cover all of it), and every file that still matches another is read in full for its content hash. No 10 MiB or 50 MiB split, and no sampling, so a reported duplicate is always byte-identical. A large file is read in full only when its size and both ends already match another file, which is when it almost certainly is a duplicate and has to be read anyway before anyone deletes it.
Question: build this? Recommendation: yes.
Model: opus-5-5
no, you don't understand the requirements. large files cannot be read in full, it would take unreasonably long. this is why the per-gigabyte sampling exists. all files should have their head and tail 64k hashed and stored. files under 100MiB should have their full content hash stored. files >100MiB should use the sampling trick.
Recorded, as the spec to build:
That replaces today's 10 MiB and 50 MiB thresholds. sfdupes' manager builds it on
next, with the README's "Duplicate detection" and "Rationale" rewritten to match, when the fleet restarts at 23:00 UTC on 25 October.Model: opus-5-5
Plan, one PR into
next:headandtail, the SHA-256 of its first and last 64 KiB, andcontent. Under 100 MiB,contentis the SHA-256 of the whole file. At 100 MiB and above it is the sampled hash (1 MiB at each gigabyte offset), so such a file is never read in full.Raw hash storage (#134) follows as its own PR.
Model: opus-5-5
Built in #136: every recorded file now gets its head, tail and content hash in the hash phase, with one 100 MiB boundary between the whole-file and the sampled content hash. The PR body has the details and disclosures.
Model: opus-5-5