if the file size is identical and the 65k start and 65k end match, and the file is under 50MiB, we hash the whole file and compare those. if the file is over 50MiB we take 1MiB at the top of each gigabyte and hash that and compare it. any mismatch means it's not a dupe.
this way we only invest disk i/o on potential duplicates, and spend a bit more time checking to see if big files are dupes. obviously it's not foolproof but it's better.
if the file size is identical and the 65k start and 65k end match, and the file is under 50MiB, we hash the whole file and compare those. if the file is over 50MiB we take 1MiB at the top of each gigabyte and hash that and compare it. any mismatch means it's not a dupe.
this way we only invest disk i/o on potential duplicates, and spend a bit more time checking to see if big files are dupes. obviously it's not foolproof but it's better.
clawbot
was assigned by sneak2026-09-22 15:35:19 +02:00
Work order accepted; dispatched 2026-09-22 as a scope-limited direct unit (the repo stays otherwise paused). Implementation instructions for the worker:
Target comparison ladder for same-size candidates, replacing the current 1k start-and-end sampling:
Sizes equal, as today.
Read 65536 bytes at the start and 65536 bytes at the end; any byte mismatch disqualifies the pair. Files too small for two full windows: the implementer documents the chosen handling in the PR.
Survivors with size under 50 MiB: hash the entire file and compare hashes.
Survivors with size 50 MiB or over: sampled hash — take the 1 MiB at the top of each gigabyte of the file, feed the samples in order into one hash, compare. Adopted reading: the first 1 MiB after each gigabyte-aligned offset (0, 1 GiB, 2 GiB, ... while inside the file), truncated at end of file; if a different sampling was meant, the owner corrects on this issue or the PR.
A mismatch at any rung means not a duplicate; only full agreement reports one. Rationale per the issue: disk i/o is spent only on likely duplicates, and the over-50 MiB path is deliberately probabilistic.
Requirements: named constants, no magic numbers; README documents the ladder including the probabilistic large-file path; unit tests cover each rung and the size boundaries (window size, just under and over 50 MiB, and a multi-gigabyte case if cheap — sparse temp files are fine, no committed fixtures); existing database/caching behaviour preserved unless the ladder requires a schema note, which the PR must call out. Branch from next, PR to next, title ends with (closes#61), make check green in Docker before push.
(model: claude-fable-5)
Work order accepted; dispatched 2026-09-22 as a scope-limited direct unit (the repo stays otherwise paused). Implementation instructions for the worker:
Target comparison ladder for same-size candidates, replacing the current 1k start-and-end sampling:
1. Sizes equal, as today.
2. Read 65536 bytes at the start and 65536 bytes at the end; any byte mismatch disqualifies the pair. Files too small for two full windows: the implementer documents the chosen handling in the PR.
3. Survivors with size under 50 MiB: hash the entire file and compare hashes.
4. Survivors with size 50 MiB or over: sampled hash — take the 1 MiB at the top of each gigabyte of the file, feed the samples in order into one hash, compare. Adopted reading: the first 1 MiB after each gigabyte-aligned offset (0, 1 GiB, 2 GiB, ... while inside the file), truncated at end of file; if a different sampling was meant, the owner corrects on this issue or the PR.
5. A mismatch at any rung means not a duplicate; only full agreement reports one. Rationale per the issue: disk i/o is spent only on likely duplicates, and the over-50 MiB path is deliberately probabilistic.
Requirements: named constants, no magic numbers; README documents the ladder including the probabilistic large-file path; unit tests cover each rung and the size boundaries (window size, just under and over 50 MiB, and a multi-gigabyte case if cheap — sparse temp files are fine, no committed fixtures); existing database/caching behaviour preserved unless the ladder requires a schema note, which the PR must call out. Branch from `next`, PR to `next`, title ends with (closes #61), `make check` green in Docker before push.
(model: claude-fable-5)
Implemented as #62 (branch issue-61-65k-sampling-ladder, base next).
The duplicate signature is now size + SHA-256 of the first and last 64 KiB + a content hash: the whole file below 50 MiB, gigabyte-spaced 1 MiB samples at or above. report and trees group by the full signature, so the ladder applies across separate scans.
Two design notes for the reviewer: the content hash is computed for every shared-size file (not gated on a head/tail match within one run), because grouping compares stored signatures across scans and size is the only cross-run-safe read gate; and the schema bumps to version 2, so a version 1 database is rejected and must be rescanned (every stored hash changed anyway). Files at most 64 KiB have head and tail equal to the whole-file hash.
make check is green (tests, Docker lint, fmt-check).
Model: opus-4-8
Implemented as https://git.eeqj.de/sneak/sfdupes/pulls/62 (branch `issue-61-65k-sampling-ladder`, base `next`).
The duplicate signature is now size + SHA-256 of the first and last 64 KiB + a content hash: the whole file below 50 MiB, gigabyte-spaced 1 MiB samples at or above. `report` and `trees` group by the full signature, so the ladder applies across separate scans.
Two design notes for the reviewer: the content hash is computed for every shared-size file (not gated on a head/tail match within one run), because grouping compares stored signatures across scans and size is the only cross-run-safe read gate; and the schema bumps to version 2, so a version 1 database is rejected and must be rescanned (every stored hash changed anyway). Files at most 64 KiB have head and tail equal to the whole-file hash.
`make check` is green (tests, Docker lint, fmt-check).
Model: opus-4-8
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
if the file size is identical and the 65k start and 65k end match, and the file is under 50MiB, we hash the whole file and compare those. if the file is over 50MiB we take 1MiB at the top of each gigabyte and hash that and compare it. any mismatch means it's not a dupe.
this way we only invest disk i/o on potential duplicates, and spend a bit more time checking to see if big files are dupes. obviously it's not foolproof but it's better.
Work order accepted; dispatched 2026-09-22 as a scope-limited direct unit (the repo stays otherwise paused). Implementation instructions for the worker:
Target comparison ladder for same-size candidates, replacing the current 1k start-and-end sampling:
Requirements: named constants, no magic numbers; README documents the ladder including the probabilistic large-file path; unit tests cover each rung and the size boundaries (window size, just under and over 50 MiB, and a multi-gigabyte case if cheap — sparse temp files are fine, no committed fixtures); existing database/caching behaviour preserved unless the ladder requires a schema note, which the PR must call out. Branch from
next, PR tonext, title ends with (closes #61),make checkgreen in Docker before push.(model: claude-fable-5)
obviously files
<128kwe just hash in full. in fact, hash any files<10MiBin full.Implemented as #62 (branch
issue-61-65k-sampling-ladder, basenext).The duplicate signature is now size + SHA-256 of the first and last 64 KiB + a content hash: the whole file below 50 MiB, gigabyte-spaced 1 MiB samples at or above.
reportandtreesgroup by the full signature, so the ladder applies across separate scans.Two design notes for the reviewer: the content hash is computed for every shared-size file (not gated on a head/tail match within one run), because grouping compares stored signatures across scans and size is the only cross-run-safe read gate; and the schema bumps to version 2, so a version 1 database is rejected and must be rescanned (every stored hash changed anyway). Files at most 64 KiB have head and tail equal to the whole-file hash.
make checkis green (tests, Docker lint, fmt-check).Model: opus-4-8