The duplicate ladder now branches on size. A file under 10 MiB is hashed
in full and compared directly, with no end-window step. A file at 10 MiB
or above is gated on the SHA-256 of its first and last 64 KiB (head, tail), then compared on a content hash — the whole file below
50 MiB, or gigabyte-spaced 1 MiB samples at or above 50 MiB. Two files
are duplicates only when size, head, tail, and content all agree. report and trees group by the full signature, so the ladder applies
across the whole database.
What head and tail hold below 10 MiB
The schema keeps three hash columns. For a file under 10 MiB there is no
separate end-window read: head, tail, and content all store the
same value — the SHA-256 of the whole file. So its signature is (size, whole, whole, whole), and grouping on the four fields reduces to
size plus whole-file content. At 10 MiB and above, head/tail hold the
first-/last-64 KiB hashes and content the whole-file or sampled hash. A
sub-10-MiB file and a larger one never share a size, so the two meanings
never collide.
Why the reader might trip
Schema is version 2; a version 1 database is rejected and must be
rescanned.
hashWhole now errors if the file shrank below its recorded size — for
the sub-10-MiB range it is the only read, so a truncated read must fail
rather than emit a hash that no longer matches the recorded size.
The large-file sampled rung is deliberately probabilistic.
Verification
make check green (tests, Docker lint, fmt-check). Tests cover the
sub-10-MiB direct hash, the 64 KiB end windows on a ≥10 MiB sparse file,
the 50 MiB content boundary, and a multi-gigabyte sampled case.
Model: opus-4-8
## What changed
The duplicate ladder now branches on size. A file under 10 MiB is hashed
in full and compared directly, with no end-window step. A file at 10 MiB
or above is gated on the SHA-256 of its first and last 64 KiB (`head`,
`tail`), then compared on a `content` hash — the whole file below
50 MiB, or gigabyte-spaced 1 MiB samples at or above 50 MiB. Two files
are duplicates only when size, head, tail, and content all agree.
`report` and `trees` group by the full signature, so the ladder applies
across the whole database.
## What `head` and `tail` hold below 10 MiB
The schema keeps three hash columns. For a file under 10 MiB there is no
separate end-window read: `head`, `tail`, and `content` all store the
same value — the SHA-256 of the whole file. So its signature is
`(size, whole, whole, whole)`, and grouping on the four fields reduces to
size plus whole-file content. At 10 MiB and above, `head`/`tail` hold the
first-/last-64 KiB hashes and `content` the whole-file or sampled hash. A
sub-10-MiB file and a larger one never share a size, so the two meanings
never collide.
## Why the reader might trip
- Schema is version 2; a version 1 database is rejected and must be
rescanned.
- `hashWhole` now errors if the file shrank below its recorded size — for
the sub-10-MiB range it is the only read, so a truncated read must fail
rather than emit a hash that no longer matches the recorded size.
- The large-file sampled rung is deliberately probabilistic.
## Verification
`make check` green (tests, Docker lint, fmt-check). Tests cover the
sub-10-MiB direct hash, the 64 KiB end windows on a ≥10 MiB sparse file,
the 50 MiB content boundary, and a multi-gigabyte sampled case.
Model: opus-4-8
Replace the 1 KiB end sampling with a ladder for same-size candidates:
SHA-256 of the first and last 64 KiB, then a content hash that is the
whole file below 50 MiB (proof of identity) and gigabyte-spaced 1 MiB
samples at or above (deliberately probabilistic). Two files are
duplicates only when size, head, tail, and content all agree.
The signature gains a content column; schema bumps to version 2, so a
version 1 database is rejected and must be rescanned (unavoidable — every
stored hash changed). Because report and trees group stored signatures
across separate scans, content is computed for every shared-size file,
not only within-run head/tail collisions; size remains the sole read
gate. README "Duplicate detection" documents each rung; tests cover the
window boundaries, the 50 MiB boundary, and a multi-gigabyte sampled
case with sparse temp files.
Model: opus-4-8
Owner amendment (issue 61, comment 99522) — REQUIRED before review: files under 10 MiB are hashed in full directly, with no 65k head/tail sampling step (under 128 KiB trivially so, since the two windows already cover the file). The ladder is therefore: equal size → under 10 MiB: the whole-file hash is the comparison → 10 MiB and over: 65k head/tail gate, then whole-file hash under 50 MiB, gigabyte-spaced 1 MiB samples at or over. Adjust implementation, constants, README and tests accordingly, and state in the PR body what the stored signature's head/tail fields hold for the sub-10-MiB range. A rework session is being dispatched; review follows the rework. (model: claude-fable-5)
Owner amendment (issue 61, comment 99522) — REQUIRED before review: files under 10 MiB are hashed in full directly, with no 65k head/tail sampling step (under 128 KiB trivially so, since the two windows already cover the file). The ladder is therefore: equal size → under 10 MiB: the whole-file hash is the comparison → 10 MiB and over: 65k head/tail gate, then whole-file hash under 50 MiB, gigabyte-spaced 1 MiB samples at or over. Adjust implementation, constants, README and tests accordingly, and state in the PR body what the stored signature's head/tail fields hold for the sub-10-MiB range. A rework session is being dispatched; review follows the rework. (model: claude-fable-5)
Amends the issue 61 ladder per the owner's design change. A file under
10 MiB (new headTailMin) is now hashed in full and compared directly,
with no end-window step: its head, tail, and content all carry the
whole-file SHA-256, so its signature is decided by size and content
alone. A file at 10 MiB or above keeps the 64 KiB head/tail gate, then
the whole-file content hash below 50 MiB or gigabyte-spaced 1 MiB
samples at or above. hashEnds drops its now-unreachable single-window
branch, and hashWhole errors if the file shrank below its recorded
size (the only read for the sub-10-MiB range). README, TODO, tests,
and the schema-version note updated to match.
Model: opus-4-8
Reworked per the owner amendment. Files under 10 MiB are now hashed in full and compared directly with no head/tail step — head, tail, and content all hold the whole-file hash. Files at 10 MiB and over keep the 64 KiB head/tail gate, then the whole-file hash under 50 MiB or gigabyte-spaced 1 MiB samples at or above. Implementation, the named constants, README, TODO, and tests are updated, and the PR body now states what the stored head/tail fields hold for the sub-10-MiB range. make check is green.
Model: opus-4-8
Reworked per the owner amendment. Files under 10 MiB are now hashed in full and compared directly with no head/tail step — `head`, `tail`, and `content` all hold the whole-file hash. Files at 10 MiB and over keep the 64 KiB head/tail gate, then the whole-file hash under 50 MiB or gigabyte-spaced 1 MiB samples at or above. Implementation, the named constants, README, TODO, and tests are updated, and the PR body now states what the stored `head`/`tail` fields hold for the sub-10-MiB range. `make check` is green.
Model: opus-4-8
Independent review against the issue instruction as amended by the owner (sub-10 MiB hashed in full; 10 MiB and over gated on 64 KiB head/tail then whole-file content under 50 MiB, gigabyte-spaced 1 MiB samples at or over; any rung mismatch means not a duplicate): the implementation matches the ladder exactly at every boundary, uses named constants with no magic numbers, bumps the schema to version 2 and rejects a version 1 database loudly, documents the ladder including the probabilistic large-file path in the README, handles a shrunk file as a loud error, carries meaningful temp-file tests for every rung and boundary with no committed fixtures, and passes make check in Docker cleanly while merging into the current next — PASS.
Model: opus-4-8
Independent review against the issue instruction as amended by the owner (sub-10 MiB hashed in full; 10 MiB and over gated on 64 KiB head/tail then whole-file content under 50 MiB, gigabyte-spaced 1 MiB samples at or over; any rung mismatch means not a duplicate): the implementation matches the ladder exactly at every boundary, uses named constants with no magic numbers, bumps the schema to version 2 and rejects a version 1 database loudly, documents the ladder including the probabilistic large-file path in the README, handles a shrunk file as a loud error, carries meaningful temp-file tests for every rung and boundary with no committed fixtures, and passes `make check` in Docker cleanly while merging into the current `next` — PASS.
Model: opus-4-8
clawbot
merged commit 29a65016d0 into next2026-09-22 16:40:43 +02:00
Squash-merged to next on the PASS above; branch deleted. Issue 61 closes when next merges to main. A verification session is re-running make check on the new next head and posts the result on PR 50, per the always-green rule — the candidate stays merge-ready while green. (model: claude-fable-5)
Squash-merged to `next` on the PASS above; branch deleted. Issue 61 closes when `next` merges to `main`. A verification session is re-running `make check` on the new `next` head and posts the result on PR 50, per the always-green rule — the candidate stays merge-ready while green. (model: claude-fable-5)
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
What changed
The duplicate ladder now branches on size. A file under 10 MiB is hashed
in full and compared directly, with no end-window step. A file at 10 MiB
or above is gated on the SHA-256 of its first and last 64 KiB (
head,tail), then compared on acontenthash — the whole file below50 MiB, or gigabyte-spaced 1 MiB samples at or above 50 MiB. Two files
are duplicates only when size, head, tail, and content all agree.
reportandtreesgroup by the full signature, so the ladder appliesacross the whole database.
What
headandtailhold below 10 MiBThe schema keeps three hash columns. For a file under 10 MiB there is no
separate end-window read:
head,tail, andcontentall store thesame value — the SHA-256 of the whole file. So its signature is
(size, whole, whole, whole), and grouping on the four fields reduces tosize plus whole-file content. At 10 MiB and above,
head/tailhold thefirst-/last-64 KiB hashes and
contentthe whole-file or sampled hash. Asub-10-MiB file and a larger one never share a size, so the two meanings
never collide.
Why the reader might trip
rescanned.
hashWholenow errors if the file shrank below its recorded size — forthe sub-10-MiB range it is the only read, so a truncated read must fail
rather than emit a hash that no longer matches the recorded size.
Verification
make checkgreen (tests, Docker lint, fmt-check). Tests cover thesub-10-MiB direct hash, the 64 KiB end windows on a ≥10 MiB sparse file,
the 50 MiB content boundary, and a multi-gigabyte sampled case.
Model: opus-4-8
Owner amendment (issue 61, comment 99522) — REQUIRED before review: files under 10 MiB are hashed in full directly, with no 65k head/tail sampling step (under 128 KiB trivially so, since the two windows already cover the file). The ladder is therefore: equal size → under 10 MiB: the whole-file hash is the comparison → 10 MiB and over: 65k head/tail gate, then whole-file hash under 50 MiB, gigabyte-spaced 1 MiB samples at or over. Adjust implementation, constants, README and tests accordingly, and state in the PR body what the stored signature's head/tail fields hold for the sub-10-MiB range. A rework session is being dispatched; review follows the rework. (model: claude-fable-5)
Reworked per the owner amendment. Files under 10 MiB are now hashed in full and compared directly with no head/tail step —
head,tail, andcontentall hold the whole-file hash. Files at 10 MiB and over keep the 64 KiB head/tail gate, then the whole-file hash under 50 MiB or gigabyte-spaced 1 MiB samples at or above. Implementation, the named constants, README, TODO, and tests are updated, and the PR body now states what the storedhead/tailfields hold for the sub-10-MiB range.make checkis green.Model: opus-4-8
Independent review against the issue instruction as amended by the owner (sub-10 MiB hashed in full; 10 MiB and over gated on 64 KiB head/tail then whole-file content under 50 MiB, gigabyte-spaced 1 MiB samples at or over; any rung mismatch means not a duplicate): the implementation matches the ladder exactly at every boundary, uses named constants with no magic numbers, bumps the schema to version 2 and rejects a version 1 database loudly, documents the ladder including the probabilistic large-file path in the README, handles a shrunk file as a loud error, carries meaningful temp-file tests for every rung and boundary with no committed fixtures, and passes
make checkin Docker cleanly while merging into the currentnext— PASS.Model: opus-4-8
Squash-merged to
nexton the PASS above; branch deleted. Issue 61 closes whennextmerges tomain. A verification session is re-runningmake checkon the newnexthead and posts the result on PR 50, per the always-green rule — the candidate stays merge-ready while green. (model: claude-fable-5)