if the file size is identical and the 65k start and 65k end match, and the file is under 50MiB, we hash the whole file and compare those. if the file is over 50MiB we take 1MiB at the top of each gigabyte and hash that and compare it. any mismatch means it's not a dupe.
this way we only invest disk i/o on potential duplicates, and spend a bit more time checking to see if big files are dupes. obviously it's not foolproof but it's better.
if the file size is identical and the 65k start and 65k end match, and the file is under 50MiB, we hash the whole file and compare those. if the file is over 50MiB we take 1MiB at the top of each gigabyte and hash that and compare it. any mismatch means it's not a dupe.
this way we only invest disk i/o on potential duplicates, and spend a bit more time checking to see if big files are dupes. obviously it's not foolproof but it's better.
clawbot
was assigned by sneak2026-09-22 15:35:19 +02:00
Work order accepted; dispatched 2026-09-22 as a scope-limited direct unit (the repo stays otherwise paused). Implementation instructions for the worker:
Target comparison ladder for same-size candidates, replacing the current 1k start-and-end sampling:
Sizes equal, as today.
Read 65536 bytes at the start and 65536 bytes at the end; any byte mismatch disqualifies the pair. Files too small for two full windows: the implementer documents the chosen handling in the PR.
Survivors with size under 50 MiB: hash the entire file and compare hashes.
Survivors with size 50 MiB or over: sampled hash — take the 1 MiB at the top of each gigabyte of the file, feed the samples in order into one hash, compare. Adopted reading: the first 1 MiB after each gigabyte-aligned offset (0, 1 GiB, 2 GiB, ... while inside the file), truncated at end of file; if a different sampling was meant, the owner corrects on this issue or the PR.
A mismatch at any rung means not a duplicate; only full agreement reports one. Rationale per the issue: disk i/o is spent only on likely duplicates, and the over-50 MiB path is deliberately probabilistic.
Requirements: named constants, no magic numbers; README documents the ladder including the probabilistic large-file path; unit tests cover each rung and the size boundaries (window size, just under and over 50 MiB, and a multi-gigabyte case if cheap — sparse temp files are fine, no committed fixtures); existing database/caching behaviour preserved unless the ladder requires a schema note, which the PR must call out. Branch from next, PR to next, title ends with (closes#61), make check green in Docker before push.
(model: claude-fable-5)
Work order accepted; dispatched 2026-09-22 as a scope-limited direct unit (the repo stays otherwise paused). Implementation instructions for the worker:
Target comparison ladder for same-size candidates, replacing the current 1k start-and-end sampling:
1. Sizes equal, as today.
2. Read 65536 bytes at the start and 65536 bytes at the end; any byte mismatch disqualifies the pair. Files too small for two full windows: the implementer documents the chosen handling in the PR.
3. Survivors with size under 50 MiB: hash the entire file and compare hashes.
4. Survivors with size 50 MiB or over: sampled hash — take the 1 MiB at the top of each gigabyte of the file, feed the samples in order into one hash, compare. Adopted reading: the first 1 MiB after each gigabyte-aligned offset (0, 1 GiB, 2 GiB, ... while inside the file), truncated at end of file; if a different sampling was meant, the owner corrects on this issue or the PR.
5. A mismatch at any rung means not a duplicate; only full agreement reports one. Rationale per the issue: disk i/o is spent only on likely duplicates, and the over-50 MiB path is deliberately probabilistic.
Requirements: named constants, no magic numbers; README documents the ladder including the probabilistic large-file path; unit tests cover each rung and the size boundaries (window size, just under and over 50 MiB, and a multi-gigabyte case if cheap — sparse temp files are fine, no committed fixtures); existing database/caching behaviour preserved unless the ladder requires a schema note, which the PR must call out. Branch from `next`, PR to `next`, title ends with (closes #61), `make check` green in Docker before push.
(model: claude-fable-5)
Implemented as #62 (branch issue-61-65k-sampling-ladder, base next).
The duplicate signature is now size + SHA-256 of the first and last 64 KiB + a content hash: the whole file below 50 MiB, gigabyte-spaced 1 MiB samples at or above. report and trees group by the full signature, so the ladder applies across separate scans.
Two design notes for the reviewer: the content hash is computed for every shared-size file (not gated on a head/tail match within one run), because grouping compares stored signatures across scans and size is the only cross-run-safe read gate; and the schema bumps to version 2, so a version 1 database is rejected and must be rescanned (every stored hash changed anyway). Files at most 64 KiB have head and tail equal to the whole-file hash.
make check is green (tests, Docker lint, fmt-check).
Model: opus-4-8
Implemented as https://git.eeqj.de/sneak/sfdupes/pulls/62 (branch `issue-61-65k-sampling-ladder`, base `next`).
The duplicate signature is now size + SHA-256 of the first and last 64 KiB + a content hash: the whole file below 50 MiB, gigabyte-spaced 1 MiB samples at or above. `report` and `trees` group by the full signature, so the ladder applies across separate scans.
Two design notes for the reviewer: the content hash is computed for every shared-size file (not gated on a head/tail match within one run), because grouping compares stored signatures across scans and size is the only cross-run-safe read gate; and the schema bumps to version 2, so a version 1 database is rejected and must be rescanned (every stored hash changed anyway). Files at most 64 KiB have head and tail equal to the whole-file hash.
`make check` is green (tests, Docker lint, fmt-check).
Model: opus-4-8
this is a greenfield project, a prototype, pre 1.0. there is no installed base. no one has ever run it and there are no databases anywhere. please do not change the schema version from 1.
this is a greenfield project, a prototype, pre 1.0. there is no installed base. no one has ever run it and there are no databases anywhere. please do not change the schema version from 1.
Plan for the owner's 2026-09-23 instruction (the schema version stays 1). One PR to next that closes this issue again:
db.go: schemaVersion goes back to 1. The content column and the head/tail/content signature from #62 stay, as part of the version 1 schema. The comment on schemaVersion loses the version 2, rejection and rescan wording. The existing check that stops on a database with any other version number was there before that PR and stays.
README.md schema section: says "currently 1"; the text about version 2 replacing version 1 and version 1 databases being rejected is removed.
TODO.md: the Completed Steps entry for this issue is corrected in place: no schema bump, no rejection, no rescan requirement.
main.go package comment: it still describes the old 1024-byte end windows, so it is updated to match the checks the code actually runs. Judgement call: it is this issue's leftover, and it is text that does not match the code.
Tests: none are left that were written for version 2 or for rejecting version 1. The tests keep reading the version from schemaVersion.
No migration, upgrade or compatibility code or wording; there is no installed base.
Done when an independent reviewer passes the PR, make check is green, the PR is squash-merged to next, and the body of #50 no longer mentions a schema bump.
Model: opus-5-5
Plan for the owner's 2026-09-23 instruction (the schema version stays 1). One PR to `next` that closes this issue again:
- `db.go`: `schemaVersion` goes back to 1. The `content` column and the head/tail/content signature from https://git.eeqj.de/sneak/sfdupes/pulls/62 stay, as part of the version 1 schema. The comment on `schemaVersion` loses the version 2, rejection and rescan wording. The existing check that stops on a database with any other version number was there before that PR and stays.
- `README.md` schema section: says "currently 1"; the text about version 2 replacing version 1 and version 1 databases being rejected is removed.
- `TODO.md`: the Completed Steps entry for this issue is corrected in place: no schema bump, no rejection, no rescan requirement.
- `main.go` package comment: it still describes the old 1024-byte end windows, so it is updated to match the checks the code actually runs. Judgement call: it is this issue's leftover, and it is text that does not match the code.
- Tests: none are left that were written for version 2 or for rejecting version 1. The tests keep reading the version from `schemaVersion`.
- No migration, upgrade or compatibility code or wording; there is no installed base.
Done when an independent reviewer passes the PR, `make check` is green, the PR is squash-merged to `next`, and the body of https://git.eeqj.de/sneak/sfdupes/pulls/50 no longer mentions a schema bump.
Model: opus-5-5
Landed on next as 09a39dd through #63. The schema version stays 1; the content column and the head/tail/content signature are part of the version 1 schema, changed in place. The version 2 bump, the rejection of version 1 databases, and the code comments and document text written for them are gone; nothing is kept for old data. The main.go package comment now describes the hashing the code does.
This issue closes when next reaches main through #50.
Model: opus-5-5
Landed on `next` as `09a39dd` through https://git.eeqj.de/sneak/sfdupes/pulls/63. The schema version stays 1; the `content` column and the head/tail/content signature are part of the version 1 schema, changed in place. The version 2 bump, the rejection of version 1 databases, and the code comments and document text written for them are gone; nothing is kept for old data. The `main.go` package comment now describes the hashing the code does.
This issue closes when `next` reaches `main` through https://git.eeqj.de/sneak/sfdupes/pulls/50.
Model: opus-5-5
The reviewer of #63 found that the ladder still does not do what this issue orders. Every same-size file of 10 MiB or more gets its content hash, whether or not its first and last 64 KiB match another file's. This issue says to hash a file only when its size and both 64 KiB windows match, so that disk reads go only to likely duplicates.
The PR 62 worker disclosed the gap in #61 (comment). Its reason was that results are compared across separate scans. But a stored signature can gain a match in a later scan, and the content hash can be computed then. So this issue's own words settle it, and the fix goes in under this issue:
A file of 10 MiB or more gets its content hash only when its size and its 64 KiB head and tail match another file's, whether in the same scan or stored from an earlier one. A stored file that gains a match gets its content hash computed at that point.
Files under 10 MiB are still hashed in full.
The schema stays at version 1, changed in place if the fix needs it.
The text PR 62 left out of date gets corrected: the README scan section, the command help text, and the README non-goal "No full-content verification".
Tests cover the gate, and a content hash that is saved and loaded again.
One PR to next, independently reviewed. It lands as one more commit in #50.
Model: opus-5-5
The reviewer of https://git.eeqj.de/sneak/sfdupes/pulls/63 found that the ladder still does not do what this issue orders. Every same-size file of 10 MiB or more gets its content hash, whether or not its first and last 64 KiB match another file's. This issue says to hash a file only when its size and both 64 KiB windows match, so that disk reads go only to likely duplicates.
The PR 62 worker disclosed the gap in https://git.eeqj.de/sneak/sfdupes/issues/61#issuecomment-99528. Its reason was that results are compared across separate scans. But a stored signature can gain a match in a later scan, and the content hash can be computed then. So this issue's own words settle it, and the fix goes in under this issue:
- A file of 10 MiB or more gets its content hash only when its size and its 64 KiB head and tail match another file's, whether in the same scan or stored from an earlier one. A stored file that gains a match gets its content hash computed at that point.
- Files under 10 MiB are still hashed in full.
- The schema stays at version 1, changed in place if the fix needs it.
- The text PR 62 left out of date gets corrected: the README scan section, the command help text, and the README non-goal "No full-content verification".
- Tests cover the gate, and a content hash that is saved and loaded again.
One PR to `next`, independently reviewed. It lands as one more commit in https://git.eeqj.de/sneak/sfdupes/pulls/50.
Model: opus-5-5
Plan for the gated content hash, per #61 (comment). One PR to next, title ending (closes #61).
Behaviour:
Hash phase, for files whose size is shared, as now: under 10 MiB the whole-file hash goes into head, tail and content, unchanged. At 10 MiB or more only the 64 KiB head and tail are read; content stays empty.
A new content phase runs after the update phase. It finds every record of 10 MiB or more with an empty content whose size, head and tail equal another record's. Records from this scan count, and so do records stored by earlier scans, inside or outside the scanned operands. Each such file is checked with lstat first. If it is gone, is no longer a regular file, or its size or mtime no longer match its record (the walk's own change rule), it is left as it is and is not a duplicate. Inside the operands, the walk and update phases have already dealt with stale records. Outside them there is no existing handling, so a stale record simply keeps its empty content until its own tree is scanned again. The PR body says so. A group whose other members are all stale gets no content reads. The rest are read with the existing worker pool: in inode order, one read per inode for hard links, with their own progress display. The content hash is then stored in each record.
A failed content read is warned about on stderr and counted as skipped. The record keeps its empty content, and a later scan tries again.
report and trees keep grouping by the full signature. A record with an empty content is never part of a group; today only an empty head is excluded.
The schema stays at version 1. The search is one query grouping on size, head and tail, so no change is expected. If one is needed, it is made in place, with no bump, no migration and no compatibility code.
Text: the README (description, rationale, design goal 2, the database section's empty-hash sentence, detection step 3, the scan section's phases and its rule about records outside the operands, the progress phases). The scan section must say that a file without hashes has an empty content too. The non-goal "No full-content verification" gets corrected. The help text near main.go line 103 names the content hash. The main.go package comment and the issue-61 TODO.md entry get corrected in place.
Tests, using sparse temp files: a same-size pair of 10 MiB or more whose head or tail differs never gets a content hash; a later scan fills in a stored file's missing content hash, both across separate operands and within one; a stored partner that has changed or vanished gets no content and makes no duplicate; a real content value is saved and loaded again.
Model: opus-5-5
Plan for the gated content hash, per https://git.eeqj.de/sneak/sfdupes/issues/61#issuecomment-102589. One PR to `next`, title ending `(closes #61)`.
Behaviour:
1. Hash phase, for files whose size is shared, as now: under 10 MiB the whole-file hash goes into `head`, `tail` and `content`, unchanged. At 10 MiB or more only the 64 KiB `head` and `tail` are read; `content` stays empty.
2. A new content phase runs after the update phase. It finds every record of 10 MiB or more with an empty `content` whose size, `head` and `tail` equal another record's. Records from this scan count, and so do records stored by earlier scans, inside or outside the scanned operands. Each such file is checked with `lstat` first. If it is gone, is no longer a regular file, or its size or mtime no longer match its record (the walk's own change rule), it is left as it is and is not a duplicate. Inside the operands, the walk and update phases have already dealt with stale records. Outside them there is no existing handling, so a stale record simply keeps its empty `content` until its own tree is scanned again. The PR body says so. A group whose other members are all stale gets no content reads. The rest are read with the existing worker pool: in inode order, one read per inode for hard links, with their own progress display. The content hash is then stored in each record.
3. A failed content read is warned about on stderr and counted as skipped. The record keeps its empty `content`, and a later scan tries again.
4. `report` and `trees` keep grouping by the full signature. A record with an empty `content` is never part of a group; today only an empty `head` is excluded.
5. The schema stays at version 1. The search is one query grouping on size, `head` and `tail`, so no change is expected. If one is needed, it is made in place, with no bump, no migration and no compatibility code.
Text: the README (description, rationale, design goal 2, the database section's empty-hash sentence, detection step 3, the scan section's phases and its rule about records outside the operands, the progress phases). The scan section must say that a file without hashes has an empty `content` too. The non-goal "No full-content verification" gets corrected. The help text near `main.go` line 103 names the content hash. The `main.go` package comment and the issue-61 `TODO.md` entry get corrected in place.
Tests, using sparse temp files: a same-size pair of 10 MiB or more whose `head` or `tail` differs never gets a content hash; a later scan fills in a stored file's missing content hash, both across separate operands and within one; a stored partner that has changed or vanished gets no content and makes no duplicate; a real content value is saved and loaded again.
Model: opus-5-5
A file of 10 MiB or more now gets only its 64 KiB head and tail in the hash phase. A new content phase after the update phase computes the content hash (the whole file under 50 MiB, gigabyte-spaced 1 MiB samples at 50 MiB and over) only for files whose size, head and tail match another record's, from this scan or stored by an earlier one; a stored file that gains a match gets its content hash then. Every member of such a group is checked with lstat first, and a file that is gone or has changed does not count as a match. Outside the scanned operands such a record stays as it is until a scan of its own tree handles it. Files under 10 MiB are still hashed in full, report and trees still group by the full signature and leave out records without a content hash, and the schema stays at version 1. The README, help text and TODO.md describe this.
This issue closes when next reaches main through #50.
Model: opus-5-5
Landed on `next` as `c737490` through https://git.eeqj.de/sneak/sfdupes/pulls/65.
A file of 10 MiB or more now gets only its 64 KiB `head` and `tail` in the hash phase. A new content phase after the update phase computes the content hash (the whole file under 50 MiB, gigabyte-spaced 1 MiB samples at 50 MiB and over) only for files whose size, `head` and `tail` match another record's, from this scan or stored by an earlier one; a stored file that gains a match gets its content hash then. Every member of such a group is checked with `lstat` first, and a file that is gone or has changed does not count as a match. Outside the scanned operands such a record stays as it is until a scan of its own tree handles it. Files under 10 MiB are still hashed in full, `report` and `trees` still group by the full signature and leave out records without a content hash, and the schema stays at version 1. The README, help text and `TODO.md` describe this.
This issue closes when `next` reaches `main` through https://git.eeqj.de/sneak/sfdupes/pulls/50.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
if the file size is identical and the 65k start and 65k end match, and the file is under 50MiB, we hash the whole file and compare those. if the file is over 50MiB we take 1MiB at the top of each gigabyte and hash that and compare it. any mismatch means it's not a dupe.
this way we only invest disk i/o on potential duplicates, and spend a bit more time checking to see if big files are dupes. obviously it's not foolproof but it's better.
Work order accepted; dispatched 2026-09-22 as a scope-limited direct unit (the repo stays otherwise paused). Implementation instructions for the worker:
Target comparison ladder for same-size candidates, replacing the current 1k start-and-end sampling:
Requirements: named constants, no magic numbers; README documents the ladder including the probabilistic large-file path; unit tests cover each rung and the size boundaries (window size, just under and over 50 MiB, and a multi-gigabyte case if cheap — sparse temp files are fine, no committed fixtures); existing database/caching behaviour preserved unless the ladder requires a schema note, which the PR must call out. Branch from
next, PR tonext, title ends with (closes #61),make checkgreen in Docker before push.(model: claude-fable-5)
obviously files
<128kwe just hash in full. in fact, hash any files<10MiBin full.Implemented as #62 (branch
issue-61-65k-sampling-ladder, basenext).The duplicate signature is now size + SHA-256 of the first and last 64 KiB + a content hash: the whole file below 50 MiB, gigabyte-spaced 1 MiB samples at or above.
reportandtreesgroup by the full signature, so the ladder applies across separate scans.Two design notes for the reviewer: the content hash is computed for every shared-size file (not gated on a head/tail match within one run), because grouping compares stored signatures across scans and size is the only cross-run-safe read gate; and the schema bumps to version 2, so a version 1 database is rejected and must be rescanned (every stored hash changed anyway). Files at most 64 KiB have head and tail equal to the whole-file hash.
make checkis green (tests, Docker lint, fmt-check).Model: opus-4-8
this is a greenfield project, a prototype, pre 1.0. there is no installed base. no one has ever run it and there are no databases anywhere. please do not change the schema version from 1.
Plan for the owner's 2026-09-23 instruction (the schema version stays 1). One PR to
nextthat closes this issue again:db.go:schemaVersiongoes back to 1. Thecontentcolumn and the head/tail/content signature from #62 stay, as part of the version 1 schema. The comment onschemaVersionloses the version 2, rejection and rescan wording. The existing check that stops on a database with any other version number was there before that PR and stays.README.mdschema section: says "currently 1"; the text about version 2 replacing version 1 and version 1 databases being rejected is removed.TODO.md: the Completed Steps entry for this issue is corrected in place: no schema bump, no rejection, no rescan requirement.main.gopackage comment: it still describes the old 1024-byte end windows, so it is updated to match the checks the code actually runs. Judgement call: it is this issue's leftover, and it is text that does not match the code.schemaVersion.Done when an independent reviewer passes the PR,
make checkis green, the PR is squash-merged tonext, and the body of #50 no longer mentions a schema bump.Model: opus-5-5
Landed on
nextas09a39ddthrough #63. The schema version stays 1; thecontentcolumn and the head/tail/content signature are part of the version 1 schema, changed in place. The version 2 bump, the rejection of version 1 databases, and the code comments and document text written for them are gone; nothing is kept for old data. Themain.gopackage comment now describes the hashing the code does.This issue closes when
nextreachesmainthrough #50.Model: opus-5-5
The reviewer of #63 found that the ladder still does not do what this issue orders. Every same-size file of 10 MiB or more gets its content hash, whether or not its first and last 64 KiB match another file's. This issue says to hash a file only when its size and both 64 KiB windows match, so that disk reads go only to likely duplicates.
The PR 62 worker disclosed the gap in #61 (comment). Its reason was that results are compared across separate scans. But a stored signature can gain a match in a later scan, and the content hash can be computed then. So this issue's own words settle it, and the fix goes in under this issue:
One PR to
next, independently reviewed. It lands as one more commit in #50.Model: opus-5-5
Plan for the gated content hash, per #61 (comment). One PR to
next, title ending(closes #61).Behaviour:
head,tailandcontent, unchanged. At 10 MiB or more only the 64 KiBheadandtailare read;contentstays empty.contentwhose size,headandtailequal another record's. Records from this scan count, and so do records stored by earlier scans, inside or outside the scanned operands. Each such file is checked withlstatfirst. If it is gone, is no longer a regular file, or its size or mtime no longer match its record (the walk's own change rule), it is left as it is and is not a duplicate. Inside the operands, the walk and update phases have already dealt with stale records. Outside them there is no existing handling, so a stale record simply keeps its emptycontentuntil its own tree is scanned again. The PR body says so. A group whose other members are all stale gets no content reads. The rest are read with the existing worker pool: in inode order, one read per inode for hard links, with their own progress display. The content hash is then stored in each record.content, and a later scan tries again.reportandtreeskeep grouping by the full signature. A record with an emptycontentis never part of a group; today only an emptyheadis excluded.headandtail, so no change is expected. If one is needed, it is made in place, with no bump, no migration and no compatibility code.Text: the README (description, rationale, design goal 2, the database section's empty-hash sentence, detection step 3, the scan section's phases and its rule about records outside the operands, the progress phases). The scan section must say that a file without hashes has an empty
contenttoo. The non-goal "No full-content verification" gets corrected. The help text nearmain.goline 103 names the content hash. Themain.gopackage comment and the issue-61TODO.mdentry get corrected in place.Tests, using sparse temp files: a same-size pair of 10 MiB or more whose
headortaildiffers never gets a content hash; a later scan fills in a stored file's missing content hash, both across separate operands and within one; a stored partner that has changed or vanished gets no content and makes no duplicate; a real content value is saved and loaded again.Model: opus-5-5
Landed on
nextasc737490through #65.A file of 10 MiB or more now gets only its 64 KiB
headandtailin the hash phase. A new content phase after the update phase computes the content hash (the whole file under 50 MiB, gigabyte-spaced 1 MiB samples at 50 MiB and over) only for files whose size,headandtailmatch another record's, from this scan or stored by an earlier one; a stored file that gains a match gets its content hash then. Every member of such a group is checked withlstatfirst, and a file that is gone or has changed does not count as a match. Outside the scanned operands such a record stays as it is until a scan of its own tree handles it. Files under 10 MiB are still hashed in full,reportandtreesstill group by the full signature and leave out records without a content hash, and the schema stays at version 1. The README, help text andTODO.mddescribe this.This issue closes when
nextreachesmainthrough #50.Model: opus-5-5