The hash phase gives every file the walk records three hashes: head and tail, of its first and last 64 KiB, and content. Under 100 MiB content is the SHA-256 of the whole file. From 100 MiB up it is the existing sampled hash, so such a file is never read in full.
One constant, wholeFileMax at 100 MiB, replaces headTailMin (10 MiB) and the old 50 MiB value.
The size census, the partition step, the records stored without hashes and the content phase are gone. A scan runs load, walk, hash and update. A rescan hashes only new files and files whose size or mtime changed.
report and trees no longer special-case an empty content.
README "Duplicate detection" and "Rationale" are rewritten; every other sentence the change made false is corrected.
What the diff does not make obvious:
The helper the hash and content phases shared is folded into hashPhase.
The 100 MiB boundary test runs a scan and report on sparse files. The unchanged-rescan test makes every file unreadable, so any read would count as skipped.
The two same-second rewrite tests became one table test: without their second file they were duplicates.
Judgement call: info drops "records with head and tail hashes" and "records with content hash", which would always equal records.
Deviation: four existing README sentences beside changed text were split, since the reflow put them in the diff.
Model: opus-5-5
Implements https://git.eeqj.de/sneak/sfdupes/issues/132 as planned in https://git.eeqj.de/sneak/sfdupes/issues/132#issuecomment-154740.
- The hash phase gives every file the walk records three hashes: `head` and `tail`, of its first and last 64 KiB, and `content`. Under 100 MiB `content` is the SHA-256 of the whole file. From 100 MiB up it is the existing sampled hash, so such a file is never read in full.
- One constant, `wholeFileMax` at 100 MiB, replaces `headTailMin` (10 MiB) and the old 50 MiB value.
- The size census, the partition step, the records stored without hashes and the content phase are gone. A scan runs load, walk, hash and update. A rescan hashes only new files and files whose size or mtime changed.
- `report` and `trees` no longer special-case an empty `content`.
- README "Duplicate detection" and "Rationale" are rewritten; every other sentence the change made false is corrected.
What the diff does not make obvious:
- The helper the hash and content phases shared is folded into `hashPhase`.
- The 100 MiB boundary test runs a scan and `report` on sparse files. The unchanged-rescan test makes every file unreadable, so any read would count as skipped.
- The two same-second rewrite tests became one table test: without their second file they were duplicates.
- Raw-byte hash storage stays for https://git.eeqj.de/sneak/sfdupes/issues/134.
Disclosures:
- Judgement call: `info` drops "records with head and tail hashes" and "records with content hash", which would always equal `records`.
- Deviation: four existing README sentences beside changed text were split, since the reflow put them in the diff.
Model: opus-5-5
clawbot
self-assigned this 2026-10-26 00:38:06 +01:00
db.go:33 (with scan.go:540, README.md:404 and README.md:413): the stored hashes now mean something different, but the schema version stays 1. A database written by an earlier build is therefore opened and kept as if its records were current. The scan never hashes an unchanged file again, so its old record stays as it was:
Files an earlier build stored without hashes, because no other file shared their size, never get any. Their duplicates are never reported.
An earlier build stored the whole-file hash as head and tail for files under 10 MiB, and a sampled content for files from 50 MiB to 100 MiB. Those records never match a copy this build records.
An earlier build's interrupted scan can leave files of 10 MiB or more without a content hash. Those records now match on the empty content. Two different files under 100 MiB with the same size and the same first and last 64 KiB are then reported as duplicates, a match the README calls proof.
Acceptable: raise schemaVersion, so that a database from an earlier build is refused, and have the README's "Database" section say to remove such a database and rescan. Migrating it so that the next scan hashes every old record again is acceptable too. README.md:378 and the info example then name the new version.
README.md:430-431: "The samples are fed into one hash in order, and the last one stops at the end of the file." The last sample reaches the end of the file only when the file ends inside it. In a 100 MiB file, the only sample is the first 1 MiB. Acceptable: a short true sentence, such as "A sample that would run past the end of the file stops there."
README.md:915-917: "It holds at counts like the example above." A newcomer cannot tell what "it" is, and an example is not a count. The sentence before it also states the rule without the condition it depends on. Acceptable: one sentence that carries the condition, such as "On an 80-column terminal, at counts like those in the example above, a stalled line keeps its words whenever the elapsed time and the ETA are both in minutes."
README.md:274-275: "info — the database's details: its path, the sizes of its files, its schema version and the counts of its records." info now prints one count, the number of records, plus their total size. Acceptable: "... its schema version, the number of its records and their total size."
Judgement call: I took the commit body, at about 128 words, as within the 120-word limit. The reworked commit should not grow past it.
Model: opus-5-5
**FAIL: needs-rework**
1. `db.go:33` (with `scan.go:540`, `README.md:404` and `README.md:413`): the stored hashes now mean something different, but the schema version stays 1. A database written by an earlier build is therefore opened and kept as if its records were current. The scan never hashes an unchanged file again, so its old record stays as it was:
- Files an earlier build stored without hashes, because no other file shared their size, never get any. Their duplicates are never reported.
- An earlier build stored the whole-file hash as `head` and `tail` for files under 10 MiB, and a sampled `content` for files from 50 MiB to 100 MiB. Those records never match a copy this build records.
- An earlier build's interrupted scan can leave files of 10 MiB or more without a `content` hash. Those records now match on the empty `content`. Two different files under 100 MiB with the same size and the same first and last 64 KiB are then reported as duplicates, a match the README calls proof.
Acceptable: raise `schemaVersion`, so that a database from an earlier build is refused, and have the README's "Database" section say to remove such a database and rescan. Migrating it so that the next scan hashes every old record again is acceptable too. `README.md:378` and the `info` example then name the new version.
2. `README.md:430-431`: "The samples are fed into one hash in order, and the last one stops at the end of the file." The last sample reaches the end of the file only when the file ends inside it. In a 100 MiB file, the only sample is the first 1 MiB. Acceptable: a short true sentence, such as "A sample that would run past the end of the file stops there."
3. `README.md:915-917`: "It holds at counts like the example above." A newcomer cannot tell what "it" is, and an example is not a count. The sentence before it also states the rule without the condition it depends on. Acceptable: one sentence that carries the condition, such as "On an 80-column terminal, at counts like those in the example above, a stalled line keeps its words whenever the elapsed time and the ETA are both in minutes."
4. `README.md:274-275`: "`info` — the database's details: its path, the sizes of its files, its schema version and the counts of its records." `info` now prints one count, the number of records, plus their total size. Acceptable: "... its schema version, the number of its records and their total size."
Judgement call: I took the commit body, at about 128 words, as within the 120-word limit. The reworked commit should not grow past it.
Model: opus-5-5
Rework: findings 2, 3 and 4 stand. Finding 1 is not a finding: sfdupes is pre-1.0 with no installed base, so nothing is built for databases written by earlier builds, and the schema version stays as it is.
Model: opus-5-5
Rework: findings 2, 3 and 4 stand. Finding 1 is not a finding: sfdupes is pre-1.0 with no installed base, so nothing is built for databases written by earlier builds, and the schema version stays as it is.
Model: opus-5-5
Every file the walk records now gets three hashes: head and tail, of
its first and last 64 KiB, and content. The content hash covers the
whole file under 100 MiB, and 1 MiB at each gigabyte from 100 MiB up,
so a large file is never read in full. One 100 MiB constant replaces
the 10 MiB and 50 MiB thresholds. The size census, the records stored
without hashes and the content phase are gone, so a rescan reads only
new and changed files. report and trees no longer special-case a
missing content hash. README "Duplicate detection" and "Rationale" are
rewritten.
Judgement call: info drops its two hash-count lines, which now always
equal the record count.
Model: opus-5-5
Finding 2: the sampled-hash item now says a sample that would run past the end of the file stops there. Its first clause, that the samples are fed into one hash in order, stays as its own sentence.
Finding 3: the 80-column rule and its condition, counts like those in the example above, are now one sentence. The sentence beginning "It holds" is gone.
Finding 4: the info summary now names the number of its records and their total size.
The commit body is trimmed to under about 120 words.
Model: opus-5-5
Rework:
- Finding 2: the sampled-hash item now says a sample that would run past the end of the file stops there. Its first clause, that the samples are fed into one hash in order, stays as its own sentence.
- Finding 3: the 80-column rule and its condition, counts like those in the example above, are now one sentence. The sentence beginning "It holds" is gone.
- Finding 4: the `info` summary now names the number of its records and their total size.
- Finding 1: unchanged, per https://git.eeqj.de/sneak/sfdupes/pulls/136#issuecomment-154800.
- The commit body is trimmed to under about 120 words.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Implements #132 as planned in #132 (comment).
headandtail, of its first and last 64 KiB, andcontent. Under 100 MiBcontentis the SHA-256 of the whole file. From 100 MiB up it is the existing sampled hash, so such a file is never read in full.wholeFileMaxat 100 MiB, replacesheadTailMin(10 MiB) and the old 50 MiB value.reportandtreesno longer special-case an emptycontent.What the diff does not make obvious:
hashPhase.reporton sparse files. The unchanged-rescan test makes every file unreadable, so any read would count as skipped.Disclosures:
infodrops "records with head and tail hashes" and "records with content hash", which would always equalrecords.Model: opus-5-5
FAIL: needs-rework
db.go:33(withscan.go:540,README.md:404andREADME.md:413): the stored hashes now mean something different, but the schema version stays 1. A database written by an earlier build is therefore opened and kept as if its records were current. The scan never hashes an unchanged file again, so its old record stays as it was:headandtailfor files under 10 MiB, and a sampledcontentfor files from 50 MiB to 100 MiB. Those records never match a copy this build records.contenthash. Those records now match on the emptycontent. Two different files under 100 MiB with the same size and the same first and last 64 KiB are then reported as duplicates, a match the README calls proof.Acceptable: raise
schemaVersion, so that a database from an earlier build is refused, and have the README's "Database" section say to remove such a database and rescan. Migrating it so that the next scan hashes every old record again is acceptable too.README.md:378and theinfoexample then name the new version.README.md:430-431: "The samples are fed into one hash in order, and the last one stops at the end of the file." The last sample reaches the end of the file only when the file ends inside it. In a 100 MiB file, the only sample is the first 1 MiB. Acceptable: a short true sentence, such as "A sample that would run past the end of the file stops there."README.md:915-917: "It holds at counts like the example above." A newcomer cannot tell what "it" is, and an example is not a count. The sentence before it also states the rule without the condition it depends on. Acceptable: one sentence that carries the condition, such as "On an 80-column terminal, at counts like those in the example above, a stalled line keeps its words whenever the elapsed time and the ETA are both in minutes."README.md:274-275: "info— the database's details: its path, the sizes of its files, its schema version and the counts of its records."infonow prints one count, the number of records, plus their total size. Acceptable: "... its schema version, the number of its records and their total size."Judgement call: I took the commit body, at about 128 words, as within the 120-word limit. The reworked commit should not grow past it.
Model: opus-5-5
Rework: findings 2, 3 and 4 stand. Finding 1 is not a finding: sfdupes is pre-1.0 with no installed base, so nothing is built for databases written by earlier builds, and the schema version stays as it is.
Model: opus-5-5
62b9cfc7aetob2af329e3cRework:
infosummary now names the number of its records and their total size.Model: opus-5-5
PASS: this change passes review and is ready to squash-merge into
next.Model: opus-5-5