Compute the content hash only when head and tail match (closes #61)
check / check (push) Successful in 1m9s

A file of 10 MiB or more now gets only its head and tail in the hash
phase. A new content phase, after the update phase, finds every record
of that size without a content hash whose size, head and tail match
another record's, anywhere in the database, checks each file with
lstat, and reads a group only while at least two members remain. It
reuses the hash worker pool, now given its hash function. report and
trees leave out records without a content hash. The README, help text
and TODO entry describe the gate; the schema stays at version 1.

Model: opus-5-5
This commit is contained in:
2026-09-23 12:18:39 +00:00
parent 09a39ddf37
commit 89fc9e4595
12 changed files with 763 additions and 221 deletions
+5 -4
View File
@@ -117,10 +117,11 @@ func collectDupeGroups(recs []scanRec) []dupeGroup {
groups := make(map[fileSig][]string)
for _, r := range recs {
// A record without hashes (its size was unique when last
// scanned) has unknown content and is never reported as a
// duplicate.
if r.head == "" {
// A record without a content hash has unknown content and is
// never reported as a duplicate: its size was unique when last
// scanned, or it is headTailMin or more and has not yet matched
// another record on size, head, and tail.
if r.content == "" {
continue
}