Compute the content hash only when head and tail match (closes #61)
check / check (push) Successful in 1m9s
check / check (push) Successful in 1m9s
A file of 10 MiB or more now gets only its head and tail in the hash phase. A new content phase, after the update phase, finds every record of that size without a content hash whose size, head and tail match another record's, anywhere in the database, checks each file with lstat, and reads a group only while at least two members remain. It reuses the hash worker pool, now given its hash function. report and trees leave out records without a content hash. The README, help text and TODO entry describe the gate; the schema stays at version 1. Model: opus-5-5
This commit is contained in:
@@ -117,10 +117,11 @@ func collectDupeGroups(recs []scanRec) []dupeGroup {
|
||||
groups := make(map[fileSig][]string)
|
||||
|
||||
for _, r := range recs {
|
||||
// A record without hashes (its size was unique when last
|
||||
// scanned) has unknown content and is never reported as a
|
||||
// duplicate.
|
||||
if r.head == "" {
|
||||
// A record without a content hash has unknown content and is
|
||||
// never reported as a duplicate: its size was unique when last
|
||||
// scanned, or it is headTailMin or more and has not yet matched
|
||||
// another record on size, head, and tail.
|
||||
if r.content == "" {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user