Compute the content hash only when head and tail match (closes #61)

Conversation 5 Commits 3 Files Changed 12

+930 -220

3 Commits

Author	SHA1	Message	Date
clawbot	3fd8ea8990	Say what the content phase's check does to a stale record check / check (push) Successful in 53s Details The README content phase item and the contentCandidates comment said a file failing the check is not a duplicate. It keeps its record and does not count as a match; with a content hash it is still reported until a scan covering its tree updates or removes it. Model: opus-5-5	2026-09-23 13:51:07 +00:00
clawbot	fff4409a24	Check every group member and show the content phase from its start check / check (push) Successful in 57s Details The content phase now checks every record sharing a size, head and tail with lstat, including those that already have a content hash, and reads those without one only when at least two pass. Only a missing file is passed over silently; any other lstat error is warned about and counted as skipped. The phase shows a running count while it queries and checks, then its bar. The README states once when content is empty. New tests cover these and hard links. Model: opus-5-5	2026-09-23 13:09:03 +00:00
clawbot	89fc9e4595	Compute the content hash only when head and tail match (closes #61 ) check / check (push) Successful in 1m9s Details A file of 10 MiB or more now gets only its head and tail in the hash phase. A new content phase, after the update phase, finds every record of that size without a content hash whose size, head and tail match another record's, anywhere in the database, checks each file with lstat, and reads a group only while at least two members remain. It reuses the hash worker pool, now given its hash function. report and trees leave out records without a content hash. The README, help text and TODO entry describe the gate; the schema stays at version 1. Model: opus-5-5	2026-09-23 12:18:39 +00:00

Author

SHA1

Message

Date

clawbot

3fd8ea8990

Say what the content phase's check does to a stale record

check / check (push) Successful in 53s

Details

The README content phase item and the contentCandidates comment said a
file failing the check is not a duplicate. It keeps its record and does
not count as a match; with a content hash it is still reported until a
scan covering its tree updates or removes it.

Model: opus-5-5

2026-09-23 13:51:07 +00:00

clawbot

fff4409a24

Check every group member and show the content phase from its start

check / check (push) Successful in 57s

Details

The content phase now checks every record sharing a size, head and
tail with lstat, including those that already have a content hash, and
reads those without one only when at least two pass. Only a missing
file is passed over silently; any other lstat error is warned about and
counted as skipped. The phase shows a running count while it queries
and checks, then its bar. The README states once when content is empty.
New tests cover these and hard links.

Model: opus-5-5

2026-09-23 13:09:03 +00:00

clawbot

89fc9e4595

Compute the content hash only when head and tail match (closes #61 )

check / check (push) Successful in 1m9s

Details

A file of 10 MiB or more now gets only its head and tail in the hash
phase. A new content phase, after the update phase, finds every record
of that size without a content hash whose size, head and tail match
another record's, anywhere in the database, checks each file with
lstat, and reads a group only while at least two members remain. It
reuses the hash worker pool, now given its hash function. report and
trees leave out records without a content hash. The README, help text
and TODO entry describe the gate; the schema stays at version 1.

Model: opus-5-5

2026-09-23 12:18:39 +00:00

Compute the content hash only when head and tail match (closes #61) #65