Compute the content hash only when head and tail match (closes #61) #65

Merged
clawbot merged 3 commits from issue-61-gate-content-hash into next 2026-09-23 16:06:09 +02:00
2 changed files with 16 additions and 10 deletions
Showing only changes of commit 3fd8ea8990 - Show all commits
+8 -5
View File
@@ -339,11 +339,14 @@ during the hash phase:
`content` hash, has its file checked with `lstat` first. A file `content` hash, has its file checked with `lstat` first. A file
that is gone, is no longer a regular file, or has changed (a that is gone, is no longer a regular file, or has changed (a
different size, or an mtime newer than recorded) keeps its record different size, or an mtime newer than recorded) keeps its record
as it is and is not a duplicate. Any other `lstat` error is warned as it is and does not count as a match for the others. Any other
about and counted as skipped, with the same result. The files that `lstat` error is warned about and counted as skipped, with the same
pass and have no `content` hash are read only if at least two of result. If such a record has no `content` hash, it stays out of
those records pass, so a file whose only matches are stale costs no duplicate groups; if it has one, it is still reported until a scan
read; a file that already has a `content` hash is never read again. covering its own tree updates or removes it. The files that pass
and have no `content` hash are read only if at least two of those
records pass, so a file whose only matches are stale costs no read;
a file that already has a `content` hash is never read again.
They are read by a worker pool as in the hash phase, in inode order They are read by a worker pool as in the hash phase, in inode order
and once per inode, and their content hashes are committed in and once per inode, and their content hashes are committed in
batches. A failed read is warned about and counted as skipped; its batches. A failed read is warned about and counted as skipped; its
+8 -5
View File
@@ -607,11 +607,14 @@ func (s *scanState) contentPhase(ctx context.Context, workers int) error {
// their records by path. Every record contentCandidatesSQL returns has // their records by path. Every record contentCandidatesSQL returns has
// its file checked with lstat, whether or not it already has a content // its file checked with lstat, whether or not it already has a content
// hash: a file that is gone, is no longer a regular file, or has // hash: a file that is gone, is no longer a regular file, or has
// changed by the walk's rule keeps its record as it is and is not a // changed by the walk's rule keeps its record as it is and does not
// duplicate, and any other lstat error is warned about and counted as // count as a match for the others, and any other lstat error is warned
// skipped. The files of a group that pass and have no content hash are // about and counted as skipped, with the same result. If such a record
// read only if at least minGroupSize of the group's files pass, so a // has no content hash, it stays out of duplicate groups; if it has one,
// group whose other members are all stale costs no reads. Only the // it is still reported until a scan covering its own tree updates or
// removes it. The files of a group that pass and have no content hash
// are read only if at least minGroupSize of the group's files pass, so
// a group whose other members are all stale costs no reads. Only the
// records to be read are kept. // records to be read are kept.
func (s *scanState) contentCandidates( func (s *scanState) contentCandidates(
ctx context.Context, ctx context.Context,