Compute the content hash only when head and tail match (closes #61)

A file of 10 MiB or more now gets only its head and tail in the hash
phase. A new content phase, after the update phase, finds every record
of that size without a content hash whose size, head and tail match
another record's, anywhere in the database, checks each file with
lstat, and reads a group only while at least two members remain. It
reuses the hash worker pool, now given its hash function. report and
trees leave out records without a content hash. The README, help text
and TODO entry describe the gate; the schema stays at version 1.

Model: opus-5-5

scan.go

@@ -21,7 +21,10 @@ import (
 // detection"). A same-size candidate below headTailMin is hashed in
 // full and compared directly; a larger one is separated first by the
 // hashes of its end windows, then by a content hash that is exact below
-// wholeFileMax and deliberately sampled at or above it.
+// wholeFileMax and deliberately sampled at or above it. The hash phase
+// reads only the end windows of a larger file; the content phase reads
+// it for its content hash only once its size, head, and tail match
+// another file's.
 
 // headTailMin is the size threshold for the end-window gate. A file
 // smaller than this is hashed in full directly, with no separate head
@@ -74,16 +77,17 @@ type fileMeta struct {
 	hashed bool
 }
 
-// runScan implements the scan subcommand: three sequential phases —
-// walk (which stats each file as it is discovered), hash, update —
-// that synchronize the persistent database with the filesystem state
-// under the PATH operands. Only files whose size at least one other
-// file shares are ever hashed: a size-unique file cannot be a
-// duplicate. Flag parsing and the at-least-one-operand check are done
-// by cobra. Errors are returned rather than exiting, so that the
-// deferred close — which checkpoints the SQLite WAL — always runs.
-// Cancelling ctx unwinds the worker pools and aborts the scan with the
-// context's error.
+// runScan implements the scan subcommand: four sequential phases —
+// walk (which stats each file as it is discovered), hash, update,
+// content — that synchronize the persistent database with the
+// filesystem state under the PATH operands. Only files whose size at
+// least one other file shares are ever hashed: a size-unique file
+// cannot be a duplicate. A file of headTailMin or more gets its content
+// hash only when its size, head, and tail match another file's. Flag
+// parsing and the at-least-one-operand check are done by cobra. Errors
+// are returned rather than exiting, so that the deferred close — which
+// checkpoints the SQLite WAL — always runs. Cancelling ctx unwinds the
+// worker pools and aborts the scan with the context's error.
 func runScan(ctx context.Context, roots []string, workers int,
 	oneFS bool,
 ) error {
@@ -196,13 +200,16 @@ type scanState struct {
 }
 
 // syncScan synchronizes the database with the filesystem under roots
-// in three sequential phases: walk (enumerate and stat every file,
+// in four sequential phases: walk (enumerate and stat every file,
 // building a complete size census), hash (read only the new or
 // changed — or previously unhashed — files whose size at least one
 // other file shares, committing results in batches as they arrive),
-// and update (record the size-unique files without reading them, and
-// delete the records the scan no longer verifies). Records outside
-// the roots are never touched.
+// update (record the size-unique files without reading them, and
+// delete the records the scan no longer verifies), and content (fill
+// in the content hash of every record of headTailMin or more whose
+// size, head, and tail match another record's). Records outside the
+// roots are never touched, except that the content phase fills in
+// their content hash.
 func syncScan(ctx context.Context, db *sql.DB, roots []string,
 	workers int, oneFS bool,
 ) (scanStats, error) {
@@ -237,7 +244,12 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
 		return s.st, err
 	}
 
-	return s.st, s.updatePhase(ctx)
+	err = s.updatePhase(ctx)
+	if err != nil {
+		return s.st, err
+	}
+
+	return s.st, s.contentPhase(ctx, workers)
 }
 
 // loadIndex indexes the database records under the scan roots for
@@ -271,7 +283,8 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 
 // walkPhase drains the walk, appending every walked file's size to
 // the census and resolving what it can immediately: an unchanged file
-// whose record already has hashes needs nothing further. It returns
+// whose record already has hashes needs nothing from the hash phase
+// (the content phase may still fill in its content hash). It returns
 // the new-or-changed files and the unchanged files whose records lack
 // hashes; both remain candidates until the census decides whether
 // their sizes are shared.
@@ -410,27 +423,40 @@ func sameInode(a, b fileRec) bool {
 	return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
 }
 
-// hashPhase hashes every queued file with the worker pool — one read
-// per inode run, in inode order — committing completed records to the
-// database in batches as results arrive, so a long scan persists its
-// progress as it goes (an interrupted scan resumes cheaply: the next
-// run skips everything already recorded). The total counts actual
-// reads, so the bar shows a real ETA. A run that fails to hash is
-// warned about and skipped; stale records for its paths, if any, are
-// deleted by the update phase.
+// hashPhase hashes every queued file with hashSignature — the head and
+// tail of a file of headTailMin or more, the whole file below that —
+// committing completed records to the database in batches as results
+// arrive, so a long scan persists its progress as it goes (an
+// interrupted scan resumes cheaply: the next run skips everything
+// already recorded). A run that fails to hash is warned about and
+// skipped; stale records for its paths, if any, are deleted by the
+// update phase.
+func (s *scanState) hashPhase(ctx context.Context, workers int) error {
+	runs := hashRuns(s.toHash)
+	s.toHash = nil
+
+	return s.readRuns(ctx, workers, "hash", runs, hashSignature, s.recordRun)
+}
+
+// readRuns reads runs with the worker pool, one read per inode run, in
+// the order given, under a progress display named label. The workers
+// compute each run's hashes with hash, and each result goes to record;
+// a run that fails to read is warned about and counted as skipped
+// instead. The total counts actual reads, so the bar shows a real ETA.
 //
 // Returning early — a failed database write, or a cancelled scan — must
 // not strand the pool: the feeder would park forever on a full jobs
 // channel and every worker on a full results channel. The deferred stop
 // is what prevents that.
-func (s *scanState) hashPhase(ctx context.Context, workers int) error {
-	runs := hashRuns(s.toHash)
-	s.toHash = nil
-
-	pool := startHashPool(ctx, runs, workers)
+func (s *scanState) readRuns(ctx context.Context, workers int,
+	label string, runs [][]fileRec,
+	hash func(path string, size int64) (string, string, string, error),
+	record func(ctx context.Context, r hashResult) error,
+) error {
+	pool := startHashPool(ctx, runs, workers, hash)
 	defer pool.stop()
 
-	prog := newProgress("hash", int64(len(runs)))
+	prog := newProgress(label, int64(len(runs)))
 	defer prog.finish()
 
 	for range runs {
@@ -447,12 +473,12 @@ func (s *scanState) hashPhase(ctx context.Context, workers int) error {
 		if r.err != nil {
 			s.st.skipped += len(r.run)
 
-			prog.warnf("hash %s: %v", r.run[0].path, r.err)
+			prog.warnf("%s %s: %v", label, r.run[0].path, r.err)
 
 			continue
 		}
 
-		err := s.recordRun(ctx, r)
+		err := record(ctx, r)
 		if err != nil {
 			return err
 		}
@@ -478,6 +504,12 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
 		})
 	}
 
+	return s.commitFullBatch(ctx)
+}
+
+// commitFullBatch commits the running batch once it holds
+// updateBatchSize records.
+func (s *scanState) commitFullBatch(ctx context.Context) error {
 	if len(s.batch) < updateBatchSize {
 		return nil
 	}
@@ -535,6 +567,112 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 	return applyChanges(ctx, s.db, nil, deletes, prog)
 }
 
+// contentPhase fills in the content hash of every record of headTailMin
+// or more that lacks one and whose size, head, and tail equal another
+// record's, anywhere in the database: records from this scan and
+// records stored by earlier scans, inside or outside the roots. Only
+// such a file can still be a duplicate, so no other file of headTailMin
+// or more is read beyond its end windows. The files are read with the
+// hash phase's worker pool and their records written back in batches. A
+// failed read is warned about and counted as skipped; the record keeps
+// its empty content, so it is never grouped, and a later scan tries
+// again.
+func (s *scanState) contentPhase(ctx context.Context, workers int) error {
+	toRead, recs, err := contentCandidates(ctx, s.db)
+	if err != nil {
+		return err
+	}
+
+	err = s.readRuns(ctx, workers, "content", hashRuns(toRead),
+		hashContentOnly, func(ctx context.Context, r hashResult) error {
+			// Every path in the run keeps its record's head and tail
+			// and gains the one content hash read for the run.
+			for _, f := range r.run {
+				rec := recs[f.path]
+				rec.content = r.content
+				s.batch = append(s.batch, rec)
+			}
+
+			return s.commitFullBatch(ctx)
+		})
+	if err != nil {
+		return err
+	}
+
+	return applyChanges(ctx, s.db, s.batch, nil, nil)
+}
+
+// contentCandidates returns the files the content phase reads, and the
+// records of the files that passed the check, by path. Every record
+// contentCandidatesSQL returns has its file checked with lstat: a file
+// that is gone, is no longer a regular file, or has changed by the
+// walk's rule keeps its record as it is and is not a duplicate. The
+// files of a group that pass are read only if the group still has at
+// least minGroupSize members, counting its records that already have a
+// content hash, so a group whose other members are all stale costs no
+// reads.
+func contentCandidates(ctx context.Context,
+	db *sql.DB,
+) ([]fileRec, map[string]scanRec, error) {
+	var (
+		toRead []fileRec
+		passed []fileRec // the current group's files that passed the check
+		first  scanRec   // the current group's first record
+		hashed int       // the current group's records with a content hash
+	)
+
+	recs := make(map[string]scanRec)
+
+	// endGroup queues the current group's files that passed the check,
+	// if the group still has at least minGroupSize members.
+	endGroup := func() {
+		if len(passed)+hashed >= minGroupSize {
+			toRead = append(toRead, passed...)
+		}
+
+		passed = nil
+	}
+
+	err := loadContentCandidates(ctx, db, func(r scanRec, groupHashed int) {
+		if r.size != first.size || r.head != first.head || r.tail != first.tail {
+			endGroup()
+
+			first, hashed = r, groupHashed
+		}
+
+		f, ok := unchangedFile(r)
+		if ok {
+			passed = append(passed, f)
+			recs[r.path] = r
+		}
+	})
+	if err != nil {
+		return nil, nil, err
+	}
+
+	endGroup()
+
+	return toRead, recs, nil
+}
+
+// unchangedFile lstats the file r names and returns it for reading if
+// it is still the regular file r records: the same size, and an mtime
+// no newer than recorded (the walk's change rule). Otherwise it
+// reports false.
+func unchangedFile(r scanRec) (fileRec, bool) {
+	fi, err := os.Lstat(r.path)
+	if err != nil || !fi.Mode().IsRegular() || fi.Size() != r.size ||
+		fi.ModTime().Unix() > r.mtime {
+		return fileRec{}, false
+	}
+
+	dev, ino := inodeOfInfo(fi)
+
+	return fileRec{
+		path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
+	}, true
+}
+
 // underAnyRoot reports whether path is any of the roots or lies under
 // one of them.
 func underAnyRoot(path string, roots []string) bool {
@@ -865,9 +1003,10 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
 	return statDev(st), st.Ino
 }
 
-// hashResult carries one inode run's signature hashes — head, tail, and
-// content — (or the error that prevented hashing it) from the hash
-// workers to the hash phase.
+// hashResult carries the hashes computed for one inode run (or the
+// error that prevented computing them) from the pool's workers to the
+// phase that started the pool: head, tail, and content from
+// hashSignature, content alone from hashContentOnly.
 type hashResult struct {
 	run     []fileRec
 	head    string
@@ -889,10 +1028,10 @@ type hashPool struct {
 }
 
 // startHashPool starts the feeder and the workers over runs. Workers
-// hash each run's first path (all paths in a run are hard links to the
-// same inode) and write one result per run.
-func startHashPool(ctx context.Context, runs [][]fileRec,
-	workers int,
+// hash each run's first path with hash (all paths in a run are hard
+// links to the same inode) and write one result per run.
+func startHashPool(ctx context.Context, runs [][]fileRec, workers int,
+	hash func(path string, size int64) (string, string, string, error),
 ) *hashPool {
 	ctx, cancel := context.WithCancel(ctx)
 
@@ -904,7 +1043,7 @@ func startHashPool(ctx context.Context, runs [][]fileRec,
 	wg.Go(func() { feedHashJobs(ctx, runs, jobs) })
 
 	for range workers {
-		wg.Go(func() { hashWorker(ctx, jobs, results) })
+		wg.Go(func() { hashWorker(ctx, jobs, results, hash) })
 	}
 
 	done := make(chan struct{})
@@ -950,20 +1089,21 @@ func feedHashJobs(ctx context.Context, runs [][]fileRec,
 	}
 }
 
-// hashWorker hashes one inode run at a time until jobs is closed or the
-// scan is cancelled. A cancelled worker drops the runs still queued
-// instead of stopping its reads of jobs: the range must run out for the
-// pool to tear down, and reading a file nobody wants the hash of only
-// delays that.
+// hashWorker hashes one inode run at a time with hash until jobs is
+// closed or the scan is cancelled. A cancelled worker drops the runs
+// still queued instead of stopping its reads of jobs: the range must
+// run out for the pool to tear down, and reading a file nobody wants
+// the hash of only delays that.
 func hashWorker(ctx context.Context, jobs <-chan []fileRec,
 	results chan<- hashResult,
+	hash func(path string, size int64) (string, string, string, error),
 ) {
 	for run := range jobs {
 		if ctx.Err() != nil {
 			continue
 		}
 
-		head, tail, content, err := hashSignature(run[0].path, run[0].size)
+		head, tail, content, err := hash(run[0].path, run[0].size)
 
 		select {
 		case results <- hashResult{
@@ -980,18 +1120,18 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
 const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" +
 	"27ae41e4649b934ca495991b7852b855"
 
-// hashSignature computes the three content hashes that, with the file
-// size, form its duplicate signature. A file below headTailMin is
-// hashed in full and its whole-file SHA-256 is returned as head, tail,
-// and content alike — that range takes no separate end-window step. For
-// a file at or above headTailMin the head and tail are the SHA-256 of
-// its first and last headTailWindow bytes, and content is the SHA-256
-// of the whole file below wholeFileMax (the exact rung) or of
-// gigabyte-spaced samples at or above it (the sampled, deliberately
-// probabilistic rung). Two files are duplicates only when all four
-// agree; any mismatch means not a duplicate. size is the value recorded
-// when the file was statted; a zero-length file has constant hashes and
-// is never opened.
+// hashSignature computes the hashes the hash phase records for a file
+// whose size is shared; with the file size they form its duplicate
+// signature. A file below headTailMin is hashed in full and its
+// whole-file SHA-256 is returned as head, tail, and content alike —
+// that range takes no separate end-window step. For a file at or above
+// headTailMin only the head and tail are computed, the SHA-256 of its
+// first and last headTailWindow bytes, and content is returned empty:
+// the content phase computes it with hashContentOnly once the file's
+// size, head, and tail match another file's. Two files are duplicates
+// only when all four agree; any mismatch means not a duplicate. size
+// is the value recorded when the file was statted; a zero-length file
+// has constant hashes and is never opened.
 func hashSignature(path string, size int64) (string, string, string, error) {
 	if size == 0 {
 		return emptyHash, emptyHash, emptyHash, nil
@@ -1021,12 +1161,25 @@ func hashSignature(path string, size int64) (string, string, string, error) {
 		return "", "", "", err
 	}
 
-	content, err := hashContent(f, size)
+	return head, tail, "", nil
+}
+
+// hashContentOnly returns the content hash of the file at path, which
+// is at least headTailMin bytes: the content phase's read. head and
+// tail are returned empty, because the content phase keeps the ones its
+// records already hold.
+func hashContentOnly(path string, size int64) (string, string, string, error) {
+	//nolint:gosec // hashing operator-supplied paths is the tool's purpose
+	f, err := os.Open(path)
 	if err != nil {
 		return "", "", "", err
 	}
 
-	return head, tail, content, nil
+	defer func() { _ = f.Close() }()
+
+	content, err := hashContent(f, size)
+
+	return "", "", content, err
 }
 
 // hashEnds returns the SHA-256 of the first and last headTailWindow