Compute the content hash only when head and tail match (closes #61)

A file of 10 MiB or more now gets only its 64 KiB head and tail in the
hash phase, so its content is read only when it can be a duplicate. A
new content phase after the update phase finds every group of records,
anywhere in the database, that share size, head and tail and include
one without a content hash. It checks every member with lstat and, when
at least two pass, reads those without a content hash through the
existing worker pool; a stale file does not count as a match. report
and trees leave out records without a content hash. The README, help
text and TODO entry describe the gate; the schema stays at version 1.

Lint suppressed: gosec on the file open in hashContentOnly, as in
hashSignature, and on one chmod in a test.

Model: opus-5-5

scan.go

@@ -6,6 +6,7 @@ import (
 	"crypto/sha256"
 	"database/sql"
 	"encoding/hex"
+	"errors"
 	"fmt"
 	"io"
 	"io/fs"
@@ -21,7 +22,10 @@ import (
 // detection"). A same-size candidate below headTailMin is hashed in
 // full and compared directly; a larger one is separated first by the
 // hashes of its end windows, then by a content hash that is exact below
-// wholeFileMax and deliberately sampled at or above it.
+// wholeFileMax and deliberately sampled at or above it. The hash phase
+// reads only the end windows of a larger file; the content phase reads
+// it for its content hash only once its size, head, and tail match
+// another file's.
 
 // headTailMin is the size threshold for the end-window gate. A file
 // smaller than this is hashed in full directly, with no separate head
@@ -74,16 +78,17 @@ type fileMeta struct {
 	hashed bool
 }
 
-// runScan implements the scan subcommand: three sequential phases —
-// walk (which stats each file as it is discovered), hash, update —
-// that synchronize the persistent database with the filesystem state
-// under the PATH operands. Only files whose size at least one other
-// file shares are ever hashed: a size-unique file cannot be a
-// duplicate. Flag parsing and the at-least-one-operand check are done
-// by cobra. Errors are returned rather than exiting, so that the
-// deferred close — which checkpoints the SQLite WAL — always runs.
-// Cancelling ctx unwinds the worker pools and aborts the scan with the
-// context's error.
+// runScan implements the scan subcommand: four sequential phases —
+// walk (which stats each file as it is discovered), hash, update,
+// content — that synchronize the persistent database with the
+// filesystem state under the PATH operands. Only files whose size at
+// least one other file shares are ever hashed: a size-unique file
+// cannot be a duplicate. A file of headTailMin or more gets its content
+// hash only when its size, head, and tail match another file's. Flag
+// parsing and the at-least-one-operand check are done by cobra. Errors
+// are returned rather than exiting, so that the deferred close — which
+// checkpoints the SQLite WAL — always runs. Cancelling ctx unwinds the
+// worker pools and aborts the scan with the context's error.
 func runScan(ctx context.Context, roots []string, workers int,
 	oneFS bool,
 ) error {
@@ -196,13 +201,16 @@ type scanState struct {
 }
 
 // syncScan synchronizes the database with the filesystem under roots
-// in three sequential phases: walk (enumerate and stat every file,
+// in four sequential phases: walk (enumerate and stat every file,
 // building a complete size census), hash (read only the new or
 // changed — or previously unhashed — files whose size at least one
 // other file shares, committing results in batches as they arrive),
-// and update (record the size-unique files without reading them, and
-// delete the records the scan no longer verifies). Records outside
-// the roots are never touched.
+// update (record the size-unique files without reading them, and
+// delete the records the scan no longer verifies), and content (fill
+// in the content hash of every record of headTailMin or more whose
+// size, head, and tail match another record's). Records outside the
+// roots are never touched, except that the content phase fills in
+// their content hash.
 func syncScan(ctx context.Context, db *sql.DB, roots []string,
 	workers int, oneFS bool,
 ) (scanStats, error) {
@@ -237,7 +245,12 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
 		return s.st, err
 	}
 
-	return s.st, s.updatePhase(ctx)
+	err = s.updatePhase(ctx)
+	if err != nil {
+		return s.st, err
+	}
+
+	return s.st, s.contentPhase(ctx, workers)
 }
 
 // loadIndex indexes the database records under the scan roots for
@@ -271,7 +284,8 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 
 // walkPhase drains the walk, appending every walked file's size to
 // the census and resolving what it can immediately: an unchanged file
-// whose record already has hashes needs nothing further. It returns
+// whose record already has hashes needs nothing from the hash phase
+// (the content phase may still fill in its content hash). It returns
 // the new-or-changed files and the unchanged files whose records lack
 // hashes; both remain candidates until the census decides whether
 // their sizes are shared.
@@ -410,27 +424,40 @@ func sameInode(a, b fileRec) bool {
 	return (a.dev != 0 || a.ino != 0) && a.dev == b.dev && a.ino == b.ino
 }
 
-// hashPhase hashes every queued file with the worker pool — one read
-// per inode run, in inode order — committing completed records to the
-// database in batches as results arrive, so a long scan persists its
-// progress as it goes (an interrupted scan resumes cheaply: the next
-// run skips everything already recorded). The total counts actual
-// reads, so the bar shows a real ETA. A run that fails to hash is
-// warned about and skipped; stale records for its paths, if any, are
-// deleted by the update phase.
+// hashPhase hashes every queued file with hashSignature — the head and
+// tail of a file of headTailMin or more, the whole file below that —
+// committing completed records to the database in batches as results
+// arrive, so a long scan persists its progress as it goes (an
+// interrupted scan resumes cheaply: the next run skips everything
+// already recorded). A run that fails to hash is warned about and
+// skipped; stale records for its paths, if any, are deleted by the
+// update phase.
+func (s *scanState) hashPhase(ctx context.Context, workers int) error {
+	runs := hashRuns(s.toHash)
+	s.toHash = nil
+
+	return s.readRuns(ctx, workers, "hash", runs, hashSignature, s.recordRun)
+}
+
+// readRuns reads runs with the worker pool, one read per inode run, in
+// the order given, under a progress display named label. The workers
+// compute each run's hashes with hash, and each result goes to record;
+// a run that fails to read is warned about and counted as skipped
+// instead. The total counts actual reads, so the bar shows a real ETA.
 //
 // Returning early — a failed database write, or a cancelled scan — must
 // not strand the pool: the feeder would park forever on a full jobs
 // channel and every worker on a full results channel. The deferred stop
 // is what prevents that.
-func (s *scanState) hashPhase(ctx context.Context, workers int) error {
-	runs := hashRuns(s.toHash)
-	s.toHash = nil
-
-	pool := startHashPool(ctx, runs, workers)
+func (s *scanState) readRuns(ctx context.Context, workers int,
+	label string, runs [][]fileRec,
+	hash func(path string, size int64) (string, string, string, error),
+	record func(ctx context.Context, r hashResult) error,
+) error {
+	pool := startHashPool(ctx, runs, workers, hash)
 	defer pool.stop()
 
-	prog := newProgress("hash", int64(len(runs)))
+	prog := newProgress(label, int64(len(runs)))
 	defer prog.finish()
 
 	for range runs {
@@ -447,12 +474,12 @@ func (s *scanState) hashPhase(ctx context.Context, workers int) error {
 		if r.err != nil {
 			s.st.skipped += len(r.run)
 
-			prog.warnf("hash %s: %v", r.run[0].path, r.err)
+			prog.warnf("%s %s: %v", label, r.run[0].path, r.err)
 
 			continue
 		}
 
-		err := s.recordRun(ctx, r)
+		err := record(ctx, r)
 		if err != nil {
 			return err
 		}
@@ -478,6 +505,12 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
 		})
 	}
 
+	return s.commitFullBatch(ctx)
+}
+
+// commitFullBatch commits the running batch once it holds
+// updateBatchSize records.
+func (s *scanState) commitFullBatch(ctx context.Context) error {
 	if len(s.batch) < updateBatchSize {
 		return nil
 	}
@@ -535,6 +568,147 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 	return applyChanges(ctx, s.db, nil, deletes, prog)
 }
 
+// contentPhase fills in the content hash of every record of headTailMin
+// or more that lacks one and whose size, head, and tail equal another
+// record's, anywhere in the database: records from this scan and
+// records stored by earlier scans, inside or outside the roots. Only
+// such a file can still be a duplicate, so no other file of headTailMin
+// or more is read beyond its end windows. The files are read with the
+// hash phase's worker pool and their records written back in batches. A
+// failed read is warned about and counted as skipped; the record keeps
+// its empty content, so it is never grouped, and a later scan tries
+// again.
+func (s *scanState) contentPhase(ctx context.Context, workers int) error {
+	toRead, recs, err := s.contentCandidates(ctx)
+	if err != nil {
+		return err
+	}
+
+	err = s.readRuns(ctx, workers, "content", hashRuns(toRead),
+		hashContentOnly, func(ctx context.Context, r hashResult) error {
+			// Every path in the run keeps its record's head and tail
+			// and gains the one content hash read for the run.
+			for _, f := range r.run {
+				rec := recs[f.path]
+				rec.content = r.content
+				s.batch = append(s.batch, rec)
+			}
+
+			return s.commitFullBatch(ctx)
+		})
+	if err != nil {
+		return err
+	}
+
+	return applyChanges(ctx, s.db, s.batch, nil, nil)
+}
+
+// contentCandidates returns the files the content phase reads, and
+// their records by path. Every record contentCandidatesSQL returns has
+// its file checked with lstat, whether or not it already has a content
+// hash: a file that is gone, is no longer a regular file, or has
+// changed by the walk's rule keeps its record as it is and does not
+// count as a match for the others, and any other lstat error is warned
+// about and counted as skipped, with the same result. If such a record
+// has no content hash, it stays out of duplicate groups; if it has one,
+// it is still reported until a scan covering its own tree updates or
+// removes it. The files of a group that pass and have no content hash
+// are read only if at least minGroupSize of the group's files pass, so
+// a group whose other members are all stale costs no reads. Only the
+// records to be read are kept.
+func (s *scanState) contentCandidates(
+	ctx context.Context,
+) ([]fileRec, map[string]scanRec, error) {
+	// The query and the checks take real time on a large database;
+	// without a display the scan looks hung before the reads begin.
+	prog := newProgress("content", -1)
+	defer prog.finish()
+
+	var (
+		toRead []fileRec
+		first  scanRec   // the current group's first record
+		passed int       // the current group's files that passed the check
+		unread []fileRec // those of them without a content hash
+	)
+
+	recs := make(map[string]scanRec)
+
+	// endGroup queues the current group's files to read if at least
+	// minGroupSize of its files passed, and drops their records if not.
+	endGroup := func() {
+		if passed >= minGroupSize {
+			toRead = append(toRead, unread...)
+		} else {
+			for _, f := range unread {
+				delete(recs, f.path)
+			}
+		}
+
+		passed, unread = 0, nil
+	}
+
+	err := loadContentCandidates(ctx, s.db, func(r scanRec, hashed bool) {
+		prog.increment()
+
+		if r.size != first.size || r.head != first.head || r.tail != first.tail {
+			endGroup()
+
+			first = r
+		}
+
+		f, ok, err := unchangedFile(r)
+		if err != nil {
+			s.st.skipped++
+
+			prog.warnf("content %s: %v", r.path, err)
+		}
+
+		if !ok {
+			return
+		}
+
+		passed++
+
+		if !hashed {
+			unread = append(unread, f)
+			recs[r.path] = r
+		}
+	})
+	if err != nil {
+		return nil, nil, err
+	}
+
+	endGroup()
+
+	return toRead, recs, nil
+}
+
+// unchangedFile lstats the file r names and returns it for reading if
+// it is still the regular file r records: the same size, and an mtime
+// no newer than recorded (the walk's change rule). A file that is gone
+// or has changed reports false; any other lstat error is returned.
+func unchangedFile(r scanRec) (fileRec, bool, error) {
+	fi, err := os.Lstat(r.path)
+	if errors.Is(err, fs.ErrNotExist) {
+		return fileRec{}, false, nil
+	}
+
+	if err != nil {
+		return fileRec{}, false, err
+	}
+
+	if !fi.Mode().IsRegular() || fi.Size() != r.size ||
+		fi.ModTime().Unix() > r.mtime {
+		return fileRec{}, false, nil
+	}
+
+	dev, ino := inodeOfInfo(fi)
+
+	return fileRec{
+		path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
+	}, true, nil
+}
+
 // underAnyRoot reports whether path is any of the roots or lies under
 // one of them.
 func underAnyRoot(path string, roots []string) bool {
@@ -865,9 +1039,10 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
 	return statDev(st), st.Ino
 }
 
-// hashResult carries one inode run's signature hashes — head, tail, and
-// content — (or the error that prevented hashing it) from the hash
-// workers to the hash phase.
+// hashResult carries the hashes computed for one inode run (or the
+// error that prevented computing them) from the pool's workers to the
+// phase that started the pool: head, tail, and content from
+// hashSignature, content alone from hashContentOnly.
 type hashResult struct {
 	run     []fileRec
 	head    string
@@ -889,10 +1064,10 @@ type hashPool struct {
 }
 
 // startHashPool starts the feeder and the workers over runs. Workers
-// hash each run's first path (all paths in a run are hard links to the
-// same inode) and write one result per run.
-func startHashPool(ctx context.Context, runs [][]fileRec,
-	workers int,
+// hash each run's first path with hash (all paths in a run are hard
+// links to the same inode) and write one result per run.
+func startHashPool(ctx context.Context, runs [][]fileRec, workers int,
+	hash func(path string, size int64) (string, string, string, error),
 ) *hashPool {
 	ctx, cancel := context.WithCancel(ctx)
 
@@ -904,7 +1079,7 @@ func startHashPool(ctx context.Context, runs [][]fileRec,
 	wg.Go(func() { feedHashJobs(ctx, runs, jobs) })
 
 	for range workers {
-		wg.Go(func() { hashWorker(ctx, jobs, results) })
+		wg.Go(func() { hashWorker(ctx, jobs, results, hash) })
 	}
 
 	done := make(chan struct{})
@@ -950,20 +1125,21 @@ func feedHashJobs(ctx context.Context, runs [][]fileRec,
 	}
 }
 
-// hashWorker hashes one inode run at a time until jobs is closed or the
-// scan is cancelled. A cancelled worker drops the runs still queued
-// instead of stopping its reads of jobs: the range must run out for the
-// pool to tear down, and reading a file nobody wants the hash of only
-// delays that.
+// hashWorker hashes one inode run at a time with hash until jobs is
+// closed or the scan is cancelled. A cancelled worker drops the runs
+// still queued instead of stopping its reads of jobs: the range must
+// run out for the pool to tear down, and reading a file nobody wants
+// the hash of only delays that.
 func hashWorker(ctx context.Context, jobs <-chan []fileRec,
 	results chan<- hashResult,
+	hash func(path string, size int64) (string, string, string, error),
 ) {
 	for run := range jobs {
 		if ctx.Err() != nil {
 			continue
 		}
 
-		head, tail, content, err := hashSignature(run[0].path, run[0].size)
+		head, tail, content, err := hash(run[0].path, run[0].size)
 
 		select {
 		case results <- hashResult{
@@ -980,18 +1156,18 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
 const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" +
 	"27ae41e4649b934ca495991b7852b855"
 
-// hashSignature computes the three content hashes that, with the file
-// size, form its duplicate signature. A file below headTailMin is
-// hashed in full and its whole-file SHA-256 is returned as head, tail,
-// and content alike — that range takes no separate end-window step. For
-// a file at or above headTailMin the head and tail are the SHA-256 of
-// its first and last headTailWindow bytes, and content is the SHA-256
-// of the whole file below wholeFileMax (the exact rung) or of
-// gigabyte-spaced samples at or above it (the sampled, deliberately
-// probabilistic rung). Two files are duplicates only when all four
-// agree; any mismatch means not a duplicate. size is the value recorded
-// when the file was statted; a zero-length file has constant hashes and
-// is never opened.
+// hashSignature computes the hashes the hash phase records for a file
+// whose size is shared; with the file size they form its duplicate
+// signature. A file below headTailMin is hashed in full and its
+// whole-file SHA-256 is returned as head, tail, and content alike —
+// that range takes no separate end-window step. For a file at or above
+// headTailMin only the head and tail are computed, the SHA-256 of its
+// first and last headTailWindow bytes, and content is returned empty:
+// the content phase computes it with hashContentOnly once the file's
+// size, head, and tail match another file's. Two files are duplicates
+// only when all four agree; any mismatch means not a duplicate. size
+// is the value recorded when the file was statted; a zero-length file
+// has constant hashes and is never opened.
 func hashSignature(path string, size int64) (string, string, string, error) {
 	if size == 0 {
 		return emptyHash, emptyHash, emptyHash, nil
@@ -1021,12 +1197,25 @@ func hashSignature(path string, size int64) (string, string, string, error) {
 		return "", "", "", err
 	}
 
-	content, err := hashContent(f, size)
+	return head, tail, "", nil
+}
+
+// hashContentOnly returns the content hash of the file at path, which
+// is at least headTailMin bytes: the content phase's read. head and
+// tail are returned empty, because the content phase keeps the ones its
+// records already hold.
+func hashContentOnly(path string, size int64) (string, string, string, error) {
+	//nolint:gosec // hashing operator-supplied paths is the tool's purpose
+	f, err := os.Open(path)
 	if err != nil {
 		return "", "", "", err
 	}
 
-	return head, tail, content, nil
+	defer func() { _ = f.Close() }()
+
+	content, err := hashContent(f, size)
+
+	return "", "", content, err
 }
 
 // hashEnds returns the SHA-256 of the first and last headTailWindow