Merge branch 'size-census': hash only files with shared sizes

Restructure scan into three phases: walk+stat, hash, update.

The stat pass is folded into the walk workers: each regular file is
lstatted as its directory is read, while the metadata is hot. The
walk builds a scan-wide size census (walked files plus records
outside the scan roots), and unchanged already-hashed files resolve
during the walk without further work.

Only files whose size at least one other file shares are ever read:
a size-unique file cannot be a duplicate, so it is recorded without
hashes (head and tail empty). When a later scan makes its size
shared, the file is hashed then, even if otherwise unchanged. report
excludes unhashed records; trees gives them a never-matching
signature so a tree containing one never compares equal to another.

Hashed records are committed in batched transactions while the hash
phase runs, so an interrupted scan keeps everything hashed so far
and the next run resumes cheaply. The hash phase total is exact,
giving a meaningful ETA.

Memory drops accordingly: the existing-record index holds only path,
size, mtime, and a hashed flag (no hash values); the walk carries one
small record per candidate file; overlapping operands are pruned up
front instead of deduplicating every walked path in a scan-wide set.
Files no bigger than one chunk are hashed with a single read.

Makefile

@@ -6,13 +6,15 @@ BINARY := sfdupes
 VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
 LDFLAGS := -X main.Version=$(VERSION)
 
-.PHONY: all build test lint fmt fmt-check check docker hooks clean
+.PHONY: sfdupes build test lint fmt fmt-check check docker hooks clean
 
-all: check build
-
-build:
+# Default target: build the binary. Phony so go build (which has its
+# own build cache) always decides what to recompile.
+sfdupes:
 	go build -ldflags "$(LDFLAGS)" -o $(BINARY)
 
+build: sfdupes
+
 test:
 	@go test -timeout 30s -cover ./... || \
 		{ echo "--- Rerunning with -v for details ---"; \

README.md

@@ -50,11 +50,13 @@ completed scan.
 
 Duplicate finders that hash entire files do not scale to the target
 environment: ~10 million files and ~150 TB on possibly slow or busy
-disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes
-a full-filesystem sweep tractable, and the signatures are kept in a
-persistent database, so the expensive filesystem pass is incremental: a
-rescan re-hashes only files whose recorded mtime or size changed, and
-all analysis happens offline from the database alone. The end goal is
+disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
+only from files whose size at least one other file shares, since a
+size-unique file cannot be a duplicate — makes a full-filesystem sweep
+tractable, and the signatures are kept in a persistent database, so
+the expensive filesystem pass is incremental: a rescan re-hashes only
+files whose recorded mtime or size changed, and all analysis happens
+offline from the database alone. The end goal is
 not individual files but whole duplicated trees — duplicate
 extractions, duplicate downloads, copied project trees — which an
 operator can consider removing as a unit.
@@ -70,10 +72,13 @@ Goals, in order:
     removing an entire subtree at once. File-level duplicate detection is
     the foundation; tree-level detection is built on top of it.
 2.  **Never read full file contents.** At most 2 KiB is read per file
-    (first and last 1024 bytes). Scale target: ~10 million files, ~150 TB
+    (first and last 1024 bytes), and only files whose size at least
+    one other file shares are read at all — a size-unique file cannot
+    be a duplicate. Scale target: tens of millions of files, ~150 TB
     filesystem, possibly slow or busy disks (ZFS pool under resilver).
-    Holding the full file list in memory is acceptable; reading file
-    contents beyond 2 KiB per file is not.
+    Holding one small record (path, size, mtime) per file in memory
+    during a scan is acceptable; holding every file's hashes is not
+    (they stay in the database).
 3.  **Scan incrementally, analyze offline.** The expensive filesystem
     scan maintains a persistent database; an unchanged file is never
     read again on a rescan. All analysis (`report`, `trees`) works from
@@ -130,12 +135,15 @@ All three subcommands operate on a single SQLite database file:
     database file is a fatal error (exit 1) telling the user to run
     `scan` first.
 -   The database uses WAL journal mode and a busy timeout, so running a
-    report while a cron `scan` is in progress is safe; the reports see
-    the last committed state. Each `PATH` operand's changes are
-    committed in a single transaction when that operand completes, so
-    a report never observes a half-finished operand, and a scan that
-    dies partway keeps every operand completed so far (the interrupted
-    operand's changes are lost, not corrupted).
+    report while a cron `scan` is in progress is safe. The filesystem
+    is authoritative; the database is an eventually-consistent
+    reflection of it. Hashed records are committed in batched
+    transactions while the scan is still running (keeping the WAL
+    small and letting concurrent reports observe progress), so a
+    report may see a scan's changes partially applied, and a scan
+    that dies partway leaves a valid database holding everything
+    hashed so far; the next scan skips those records and converges
+    toward the filesystem.
 -   Schema (`PRAGMA user_version` is the schema version, currently 1; a
     database with any other version is a fatal error):
 
@@ -151,7 +159,11 @@ All three subcommands operate on a single SQLite database file:
 
     Paths are stored as BLOBs because Unix paths are raw bytes, not
     guaranteed UTF-8. `mtime` is used only for change detection; it is
-    not part of the duplicate key.
+    not part of the duplicate key. `head` and `tail` are empty strings
+    when the file has never been hashed because its size was unique as
+    of the last scan that covered it; such records still define the
+    file for tree reconstruction but never participate in duplicate
+    groups.
 
 ### `scan` mode
 
@@ -162,22 +174,34 @@ or a regular file; an operand that does not exist is a fatal error
 (exit 1). Because database records persist between runs and are keyed
 by absolute path, each operand is resolved to an absolute, lexically
 cleaned path (symlinks are not resolved) before walking, so results do
-not depend on the working directory. Operands are processed in the
-order given, each one walked and committed independently; overlapping
-operands (one containing another) are harmless — a file reached via
-multiple operands produces one database record (later operands see the
-records committed by earlier ones and reuse them unchanged).
+not depend on the working directory. All operands belong to a single
+scan and are enumerated concurrently: every operand seeds the shared
+walk worker pool. Overlapping operands are harmless — an operand that
+duplicates another or lies under another is dropped before walking,
+so every file is reached exactly once and produces one database
+record.
 
 `scan` synchronizes the database with the filesystem state under the
 scanned operands:
 
--   A file not yet in the database is hashed and inserted.
+-   Only a file whose size at least one other file shares is ever
+    read: a size-unique file cannot be a duplicate, so it is recorded
+    without hashes (`head` and `tail` empty). The size census covers
+    every file walked this scan plus every database record outside
+    the scanned operands, so a possible duplicate of a separately
+    scanned tree is still recognized.
+-   A file not yet in the database is inserted: hashed when its size
+    is shared, without hashes otherwise.
 -   A file already in the database is **skipped without reading its
     contents** when its lstat size equals the recorded size and its
     lstat mtime is not newer than the recorded mtime. This is what
-    makes a daily rescan cheap.
+    makes a daily rescan cheap. Exception: an unchanged file whose
+    record lacks hashes is hashed — and its record updated — once its
+    size becomes shared, so hashing deferred by size-uniqueness
+    happens as soon as it could matter.
 -   A file whose mtime is newer than recorded, or whose size differs,
-    is re-hashed and its record updated.
+    is processed as if new: re-hashed, or recorded without hashes,
+    per the shared-size rule.
 -   A database record whose path lies under one of the scanned operands
     but was not successfully processed this run is deleted. This
     removes records for deleted files. It also removes records for
@@ -188,35 +212,39 @@ scanned operands:
     disjoint trees can be scanned on different schedules into the same
     database.
 
-`scan` runs **three sequential passes per operand**, committing each
-operand before starting the next:
+`scan` runs **three sequential phases over the whole scan**.
+Parallelism lives inside each phase; batched database writes begin
+during the hash phase:
 
-1.  **walk** — enumerate the tree with the worker pool: each worker
-    reads one directory at a time, records size and mtime for every
-    regular-file entry (`lstat` while the directory is fresh in
-    cache), and hands discovered subdirectories back to the shared
-    queue. Sequential directory enumeration is metadata-latency-bound
-    and takes hours at tens of millions of files; per-directory
-    parallelism is what makes the walk tractable on large or busy
-    pools. Total unknown while running: show a live count, not a
-    percentage.
-2.  **hash** — for each new or changed file (per the rules above), read
-    the first `min(1024, size)` bytes and the last `min(1024, size)`
-    bytes (the two reads overlap when `size < 2048`; for `size == 0`
-    hash the empty input) and compute the SHA-256 of each. Unchanged
-    files are not read and do not appear in this pass's total, which
-    is therefore exact for meaningful progress and ETA.
-3.  **update** — apply the operand's insertions, updates, and
-    deletions to the database in a single transaction.
-
-Because every operand runs its own three passes with its own totals,
-`scan` announces each operand on stderr before starting it, so a
-multi-operand invocation (e.g. a shell glob) is not mistaken for a
-nearly-finished run:
-
-```
-scan: /srv/media (operand 3 of 14)
-```
+1.  **walk + stat** — enumerate the trees under all `PATH` operands
+    concurrently with the walk worker pool: every operand seeds the
+    shared queue, and each worker reads one directory at a time,
+    handing discovered subdirectories back to the queue and running
+    `lstat` on each regular file as it is discovered (while the
+    directory's metadata is still hot). Sequential directory
+    enumeration is metadata-latency-bound and takes hours at tens of
+    millions of files; per-directory parallelism is what makes the
+    walk tractable on large or busy pools. The walk builds the size
+    census and resolves unchanged already-hashed files on the fly;
+    every other file is carried to the hash phase as a (path, size,
+    mtime) record.
+2.  **hash** — with the census complete, each carried file's size
+    decides its fate. Size-unique files are never read: new or
+    changed ones are recorded without hashes in the update phase,
+    unchanged unhashed ones simply keep their records. Every file
+    with a shared size is hashed by the worker pool: read the first
+    `min(1024, size)` bytes and the last `min(1024, size)` bytes
+    (one read when `size <= 1024`, since the two windows coincide;
+    for `size == 0` hash the empty input) and compute the SHA-256 of
+    each. The phase total is exact, so progress and ETA are
+    meaningful. Completed records are committed in batched
+    transactions **while hashing runs**, so a scan interrupted after
+    hours keeps everything hashed so far and the next scan resumes
+    cheaply, skipping records already written.
+3.  **update** — commit the final partial batch, the hash-less
+    records for size-unique new and changed files, and the deletions
+    for records the scan did not verify (vanished files, plus paths
+    that failed to stat or hash).
 
 Rules for the walk:
 
@@ -238,11 +266,14 @@ Rules for the walk:
     records (accepted: the database mirrors what the latest scan could
     actually verify).
 
-Concurrency: the walk and hash passes use a worker pool (`--workers`,
-default `runtime.NumCPU()`); the walk parallelizes across directories,
-so raising `--workers` can speed up metadata-bound walks on busy
-pools. The main goroutine owns database writes and progress rendering;
-progress display must never block the workers.
+Concurrency: the walk phase (which also stats files) and the hash
+phase each use a worker pool of `--workers` workers (default
+`runtime.NumCPU()`); the walk parallelizes across directories,
+hashing across files. Both phases are seek-bound on spinning disks,
+so raising `--workers` well past the core count can help on pools
+with many spindles. The main goroutine owns partitioning, database
+writes, and progress rendering; progress display must never block
+the workers.
 
 `scan` writes nothing to stdout. The summary line on stderr reports the
 files seen this run broken down by disposition, plus skips:
@@ -267,7 +298,11 @@ mounted.
 
 Processing:
 
--   Group records by the key `(size, head_hash, tail_hash)`.
+-   Records without hashes (size-unique when last scanned) are
+    excluded: their content is unknown, so they are never reported as
+    duplicates.
+-   Group the remaining records by the key
+    `(size, head_hash, tail_hash)`.
 -   Every group with two or more paths is a duplicate group.
 -   Within each group, sort paths lexicographically (byte order). The
     first path is the group's `first`; every other path is a `dupe`.
@@ -304,7 +339,10 @@ the paths in the records, split on `/`.
 Definitions:
 
 -   A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
-    informational and excluded.
+    informational and excluded. An unhashed record (empty hashes) has
+    unknown content: its signature is treated as unique to that file,
+    so a tree containing an unhashed file never compares equal to any
+    other tree.
 -   A directory's **digest** is a SHA-256 Merkle digest computed
     bottom-up: serialize the directory's child entries — for a file
     child, its name and signature; for a subdirectory child, its name
@@ -354,11 +392,15 @@ all dupe rows) in human units.
 
 ### Progress
 
-Use the progress-bar library for all scan-pass progress; rendering in the
+Use the progress-bar library for all scan progress; rendering in the
 style of `pv` is the model. All progress goes to stderr.
 
-Each scan pass gets its own bar, repeated per operand. Required
-elements for the hash and update passes (known totals):
+Each phase gets its own display. The walk has no known total while
+running: show a live file count, rate, and elapsed time
+(spinner-style, no percentage or ETA). The hash and update phases
+have exact totals — only files that actually need hashing appear in
+the hash total, so its ETA is meaningful. Required elements for the
+bars with known totals:
 
 -   elapsed time
 -   estimated time remaining
@@ -368,12 +410,9 @@ elements for the hash and update passes (known totals):
 Example shape (exact layout is flexible, content is not):
 
 ```
-hash: [1234567/9876543] 12% |████      | 8123 files/s elapsed 2:32 eta 17:54
+hash: [12345/98765] 12% |████      | 92 files/s elapsed 2:32 eta 17:54
 ```
 
-The walk pass has no known total: show a live file count and elapsed time
-(spinner-style, no percentage or ETA).
-
 Additional requirements:
 
 -   When stderr is not a TTY, do not emit ANSI redraws: print a plain
@@ -396,7 +435,8 @@ Additional requirements:
 
 The `Makefile` is the single source of truth for all operations:
 
--   `make build` — build the `sfdupes` binary (cgo disabled).
+-   `make` / `make build` — build the `sfdupes` binary (cgo
+    disabled); building is the default target.
 -   `make test` — run the test suite (30-second timeout; reruns with
     `-v` on failure).
 -   `make lint` — run `golangci-lint` with the repo config.

TODO.md

@@ -19,6 +19,24 @@
 
 # Completed Steps
 
+- make the binary the default Make target (2026-07-24, branch
+  `make-default-target`): plain `make` now builds `sfdupes`
+  (previously it ran `check` plus `build`); `make build` remains as
+  an alias
+- scan-wide phases, concurrent operands, batched updates (2026-07-24,
+  branch `scan-wide-phases`): all operands seed the shared walk pool
+  and every pass runs once over the whole scan, so totals and ETAs
+  are scan-global; the per-operand walk/hash/update cycles and their
+  stderr announcements are gone; the update pass commits in batched
+  transactions — the filesystem is authoritative and the database an
+  eventually-consistent reflection, so scan-level atomicity is not
+  required
+- split the stat pass back out of the walk (2026-07-24, branch
+  `parallel-phases`): phases are strictly sequential again — walk,
+  stat, hash, update per operand — with parallelism only inside each
+  phase; the walk enumerates paths with per-directory workers and the
+  stat pass lstats them with per-file workers, restoring the exact
+  total/ETA stat bar
 - announce each operand on stderr before its passes (2026-07-24,
   branch `scan-operand-progress`): with per-operand walk/hash/update
   cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals

db.go

@@ -8,6 +8,7 @@ import (
 	"io/fs"
 	"os"
 	"path/filepath"
+	"slices"
 	"strconv"
 
 	// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
@@ -237,12 +238,78 @@ func loadFileRows(db *sql.DB) ([]scanRec, error) {
 	return recs, nil
 }
 
+// loadFileMeta streams every record's path, size, mtime, and whether
+// it carries hashes to fn. Scan change detection needs no hash
+// values, and skipping the hash columns keeps the scan's in-memory
+// index small on multi-million-file databases.
+func loadFileMeta(db *sql.DB,
+	fn func(path string, size, mtime int64, hashed bool),
+) error {
+	rows, err := db.QueryContext(context.Background(),
+		"SELECT path, size, mtime, head <> '' FROM files")
+	if err != nil {
+		return fmt.Errorf("read records: %w", err)
+	}
+
+	defer func() { _ = rows.Close() }()
+
+	for rows.Next() {
+		var (
+			path        []byte
+			size, mtime int64
+			hashed      int64
+		)
+
+		err = rows.Scan(&path, &size, &mtime, &hashed)
+		if err != nil {
+			return fmt.Errorf("read record: %w", err)
+		}
+
+		fn(string(path), size, mtime, hashed != 0)
+	}
+
+	err = rows.Err()
+	if err != nil {
+		return fmt.Errorf("read records: %w", err)
+	}
+
+	return nil
+}
+
+// updateBatchSize is the number of record changes committed per
+// transaction during the update pass. The filesystem is authoritative
+// and the database an eventually-consistent reflection of it, so
+// scan-level atomicity is not required; smaller transactions keep the
+// WAL small and let concurrent reports observe progress.
+const updateBatchSize = 10000
+
 // applyChanges writes one scan's database changes — upserts for new and
-// changed files, deletes for vanished ones — in a single transaction,
-// so a concurrent report never observes a half-finished scan. Progress
-// is rendered on prog (one increment per change).
+// changed files, deletes for vanished ones — in batched transactions.
+// Progress is rendered on prog (one increment per change).
 func applyChanges(db *sql.DB, upserts []scanRec, deletes []string,
 	prog *progress,
+) error {
+	for batch := range slices.Chunk(upserts, updateBatchSize) {
+		err := applyBatch(db, batch, nil, prog)
+		if err != nil {
+			return err
+		}
+	}
+
+	for batch := range slices.Chunk(deletes, updateBatchSize) {
+		err := applyBatch(db, nil, batch, prog)
+		if err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+// applyBatch commits one batch of upserts and deletes in a single
+// transaction.
+func applyBatch(db *sql.DB, upserts []scanRec, deletes []string,
+	prog *progress,
 ) error {
 	ctx := context.Background()
 

db_test.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"database/sql"
 	"errors"
+	"fmt"
 	"path/filepath"
 	"slices"
 	"strings"
@@ -178,3 +179,46 @@ func TestApplyChangesRoundTrip(t *testing.T) {
 		t.Fatalf("rows = %+v, want just %+v", got, upd)
 	}
 }
+
+func TestApplyChangesBatching(t *testing.T) {
+	t.Parallel()
+
+	db := openTestDB(t)
+
+	// One more change than the batch size, so the update spans two
+	// transactions.
+	n := updateBatchSize + 1
+
+	recs := make([]scanRec, 0, n)
+	for i := range n {
+		recs = append(recs, scanRec{
+			size: int64(i), mtime: 1, head: "h", tail: "t",
+			path: fmt.Sprintf("/batch/%07d", i),
+		})
+	}
+
+	err := applyChanges(db, recs, nil, newProgress("update", int64(n)))
+	if err != nil {
+		t.Fatalf("applyChanges: %v", err)
+	}
+
+	got, err := loadFileRows(db)
+	if err != nil || len(got) != n {
+		t.Fatalf("loadFileRows = %d rows, %v; want %d", len(got), err, n)
+	}
+
+	deletes := make([]string, 0, n)
+	for _, r := range recs {
+		deletes = append(deletes, r.path)
+	}
+
+	err = applyChanges(db, nil, deletes, newProgress("update", int64(n)))
+	if err != nil {
+		t.Fatalf("applyChanges deletes: %v", err)
+	}
+
+	got, err = loadFileRows(db)
+	if err != nil || len(got) != 0 {
+		t.Fatalf("loadFileRows = %d rows, %v; want 0", len(got), err)
+	}
+}

main.go

@@ -69,7 +69,7 @@ func main() {
 		},
 	}
 	scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
-		"concurrent workers for the stat and hash passes")
+		"concurrent workers for the walk and hash phases")
 	scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
 		"do not cross filesystem boundaries")
 

progress.go

@@ -38,7 +38,10 @@ func stderrIsTTY() bool {
 // stderr is not a TTY it emits no ANSI redraws: it prints a plain
 // one-line update no more often than every plainInterval.
 //
-// All methods must be called from the main goroutine only.
+// All methods must be called from the main goroutine only. A nil
+// *progress is a valid no-display receiver: every method is a no-op,
+// so batched database flushes during the streaming pass can reuse the
+// update-pass helpers without rendering anything.
 type progress struct {
 	label string
 	total int64 // -1 when unknown (walk pass)
@@ -82,6 +85,10 @@ func newProgress(label string, total int64) *progress {
 
 // increment records one completed item and refreshes the display.
 func (p *progress) increment() {
+	if p == nil {
+		return
+	}
+
 	p.count++
 	if p.bar != nil {
 		_ = p.bar.Add(1)
@@ -97,6 +104,10 @@ func (p *progress) increment() {
 
 // warnf prints a one-line warning to stderr without corrupting the bar.
 func (p *progress) warnf(format string, args ...any) {
+	if p == nil {
+		return
+	}
+
 	if p.bar != nil {
 		_ = p.bar.Clear()
 	}
@@ -106,6 +117,10 @@ func (p *progress) warnf(format string, args ...any) {
 
 // finish terminates the pass's display.
 func (p *progress) finish() {
+	if p == nil {
+		return
+	}
+
 	if p.bar != nil {
 		_ = p.bar.Finish()
 

report.go

@@ -105,6 +105,13 @@ func collectDupeGroups(recs []scanRec) []dupeGroup {
 	groups := make(map[fileSig][]string)
 
 	for _, r := range recs {
+		// A record without hashes (its size was unique when last
+		// scanned) has unknown content and is never reported as a
+		// duplicate.
+		if r.head == "" {
+			continue
+		}
+
 		k := fileSig{size: r.size, head: r.head, tail: r.tail}
 		groups[k] = append(groups[k], r.path)
 	}

scan.go

@@ -4,7 +4,6 @@ import (
 	"crypto/sha256"
 	"database/sql"
 	"encoding/hex"
-	"errors"
 	"fmt"
 	"io/fs"
 	"os"
@@ -18,25 +17,34 @@ import (
 // chunk is the number of bytes hashed from each end of a file.
 const chunk = 1024
 
-// workQueueDepth bounds the job and result channels feeding the stat
+// workQueueDepth bounds the job and result channels feeding the walk
 // and hash worker pools.
 const workQueueDepth = 1024
 
-// errNotRegular reports a path whose type changed between the
-// directory read and its lstat.
-var errNotRegular = errors.New("no longer a regular file")
-
-// fileRec carries one file between the stat and hash passes.
+// fileRec carries one statted file between the scan phases.
 type fileRec struct {
 	path  string
 	size  int64
 	mtime int64
 }
 
-// runScan implements the scan subcommand: four sequential passes
-// (walk, stat, hash, update) that synchronize the persistent database
-// with the filesystem state under the PATH operands. Flag parsing and
-// the at-least-one-operand check are done by cobra.
+// fileMeta is the in-memory index entry for one existing database
+// record: just enough for change detection, plus whether the record
+// carries hashes. Hashes stay on disk; at tens of millions of records
+// they would dominate the scan's memory.
+type fileMeta struct {
+	size   int64
+	mtime  int64
+	hashed bool
+}
+
+// runScan implements the scan subcommand: three sequential phases —
+// walk (which stats each file as it is discovered), hash, update —
+// that synchronize the persistent database with the filesystem state
+// under the PATH operands. Only files whose size at least one other
+// file shares are ever hashed: a size-unique file cannot be a
+// duplicate. Flag parsing and the at-least-one-operand check are done
+// by cobra.
 func runScan(roots []string, workers int, oneFS bool) {
 	if workers < 1 {
 		workers = 1
@@ -89,6 +97,33 @@ func resolveRoots(roots []string) []string {
 	return abs
 }
 
+// pruneRoots drops operands already covered by another operand:
+// duplicates and any operand lying under another one. Every remaining
+// file is then reachable through exactly one root, so walked paths are
+// unique without keeping a scan-wide set of every path seen.
+func pruneRoots(roots []string) []string {
+	// Shorter-first ordering guarantees an ancestor is kept before any
+	// operand under it is considered.
+	sorted := slices.Clone(roots)
+	slices.SortFunc(sorted, func(a, b string) int {
+		if len(a) != len(b) {
+			return len(a) - len(b)
+		}
+
+		return strings.Compare(a, b)
+	})
+
+	kept := make([]string, 0, len(sorted))
+
+	for _, r := range sorted {
+		if !underAnyRoot(r, kept) {
+			kept = append(kept, r)
+		}
+	}
+
+	return kept
+}
+
 // scanStats summarizes one scan's database synchronization for the
 // final stderr summary.
 type scanStats struct {
@@ -99,81 +134,301 @@ type scanStats struct {
 	skipped   int
 }
 
-// syncScan synchronizes the database with the filesystem under roots.
-// Operands are processed in order, each one walked, hashed, and
-// committed independently, so an interrupted scan keeps every operand
-// completed so far. Records outside the roots are never touched.
+// scanState carries one scan's evolving state across its phases.
+// Entries consumed from existing mark files verified this run;
+// whatever remains after the walk and hash phases is deleted by the
+// update phase.
+type scanState struct {
+	db        *sql.DB
+	existing  map[string]fileMeta
+	sizes     []int64   // size census: every walked file, plus records outside the roots
+	toHash    []fileRec // files whose size is shared: must be read
+	sentinels []fileRec // new/changed size-unique files: recorded without hashes
+	batch     []scanRec
+	st        scanStats
+}
+
+// syncScan synchronizes the database with the filesystem under roots
+// in three sequential phases: walk (enumerate and stat every file,
+// building a complete size census), hash (read only the new or
+// changed — or previously unhashed — files whose size at least one
+// other file shares, committing results in batches as they arrive),
+// and update (record the size-unique files without reading them, and
+// delete the records the scan no longer verifies). Records outside
+// the roots are never touched.
 func syncScan(db *sql.DB, roots []string, workers int,
 	oneFS bool,
 ) (scanStats, error) {
-	var st scanStats
+	roots = pruneRoots(roots)
 
-	for i, root := range roots {
-		// Each operand runs its own walk/hash/update sequence, so
-		// announce it: without this, the per-operand pass totals look
-		// like the whole run's.
-		fmt.Fprintf(os.Stderr, "scan: %s (operand %d of %d)\n",
-			root, i+1, len(roots))
+	s := &scanState{db: db}
 
-		err := syncRoot(db, root, workers, oneFS, &st)
+	err := s.loadIndex(roots)
 	if err != nil {
-			return st, err
+		return s.st, err
+	}
+
+	changed, unhashed := s.walkPhase(startWalk(roots, oneFS, workers))
+
+	s.partition(changed, unhashed)
+
+	err = s.hashPhase(workers)
+	if err != nil {
+		return s.st, err
+	}
+
+	return s.st, s.updatePhase()
+}
+
+// loadIndex indexes the database records under the scan roots for
+// change detection and collects the sizes of every record outside
+// them: out-of-scope records join the size census so a scanned file
+// can be recognized as a possible duplicate of a tree scanned
+// separately into the same database.
+func (s *scanState) loadIndex(roots []string) error {
+	s.existing = make(map[string]fileMeta)
+
+	return loadFileMeta(s.db,
+		func(path string, size, mtime int64, hashed bool) {
+			if underAnyRoot(path, roots) {
+				s.existing[path] = fileMeta{
+					size: size, mtime: mtime, hashed: hashed,
+				}
+
+				return
+			}
+
+			s.sizes = append(s.sizes, size)
+		})
+}
+
+// walkPhase drains the walk, appending every walked file's size to
+// the census and resolving what it can immediately: an unchanged file
+// whose record already has hashes needs nothing further. It returns
+// the new-or-changed files and the unchanged files whose records lack
+// hashes; both remain candidates until the census decides whether
+// their sizes are shared.
+func (s *scanState) walkPhase(
+	events <-chan walkEvent,
+) ([]fileRec, []fileRec) {
+	prog := newProgress("walk", -1)
+
+	var changed, unhashed []fileRec
+
+	for ev := range events {
+		if ev.fail {
+			s.st.skipped++
+
+			prog.warnf("%s", ev.warn)
+
+			continue
+		}
+
+		s.sizes = append(s.sizes, ev.rec.size)
+
+		prog.increment()
+
+		old, ok := s.existing[ev.rec.path]
+
+		switch {
+		case !ok || old.size != ev.rec.size || old.mtime < ev.rec.mtime:
+			changed = append(changed, ev.rec)
+		case old.hashed:
+			delete(s.existing, ev.rec.path)
+
+			s.st.unchanged++
+		default:
+			unhashed = append(unhashed, ev.rec)
 		}
 	}
 
-	return st, nil
+	prog.finish()
+
+	return changed, unhashed
 }
 
-// syncRoot walks one PATH operand, hashes its new and changed files,
-// and commits the operand's record insertions, updates, and deletions
-// in a single transaction.
-func syncRoot(db *sql.DB, root string, workers int, oneFS bool,
-	st *scanStats,
-) error {
-	existing, err := loadScopedRows(db, root)
+// partition decides each candidate's disposition now that the size
+// census is complete. A file whose size no other file shares cannot
+// be a duplicate and is never read: a new or changed one is recorded
+// without hashes, an unchanged unhashed one keeps its record. Every
+// file with a shared size queues for the hash phase.
+func (s *scanState) partition(changed, unhashed []fileRec) {
+	slices.Sort(s.sizes)
+
+	for _, rec := range changed {
+		if s.sizeShared(rec.size) {
+			s.toHash = append(s.toHash, rec)
+
+			continue
+		}
+
+		s.sentinels = append(s.sentinels, rec)
+		s.resolve(rec.path)
+	}
+
+	for _, rec := range unhashed {
+		if s.sizeShared(rec.size) {
+			s.toHash = append(s.toHash, rec)
+
+			continue
+		}
+
+		delete(s.existing, rec.path)
+
+		s.st.unchanged++
+	}
+
+	s.sizes = nil
+}
+
+// sizeShared reports whether at least two census entries have this
+// size. Every candidate's own size is in the census exactly once, so
+// a second entry means another file (or a record outside the scan
+// roots) could share its content.
+func (s *scanState) sizeShared(size int64) bool {
+	i, found := slices.BinarySearch(s.sizes, size)
+
+	return found && i+1 < len(s.sizes) && s.sizes[i+1] == size
+}
+
+// resolve counts one written record as added or updated and marks its
+// path verified.
+func (s *scanState) resolve(path string) {
+	if _, ok := s.existing[path]; ok {
+		s.st.updated++
+
+		delete(s.existing, path)
+
+		return
+	}
+
+	s.st.added++
+}
+
+// hashPhase hashes every queued file with the worker pool, committing
+// completed records to the database in batches as results arrive, so
+// a long scan persists its progress as it goes (an interrupted scan
+// resumes cheaply: the next run skips everything already recorded).
+// The total is exact, so the bar shows a real ETA. Files that fail to
+// hash are warned about and skipped; their stale records, if any, are
+// deleted by the update phase.
+func (s *scanState) hashPhase(workers int) error {
+	jobs := make(chan fileRec, workQueueDepth)
+	results := make(chan hashResult, workQueueDepth)
+
+	startHashWorkers(jobs, results, workers)
+
+	// The feeder ranges over its own reference: s.toHash is released
+	// below while the feeder may still be running.
+	toHash := s.toHash
+	s.toHash = nil
+
+	go func() {
+		for _, rec := range toHash {
+			jobs <- rec
+		}
+
+		close(jobs)
+	}()
+
+	prog := newProgress("hash", int64(len(toHash)))
+	defer prog.finish()
+
+	for range toHash {
+		r := <-results
+
+		prog.increment()
+
+		if r.err != nil {
+			s.st.skipped++
+
+			prog.warnf("hash %s: %v", r.rec.path, r.err)
+
+			continue
+		}
+
+		s.resolve(r.rec.path)
+
+		s.batch = append(s.batch, scanRec{
+			size:  r.rec.size,
+			mtime: r.rec.mtime,
+			head:  r.head,
+			tail:  r.tail,
+			path:  r.rec.path,
+		})
+
+		if len(s.batch) < updateBatchSize {
+			continue
+		}
+
+		err := applyBatch(s.db, s.batch, nil, nil)
 		if err != nil {
 			return err
 		}
 
-	walked, walkErrs := walkPass(root, oneFS, workers)
-	toHash, unchanged := partitionChanged(walked, existing)
-	hashed, hashErrs := hashPass(toHash, workers)
-
-	st.unchanged += len(unchanged)
-	st.skipped += walkErrs + hashErrs
-
-	for _, r := range hashed {
-		if _, ok := existing[r.path]; ok {
-			st.updated++
-		} else {
-			st.added++
-		}
+		s.batch = s.batch[:0]
 	}
 
-	deletes := collectDeletes(existing, unchanged, hashed)
-	st.removed += len(deletes)
-
-	return applyPass(db, hashed, deletes)
+	return nil
 }
 
-// loadScopedRows loads the database records whose paths lie under
-// root, keyed by path. Records outside the scanned operands belong to
-// other trees and are left untouched.
-func loadScopedRows(db *sql.DB, root string) (map[string]scanRec, error) {
-	all, err := loadFileRows(db)
+// updatePhase writes the scan's tail under one progress display: the
+// final partial batch of hashed records, a hash-less record for every
+// size-unique new or changed file, and deletions for every record the
+// scan did not verify (vanished files, plus paths that failed to stat
+// or hash).
+func (s *scanState) updatePhase() error {
+	deletes := make([]string, 0, len(s.existing))
+	for path := range s.existing {
+		deletes = append(deletes, path)
+	}
+
+	// Sorted deletes keep the update phase deterministic.
+	slices.Sort(deletes)
+
+	s.st.removed = len(deletes)
+
+	total := len(s.batch) + len(s.sentinels) + len(deletes)
+	prog := newProgress("update", int64(total))
+
+	defer prog.finish()
+
+	err := applyChanges(s.db, s.batch, nil, prog)
 	if err != nil {
-		return nil, err
+		return err
 	}
 
-	scoped := make(map[string]scanRec)
+	s.batch = nil
 
-	for _, r := range all {
-		if underRoot(r.path, root) {
-			scoped[r.path] = r
+	// Sentinel records are converted in batch-sized chunks rather than
+	// materialized all at once; a first scan can have millions.
+	for chunk := range slices.Chunk(s.sentinels, updateBatchSize) {
+		recs := make([]scanRec, 0, len(chunk))
+		for _, rec := range chunk {
+			recs = append(recs, scanRec{
+				size: rec.size, mtime: rec.mtime, path: rec.path,
+			})
+		}
+
+		err = applyBatch(s.db, recs, nil, prog)
+		if err != nil {
+			return err
 		}
 	}
 
-	return scoped, nil
+	return applyChanges(s.db, nil, deletes, prog)
+}
+
+// underAnyRoot reports whether path is any of the roots or lies under
+// one of them.
+func underAnyRoot(path string, roots []string) bool {
+	for _, root := range roots {
+		if underRoot(path, root) {
+			return true
+		}
+	}
+
+	return false
 }
 
 // underRoot reports whether path is root itself or lies under it. Both
@@ -191,75 +446,6 @@ func underRoot(path, root string) bool {
 	return strings.HasPrefix(path, prefix)
 }
 
-// partitionChanged splits the stat results into files that must be
-// hashed (new, or changed) and files whose existing records are reused
-// without reading them: a file is unchanged when its statted size
-// equals the recorded size and its statted mtime is not newer than the
-// recorded mtime.
-func partitionChanged(recs []fileRec,
-	existing map[string]scanRec,
-) ([]fileRec, []scanRec) {
-	var (
-		toHash    []fileRec
-		unchanged []scanRec
-	)
-
-	for _, rec := range recs {
-		old, ok := existing[rec.path]
-		if ok && old.size == rec.size && old.mtime >= rec.mtime {
-			unchanged = append(unchanged, old)
-
-			continue
-		}
-
-		toHash = append(toHash, rec)
-	}
-
-	return toHash, unchanged
-}
-
-// collectDeletes returns the existing record paths that were not
-// successfully processed this run: vanished files, plus paths that
-// failed to stat or hash. The database keeps only records verified by
-// the latest scan covering them. The result is sorted so the update
-// pass is deterministic.
-func collectDeletes(existing map[string]scanRec,
-	unchanged, hashed []scanRec,
-) []string {
-	kept := make(map[string]bool, len(unchanged)+len(hashed))
-
-	for _, r := range unchanged {
-		kept[r.path] = true
-	}
-
-	for _, r := range hashed {
-		kept[r.path] = true
-	}
-
-	var deletes []string
-
-	for p := range existing {
-		if !kept[p] {
-			deletes = append(deletes, p)
-		}
-	}
-
-	slices.Sort(deletes)
-
-	return deletes
-}
-
-// applyPass writes the scan's changes to the database under an update
-// progress display (one item per insertion, update, or deletion).
-func applyPass(db *sql.DB, upserts []scanRec, deletes []string) error {
-	prog := newProgress("update", int64(len(upserts)+len(deletes)))
-	err := applyChanges(db, upserts, deletes, prog)
-
-	prog.finish()
-
-	return err
-}
-
 // dirJob is one directory awaiting traversal by the walk workers. It
 // carries its operand's filesystem device so -x can stop at
 // filesystem boundaries.
@@ -269,77 +455,68 @@ type dirJob struct {
 	rootDevOK bool
 }
 
-// walkEvent is one walk result delivered to the main goroutine: a
-// regular file's stat record, or a warning when fail is set.
+// walkEvent is one walk result delivered to the walk phase: a regular
+// file's statted record, or a warning when fail is set.
 type walkEvent struct {
 	rec  fileRec
 	warn string
 	fail bool
 }
 
-// walkPass enumerates every regular file under root with a
-// per-directory worker pool, recording size and mtime from lstat
-// while each directory is fresh in cache. It never follows symlinks,
-// never descends into directories named .zfs, and warns and continues
-// on any per-path error. With oneFS set it never descends into a
-// directory on a different filesystem than root.
-func walkPass(root string, oneFS bool, workers int) ([]fileRec, int) {
-	prog := newProgress("walk", -1)
-
-	recs, initial, errs := seedRoot(root, prog)
+// startWalk seeds every root into the shared walk worker pool and
+// returns the event stream: one record per regular file, one warning
+// event per per-path error. The channel is closed when the walk
+// completes.
+func startWalk(roots []string, oneFS bool, workers int) <-chan walkEvent {
 	jobs, subdirs, events := startWalkWorkers(workers, oneFS)
 
+	go func() {
+		initial := make([]dirJob, 0, len(roots))
+		for _, root := range roots {
+			initial = append(initial, seedRoot(root, events)...)
+		}
+
 		dispatchDirs(initial, jobs, subdirs)
+	}()
 
-	for ev := range events {
-		if ev.fail {
-			errs++
-
-			prog.warnf("%s", ev.warn)
-
-			continue
+	return events
 }
 
-		recs = append(recs, ev.rec)
-
-		prog.increment()
-	}
-
-	prog.finish()
-
-	return recs, errs
-}
-
-// seedRoot turns the PATH operand into the walk's starting state: a
-// regular-file operand becomes a record directly, a directory operand
-// becomes the initial job, and a symlink or other non-regular operand
-// yields nothing (symlinks are never followed, including as
+// seedRoot turns one PATH operand into the walk's starting state: a
+// regular-file operand is statted and emitted directly, a directory
+// operand becomes an initial job, and a symlink or other non-regular
+// operand yields nothing (symlinks are never followed, including as
 // operands).
-func seedRoot(root string, prog *progress) ([]fileRec, []dirJob, int) {
+func seedRoot(root string, events chan<- walkEvent) []dirJob {
 	fi, err := os.Lstat(root)
 	if err != nil {
-		prog.warnf("walk %s: %v", root, err)
+		events <- walkEvent{
+			warn: fmt.Sprintf("walk %s: %v", root, err),
+			fail: true,
+		}
 
-		return nil, nil, 1
+		return nil
 	}
 
 	switch {
 	case fi.IsDir():
 		if filepath.Base(root) == ".zfs" {
-			return nil, nil, 0
+			return nil
 		}
 
 		dev, ok := deviceOfInfo(fi)
 
-		return nil, []dirJob{{path: root, rootDev: dev, rootDevOK: ok}}, 0
+		return []dirJob{{path: root, rootDev: dev, rootDevOK: ok}}
 	case fi.Mode().IsRegular():
-		rec := fileRec{path: root, size: fi.Size(), mtime: fi.ModTime().Unix()}
+		events <- walkEvent{rec: fileRec{
+			path:  root,
+			size:  fi.Size(),
+			mtime: fi.ModTime().Unix(),
+		}}
 
-		prog.increment()
-
-		return []fileRec{rec}, nil, 0
+		return nil
 	default:
-		return nil, nil, 0
+		return nil
 	}
 }
 
@@ -439,36 +616,38 @@ func walkOneDir(job dirJob, oneFS bool, events chan<- walkEvent) []dirJob {
 			continue
 		}
 
-		events <- fileEvent(p, e)
+		emitFile(p, e, events)
 	}
 
 	return subs
 }
 
-// fileEvent lstats one regular-file directory entry into its walk
-// event.
-func fileEvent(p string, e fs.DirEntry) walkEvent {
-	fi, err := e.Info()
+// emitFile stats one regular directory entry and emits its record.
+// The lstat happens here in the walk worker, while the directory's
+// metadata is still hot; a path that fails to stat (or stops being a
+// regular file) between the directory read and the lstat is warned
+// about and skipped.
+func emitFile(p string, e fs.DirEntry, events chan<- walkEvent) {
+	info, err := e.Info()
+	if err != nil {
+		events <- walkEvent{
+			warn: fmt.Sprintf("stat %s: %v", p, err),
+			fail: true,
+		}
 
-	switch {
-	case err != nil:
-		return walkEvent{
-			warn: fmt.Sprintf("walk %s: %v", p, err),
-			fail: true,
+		return
 	}
-	case !fi.Mode().IsRegular():
-		return walkEvent{
-			warn: fmt.Sprintf("walk %s: %v", p, errNotRegular),
-			fail: true,
+
+	if !info.Mode().IsRegular() {
+		return
 	}
-	default:
-		return walkEvent{rec: fileRec{
+
+	events <- walkEvent{rec: fileRec{
 		path:  p,
-			size:  fi.Size(),
-			mtime: fi.ModTime().Unix(),
+		size:  info.Size(),
+		mtime: info.ModTime().Unix(),
 	}}
 }
-}
 
 // subdirJob applies the descent rules to directory p: never enter
 // .zfs (ZFS snapshot pseudo-dirs would list every file once per
@@ -515,7 +694,7 @@ func deviceOfInfo(fi fs.FileInfo) (uint64, bool) {
 }
 
 // hashResult carries one file's head/tail hashes (or the error that
-// prevented hashing it) from the hash workers to the main goroutine.
+// prevented hashing it) from the hash workers to the hash phase.
 type hashResult struct {
 	rec  fileRec
 	head string
@@ -523,76 +702,28 @@ type hashResult struct {
 	err  error
 }
 
-// startHashWorkers starts the hash worker pool over recs and returns
-// the channel its results arrive on (one per record, in completion
-// order).
-func startHashWorkers(recs []fileRec, workers int) <-chan hashResult {
-	jobs := make(chan fileRec, workQueueDepth)
-	results := make(chan hashResult, workQueueDepth)
-
+// startHashWorkers starts the hash worker pool: workers read jobs,
+// write one result per record, and exit when jobs is closed.
+func startHashWorkers(jobs <-chan fileRec, results chan<- hashResult,
+	workers int,
+) {
 	for range workers {
 		go func() {
 			for rec := range jobs {
 				head, tail, err := hashHeadTail(rec.path, rec.size)
-				results <- hashResult{rec: rec, head: head, tail: tail, err: err}
+				results <- hashResult{
+					rec: rec, head: head, tail: tail, err: err,
+				}
 			}
 		}()
 	}
-
-	go func() {
-		for _, rec := range recs {
-			jobs <- rec
-		}
-
-		close(jobs)
-	}()
-
-	return results
-}
-
-// hashPass hashes the first and last chunk bytes of every file in a
-// worker pool and collects the resulting records on the main
-// goroutine. Files that fail to open or read are warned about and
-// dropped. Only new or changed files reach this pass.
-func hashPass(recs []fileRec, workers int) ([]scanRec, int) {
-	results := startHashWorkers(recs, workers)
-	prog := newProgress("hash", int64(len(recs)))
-	out := make([]scanRec, 0, len(recs))
-
-	var errs int
-
-	for range recs {
-		r := <-results
-		if r.err != nil {
-			errs++
-
-			prog.warnf("hash %s: %v", r.rec.path, r.err)
-			prog.increment()
-
-			continue
-		}
-
-		out = append(out, scanRec{
-			size:  r.rec.size,
-			mtime: r.rec.mtime,
-			head:  r.head,
-			tail:  r.tail,
-			path:  r.rec.path,
-		})
-
-		prog.increment()
-	}
-
-	prog.finish()
-
-	return out, errs
 }
 
 // hashHeadTail returns the lowercase-hex SHA-256 of the first
 // min(chunk, size) bytes and of the last min(chunk, size) bytes of the
 // file at path. The two reads overlap when size < 2*chunk; for
 // size == 0 both hashes are of the empty input. size is the value
-// recorded by the stat pass.
+// recorded when the file was statted.
 func hashHeadTail(path string, size int64) (string, string, error) {
 	//nolint:gosec // hashing operator-supplied paths is the tool's purpose
 	f, err := os.Open(path)
@@ -614,12 +745,19 @@ func hashHeadTail(path string, size int64) (string, string, error) {
 
 	h := sha256.Sum256(buf)
 
-	if n > 0 {
+	// When the whole file fits in one chunk the tail window is exactly
+	// the bytes just read: reuse the head hash instead of issuing a
+	// second read for every small file.
+	if size <= int64(chunk) {
+		hh := hex.EncodeToString(h[:])
+
+		return hh, hh, nil
+	}
+
 	_, err = f.ReadAt(buf, size-n)
 	if err != nil {
 		return "", "", err
 	}
-	}
 
 	t := sha256.Sum256(buf)
 

scan_test.go

@@ -110,7 +110,32 @@ func TestHashHeadTailErrors(t *testing.T) {
 	}
 }
 
-// walkedPaths returns the sorted paths of walked records.
+// collectWalk runs a walk over roots and returns the emitted records
+// and the number of warning events.
+func collectWalk(t *testing.T, roots []string, oneFS bool,
+	workers int,
+) ([]fileRec, int) {
+	t.Helper()
+
+	var (
+		recs []fileRec
+		errs int
+	)
+
+	for ev := range startWalk(roots, oneFS, workers) {
+		if ev.fail {
+			errs++
+
+			continue
+		}
+
+		recs = append(recs, ev.rec)
+	}
+
+	return recs, errs
+}
+
+// walkedPaths returns the sorted paths of the walked records.
 func walkedPaths(recs []fileRec) []string {
 	paths := make([]string, 0, len(recs))
 	for _, r := range recs {
@@ -122,29 +147,14 @@ func walkedPaths(recs []fileRec) []string {
 	return paths
 }
 
-// walkedByPath finds the walked record with the given path.
-func walkedByPath(t *testing.T, recs []fileRec, path string) fileRec {
-	t.Helper()
-
-	for _, r := range recs {
-		if r.path == path {
-			return r
-		}
-	}
-
-	t.Fatalf("no walked record for %q", path)
-
-	return fileRec{}
-}
-
-func TestWalkPass(t *testing.T) {
+func TestWalk(t *testing.T) {
 	t.Parallel()
 
 	dir := t.TempDir()
 	want := []string{
 		writeFile(t, dir, "a.txt", []byte("a")),
-		writeFile(t, dir, "sub/b.txt", []byte("b")),
-		writeFile(t, dir, "sub/deeper/c.txt", []byte("c")),
+		writeFile(t, dir, "sub/b.txt", []byte("bb")),
+		writeFile(t, dir, "sub/deeper/c.txt", []byte("ccc")),
 	}
 
 	slices.Sort(want)
@@ -158,7 +168,7 @@ func TestWalkPass(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	recs, errs := walkPass(dir, false, 4)
+	recs, errs := collectWalk(t, []string{dir}, false, 4)
 	if errs != 0 {
 		t.Fatalf("errs = %d, want 0", errs)
 	}
@@ -167,13 +177,20 @@ func TestWalkPass(t *testing.T) {
 		t.Fatalf("paths = %q, want %q", got, want)
 	}
 
-	// The walk records lstat sizes and mtimes.
-	if r := walkedByPath(t, recs, want[0]); r.size != 1 || r.mtime <= 0 {
-		t.Fatalf("rec = %+v, want size 1 and a positive mtime", r)
+	// The walk stats each file as it is discovered: every record must
+	// carry the real size and a plausible mtime.
+	for _, r := range recs {
+		if r.size < 1 || r.size > 3 {
+			t.Errorf("%s: size = %d, want 1..3", r.path, r.size)
+		}
+
+		if r.mtime <= 0 {
+			t.Errorf("%s: mtime = %d, want positive", r.path, r.mtime)
+		}
 	}
 }
 
-func TestWalkPassDeepAndWide(t *testing.T) {
+func TestWalkDeepAndWide(t *testing.T) {
 	t.Parallel()
 
 	// Exercise the dispatcher with more directories than workers and
@@ -191,7 +208,7 @@ func TestWalkPassDeepAndWide(t *testing.T) {
 
 	slices.Sort(want)
 
-	recs, errs := walkPass(dir, false, 8)
+	recs, errs := collectWalk(t, []string{dir}, false, 8)
 	if errs != 0 {
 		t.Fatalf("errs = %d, want 0", errs)
 	}
@@ -201,7 +218,32 @@ func TestWalkPassDeepAndWide(t *testing.T) {
 	}
 }
 
-func TestWalkPassFileAndSymlinkOperands(t *testing.T) {
+func TestWalkMultipleRoots(t *testing.T) {
+	t.Parallel()
+
+	rootA := t.TempDir()
+	rootB := t.TempDir()
+	want := []string{
+		writeFile(t, rootA, "a1", []byte("1")),
+		writeFile(t, rootA, "sub/a2", []byte("2")),
+		writeFile(t, rootB, "b1", []byte("3")),
+	}
+
+	slices.Sort(want)
+
+	// Operands are enumerated concurrently by the shared pool; order
+	// is unspecified.
+	recs, errs := collectWalk(t, []string{rootA, rootB}, false, 4)
+	if errs != 0 {
+		t.Fatalf("errs = %d, want 0", errs)
+	}
+
+	if got := walkedPaths(recs); !slices.Equal(got, want) {
+		t.Fatalf("paths = %q, want %q", got, want)
+	}
+}
+
+func TestWalkFileAndSymlinkOperands(t *testing.T) {
 	t.Parallel()
 
 	dir := t.TempDir()
@@ -214,20 +256,20 @@ func TestWalkPassFileAndSymlinkOperands(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	// A regular-file operand is emitted as itself.
-	recs, errs := walkPass(f, false, 2)
+	// A regular-file operand is emitted as itself, statted.
+	recs, errs := collectWalk(t, []string{f}, false, 2)
 	if errs != 0 || len(recs) != 1 || recs[0].path != f || recs[0].size != 4 {
 		t.Fatalf("file operand: recs = %+v, errs = %d", recs, errs)
 	}
 
 	// A symlink operand is not followed and yields nothing.
-	recs, errs = walkPass(link, false, 2)
+	recs, errs = collectWalk(t, []string{link}, false, 2)
 	if errs != 0 || len(recs) != 0 {
 		t.Fatalf("symlink operand: recs = %+v, errs = %d", recs, errs)
 	}
 }
 
-func TestWalkPassOneFilesystemSameFS(t *testing.T) {
+func TestWalkOneFilesystemSameFS(t *testing.T) {
 	t.Parallel()
 
 	// Everything in one filesystem: -x must not skip anything.
@@ -237,7 +279,7 @@ func TestWalkPassOneFilesystemSameFS(t *testing.T) {
 		writeFile(t, dir, "sub/deep/b", []byte("b")),
 	}
 
-	recs, errs := walkPass(dir, true, 4)
+	recs, errs := collectWalk(t, []string{dir}, true, 4)
 	if errs != 0 {
 		t.Fatalf("errs = %d, want 0", errs)
 	}
@@ -623,13 +665,11 @@ func TestSyncScanOverlappingRoots(t *testing.T) {
 
 	writeFile(t, dir, "sub/f", pattern(1, 10))
 
-	// A file reachable via two overlapping operands yields one
-	// record: the second operand sees the rows the first operand
-	// committed and reuses them unchanged, which also proves each
-	// operand commits before the next starts.
+	// A file reachable via two overlapping operands is deduplicated
+	// by path in the shared walk and processed once.
 	st := syncTree(t, db, dir, filepath.Join(dir, "sub"))
-	if st != (scanStats{added: 1, unchanged: 1}) {
-		t.Fatalf("stats = %+v, want 1 added 1 unchanged", st)
+	if st != (scanStats{added: 1}) {
+		t.Fatalf("stats = %+v, want 1 added", st)
 	}
 
 	if got := recordPaths(dbRecords(t, db)); len(got) != 1 {
@@ -637,6 +677,91 @@ func TestSyncScanOverlappingRoots(t *testing.T) {
 	}
 }
 
+func TestScanSkipsUniqueSizes(t *testing.T) {
+	t.Parallel()
+
+	dir := t.TempDir()
+	db := openTestDB(t)
+	a := writeFile(t, dir, "a.bin", pattern(1, 500))
+
+	writeFile(t, dir, "b.bin", pattern(2, 600))
+
+	// Neither size is shared, so neither file is read: both records
+	// are written without hashes and no duplicates are reported.
+	st := syncTree(t, db, dir)
+	if st != (scanStats{added: 2}) {
+		t.Fatalf("stats = %+v, want 2 added", st)
+	}
+
+	recs := dbRecords(t, db)
+	for _, r := range recs {
+		if r.head != "" || r.tail != "" {
+			t.Errorf("%s: head = %q tail = %q, want unhashed",
+				r.path, r.head, r.tail)
+		}
+	}
+
+	if groups := collectDupeGroups(recs); len(groups) != 0 {
+		t.Fatalf("groups = %+v, want none from unhashed records", groups)
+	}
+
+	// A new same-size file makes 500 a shared size: the next scan
+	// hashes both the new file and the previously unhashed unchanged
+	// one, and they group as duplicates.
+	c := writeFile(t, dir, "c.bin", pattern(1, 500))
+
+	st = syncTree(t, db, dir)
+	if st != (scanStats{added: 1, updated: 1, unchanged: 1}) {
+		t.Fatalf("rescan stats = %+v, want 1 added 1 updated 1 unchanged",
+			st)
+	}
+
+	groups := collectDupeGroups(dbRecords(t, db))
+	if len(groups) != 1 {
+		t.Fatalf("groups = %+v, want the a/c pair", groups)
+	}
+
+	if want := []string{a, c}; !slices.Equal(groups[0].paths, want) {
+		t.Fatalf("group paths = %q, want %q", groups[0].paths, want)
+	}
+}
+
+func TestTreesUnhashedNeverEqual(t *testing.T) {
+	t.Parallel()
+
+	// Two trees identical except for unhashed same-name, same-size
+	// files (possible when the trees were scanned separately) must not
+	// compare equal: unhashed content is unknown.
+	shared := pattern(1, 100)
+	recs := []scanRec{
+		{path: "/x/t1/f1", size: 100, head: hexSum(shared), tail: hexSum(shared)},
+		{path: "/x/t2/f1", size: 100, head: hexSum(shared), tail: hexSum(shared)},
+		{path: "/x/t1/u", size: 50},
+		{path: "/x/t2/u", size: 50},
+	}
+
+	super, dirs := buildHierarchy(recs)
+	super.compute()
+
+	if tg := collectTreeGroups(dirs, super); len(tg) != 0 {
+		t.Fatalf("tree groups = %d, want 0 (unhashed files differ)",
+			len(tg))
+	}
+}
+
+func TestPruneRoots(t *testing.T) {
+	t.Parallel()
+
+	// Duplicates and operands under other operands are dropped; /cc is
+	// not under /c (sibling with a shared prefix).
+	got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"})
+
+	want := []string{"/a", "/c", "/cc"}
+	if !slices.Equal(got, want) {
+		t.Fatalf("pruneRoots = %q, want %q", got, want)
+	}
+}
+
 func TestReportsNeverTouchFilesystem(t *testing.T) {
 	t.Parallel()
 

trees.go

@@ -116,9 +116,17 @@ func buildHierarchy(recs []scanRec) (*treeNode, []*treeNode) {
 			node.files = make(map[string]fileSig)
 		}
 
-		node.files[comps[len(comps)-1]] = fileSig{
-			size: r.size, head: r.head, tail: r.tail,
+		sig := fileSig{size: r.size, head: r.head, tail: r.tail}
+
+		// An unhashed record (its size was unique when last scanned)
+		// has unknown content: give it a signature no other file can
+		// share, so trees containing it never compare equal. Real
+		// heads are hex, so the NUL-prefixed form cannot collide.
+		if sig.head == "" {
+			sig.head = "unhashed\x00" + r.path
 		}
+
+		node.files[comps[len(comps)-1]] = sig
 	}
 
 	return super, allDirs