Stat in the walk, hash only shared sizes, flush batches mid-scan

Restructure scan into three phases: walk+stat, hash, update.

The stat pass is folded into the walk workers: each regular file is
lstatted as its directory is read, while the metadata is hot. The
walk builds a scan-wide size census (walked files plus records
outside the scan roots), and unchanged already-hashed files resolve
during the walk without further work.

Only files whose size at least one other file shares are ever read:
a size-unique file cannot be a duplicate, so it is recorded without
hashes (head and tail empty). When a later scan makes its size
shared, the file is hashed then, even if otherwise unchanged. report
excludes unhashed records; trees gives them a never-matching
signature so a tree containing one never compares equal to another.

Hashed records are committed in batched transactions while the hash
phase runs, so an interrupted scan keeps everything hashed so far
and the next run resumes cheaply. The hash phase total is exact,
giving a meaningful ETA.

Memory drops accordingly: the existing-record index holds only path,
size, mtime, and a hashed flag (no hash values); the walk carries one
small record per candidate file; overlapping operands are pruned up
front instead of deduplicating every walked path in a scan-wide set.
Files no bigger than one chunk are hashed with a single read.

README.md

@@ -50,11 +50,13 @@ completed scan.
 
 Duplicate finders that hash entire files do not scale to the target
 environment: ~10 million files and ~150 TB on possibly slow or busy
-disks (a ZFS pool under resilver). Reading at most 2 KiB per file makes
-a full-filesystem sweep tractable, and the signatures are kept in a
-persistent database, so the expensive filesystem pass is incremental: a
-rescan re-hashes only files whose recorded mtime or size changed, and
-all analysis happens offline from the database alone. The end goal is
+disks (a ZFS pool under resilver). Reading at most 2 KiB per file — and
+only from files whose size at least one other file shares, since a
+size-unique file cannot be a duplicate — makes a full-filesystem sweep
+tractable, and the signatures are kept in a persistent database, so
+the expensive filesystem pass is incremental: a rescan re-hashes only
+files whose recorded mtime or size changed, and all analysis happens
+offline from the database alone. The end goal is
 not individual files but whole duplicated trees — duplicate
 extractions, duplicate downloads, copied project trees — which an
 operator can consider removing as a unit.
@@ -70,10 +72,13 @@ Goals, in order:
     removing an entire subtree at once. File-level duplicate detection is
     the foundation; tree-level detection is built on top of it.
 2.  **Never read full file contents.** At most 2 KiB is read per file
-    (first and last 1024 bytes). Scale target: ~10 million files, ~150 TB
+    (first and last 1024 bytes), and only files whose size at least
+    one other file shares are read at all — a size-unique file cannot
+    be a duplicate. Scale target: tens of millions of files, ~150 TB
     filesystem, possibly slow or busy disks (ZFS pool under resilver).
-    Holding the full file list in memory is acceptable; reading file
-    contents beyond 2 KiB per file is not.
+    Holding one small record (path, size, mtime) per file in memory
+    during a scan is acceptable; holding every file's hashes is not
+    (they stay in the database).
 3.  **Scan incrementally, analyze offline.** The expensive filesystem
     scan maintains a persistent database; an unchanged file is never
     read again on a rescan. All analysis (`report`, `trees`) works from
@@ -132,11 +137,13 @@ All three subcommands operate on a single SQLite database file:
 -   The database uses WAL journal mode and a busy timeout, so running a
     report while a cron `scan` is in progress is safe. The filesystem
     is authoritative; the database is an eventually-consistent
-    reflection of it. The update pass applies changes in batched
-    transactions (keeping the WAL small and letting concurrent reports
-    observe progress), so a report may see a scan's changes partially
-    applied, and a scan that dies partway leaves a valid database that
-    the next scan converges toward the filesystem.
+    reflection of it. Hashed records are committed in batched
+    transactions while the scan is still running (keeping the WAL
+    small and letting concurrent reports observe progress), so a
+    report may see a scan's changes partially applied, and a scan
+    that dies partway leaves a valid database holding everything
+    hashed so far; the next scan skips those records and converges
+    toward the filesystem.
 -   Schema (`PRAGMA user_version` is the schema version, currently 1; a
     database with any other version is a fatal error):
 
@@ -152,7 +159,11 @@ All three subcommands operate on a single SQLite database file:
 
     Paths are stored as BLOBs because Unix paths are raw bytes, not
     guaranteed UTF-8. `mtime` is used only for change detection; it is
-    not part of the duplicate key.
+    not part of the duplicate key. `head` and `tail` are empty strings
+    when the file has never been hashed because its size was unique as
+    of the last scan that covered it; such records still define the
+    file for tree reconstruction but never participate in duplicate
+    groups.
 
 ### `scan` mode
 
@@ -165,21 +176,32 @@ by absolute path, each operand is resolved to an absolute, lexically
 cleaned path (symlinks are not resolved) before walking, so results do
 not depend on the working directory. All operands belong to a single
 scan and are enumerated concurrently: every operand seeds the shared
-walk worker pool, and every pass runs once over the whole scan, so
-pass totals and ETAs are scan-wide. Overlapping operands (one
-containing another) are harmless — a file reached via multiple
-operands is deduplicated by path and produces one database record.
+walk worker pool. Overlapping operands are harmless — an operand that
+duplicates another or lies under another is dropped before walking,
+so every file is reached exactly once and produces one database
+record.
 
 `scan` synchronizes the database with the filesystem state under the
 scanned operands:
 
--   A file not yet in the database is hashed and inserted.
+-   Only a file whose size at least one other file shares is ever
+    read: a size-unique file cannot be a duplicate, so it is recorded
+    without hashes (`head` and `tail` empty). The size census covers
+    every file walked this scan plus every database record outside
+    the scanned operands, so a possible duplicate of a separately
+    scanned tree is still recognized.
+-   A file not yet in the database is inserted: hashed when its size
+    is shared, without hashes otherwise.
 -   A file already in the database is **skipped without reading its
     contents** when its lstat size equals the recorded size and its
     lstat mtime is not newer than the recorded mtime. This is what
-    makes a daily rescan cheap.
+    makes a daily rescan cheap. Exception: an unchanged file whose
+    record lacks hashes is hashed — and its record updated — once its
+    size becomes shared, so hashing deferred by size-uniqueness
+    happens as soon as it could matter.
 -   A file whose mtime is newer than recorded, or whose size differs,
-    is re-hashed and its record updated.
+    is processed as if new: re-hashed, or recorded without hashes,
+    per the shared-size rule.
 -   A database record whose path lies under one of the scanned operands
     but was not successfully processed this run is deleted. This
     removes records for deleted files. It also removes records for
@@ -190,29 +212,39 @@ scanned operands:
     disjoint trees can be scanned on different schedules into the same
     database.
 
-`scan` runs **four sequential passes over the whole scan**.
-Parallelism lives strictly inside each pass; the passes themselves
-never overlap:
+`scan` runs **three sequential phases over the whole scan**.
+Parallelism lives inside each phase; batched database writes begin
+during the hash phase:
 
-1.  **walk** — enumerate the trees under all `PATH` operands
-    concurrently with the worker pool: every operand seeds the shared
-    queue, and each worker reads one directory at a time, collecting
-    regular-file paths and handing discovered subdirectories back to
-    the queue. Sequential directory enumeration is
-    metadata-latency-bound and takes hours at tens of millions of
-    files; per-directory parallelism is what makes the walk tractable
-    on large or busy pools. Total unknown while running: show a live
-    count, not a percentage.
-2.  **stat** — `lstat` every collected path with the worker pool
-    (per-file parallelism), recording size and mtime.
-3.  **hash** — for each new or changed file (per the rules above), read
-    the first `min(1024, size)` bytes and the last `min(1024, size)`
-    bytes (the two reads overlap when `size < 2048`; for `size == 0`
-    hash the empty input) and compute the SHA-256 of each. Unchanged
-    files are not read and do not appear in this pass's total, which
-    is therefore exact for meaningful progress and ETA.
-4.  **update** — apply the scan's insertions, updates, and deletions
-    to the database in batched transactions.
+1.  **walk + stat** — enumerate the trees under all `PATH` operands
+    concurrently with the walk worker pool: every operand seeds the
+    shared queue, and each worker reads one directory at a time,
+    handing discovered subdirectories back to the queue and running
+    `lstat` on each regular file as it is discovered (while the
+    directory's metadata is still hot). Sequential directory
+    enumeration is metadata-latency-bound and takes hours at tens of
+    millions of files; per-directory parallelism is what makes the
+    walk tractable on large or busy pools. The walk builds the size
+    census and resolves unchanged already-hashed files on the fly;
+    every other file is carried to the hash phase as a (path, size,
+    mtime) record.
+2.  **hash** — with the census complete, each carried file's size
+    decides its fate. Size-unique files are never read: new or
+    changed ones are recorded without hashes in the update phase,
+    unchanged unhashed ones simply keep their records. Every file
+    with a shared size is hashed by the worker pool: read the first
+    `min(1024, size)` bytes and the last `min(1024, size)` bytes
+    (one read when `size <= 1024`, since the two windows coincide;
+    for `size == 0` hash the empty input) and compute the SHA-256 of
+    each. The phase total is exact, so progress and ETA are
+    meaningful. Completed records are committed in batched
+    transactions **while hashing runs**, so a scan interrupted after
+    hours keeps everything hashed so far and the next scan resumes
+    cheaply, skipping records already written.
+3.  **update** — commit the final partial batch, the hash-less
+    records for size-unique new and changed files, and the deletions
+    for records the scan did not verify (vanished files, plus paths
+    that failed to stat or hash).
 
 Rules for the walk:
 
@@ -234,12 +266,14 @@ Rules for the walk:
     records (accepted: the database mirrors what the latest scan could
     actually verify).
 
-Concurrency: the walk, stat, and hash passes each use a worker pool
-(`--workers`, default `runtime.NumCPU()`); the walk parallelizes
-across directories, stat and hash across files, so raising
-`--workers` can speed up metadata-bound passes on busy pools. The
-main goroutine owns database writes and progress rendering; progress
-display must never block the workers.
+Concurrency: the walk phase (which also stats files) and the hash
+phase each use a worker pool of `--workers` workers (default
+`runtime.NumCPU()`); the walk parallelizes across directories,
+hashing across files. Both phases are seek-bound on spinning disks,
+so raising `--workers` well past the core count can help on pools
+with many spindles. The main goroutine owns partitioning, database
+writes, and progress rendering; progress display must never block
+the workers.
 
 `scan` writes nothing to stdout. The summary line on stderr reports the
 files seen this run broken down by disposition, plus skips:
@@ -264,7 +298,11 @@ mounted.
 
 Processing:
 
--   Group records by the key `(size, head_hash, tail_hash)`.
+-   Records without hashes (size-unique when last scanned) are
+    excluded: their content is unknown, so they are never reported as
+    duplicates.
+-   Group the remaining records by the key
+    `(size, head_hash, tail_hash)`.
 -   Every group with two or more paths is a duplicate group.
 -   Within each group, sort paths lexicographically (byte order). The
     first path is the group's `first`; every other path is a `dupe`.
@@ -301,7 +339,10 @@ the paths in the records, split on `/`.
 Definitions:
 
 -   A file's **signature** is `(size, head_hash, tail_hash)` — mtime is
-    informational and excluded.
+    informational and excluded. An unhashed record (empty hashes) has
+    unknown content: its signature is treated as unique to that file,
+    so a tree containing an unhashed file never compares equal to any
+    other tree.
 -   A directory's **digest** is a SHA-256 Merkle digest computed
     bottom-up: serialize the directory's child entries — for a file
     child, its name and signature; for a subdirectory child, its name
@@ -351,11 +392,15 @@ all dupe rows) in human units.
 
 ### Progress
 
-Use the progress-bar library for all scan-pass progress; rendering in the
+Use the progress-bar library for all scan progress; rendering in the
 style of `pv` is the model. All progress goes to stderr.
 
-Each scan pass gets its own bar. Required elements for the stat,
-hash, and update passes (known totals):
+Each phase gets its own display. The walk has no known total while
+running: show a live file count, rate, and elapsed time
+(spinner-style, no percentage or ETA). The hash and update phases
+have exact totals — only files that actually need hashing appear in
+the hash total, so its ETA is meaningful. Required elements for the
+bars with known totals:
 
 -   elapsed time
 -   estimated time remaining
@@ -365,12 +410,9 @@ hash, and update passes (known totals):
 Example shape (exact layout is flexible, content is not):
 
 ```
-hash: [1234567/9876543] 12% |████      | 8123 files/s elapsed 2:32 eta 17:54
+hash: [12345/98765] 12% |████      | 92 files/s elapsed 2:32 eta 17:54
 ```
 
-The walk pass has no known total: show a live file count and elapsed time
-(spinner-style, no percentage or ETA).
-
 Additional requirements:
 
 -   When stderr is not a TTY, do not emit ANSI redraws: print a plain