Specify parallel per-directory walk and per-operand commits

The walk pass is a single goroutine; on a busy ZFS pool it manages
only a few thousand directory entries per second and takes hours at
~20M files. Respecify it as a worker-pool traversal that reads
directories concurrently and records size/mtime during the walk,
folding away the separate stat pass and halving metadata I/O. Each
PATH operand now commits in its own transaction so an interrupted
scan keeps the operands completed so far.

README.md

@@ -131,9 +131,11 @@ All three subcommands operate on a single SQLite database file:
     `scan` first.
 -   The database uses WAL journal mode and a busy timeout, so running a
     report while a cron `scan` is in progress is safe; the reports see
-    the last committed state. Each scan commits its changes in a single
-    transaction, so a report never observes a half-finished scan and a
-    scan that dies partway leaves the previous state intact.
+    the last committed state. Each `PATH` operand's changes are
+    committed in a single transaction when that operand completes, so
+    a report never observes a half-finished operand, and a scan that
+    dies partway keeps every operand completed so far (the interrupted
+    operand's changes are lost, not corrupted).
 -   Schema (`PRAGMA user_version` is the schema version, currently 1; a
     database with any other version is a fatal error):
 
@@ -160,9 +162,11 @@ or a regular file; an operand that does not exist is a fatal error
 (exit 1). Because database records persist between runs and are keyed
 by absolute path, each operand is resolved to an absolute, lexically
 cleaned path (symlinks are not resolved) before walking, so results do
-not depend on the working directory. Operands are walked in the order
-given; overlapping operands (one containing another) are harmless — a
-file reached via multiple operands produces one database record.
+not depend on the working directory. Operands are processed in the
+order given, each one walked and committed independently; overlapping
+operands (one containing another) are harmless — a file reached via
+multiple operands produces one database record (later operands see the
+records committed by earlier ones and reuse them unchanged).
 
 `scan` synchronizes the database with the filesystem state under the
 scanned operands:
@@ -184,20 +188,26 @@ scanned operands:
     disjoint trees can be scanned on different schedules into the same
     database.
 
-`scan` runs **four sequential passes**, in this order, so that every
-expensive pass has an exact total for meaningful progress and ETA:
+`scan` runs **three sequential passes per operand**, committing each
+operand before starting the next:
 
-1.  **walk** — recursively enumerate the tree under each `PATH` in
-    turn, collecting the list of regular-file paths. Total unknown
-    while running: show a live count, not a percentage.
-2.  **stat** — `lstat` every collected path, recording size and mtime.
-3.  **hash** — for each new or changed file (per the rules above), read
+1.  **walk** — enumerate the tree with the worker pool: each worker
+    reads one directory at a time, records size and mtime for every
+    regular-file entry (`lstat` while the directory is fresh in
+    cache), and hands discovered subdirectories back to the shared
+    queue. Sequential directory enumeration is metadata-latency-bound
+    and takes hours at tens of millions of files; per-directory
+    parallelism is what makes the walk tractable on large or busy
+    pools. Total unknown while running: show a live count, not a
+    percentage.
+2.  **hash** — for each new or changed file (per the rules above), read
     the first `min(1024, size)` bytes and the last `min(1024, size)`
     bytes (the two reads overlap when `size < 2048`; for `size == 0`
     hash the empty input) and compute the SHA-256 of each. Unchanged
-    files are not read and do not appear in this pass's total.
-4.  **update** — apply all insertions, updates, and deletions to the
-    database in a single transaction.
+    files are not read and do not appear in this pass's total, which
+    is therefore exact for meaningful progress and ETA.
+3.  **update** — apply the operand's insertions, updates, and
+    deletions to the database in a single transaction.
 
 Rules for the walk:
 
@@ -219,9 +229,11 @@ Rules for the walk:
     records (accepted: the database mirrors what the latest scan could
     actually verify).
 
-Concurrency: the stat and hash passes use a worker pool (`--workers`,
-default `runtime.NumCPU()`). The main goroutine owns database writes
-and progress rendering; progress display must never block the workers.
+Concurrency: the walk and hash passes use a worker pool (`--workers`,
+default `runtime.NumCPU()`); the walk parallelizes across directories,
+so raising `--workers` can speed up metadata-bound walks on busy
+pools. The main goroutine owns database writes and progress rendering;
+progress display must never block the workers.
 
 `scan` writes nothing to stdout. The summary line on stderr reports the
 files seen this run broken down by disposition, plus skips:
@@ -336,8 +348,8 @@ all dupe rows) in human units.
 Use the progress-bar library for all scan-pass progress; rendering in the
 style of `pv` is the model. All progress goes to stderr.
 
-Each scan pass gets its own bar. Required elements for the stat, hash,
-and update passes (known totals):
+Each scan pass gets its own bar, repeated per operand. Required
+elements for the hash and update passes (known totals):
 
 -   elapsed time
 -   estimated time remaining

TODO.md

@@ -14,8 +14,13 @@
 
 # Next Step
 
-- convert Makefile targets to scripts-to-rule-them-all `script/`
-  entrypoints like the other managed repos
+- parallel walk (branch `parallel-walk`): the walk pass is a single
+  goroutine and takes hours at ~20M files on a busy pool (observed:
+  22M files in 4h on a ZFS server); make it a per-directory
+  worker-pool traversal that records size/mtime during the walk
+  (folding away the separate stat pass, halving metadata I/O), and
+  commit each `PATH` operand in its own transaction so an interrupted
+  scan keeps completed operands
 
 # Completed Steps
 
@@ -44,6 +49,8 @@
 
 # Future Steps
 
+- convert Makefile targets to scripts-to-rule-them-all `script/`
+  entrypoints like the other managed repos
 - possible later features (explicitly out of scope per README):
   full-content verification of candidates, removal-script helpers