Specify parallel per-directory walk and per-operand commits

The walk pass is a single goroutine; on a busy ZFS pool it manages
only a few thousand directory entries per second and takes hours at
~20M files. Respecify it as a worker-pool traversal that reads
directories concurrently and records size/mtime during the walk,
folding away the separate stat pass and halving metadata I/O. Each
PATH operand now commits in its own transaction so an interrupted
scan keeps the operands completed so far.

README.md

@@ -131,9 +131,11 @@ All three subcommands operate on a single SQLite database file:
     `scan` first.
 -   The database uses WAL journal mode and a busy timeout, so running a
     report while a cron `scan` is in progress is safe; the reports see
-    the last committed state. Each scan commits its changes in a single
+    the last committed state. Each `PATH` operand's changes are
-    transaction, so a report never observes a half-finished scan and a
+    committed in a single transaction when that operand completes, so
-    scan that dies partway leaves the previous state intact.
+    a report never observes a half-finished operand, and a scan that
+    dies partway keeps every operand completed so far (the interrupted
+    operand's changes are lost, not corrupted).
 -   Schema (`PRAGMA user_version` is the schema version, currently 1; a
     database with any other version is a fatal error):
 
@@ -160,9 +162,11 @@ or a regular file; an operand that does not exist is a fatal error
 (exit 1). Because database records persist between runs and are keyed
 by absolute path, each operand is resolved to an absolute, lexically
 cleaned path (symlinks are not resolved) before walking, so results do
-not depend on the working directory. Operands are walked in the order
+not depend on the working directory. Operands are processed in the
-given; overlapping operands (one containing another) are harmless — a
+order given, each one walked and committed independently; overlapping
-file reached via multiple operands produces one database record.
+operands (one containing another) are harmless — a file reached via
+multiple operands produces one database record (later operands see the
+records committed by earlier ones and reuse them unchanged).
 
 `scan` synchronizes the database with the filesystem state under the
 scanned operands:
@@ -184,20 +188,26 @@ scanned operands:
     disjoint trees can be scanned on different schedules into the same
     database.
 
-`scan` runs **four sequential passes**, in this order, so that every
+`scan` runs **three sequential passes per operand**, committing each
-expensive pass has an exact total for meaningful progress and ETA:
+operand before starting the next:
 
-1.  **walk** — recursively enumerate the tree under each `PATH` in
+1.  **walk** — enumerate the tree with the worker pool: each worker
-    turn, collecting the list of regular-file paths. Total unknown
+    reads one directory at a time, records size and mtime for every
-    while running: show a live count, not a percentage.
+    regular-file entry (`lstat` while the directory is fresh in
-2.  **stat** — `lstat` every collected path, recording size and mtime.
+    cache), and hands discovered subdirectories back to the shared
-3.  **hash** — for each new or changed file (per the rules above), read
+    queue. Sequential directory enumeration is metadata-latency-bound
+    and takes hours at tens of millions of files; per-directory
+    parallelism is what makes the walk tractable on large or busy
+    pools. Total unknown while running: show a live count, not a
+    percentage.
+2.  **hash** — for each new or changed file (per the rules above), read
     the first `min(1024, size)` bytes and the last `min(1024, size)`
     bytes (the two reads overlap when `size < 2048`; for `size == 0`
     hash the empty input) and compute the SHA-256 of each. Unchanged
-    files are not read and do not appear in this pass's total.
+    files are not read and do not appear in this pass's total, which
-4.  **update** — apply all insertions, updates, and deletions to the
+    is therefore exact for meaningful progress and ETA.
-    database in a single transaction.
+3.  **update** — apply the operand's insertions, updates, and
+    deletions to the database in a single transaction.
 
 Rules for the walk:
 
@@ -219,9 +229,11 @@ Rules for the walk:
     records (accepted: the database mirrors what the latest scan could
     actually verify).
 
-Concurrency: the stat and hash passes use a worker pool (`--workers`,
+Concurrency: the walk and hash passes use a worker pool (`--workers`,
-default `runtime.NumCPU()`). The main goroutine owns database writes
+default `runtime.NumCPU()`); the walk parallelizes across directories,
-and progress rendering; progress display must never block the workers.
+so raising `--workers` can speed up metadata-bound walks on busy
+pools. The main goroutine owns database writes and progress rendering;
+progress display must never block the workers.
 
 `scan` writes nothing to stdout. The summary line on stderr reports the
 files seen this run broken down by disposition, plus skips:
@@ -336,8 +348,8 @@ all dupe rows) in human units.
 Use the progress-bar library for all scan-pass progress; rendering in the
 style of `pv` is the model. All progress goes to stderr.
 
-Each scan pass gets its own bar. Required elements for the stat, hash,
+Each scan pass gets its own bar, repeated per operand. Required
-and update passes (known totals):
+elements for the hash and update passes (known totals):
 
 -   elapsed time
 -   estimated time remaining

TODO.md

@@ -14,8 +14,13 @@
 
 # Next Step
 
-- convert Makefile targets to scripts-to-rule-them-all `script/`
+- parallel walk (branch `parallel-walk`): the walk pass is a single
-  entrypoints like the other managed repos
+  goroutine and takes hours at ~20M files on a busy pool (observed:
+  22M files in 4h on a ZFS server); make it a per-directory
+  worker-pool traversal that records size/mtime during the walk
+  (folding away the separate stat pass, halving metadata I/O), and
+  commit each `PATH` operand in its own transaction so an interrupted
+  scan keeps completed operands
 
 # Completed Steps
 
@@ -44,6 +49,8 @@
 
 # Future Steps
 
+- convert Makefile targets to scripts-to-rule-them-all `script/`
+  entrypoints like the other managed repos
 - possible later features (explicitly out of scope per README):
   full-content verification of candidates, removal-script helpers