Specify parallel per-directory walk and per-operand commits

The walk pass is a single goroutine; on a busy ZFS pool it manages
only a few thousand directory entries per second and takes hours at
~20M files. Respecify it as a worker-pool traversal that reads
directories concurrently and records size/mtime during the walk,
folding away the separate stat pass and halving metadata I/O. Each
PATH operand now commits in its own transaction so an interrupted
scan keeps the operands completed so far.
This commit is contained in:
2026-07-24 06:28:12 +07:00
parent abea945730
commit 3ebf98940a
2 changed files with 42 additions and 23 deletions

11
TODO.md
View File

@@ -14,8 +14,13 @@
# Next Step
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos
- parallel walk (branch `parallel-walk`): the walk pass is a single
goroutine and takes hours at ~20M files on a busy pool (observed:
22M files in 4h on a ZFS server); make it a per-directory
worker-pool traversal that records size/mtime during the walk
(folding away the separate stat pass, halving metadata I/O), and
commit each `PATH` operand in its own transaction so an interrupted
scan keeps completed operands
# Completed Steps
@@ -44,6 +49,8 @@
# Future Steps
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos
- possible later features (explicitly out of scope per README):
full-content verification of candidates, removal-script helpers