Specify parallel per-directory walk and per-operand commits
The walk pass is a single goroutine; on a busy ZFS pool it manages only a few thousand directory entries per second and takes hours at ~20M files. Respecify it as a worker-pool traversal that reads directories concurrently and records size/mtime during the walk, folding away the separate stat pass and halving metadata I/O. Each PATH operand now commits in its own transaction so an interrupted scan keeps the operands completed so far.
This commit is contained in:
11
TODO.md
11
TODO.md
@@ -14,8 +14,13 @@
|
||||
|
||||
# Next Step
|
||||
|
||||
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
||||
entrypoints like the other managed repos
|
||||
- parallel walk (branch `parallel-walk`): the walk pass is a single
|
||||
goroutine and takes hours at ~20M files on a busy pool (observed:
|
||||
22M files in 4h on a ZFS server); make it a per-directory
|
||||
worker-pool traversal that records size/mtime during the walk
|
||||
(folding away the separate stat pass, halving metadata I/O), and
|
||||
commit each `PATH` operand in its own transaction so an interrupted
|
||||
scan keeps completed operands
|
||||
|
||||
# Completed Steps
|
||||
|
||||
@@ -44,6 +49,8 @@
|
||||
|
||||
# Future Steps
|
||||
|
||||
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
||||
entrypoints like the other managed repos
|
||||
- possible later features (explicitly out of scope per README):
|
||||
full-content verification of candidates, removal-script helpers
|
||||
|
||||
|
||||
Reference in New Issue
Block a user