Parallelize the walk and commit per operand
All checks were successful
check / check (push) Successful in 5s
All checks were successful
check / check (push) Successful in 5s
Replace the single-goroutine WalkDir traversal with a per-directory worker pool: workers read directories concurrently and lstat entries while each directory is fresh in cache, recording size and mtime during the walk. This folds the separate stat pass away (halving metadata I/O per run) and overlaps metadata latency, which dominated on busy pools — a sequential walk of a ~22M-file tree was observed taking over 4 hours. Each PATH operand now loads its scope, walks, hashes, and commits in its own transaction, so an interrupted scan keeps every operand completed so far; a later overlapping operand sees the records committed by earlier ones and reuses them unchanged.
This commit is contained in:
19
TODO.md
19
TODO.md
@@ -14,16 +14,19 @@
|
||||
|
||||
# Next Step
|
||||
|
||||
- parallel walk (branch `parallel-walk`): the walk pass is a single
|
||||
goroutine and takes hours at ~20M files on a busy pool (observed:
|
||||
22M files in 4h on a ZFS server); make it a per-directory
|
||||
worker-pool traversal that records size/mtime during the walk
|
||||
(folding away the separate stat pass, halving metadata I/O), and
|
||||
commit each `PATH` operand in its own transaction so an interrupted
|
||||
scan keeps completed operands
|
||||
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
||||
entrypoints like the other managed repos
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
|
||||
was a single goroutine and took hours at ~20M files on a busy pool
|
||||
(observed: 22M files in 4h on a ZFS server); it is now a
|
||||
per-directory worker-pool traversal that records size/mtime during
|
||||
the walk (folding away the separate stat pass, halving metadata
|
||||
I/O), and each `PATH` operand commits in its own transaction so an
|
||||
interrupted scan keeps completed operands
|
||||
|
||||
- persistent scan database (2026-07-24, branch `persistent-database`):
|
||||
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
|
||||
Go, cgo stays disabled) keyed by absolute path that survives between
|
||||
@@ -49,8 +52,6 @@
|
||||
|
||||
# Future Steps
|
||||
|
||||
- convert Makefile targets to scripts-to-rule-them-all `script/`
|
||||
entrypoints like the other managed repos
|
||||
- possible later features (explicitly out of scope per README):
|
||||
full-content verification of candidates, removal-script helpers
|
||||
|
||||
|
||||
Reference in New Issue
Block a user