Parallelize the walk and commit per operand
All checks were successful
check / check (push) Successful in 5s

Replace the single-goroutine WalkDir traversal with a per-directory
worker pool: workers read directories concurrently and lstat entries
while each directory is fresh in cache, recording size and mtime
during the walk. This folds the separate stat pass away (halving
metadata I/O per run) and overlaps metadata latency, which dominated
on busy pools — a sequential walk of a ~22M-file tree was observed
taking over 4 hours.

Each PATH operand now loads its scope, walks, hashes, and commits in
its own transaction, so an interrupted scan keeps every operand
completed so far; a later overlapping operand sees the records
committed by earlier ones and reuses them unchanged.
This commit is contained in:
2026-07-24 07:44:18 +07:00
parent 3ebf98940a
commit dced5cf0d2
3 changed files with 344 additions and 271 deletions

19
TODO.md
View File

@@ -14,16 +14,19 @@
# Next Step
- parallel walk (branch `parallel-walk`): the walk pass is a single
goroutine and takes hours at ~20M files on a busy pool (observed:
22M files in 4h on a ZFS server); make it a per-directory
worker-pool traversal that records size/mtime during the walk
(folding away the separate stat pass, halving metadata I/O), and
commit each `PATH` operand in its own transaction so an interrupted
scan keeps completed operands
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos
# Completed Steps
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
was a single goroutine and took hours at ~20M files on a busy pool
(observed: 22M files in 4h on a ZFS server); it is now a
per-directory worker-pool traversal that records size/mtime during
the walk (folding away the separate stat pass, halving metadata
I/O), and each `PATH` operand commits in its own transaction so an
interrupted scan keeps completed operands
- persistent scan database (2026-07-24, branch `persistent-database`):
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
Go, cgo stays disabled) keyed by absolute path that survives between
@@ -49,8 +52,6 @@
# Future Steps
- convert Makefile targets to scripts-to-rule-them-all `script/`
entrypoints like the other managed repos
- possible later features (explicitly out of scope per README):
full-content verification of candidates, removal-script helpers