Split the stat pass back out of the walk
All checks were successful
check / check (push) Successful in 4s

Phases are strictly sequential again — walk, stat, hash, update per
operand — with parallelism only inside each phase. The walk
enumerates paths with per-directory workers (no lstat of file
entries); the stat pass lstats every collected path with per-file
workers, restoring its exact-total/ETA progress bar and per-file
parallelism inside wide flat directories.
This commit is contained in:
2026-07-24 08:12:47 +07:00
parent 09ff9b5f30
commit 1e7a519608
4 changed files with 153 additions and 105 deletions

View File

@@ -188,25 +188,27 @@ scanned operands:
disjoint trees can be scanned on different schedules into the same
database.
`scan` runs **three sequential passes per operand**, committing each
operand before starting the next:
`scan` runs **four sequential passes per operand**, committing each
operand before starting the next. Parallelism lives strictly inside
each pass; the passes themselves never overlap:
1. **walk** — enumerate the tree with the worker pool: each worker
reads one directory at a time, records size and mtime for every
regular-file entry (`lstat` while the directory is fresh in
cache), and hands discovered subdirectories back to the shared
queue. Sequential directory enumeration is metadata-latency-bound
and takes hours at tens of millions of files; per-directory
reads one directory at a time, collecting regular-file paths and
handing discovered subdirectories back to the shared queue.
Sequential directory enumeration is metadata-latency-bound and
takes hours at tens of millions of files; per-directory
parallelism is what makes the walk tractable on large or busy
pools. Total unknown while running: show a live count, not a
percentage.
2. **hash** — for each new or changed file (per the rules above), read
2. **stat** — `lstat` every collected path with the worker pool
(per-file parallelism), recording size and mtime.
3. **hash** — for each new or changed file (per the rules above), read
the first `min(1024, size)` bytes and the last `min(1024, size)`
bytes (the two reads overlap when `size < 2048`; for `size == 0`
hash the empty input) and compute the SHA-256 of each. Unchanged
files are not read and do not appear in this pass's total, which
is therefore exact for meaningful progress and ETA.
3. **update** — apply the operand's insertions, updates, and
4. **update** — apply the operand's insertions, updates, and
deletions to the database in a single transaction.
Because every operand runs its own three passes with its own totals,
@@ -238,11 +240,12 @@ Rules for the walk:
records (accepted: the database mirrors what the latest scan could
actually verify).
Concurrency: the walk and hash passes use a worker pool (`--workers`,
default `runtime.NumCPU()`); the walk parallelizes across directories,
so raising `--workers` can speed up metadata-bound walks on busy
pools. The main goroutine owns database writes and progress rendering;
progress display must never block the workers.
Concurrency: the walk, stat, and hash passes each use a worker pool
(`--workers`, default `runtime.NumCPU()`); the walk parallelizes
across directories, stat and hash across files, so raising
`--workers` can speed up metadata-bound passes on busy pools. The
main goroutine owns database writes and progress rendering; progress
display must never block the workers.
`scan` writes nothing to stdout. The summary line on stderr reports the
files seen this run broken down by disposition, plus skips:
@@ -358,7 +361,7 @@ Use the progress-bar library for all scan-pass progress; rendering in the
style of `pv` is the model. All progress goes to stderr.
Each scan pass gets its own bar, repeated per operand. Required
elements for the hash and update passes (known totals):
elements for the stat, hash, and update passes (known totals):
- elapsed time
- estimated time remaining