Split the stat pass back out of the walk
All checks were successful
check / check (push) Successful in 4s
All checks were successful
check / check (push) Successful in 4s
Phases are strictly sequential again — walk, stat, hash, update per operand — with parallelism only inside each phase. The walk enumerates paths with per-directory workers (no lstat of file entries); the stat pass lstats every collected path with per-file workers, restoring its exact-total/ETA progress bar and per-file parallelism inside wide flat directories.
This commit is contained in:
33
README.md
33
README.md
@@ -188,25 +188,27 @@ scanned operands:
|
||||
disjoint trees can be scanned on different schedules into the same
|
||||
database.
|
||||
|
||||
`scan` runs **three sequential passes per operand**, committing each
|
||||
operand before starting the next:
|
||||
`scan` runs **four sequential passes per operand**, committing each
|
||||
operand before starting the next. Parallelism lives strictly inside
|
||||
each pass; the passes themselves never overlap:
|
||||
|
||||
1. **walk** — enumerate the tree with the worker pool: each worker
|
||||
reads one directory at a time, records size and mtime for every
|
||||
regular-file entry (`lstat` while the directory is fresh in
|
||||
cache), and hands discovered subdirectories back to the shared
|
||||
queue. Sequential directory enumeration is metadata-latency-bound
|
||||
and takes hours at tens of millions of files; per-directory
|
||||
reads one directory at a time, collecting regular-file paths and
|
||||
handing discovered subdirectories back to the shared queue.
|
||||
Sequential directory enumeration is metadata-latency-bound and
|
||||
takes hours at tens of millions of files; per-directory
|
||||
parallelism is what makes the walk tractable on large or busy
|
||||
pools. Total unknown while running: show a live count, not a
|
||||
percentage.
|
||||
2. **hash** — for each new or changed file (per the rules above), read
|
||||
2. **stat** — `lstat` every collected path with the worker pool
|
||||
(per-file parallelism), recording size and mtime.
|
||||
3. **hash** — for each new or changed file (per the rules above), read
|
||||
the first `min(1024, size)` bytes and the last `min(1024, size)`
|
||||
bytes (the two reads overlap when `size < 2048`; for `size == 0`
|
||||
hash the empty input) and compute the SHA-256 of each. Unchanged
|
||||
files are not read and do not appear in this pass's total, which
|
||||
is therefore exact for meaningful progress and ETA.
|
||||
3. **update** — apply the operand's insertions, updates, and
|
||||
4. **update** — apply the operand's insertions, updates, and
|
||||
deletions to the database in a single transaction.
|
||||
|
||||
Because every operand runs its own three passes with its own totals,
|
||||
@@ -238,11 +240,12 @@ Rules for the walk:
|
||||
records (accepted: the database mirrors what the latest scan could
|
||||
actually verify).
|
||||
|
||||
Concurrency: the walk and hash passes use a worker pool (`--workers`,
|
||||
default `runtime.NumCPU()`); the walk parallelizes across directories,
|
||||
so raising `--workers` can speed up metadata-bound walks on busy
|
||||
pools. The main goroutine owns database writes and progress rendering;
|
||||
progress display must never block the workers.
|
||||
Concurrency: the walk, stat, and hash passes each use a worker pool
|
||||
(`--workers`, default `runtime.NumCPU()`); the walk parallelizes
|
||||
across directories, stat and hash across files, so raising
|
||||
`--workers` can speed up metadata-bound passes on busy pools. The
|
||||
main goroutine owns database writes and progress rendering; progress
|
||||
display must never block the workers.
|
||||
|
||||
`scan` writes nothing to stdout. The summary line on stderr reports the
|
||||
files seen this run broken down by disposition, plus skips:
|
||||
@@ -358,7 +361,7 @@ Use the progress-bar library for all scan-pass progress; rendering in the
|
||||
style of `pv` is the model. All progress goes to stderr.
|
||||
|
||||
Each scan pass gets its own bar, repeated per operand. Required
|
||||
elements for the hash and update passes (known totals):
|
||||
elements for the stat, hash, and update passes (known totals):
|
||||
|
||||
- elapsed time
|
||||
- estimated time remaining
|
||||
|
||||
Reference in New Issue
Block a user