Make phases scan-wide, walk operands concurrently, batch updates
All checks were successful
check / check (push) Successful in 5s
All checks were successful
check / check (push) Successful in 5s
All PATH operands belong to a single scan: every operand seeds the shared walk worker pool, and each pass (walk, stat, hash, update) runs exactly once over the whole scan, so pass totals, percentages, and ETAs are scan-global. The per-operand walk/hash/update cycles and their stderr operand announcements are gone; duplicate paths from overlapping operands are deduplicated before stat. The update pass now commits in batched transactions (10k changes per batch) instead of one scan-wide transaction: the filesystem is authoritative and the database is an eventually-consistent reflection of it, so scan-level atomicity buys nothing, while batches keep the WAL small and let concurrent reports observe progress.
This commit is contained in:
64
README.md
64
README.md
@@ -130,12 +130,13 @@ All three subcommands operate on a single SQLite database file:
|
||||
database file is a fatal error (exit 1) telling the user to run
|
||||
`scan` first.
|
||||
- The database uses WAL journal mode and a busy timeout, so running a
|
||||
report while a cron `scan` is in progress is safe; the reports see
|
||||
the last committed state. Each `PATH` operand's changes are
|
||||
committed in a single transaction when that operand completes, so
|
||||
a report never observes a half-finished operand, and a scan that
|
||||
dies partway keeps every operand completed so far (the interrupted
|
||||
operand's changes are lost, not corrupted).
|
||||
report while a cron `scan` is in progress is safe. The filesystem
|
||||
is authoritative; the database is an eventually-consistent
|
||||
reflection of it. The update pass applies changes in batched
|
||||
transactions (keeping the WAL small and letting concurrent reports
|
||||
observe progress), so a report may see a scan's changes partially
|
||||
applied, and a scan that dies partway leaves a valid database that
|
||||
the next scan converges toward the filesystem.
|
||||
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
|
||||
database with any other version is a fatal error):
|
||||
|
||||
@@ -162,11 +163,12 @@ or a regular file; an operand that does not exist is a fatal error
|
||||
(exit 1). Because database records persist between runs and are keyed
|
||||
by absolute path, each operand is resolved to an absolute, lexically
|
||||
cleaned path (symlinks are not resolved) before walking, so results do
|
||||
not depend on the working directory. Operands are processed in the
|
||||
order given, each one walked and committed independently; overlapping
|
||||
operands (one containing another) are harmless — a file reached via
|
||||
multiple operands produces one database record (later operands see the
|
||||
records committed by earlier ones and reuse them unchanged).
|
||||
not depend on the working directory. All operands belong to a single
|
||||
scan and are enumerated concurrently: every operand seeds the shared
|
||||
walk worker pool, and every pass runs once over the whole scan, so
|
||||
pass totals and ETAs are scan-wide. Overlapping operands (one
|
||||
containing another) are harmless — a file reached via multiple
|
||||
operands is deduplicated by path and produces one database record.
|
||||
|
||||
`scan` synchronizes the database with the filesystem state under the
|
||||
scanned operands:
|
||||
@@ -188,18 +190,19 @@ scanned operands:
|
||||
disjoint trees can be scanned on different schedules into the same
|
||||
database.
|
||||
|
||||
`scan` runs **four sequential passes per operand**, committing each
|
||||
operand before starting the next. Parallelism lives strictly inside
|
||||
each pass; the passes themselves never overlap:
|
||||
`scan` runs **four sequential passes over the whole scan**.
|
||||
Parallelism lives strictly inside each pass; the passes themselves
|
||||
never overlap:
|
||||
|
||||
1. **walk** — enumerate the tree with the worker pool: each worker
|
||||
reads one directory at a time, collecting regular-file paths and
|
||||
handing discovered subdirectories back to the shared queue.
|
||||
Sequential directory enumeration is metadata-latency-bound and
|
||||
takes hours at tens of millions of files; per-directory
|
||||
parallelism is what makes the walk tractable on large or busy
|
||||
pools. Total unknown while running: show a live count, not a
|
||||
percentage.
|
||||
1. **walk** — enumerate the trees under all `PATH` operands
|
||||
concurrently with the worker pool: every operand seeds the shared
|
||||
queue, and each worker reads one directory at a time, collecting
|
||||
regular-file paths and handing discovered subdirectories back to
|
||||
the queue. Sequential directory enumeration is
|
||||
metadata-latency-bound and takes hours at tens of millions of
|
||||
files; per-directory parallelism is what makes the walk tractable
|
||||
on large or busy pools. Total unknown while running: show a live
|
||||
count, not a percentage.
|
||||
2. **stat** — `lstat` every collected path with the worker pool
|
||||
(per-file parallelism), recording size and mtime.
|
||||
3. **hash** — for each new or changed file (per the rules above), read
|
||||
@@ -208,17 +211,8 @@ each pass; the passes themselves never overlap:
|
||||
hash the empty input) and compute the SHA-256 of each. Unchanged
|
||||
files are not read and do not appear in this pass's total, which
|
||||
is therefore exact for meaningful progress and ETA.
|
||||
4. **update** — apply the operand's insertions, updates, and
|
||||
deletions to the database in a single transaction.
|
||||
|
||||
Because every operand runs its own three passes with its own totals,
|
||||
`scan` announces each operand on stderr before starting it, so a
|
||||
multi-operand invocation (e.g. a shell glob) is not mistaken for a
|
||||
nearly-finished run:
|
||||
|
||||
```
|
||||
scan: /srv/media (operand 3 of 14)
|
||||
```
|
||||
4. **update** — apply the scan's insertions, updates, and deletions
|
||||
to the database in batched transactions.
|
||||
|
||||
Rules for the walk:
|
||||
|
||||
@@ -360,8 +354,8 @@ all dupe rows) in human units.
|
||||
Use the progress-bar library for all scan-pass progress; rendering in the
|
||||
style of `pv` is the model. All progress goes to stderr.
|
||||
|
||||
Each scan pass gets its own bar, repeated per operand. Required
|
||||
elements for the stat, hash, and update passes (known totals):
|
||||
Each scan pass gets its own bar. Required elements for the stat,
|
||||
hash, and update passes (known totals):
|
||||
|
||||
- elapsed time
|
||||
- estimated time remaining
|
||||
|
||||
Reference in New Issue
Block a user