Make phases scan-wide, walk operands concurrently, batch updates
All checks were successful
check / check (push) Successful in 5s

All PATH operands belong to a single scan: every operand seeds the
shared walk worker pool, and each pass (walk, stat, hash, update)
runs exactly once over the whole scan, so pass totals, percentages,
and ETAs are scan-global. The per-operand walk/hash/update cycles and
their stderr operand announcements are gone; duplicate paths from
overlapping operands are deduplicated before stat.

The update pass now commits in batched transactions (10k changes per
batch) instead of one scan-wide transaction: the filesystem is
authoritative and the database is an eventually-consistent reflection
of it, so scan-level atomicity buys nothing, while batches keep the
WAL small and let concurrent reports observe progress.
This commit is contained in:
2026-07-24 10:26:52 +07:00
parent 732fc351d7
commit 3ecf73c80a
6 changed files with 229 additions and 95 deletions

View File

@@ -130,12 +130,13 @@ All three subcommands operate on a single SQLite database file:
database file is a fatal error (exit 1) telling the user to run
`scan` first.
- The database uses WAL journal mode and a busy timeout, so running a
report while a cron `scan` is in progress is safe; the reports see
the last committed state. Each `PATH` operand's changes are
committed in a single transaction when that operand completes, so
a report never observes a half-finished operand, and a scan that
dies partway keeps every operand completed so far (the interrupted
operand's changes are lost, not corrupted).
report while a cron `scan` is in progress is safe. The filesystem
is authoritative; the database is an eventually-consistent
reflection of it. The update pass applies changes in batched
transactions (keeping the WAL small and letting concurrent reports
observe progress), so a report may see a scan's changes partially
applied, and a scan that dies partway leaves a valid database that
the next scan converges toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error):
@@ -162,11 +163,12 @@ or a regular file; an operand that does not exist is a fatal error
(exit 1). Because database records persist between runs and are keyed
by absolute path, each operand is resolved to an absolute, lexically
cleaned path (symlinks are not resolved) before walking, so results do
not depend on the working directory. Operands are processed in the
order given, each one walked and committed independently; overlapping
operands (one containing another) are harmless — a file reached via
multiple operands produces one database record (later operands see the
records committed by earlier ones and reuse them unchanged).
not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared
walk worker pool, and every pass runs once over the whole scan, so
pass totals and ETAs are scan-wide. Overlapping operands (one
containing another) are harmless — a file reached via multiple
operands is deduplicated by path and produces one database record.
`scan` synchronizes the database with the filesystem state under the
scanned operands:
@@ -188,18 +190,19 @@ scanned operands:
disjoint trees can be scanned on different schedules into the same
database.
`scan` runs **four sequential passes per operand**, committing each
operand before starting the next. Parallelism lives strictly inside
each pass; the passes themselves never overlap:
`scan` runs **four sequential passes over the whole scan**.
Parallelism lives strictly inside each pass; the passes themselves
never overlap:
1. **walk** — enumerate the tree with the worker pool: each worker
reads one directory at a time, collecting regular-file paths and
handing discovered subdirectories back to the shared queue.
Sequential directory enumeration is metadata-latency-bound and
takes hours at tens of millions of files; per-directory
parallelism is what makes the walk tractable on large or busy
pools. Total unknown while running: show a live count, not a
percentage.
1. **walk** — enumerate the trees under all `PATH` operands
concurrently with the worker pool: every operand seeds the shared
queue, and each worker reads one directory at a time, collecting
regular-file paths and handing discovered subdirectories back to
the queue. Sequential directory enumeration is
metadata-latency-bound and takes hours at tens of millions of
files; per-directory parallelism is what makes the walk tractable
on large or busy pools. Total unknown while running: show a live
count, not a percentage.
2. **stat** — `lstat` every collected path with the worker pool
(per-file parallelism), recording size and mtime.
3. **hash** — for each new or changed file (per the rules above), read
@@ -208,17 +211,8 @@ each pass; the passes themselves never overlap:
hash the empty input) and compute the SHA-256 of each. Unchanged
files are not read and do not appear in this pass's total, which
is therefore exact for meaningful progress and ETA.
4. **update** — apply the operand's insertions, updates, and
deletions to the database in a single transaction.
Because every operand runs its own three passes with its own totals,
`scan` announces each operand on stderr before starting it, so a
multi-operand invocation (e.g. a shell glob) is not mistaken for a
nearly-finished run:
```
scan: /srv/media (operand 3 of 14)
```
4. **update** — apply the scan's insertions, updates, and deletions
to the database in batched transactions.
Rules for the walk:
@@ -360,8 +354,8 @@ all dupe rows) in human units.
Use the progress-bar library for all scan-pass progress; rendering in the
style of `pv` is the model. All progress goes to stderr.
Each scan pass gets its own bar, repeated per operand. Required
elements for the stat, hash, and update passes (known totals):
Each scan pass gets its own bar. Required elements for the stat,
hash, and update passes (known totals):
- elapsed time
- estimated time remaining