scan: show each phase as N of M, with progress, elapsed time and ETA, also while it waits (closes #112)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
Every scan display now starts with its phase and its place among the five, as "[2/5] walk", and always shows the rate, elapsed time and ETA. load counts the records before reading them, and the content phase counts what its search finds before checking it, so both have totals; until then the line says what the phase waits for. The walk's total is estimated from the records under the operands, with the ETA unknown past the estimate or without one. Each display draws from its own goroutine under one mutex: several times a second on a terminal, every 5 seconds in plain mode. scanPhase holds the phase list. Judgement call: on a terminal the line is the plain line behind the library's spinner; the bar graphic is gone. Judgement call: the content phase runs its search query twice, once to count its rows. Model: opus-5-5
This commit is contained in:
@@ -97,10 +97,11 @@ As a file `/etc/cron.d/sfdupes`:
|
||||
`SFDUPES_DATABASE=/path/to/db.sqlite` before the command on the same line.
|
||||
- `scan` writes nothing to stdout. Its stderr, appended here to
|
||||
`/var/log/sfdupes.log`, holds a first line naming the database (see
|
||||
"Database"), a plain progress line as each phase starts and then at most every
|
||||
5 seconds, a warning for each path it skips, and the summary line (see
|
||||
"Progress" and "`scan` mode"). The log grows with every scan; rotate it like
|
||||
any other.
|
||||
"Database"), plain progress lines naming each phase and its place among the
|
||||
five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
|
||||
runs, and one as it ends), a warning for each path it skips, and the summary
|
||||
line (see "Progress" and "`scan` mode"). The log grows with every scan; rotate
|
||||
it like any other.
|
||||
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
|
||||
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
|
||||
reason among the last lines of the log; `tail` prints them, and cron mails
|
||||
@@ -492,10 +493,15 @@ operands:
|
||||
schedule into the same database. Records below an operand at or inside such a
|
||||
directory are not kept this way, since that operand's own walk verifies them.
|
||||
|
||||
`scan` runs **four sequential phases over the whole scan**. Parallelism lives
|
||||
inside each phase; batched database writes begin during the hash phase:
|
||||
`scan` runs **five sequential phases over the whole scan**. Parallelism lives
|
||||
inside the walk, hash and content phases; batched database writes begin during
|
||||
the hash phase:
|
||||
|
||||
1. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
|
||||
1. **load** — read the existing records. Each record under the operands is kept
|
||||
in memory, by path, with its size, its mtime and whether it has hashes, for
|
||||
the walk to compare each file with; the size of every other record joins the
|
||||
size census.
|
||||
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
|
||||
with the walk worker pool: every operand seeds the shared queue, and each
|
||||
worker reads one directory at a time, handing discovered subdirectories back
|
||||
to the queue and running `lstat` on each regular file as it is discovered
|
||||
@@ -505,7 +511,7 @@ inside each phase; batched database writes begin during the hash phase:
|
||||
or busy pools. The walk builds the size census and resolves unchanged
|
||||
already-hashed files on the fly; every other file is carried to the hash
|
||||
phase as a (path, size, mtime) record.
|
||||
2. **hash** — with the census complete, each carried file's size decides its
|
||||
3. **hash** — with the census complete, each carried file's size decides its
|
||||
fate. Size-unique files are never read: new or changed ones are recorded
|
||||
without hashes in the update phase, unchanged unhashed ones simply keep
|
||||
their records. Every file with a shared size is hashed by the worker pool as
|
||||
@@ -521,10 +527,10 @@ inside each phase; batched database writes begin during the hash phase:
|
||||
**while hashing runs**, so a scan interrupted after hours keeps everything
|
||||
hashed so far and the next scan resumes cheaply, skipping records already
|
||||
written.
|
||||
3. **update** — commit the final partial batch, the hash-less records for
|
||||
4. **update** — commit the final partial batch, the hash-less records for
|
||||
size-unique new and changed files, and the deletions for records the scan
|
||||
did not verify (vanished files, plus paths that failed to stat or hash).
|
||||
4. **content** — find every record of 10 MiB or more without a `content` hash
|
||||
5. **content** — find every record of 10 MiB or more without a `content` hash
|
||||
whose size, `head`, and `tail` equal another record's, anywhere in the
|
||||
database: records from this scan and records stored by earlier scans, inside
|
||||
or outside the scanned operands. SQLite finds them, so only the records to
|
||||
@@ -578,8 +584,9 @@ files. `--workers` must be at least 1: a smaller value is a usage error,
|
||||
reported in one line on stderr with exit 2 before anything is scanned. All three
|
||||
phases are seek-bound on spinning disks, so raising `--workers` well past the
|
||||
core count can help on pools with many spindles. The main goroutine owns
|
||||
partitioning, database writes, and progress rendering; progress display must
|
||||
never block the workers.
|
||||
partitioning, database writes, and the progress counts, which each progress
|
||||
display draws from a goroutine of its own; progress display must never block the
|
||||
workers.
|
||||
|
||||
`scan` writes nothing to stdout. The summary line on stderr reports the files
|
||||
seen this run broken down by disposition, plus skips:
|
||||
@@ -711,40 +718,56 @@ units.
|
||||
Use the progress-bar library for all scan progress; rendering in the style of
|
||||
`pv` is the model. All progress goes to stderr.
|
||||
|
||||
Each phase gets its own display, rendered the moment the phase starts — a scan
|
||||
must never look hung. Loading the existing-record index (`load`) and the walk
|
||||
have no known totals while running: show a live count, rate, and elapsed time
|
||||
(spinner-style, no percentage or ETA). The content phase's display (`content`)
|
||||
starts the same way, counting the records checked while SQLite finds the files
|
||||
to read and `lstat` checks them, then shows a bar once reading starts. The hash
|
||||
and update phases, and the content phase's reads, have exact totals — only files
|
||||
that actually need hashing appear in the hash and content totals, so their ETAs
|
||||
are meaningful. Required elements for the bars with known totals:
|
||||
A scan must always show that it is working and not hung. Each phase gets its own
|
||||
display from the moment it starts, naming the phase and its place among the five
|
||||
(`[1/5] load`, `[2/5] walk`, `[3/5] hash`, `[4/5] update`, `[5/5] content`) and
|
||||
showing its progress, its elapsed time and its estimated time remaining (ETA):
|
||||
|
||||
- elapsed time
|
||||
- estimated time remaining
|
||||
- a `[m/n] x%` display (items processed / total items, percent)
|
||||
- `load` counts the records first, then reads them against that count. While the
|
||||
count runs, the display says it is counting the records.
|
||||
- `walk` has no exact total. Its estimate is the number of records the last scan
|
||||
left under the operands, known once `load` is done and shown with a `~`. Once
|
||||
the walk passes the estimate, or when there are no such records, as on a first
|
||||
scan, the ETA is unknown.
|
||||
- `hash` and `update` have exact totals. Only files that actually need hashing
|
||||
appear in the hash total, so its ETA is meaningful.
|
||||
- `content` counts the records whose files it checks with `lstat`, saying it is
|
||||
searching the database while SQLite finds them, then checks them against that
|
||||
count. The files to read then get a display of their own, with their count as
|
||||
its total.
|
||||
|
||||
Each display is one line holding:
|
||||
|
||||
- the phase and its place among the phases
|
||||
- a `[m/n] x%` display (items processed / total items, percent); before a phase
|
||||
has a total, the items processed, or what the phase is waiting for
|
||||
- current rate (items/s)
|
||||
- elapsed time
|
||||
- estimated time remaining, or `unknown`
|
||||
|
||||
Example shape (exact layout is flexible, content is not):
|
||||
|
||||
```
|
||||
hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
|
||||
[3/5] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, eta 17m54s
|
||||
[2/5] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
|
||||
```
|
||||
|
||||
Additional requirements:
|
||||
|
||||
- On a terminal the progress-bar library draws the line behind a spinner, and
|
||||
draws it again several times a second, so the elapsed time and ETA stay
|
||||
current while a phase waits for its next item.
|
||||
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
|
||||
redraws: print a plain one-line progress update the moment each phase starts,
|
||||
then no more often than every 5 seconds.
|
||||
- Progress updates are driven from the main goroutine and must be non-blocking
|
||||
with respect to the worker pool. On a terminal the spinner-style displays also
|
||||
redraw on their own several times a second, so their count and elapsed time
|
||||
stay current while a phase waits for its next item.
|
||||
redraws: print the line the moment each phase starts, every 5 seconds while it
|
||||
runs (whether or not an item has completed, and never more often), and once
|
||||
more as the phase ends.
|
||||
- Progress must be non-blocking with respect to the worker pool: the main
|
||||
goroutine counts the items, and each display draws them from a goroutine of
|
||||
its own.
|
||||
- A warning printed during a phase always lands on a line of its own, never
|
||||
inside the progress display.
|
||||
- A bar whose phase stops short of its total, as an interrupted one does, is
|
||||
left as last drawn rather than filled up.
|
||||
- A phase that stops short of its total, as an interrupted one does, is left
|
||||
showing the count it stopped at rather than filled up.
|
||||
- `report` and `trees` modes need no progress display, only their stderr
|
||||
summaries.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user