Skip removed paths silently and fix the progress line (closes #133)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
A path removed while the scan runs gets no warning and no skipped count, in the walk or the hash phase. If the first of several hard links is gone, the next is read. Other errors are still warned about. On a terminal every display is the library's spinner, then the line. A phase with an exact total ends the padded line with its percent and a bar at a fixed column, drawn by sfdupes, as the library shows no spinner beside its own bar. A too-wide line loses its rate first. The time remaining is labelled remain, not eta. Judgement call: an operand removed mid-scan is skipped silently too. Deviation: README's stalled-line example keeps its words on 100 columns, not 80. Model: opus-5-5
This commit was merged in pull request #139.
This commit is contained in:
@@ -525,11 +525,11 @@ hash phase:
|
||||
Files are hashed in **inode order**, which minimizes seeks on spinning
|
||||
disks. Paths that are hard links to the same inode are **read once**, and
|
||||
all share the one result. A hard-link backup farm therefore costs one read
|
||||
per inode, not per path. The phase total counts actual reads, so progress
|
||||
and ETA are meaningful. Completed records are committed in batched
|
||||
transactions **while hashing runs**. A scan interrupted after hours
|
||||
therefore keeps everything hashed so far, and the next scan resumes cheaply,
|
||||
skipping records already written.
|
||||
per inode, not per path. The phase total counts actual reads, so its
|
||||
progress and its time remaining are meaningful. Completed records are
|
||||
committed in batched transactions **while hashing runs**. A scan interrupted
|
||||
after hours therefore keeps everything hashed so far, and the next scan
|
||||
resumes cheaply, skipping records already written.
|
||||
4. **update** — commit the final partial batch of hashed records. Then delete
|
||||
the records the scan did not verify: those of vanished files, and of paths
|
||||
that failed to stat or hash.
|
||||
@@ -625,14 +625,19 @@ Rules for the walk:
|
||||
as described in "`scan` mode" above: they are not deleted. That does not
|
||||
hold below an operand at or inside it.
|
||||
|
||||
- On any per-path error (permission denied, file vanished between passes,
|
||||
unreadable): print a one-line warning to stderr, skip the path, and continue.
|
||||
Per-file errors never abort the run; the final summary reports how many were
|
||||
skipped. As specified above, a skipped path that has a database record from an
|
||||
earlier scan loses that record. An unreadable directory subtree likewise loses
|
||||
its records. This is accepted: the database mirrors what the latest scan could
|
||||
actually verify. The exception is an operand dropped before the database was
|
||||
read, whose record counts as outside the scanned operands.
|
||||
- On a per-path error, such as permission denied or an unreadable file: print a
|
||||
one-line warning to stderr, skip the path, and continue. Per-file errors never
|
||||
abort the run; the final summary reports how many were skipped. As specified
|
||||
above, a skipped path that has a database record from an earlier scan loses
|
||||
that record. An unreadable directory subtree likewise loses its records. This
|
||||
is accepted: the database mirrors what the latest scan could actually verify.
|
||||
The exception is an operand dropped before the database was read, whose record
|
||||
counts as outside the scanned operands.
|
||||
- A file or directory removed while the scan runs is not an error. The scan
|
||||
prints no line for it and does not count it as skipped. A path gone when the
|
||||
walk reaches it gets no record. A file gone when the hash phase opens it gets
|
||||
none either. The records an earlier scan left at or below such a path are
|
||||
deleted.
|
||||
|
||||
Concurrency: the walk phase, which also stats files, and the hash phase each use
|
||||
a worker pool of `--workers` workers (default `runtime.NumCPU()`). The walk
|
||||
@@ -822,9 +827,9 @@ Use the progress-bar library for all progress of `scan`, `report`, `trees` and
|
||||
stderr.
|
||||
|
||||
`scan`, `report`, `trees` and `info` must always show that they are working and
|
||||
not hung. Each phase gets its own display from the moment it starts, naming the
|
||||
phase and its place among the command's phases and showing its progress, its
|
||||
elapsed time and its estimated time remaining (ETA).
|
||||
not hung. Each phase gets its own display from the moment it starts. The display
|
||||
names the phase and its place among the command's phases. It shows the phase's
|
||||
progress, its elapsed time and its time remaining.
|
||||
|
||||
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
|
||||
`[4/4] update`):
|
||||
@@ -832,11 +837,11 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
|
||||
- `load` counts the records first, then reads them against that count. While the
|
||||
count runs, the display says it is counting the records.
|
||||
- `walk` has no exact total. Its estimate is the number of records the last scan
|
||||
left under the operands, known once `load` is done and shown with a `~`. Once
|
||||
the walk passes the estimate, or when there are no such records, as on a first
|
||||
scan, the ETA is unknown.
|
||||
left under the operands, known once `load` is done and shown with a `~`. The
|
||||
time remaining is unknown once the walk passes the estimate, and when there
|
||||
are no such records, as on a first scan.
|
||||
- `hash` and `update` have exact totals. Only files that actually need hashing
|
||||
appear in the hash total, so its ETA is meaningful.
|
||||
appear in the hash total, so its time remaining is meaningful.
|
||||
|
||||
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
|
||||
`[4/4] write`, see "`trees` mode"):
|
||||
@@ -861,8 +866,8 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
|
||||
- `open` has no total. While it opens the database, waiting on a busy one
|
||||
included, the display says it is opening the database.
|
||||
- `count` has no total either, because one query counts everything: the display
|
||||
says it is counting the records until the query returns, and its ETA is
|
||||
unknown.
|
||||
says it is counting the records until the query returns, and its time
|
||||
remaining is unknown.
|
||||
|
||||
Each display is one line holding:
|
||||
|
||||
@@ -875,16 +880,17 @@ Each display is one line holding:
|
||||
while nothing completes, and a burst of items after a wait shows as a burst
|
||||
rather than as a steady rate. Once no item has completed for 10 seconds, the
|
||||
word `stalled` and where the phase waits take the rate's place (see below).
|
||||
- elapsed time
|
||||
- estimated time remaining, or `unknown`. It comes from the phase's average rate
|
||||
since it started counting, so that it does not swing with each burst.
|
||||
- elapsed time, labelled `elapsed`
|
||||
- time remaining, labelled `remain`: a duration such as `12m9s`, or `unknown`.
|
||||
It comes from the phase's average rate since it started counting, so that it
|
||||
does not swing with each burst.
|
||||
|
||||
Example shape (exact layout is flexible, content is not):
|
||||
Example lines, as printed when stderr is not a terminal:
|
||||
|
||||
```
|
||||
[3/4] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
|
||||
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
|
||||
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
|
||||
[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s
|
||||
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown
|
||||
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s
|
||||
```
|
||||
|
||||
When no item has completed for 10 seconds, as while one large file is read, a
|
||||
@@ -900,27 +906,43 @@ for its total already says what it waits for, and says nothing more.
|
||||
|
||||
Additional requirements:
|
||||
|
||||
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
|
||||
the `open` phase of `report` and `trees`, and both phases of `info`, once it
|
||||
has counted what it will do) is drawn as its line followed by the progress-bar
|
||||
library's bar, 10 columns wide, which shows the percent in place of the line's
|
||||
own. Any other phase, and a phase whose bar has no room, is drawn as the
|
||||
library's spinner followed by its line. Each display is drawn again several
|
||||
times a second, so its elapsed time keeps moving while a phase waits for its
|
||||
next item.
|
||||
- On a terminal, every display starts with the progress-bar library's spinner in
|
||||
its first column, followed by its line. Each display is drawn again several
|
||||
times a second, so its spinner turns and its elapsed time keeps moving while a
|
||||
phase waits for its next item.
|
||||
- On a terminal, a phase with an exact total ends its line with its percent and
|
||||
a bar 10 columns wide. This percent takes the place of the line's own. The bar
|
||||
stays at the terminal's right edge, so it does not move as the line grows or
|
||||
shrinks. A phase with nothing to do shows no bar. Every phase has an exact
|
||||
total once it has counted what it will do, except these:
|
||||
- `scan`'s `walk`;
|
||||
- the `open` phase of `report` and `trees`;
|
||||
- both phases of `info`.
|
||||
|
||||
On a terminal 100 columns wide:
|
||||
|
||||
```
|
||||
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
||||
```
|
||||
|
||||
- On a terminal no line reaches the terminal's last column, however narrow the
|
||||
terminal is. A line too wide for the terminal first loses its bar, which gives
|
||||
way to the spinner, then its rate, the words of a stalled phase, or what the
|
||||
phase is waiting for. The phase and its place, the progress, the elapsed time
|
||||
and the ETA are kept whole. On an 80-column terminal, at counts like those in
|
||||
the example above, a stalled line keeps its words whenever the elapsed time
|
||||
and the ETA are both in minutes. Where the words and the ETA do not both fit,
|
||||
as at millions of files and times in hours, the words give way. Only when
|
||||
those four alone are too wide, as on a terminal narrower than about 70 columns
|
||||
hours into a scan of millions of files, is the line cut short at the
|
||||
terminal's edge, so the end of the ETA is lost first. A terminal made narrower
|
||||
while a phase runs gets the same: the next draw erases the line within the new
|
||||
width and fits the line to it, and the display goes on redrawing in place.
|
||||
terminal is. A line too wide for the terminal first loses whichever of these
|
||||
it shows: its rate, the words of a stalled phase, or what the phase is waiting
|
||||
for. The other four parts are kept whole:
|
||||
- the phase and its place;
|
||||
- the progress;
|
||||
- the elapsed time;
|
||||
- the time remaining.
|
||||
|
||||
The line loses its bar only when those four and the bar do not fit. It is
|
||||
cut short at the terminal's edge only when those four alone are too wide,
|
||||
and then the end of the time remaining is lost first. That happens hours
|
||||
into a scan of millions of files, on a terminal narrower than about 75
|
||||
columns. On a terminal 100 columns wide, the stalled line of the example
|
||||
above keeps its words and its bar. The next draw fits the line to a terminal
|
||||
made narrower while a phase runs. It erases the old line within the new
|
||||
width, and the display goes on redrawing in place.
|
||||
|
||||
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
|
||||
redraws: print the line the moment each phase starts, every 5 seconds while it
|
||||
runs (whether or not an item has completed, and never more often), and once
|
||||
|
||||
Reference in New Issue
Block a user