Skip removed paths silently and fix the progress line (closes #133)
check / check (push) Canceled after 0s

A path removed while the scan runs gets no warning and no skipped
count, in the walk or the hash phase. If the first of several hard
links is gone, the next is read. Other errors are still warned about.

On a terminal every display is the library's spinner, then the line. A
phase with an exact total ends the padded line with its percent and a
bar at a fixed column, drawn by sfdupes, as the library shows no
spinner beside its own bar. A too-wide line loses its rate first.

The time remaining is labelled remain, not eta.

Judgement call: an operand removed mid-scan is skipped silently too.
Deviation: README's stalled-line example keeps its words on 100
columns, not 80.

Model: opus-5-5
This commit was merged in pull request #139.
This commit is contained in:
2026-10-26 06:32:38 +01:00
parent 68881f04f2
commit aa99b37b99
9 changed files with 664 additions and 270 deletions
+71 -49
View File
@@ -525,11 +525,11 @@ hash phase:
Files are hashed in **inode order**, which minimizes seeks on spinning
disks. Paths that are hard links to the same inode are **read once**, and
all share the one result. A hard-link backup farm therefore costs one read
per inode, not per path. The phase total counts actual reads, so progress
and ETA are meaningful. Completed records are committed in batched
transactions **while hashing runs**. A scan interrupted after hours
therefore keeps everything hashed so far, and the next scan resumes cheaply,
skipping records already written.
per inode, not per path. The phase total counts actual reads, so its
progress and its time remaining are meaningful. Completed records are
committed in batched transactions **while hashing runs**. A scan interrupted
after hours therefore keeps everything hashed so far, and the next scan
resumes cheaply, skipping records already written.
4. **update** — commit the final partial batch of hashed records. Then delete
the records the scan did not verify: those of vanished files, and of paths
that failed to stat or hash.
@@ -625,14 +625,19 @@ Rules for the walk:
as described in "`scan` mode" above: they are not deleted. That does not
hold below an operand at or inside it.
- On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were
skipped. As specified above, a skipped path that has a database record from an
earlier scan loses that record. An unreadable directory subtree likewise loses
its records. This is accepted: the database mirrors what the latest scan could
actually verify. The exception is an operand dropped before the database was
read, whose record counts as outside the scanned operands.
- On a per-path error, such as permission denied or an unreadable file: print a
one-line warning to stderr, skip the path, and continue. Per-file errors never
abort the run; the final summary reports how many were skipped. As specified
above, a skipped path that has a database record from an earlier scan loses
that record. An unreadable directory subtree likewise loses its records. This
is accepted: the database mirrors what the latest scan could actually verify.
The exception is an operand dropped before the database was read, whose record
counts as outside the scanned operands.
- A file or directory removed while the scan runs is not an error. The scan
prints no line for it and does not count it as skipped. A path gone when the
walk reaches it gets no record. A file gone when the hash phase opens it gets
none either. The records an earlier scan left at or below such a path are
deleted.
Concurrency: the walk phase, which also stats files, and the hash phase each use
a worker pool of `--workers` workers (default `runtime.NumCPU()`). The walk
@@ -822,9 +827,9 @@ Use the progress-bar library for all progress of `scan`, `report`, `trees` and
stderr.
`scan`, `report`, `trees` and `info` must always show that they are working and
not hung. Each phase gets its own display from the moment it starts, naming the
phase and its place among the command's phases and showing its progress, its
elapsed time and its estimated time remaining (ETA).
not hung. Each phase gets its own display from the moment it starts. The display
names the phase and its place among the command's phases. It shows the phase's
progress, its elapsed time and its time remaining.
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
`[4/4] update`):
@@ -832,11 +837,11 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
- `load` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `walk` has no exact total. Its estimate is the number of records the last scan
left under the operands, known once `load` is done and shown with a `~`. Once
the walk passes the estimate, or when there are no such records, as on a first
scan, the ETA is unknown.
left under the operands, known once `load` is done and shown with a `~`. The
time remaining is unknown once the walk passes the estimate, and when there
are no such records, as on a first scan.
- `hash` and `update` have exact totals. Only files that actually need hashing
appear in the hash total, so its ETA is meaningful.
appear in the hash total, so its time remaining is meaningful.
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
`[4/4] write`, see "`trees` mode"):
@@ -861,8 +866,8 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `count` has no total either, because one query counts everything: the display
says it is counting the records until the query returns, and its ETA is
unknown.
says it is counting the records until the query returns, and its time
remaining is unknown.
Each display is one line holding:
@@ -875,16 +880,17 @@ Each display is one line holding:
while nothing completes, and a burst of items after a wait shows as a burst
rather than as a steady rate. Once no item has completed for 10 seconds, the
word `stalled` and where the phase waits take the rate's place (see below).
- elapsed time
- estimated time remaining, or `unknown`. It comes from the phase's average rate
since it started counting, so that it does not swing with each burst.
- elapsed time, labelled `elapsed`
- time remaining, labelled `remain`: a duration such as `12m9s`, or `unknown`.
It comes from the phase's average rate since it started counting, so that it
does not swing with each burst.
Example shape (exact layout is flexible, content is not):
Example lines, as printed when stderr is not a terminal:
```
[3/4] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s
```
When no item has completed for 10 seconds, as while one large file is read, a
@@ -900,27 +906,43 @@ for its total already says what it waits for, and says nothing more.
Additional requirements:
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
the `open` phase of `report` and `trees`, and both phases of `info`, once it
has counted what it will do) is drawn as its line followed by the progress-bar
library's bar, 10 columns wide, which shows the percent in place of the line's
own. Any other phase, and a phase whose bar has no room, is drawn as the
library's spinner followed by its line. Each display is drawn again several
times a second, so its elapsed time keeps moving while a phase waits for its
next item.
- On a terminal, every display starts with the progress-bar library's spinner in
its first column, followed by its line. Each display is drawn again several
times a second, so its spinner turns and its elapsed time keeps moving while a
phase waits for its next item.
- On a terminal, a phase with an exact total ends its line with its percent and
a bar 10 columns wide. This percent takes the place of the line's own. The bar
stays at the terminal's right edge, so it does not move as the line grows or
shrinks. A phase with nothing to do shows no bar. Every phase has an exact
total once it has counted what it will do, except these:
- `scan`'s `walk`;
- the `open` phase of `report` and `trees`;
- both phases of `info`.
On a terminal 100 columns wide:
```
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
```
- On a terminal no line reaches the terminal's last column, however narrow the
terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, at counts like those in
the example above, a stalled line keeps its words whenever the elapsed time
and the ETA are both in minutes. Where the words and the ETA do not both fit,
as at millions of files and times in hours, the words give way. Only when
those four alone are too wide, as on a terminal narrower than about 70 columns
hours into a scan of millions of files, is the line cut short at the
terminal's edge, so the end of the ETA is lost first. A terminal made narrower
while a phase runs gets the same: the next draw erases the line within the new
width and fits the line to it, and the display goes on redrawing in place.
terminal is. A line too wide for the terminal first loses whichever of these
it shows: its rate, the words of a stalled phase, or what the phase is waiting
for. The other four parts are kept whole:
- the phase and its place;
- the progress;
- the elapsed time;
- the time remaining.
The line loses its bar only when those four and the bar do not fit. It is
cut short at the terminal's edge only when those four alone are too wide,
and then the end of the time remaining is lost first. That happens hours
into a scan of millions of files, on a terminal narrower than about 75
columns. On a terminal 100 columns wide, the stalled line of the example
above keeps its words and its bar. The next draw fits the line to a terminal
made narrower while a phase runs. It erases the old line within the new
width, and the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once