Skip removed paths silently and fix the progress line (closes #133)
check / check (push) Canceled after 0s

A path removed before the scan reaches it, in the walk or the hash
phase, gets no warning, no skipped count and no record. If the first of
several hard links is gone, the next is read.

On a terminal every display is the library's spinner, then the line. A
phase with an exact total ends the padded line with its percent and a
bar at a fixed column. sfdupes draws that bar: the library shows no
spinner beside its own.

The time remaining is labelled remain, not eta.

Judgement call: an operand removed after the scan starts is passed over
silently too.
Deviation: README's 80-column promise for stalled lines is narrowed to
its example.

Model: opus-5-5
This commit is contained in:
2026-10-26 02:35:54 +00:00
parent 68881f04f2
commit c4aeec355a
9 changed files with 465 additions and 230 deletions
+70 -47
View File
@@ -525,11 +525,11 @@ hash phase:
Files are hashed in **inode order**, which minimizes seeks on spinning
disks. Paths that are hard links to the same inode are **read once**, and
all share the one result. A hard-link backup farm therefore costs one read
per inode, not per path. The phase total counts actual reads, so progress
and ETA are meaningful. Completed records are committed in batched
transactions **while hashing runs**. A scan interrupted after hours
therefore keeps everything hashed so far, and the next scan resumes cheaply,
skipping records already written.
per inode, not per path. The phase total counts actual reads, so its
progress and its time remaining are meaningful. Completed records are
committed in batched transactions **while hashing runs**. A scan interrupted
after hours therefore keeps everything hashed so far, and the next scan
resumes cheaply, skipping records already written.
4. **update** — commit the final partial batch of hashed records. Then delete
the records the scan did not verify: those of vanished files, and of paths
that failed to stat or hash.
@@ -625,14 +625,19 @@ Rules for the walk:
as described in "`scan` mode" above: they are not deleted. That does not
hold below an operand at or inside it.
- On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were
skipped. As specified above, a skipped path that has a database record from an
earlier scan loses that record. An unreadable directory subtree likewise loses
its records. This is accepted: the database mirrors what the latest scan could
actually verify. The exception is an operand dropped before the database was
read, whose record counts as outside the scanned operands.
- On a per-path error, such as permission denied or an unreadable file: print a
one-line warning to stderr, skip the path, and continue. Per-file errors never
abort the run; the final summary reports how many were skipped. As specified
above, a skipped path that has a database record from an earlier scan loses
that record. An unreadable directory subtree likewise loses its records. This
is accepted: the database mirrors what the latest scan could actually verify.
The exception is an operand dropped before the database was read, whose record
counts as outside the scanned operands.
- A file or directory removed before the scan reaches it is not an error. The
scan prints no line for it and does not count it as skipped. That holds in the
walk, and in the hash phase for a file removed since the walk. Such a path
gets no record, and the records an earlier scan left at or below it are
deleted.
Concurrency: the walk phase, which also stats files, and the hash phase each use
a worker pool of `--workers` workers (default `runtime.NumCPU()`). The walk
@@ -822,9 +827,9 @@ Use the progress-bar library for all progress of `scan`, `report`, `trees` and
stderr.
`scan`, `report`, `trees` and `info` must always show that they are working and
not hung. Each phase gets its own display from the moment it starts, naming the
phase and its place among the command's phases and showing its progress, its
elapsed time and its estimated time remaining (ETA).
not hung. Each phase gets its own display from the moment it starts. The display
names the phase and its place among the command's phases. It shows the phase's
progress, its elapsed time and its time remaining.
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
`[4/4] update`):
@@ -834,9 +839,9 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
- `walk` has no exact total. Its estimate is the number of records the last scan
left under the operands, known once `load` is done and shown with a `~`. Once
the walk passes the estimate, or when there are no such records, as on a first
scan, the ETA is unknown.
scan, the time remaining is unknown.
- `hash` and `update` have exact totals. Only files that actually need hashing
appear in the hash total, so its ETA is meaningful.
appear in the hash total, so its time remaining is meaningful.
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
`[4/4] write`, see "`trees` mode"):
@@ -861,8 +866,8 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `count` has no total either, because one query counts everything: the display
says it is counting the records until the query returns, and its ETA is
unknown.
says it is counting the records until the query returns, and its time
remaining is unknown.
Each display is one line holding:
@@ -875,16 +880,17 @@ Each display is one line holding:
while nothing completes, and a burst of items after a wait shows as a burst
rather than as a steady rate. Once no item has completed for 10 seconds, the
word `stalled` and where the phase waits take the rate's place (see below).
- elapsed time
- estimated time remaining, or `unknown`. It comes from the phase's average rate
since it started counting, so that it does not swing with each burst.
- elapsed time, labelled `elapsed`
- time remaining, labelled `remain`: a duration such as `12m9s`, or `unknown`.
It comes from the phase's average rate since it started counting, so that it
does not swing with each burst.
Example shape (exact layout is flexible, content is not):
Example lines, as printed when stderr is not a terminal:
```
[3/4] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s
```
When no item has completed for 10 seconds, as while one large file is read, a
@@ -900,27 +906,44 @@ for its total already says what it waits for, and says nothing more.
Additional requirements:
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
the `open` phase of `report` and `trees`, and both phases of `info`, once it
has counted what it will do) is drawn as its line followed by the progress-bar
library's bar, 10 columns wide, which shows the percent in place of the line's
own. Any other phase, and a phase whose bar has no room, is drawn as the
library's spinner followed by its line. Each display is drawn again several
times a second, so its elapsed time keeps moving while a phase waits for its
next item.
- On a terminal, every display starts with the progress-bar library's spinner in
its first column, followed by its line. Each display is drawn again several
times a second, so its spinner turns and its elapsed time keeps moving while a
phase waits for its next item.
- On a terminal, a phase with an exact total ends its line with its percent and
a bar 10 columns wide. This percent takes the place of the line's own. The bar
stays at the terminal's right edge, so it does not move as the line grows or
shrinks. Every phase has an exact total once it has counted what it will do,
except these:
- `scan`'s `walk`;
- the `open` phase of `report` and `trees`;
- both phases of `info`.
On a terminal 100 columns wide:
```
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
```
- On a terminal no line reaches the terminal's last column, however narrow the
terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, at counts like those in
the example above, a stalled line keeps its words whenever the elapsed time
and the ETA are both in minutes. Where the words and the ETA do not both fit,
as at millions of files and times in hours, the words give way. Only when
those four alone are too wide, as on a terminal narrower than about 70 columns
hours into a scan of millions of files, is the line cut short at the
terminal's edge, so the end of the ETA is lost first. A terminal made narrower
while a phase runs gets the same: the next draw erases the line within the new
width and fits the line to it, and the display goes on redrawing in place.
terminal is. A line too wide for the terminal first loses its bar. Next it
loses whichever of these it shows: its rate, the words of a stalled phase, or
what the phase is waiting for. The other four parts are kept whole:
- the phase and its place;
- the progress;
- the elapsed time;
- the time remaining.
The line is cut short at the terminal's edge only when those four alone are
too wide, and then the end of the time remaining is lost first. That happens
hours into a scan of millions of files, on a terminal narrower than about 75
columns. On an 80-column terminal, the stalled line of the example above
keeps its words. Where the words and the time remaining do not both fit, as
at millions of files and times in hours, the words give way. A terminal made
narrower while a phase runs gets the same: the next draw erases the line
within the new width and fits the line to it, and the display goes on
redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once