sneak/sfdupes
1
0
Fork 0
Code Issues 4 Pull Requests 3 Actions 9 Packages Projects Releases Wiki Activity

Skip removed paths silently and fix the progress line (closes #133) #139

Merged
clawbot merged 1 commits from issue-133-display-fixes into next 2026-10-26 06:32:38 +01:00
Conversation 6 Commits 1 Files Changed 9
+664 -270
9 changed files with 664 additions and 270 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+71 -49
View File
@@ -525,11 +525,11 @@ hash phase:
Files are hashed in **inode order**, which minimizes seeks on spinning
disks. Paths that are hard links to the same inode are **read once**, and
all share the one result. A hard-link backup farm therefore costs one read
per inode, not per path. The phase total counts actual reads, so progress
and ETA are meaningful. Completed records are committed in batched
transactions **while hashing runs**. A scan interrupted after hours
therefore keeps everything hashed so far, and the next scan resumes cheaply,
skipping records already written.
per inode, not per path. The phase total counts actual reads, so its
progress and its time remaining are meaningful. Completed records are
committed in batched transactions **while hashing runs**. A scan interrupted
after hours therefore keeps everything hashed so far, and the next scan
resumes cheaply, skipping records already written.
4. **update** — commit the final partial batch of hashed records. Then delete
the records the scan did not verify: those of vanished files, and of paths
that failed to stat or hash.
@@ -625,14 +625,19 @@ Rules for the walk:
as described in "`scan` mode" above: they are not deleted. That does not
hold below an operand at or inside it.
- On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were
skipped. As specified above, a skipped path that has a database record from an
earlier scan loses that record. An unreadable directory subtree likewise loses
its records. This is accepted: the database mirrors what the latest scan could
actually verify. The exception is an operand dropped before the database was
read, whose record counts as outside the scanned operands.
- On a per-path error, such as permission denied or an unreadable file: print a
one-line warning to stderr, skip the path, and continue. Per-file errors never
abort the run; the final summary reports how many were skipped. As specified
above, a skipped path that has a database record from an earlier scan loses
that record. An unreadable directory subtree likewise loses its records. This
is accepted: the database mirrors what the latest scan could actually verify.
The exception is an operand dropped before the database was read, whose record
counts as outside the scanned operands.
- A file or directory removed while the scan runs is not an error. The scan
prints no line for it and does not count it as skipped. A path gone when the
walk reaches it gets no record. A file gone when the hash phase opens it gets
none either. The records an earlier scan left at or below such a path are
deleted.
Concurrency: the walk phase, which also stats files, and the hash phase each use
a worker pool of `--workers` workers (default `runtime.NumCPU()`). The walk
@@ -822,9 +827,9 @@ Use the progress-bar library for all progress of `scan`, `report`, `trees` and
stderr.
`scan`, `report`, `trees` and `info` must always show that they are working and
not hung. Each phase gets its own display from the moment it starts, naming the
phase and its place among the command's phases and showing its progress, its
elapsed time and its estimated time remaining (ETA).
not hung. Each phase gets its own display from the moment it starts. The display
names the phase and its place among the command's phases. It shows the phase's
progress, its elapsed time and its time remaining.
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
`[4/4] update`):
@@ -832,11 +837,11 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
- `load` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `walk` has no exact total. Its estimate is the number of records the last scan
left under the operands, known once `load` is done and shown with a `~`. Once
the walk passes the estimate, or when there are no such records, as on a first
scan, the ETA is unknown.
left under the operands, known once `load` is done and shown with a `~`. The
time remaining is unknown once the walk passes the estimate, and when there
are no such records, as on a first scan.
- `hash` and `update` have exact totals. Only files that actually need hashing
appear in the hash total, so its ETA is meaningful.
appear in the hash total, so its time remaining is meaningful.
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
`[4/4] write`, see "`trees` mode"):
@@ -861,8 +866,8 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `count` has no total either, because one query counts everything: the display
says it is counting the records until the query returns, and its ETA is
unknown.
says it is counting the records until the query returns, and its time
remaining is unknown.
Each display is one line holding:
@@ -875,16 +880,17 @@ Each display is one line holding:
while nothing completes, and a burst of items after a wait shows as a burst
rather than as a steady rate. Once no item has completed for 10 seconds, the
word `stalled` and where the phase waits take the rate's place (see below).
- elapsed time
- estimated time remaining, or `unknown`. It comes from the phase's average rate
since it started counting, so that it does not swing with each burst.
- elapsed time, labelled `elapsed`
- time remaining, labelled `remain`: a duration such as `12m9s`, or `unknown`.
It comes from the phase's average rate since it started counting, so that it
does not swing with each burst.
Example shape (exact layout is flexible, content is not):
Example lines, as printed when stderr is not a terminal:
```
[3/4] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s
```
When no item has completed for 10 seconds, as while one large file is read, a
@@ -900,27 +906,43 @@ for its total already says what it waits for, and says nothing more.
Additional requirements:
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`,
the `open` phase of `report` and `trees`, and both phases of `info`, once it
has counted what it will do) is drawn as its line followed by the progress-bar
library's bar, 10 columns wide, which shows the percent in place of the line's
own. Any other phase, and a phase whose bar has no room, is drawn as the
library's spinner followed by its line. Each display is drawn again several
times a second, so its elapsed time keeps moving while a phase waits for its
next item.
- On a terminal, every display starts with the progress-bar library's spinner in
its first column, followed by its line. Each display is drawn again several
times a second, so its spinner turns and its elapsed time keeps moving while a
phase waits for its next item.
- On a terminal, a phase with an exact total ends its line with its percent and
a bar 10 columns wide. This percent takes the place of the line's own. The bar
stays at the terminal's right edge, so it does not move as the line grows or
shrinks. A phase with nothing to do shows no bar. Every phase has an exact
total once it has counted what it will do, except these:
- `scan`'s `walk`;
- the `open` phase of `report` and `trees`;
- both phases of `info`.
On a terminal 100 columns wide:
```
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
```
- On a terminal no line reaches the terminal's last column, however narrow the
terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, at counts like those in
the example above, a stalled line keeps its words whenever the elapsed time
and the ETA are both in minutes. Where the words and the ETA do not both fit,
as at millions of files and times in hours, the words give way. Only when
those four alone are too wide, as on a terminal narrower than about 70 columns
hours into a scan of millions of files, is the line cut short at the
terminal's edge, so the end of the ETA is lost first. A terminal made narrower
while a phase runs gets the same: the next draw erases the line within the new
width and fits the line to it, and the display goes on redrawing in place.
terminal is. A line too wide for the terminal first loses whichever of these
it shows: its rate, the words of a stalled phase, or what the phase is waiting
for. The other four parts are kept whole:
- the phase and its place;
- the progress;
- the elapsed time;
- the time remaining.
The line loses its bar only when those four and the bar do not fit. It is
cut short at the terminal's edge only when those four alone are too wide,
and then the end of the time remaining is lost first. That happens hours
into a scan of millions of files, on a terminal narrower than about 75
columns. On a terminal 100 columns wide, the stalled line of the example
above keeps its words and its bar. The next draw fits the line to a terminal
made narrower while a phase runs. It erases the old line within the new
width, and the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once
TODO.md
+10
View File
@@ -29,6 +29,16 @@
# Completed Steps
- a file or directory removed while the scan runs is not an error, in the walk
or in the hash phase. The scan prints no line for it and does not count it as
skipped. When the first of several hard links to a file is gone, the hash
phase reads the next one instead. On a terminal every progress display starts
with the spinner, the hash phase's included. A phase with an exact total ends
its line with its percent and a 10-column bar at the terminal's right edge,
which no longer moves as the line's length changes. A line too wide for the
terminal now loses its rate before its bar. The time remaining is labelled
`remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133)
- the `head`, `tail` and `content` columns are BLOBs, each holding the raw 32
bytes of a SHA-256, half the size of the hex text they held. The hashes stay
raw from the hash phase through the database to `report` and `trees`. Reading
info_test.go
+2 -2
View File
@@ -12,13 +12,13 @@ import (
// infoOpenLine is the plain line of info's open phase when it opens the
// database at once; it prints it as the phase starts and as it ends.
const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown"
"remain unknown"
// infoCountLine is the plain line of info's count phase when the count
// takes under a second; it prints it as the phase starts and as it
// ends.
const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " +
"eta unknown"
"remain unknown"
// infoRecs are four records: a duplicate pair, a file of 20 MiB and
// another small file.
progress.go
+102 -90
View File
@@ -9,6 +9,7 @@ import (
"strings"
"sync"
"time"
"unicode/utf8"
"github.com/schollz/progressbar/v3"
"golang.org/x/term"
@@ -34,10 +35,17 @@ const stallTime = 10 * time.Second
// spinnerType selects the progressbar library's spinner style.
const spinnerType = 14
// barWidth is the width of the progressbar library's bar between its
// two ends, as in the example line in README "Progress".
// barWidth is the width of a phase's bar between its two ends, as in
// the example line in README "Progress".
const barWidth = 10
// barCell is one column of the filled part of a bar.
const barCell = "█"
// barColumns is the width of what percentBar returns: the percent in
// four columns and its sign, a space, and the bar with its two ends.
const barColumns = barWidth + 8
// percentScale converts a fraction to a percentage.
const percentScale = 100
@@ -45,17 +53,12 @@ const percentScale = 100
// read, as the progressbar library takes it.
const fallbackWidth = 80
// spinnerMargin is the columns of a spinner's line besides its text:
// the spinner and the three spaces the library draws around the text,
// and the terminal's last column, which every line stays clear of so
// that no terminal wraps it.
// spinnerMargin is the columns of a line besides its text: the spinner
// and the three spaces the library draws around the text, and the
// terminal's last column, which every line stays clear of so that no
// terminal wraps it.
const spinnerMargin = 5
// barMargin is the columns of a bar's line besides its text: the
// percent and the bar the library draws after the text, the spaces
// around them, and the terminal's last column.
const barMargin = barWidth + 11
// countingRecords is what the display of a phase that reads the records
// says while it counts them, before it has its total.
const countingRecords = "counting the records"
@@ -109,13 +112,13 @@ type drawnCount struct {
}
// progress renders one phase's progress on stderr. On a TTY the
// progressbar library draws line, fitted to the terminal's width: after
// the library's spinner, or, for a phase with an exact total, before the
// library's bar when the terminal has room for the bar beside it. It is
// drawn again every redrawInterval, so its elapsed time keeps moving
// while the phase waits for its next item. When stderr is not a TTY it
// emits no ANSI redraws: it prints line as the phase starts, every
// plainInterval while it runs, and as it ends.
// progressbar library draws its spinner and then line, fitted to the
// terminal's width. A phase with an exact total ends the line with its
// percent and bar, at the terminal's right edge, when the terminal has
// room for them. It is drawn again every redrawInterval, so its elapsed
// time keeps moving while the phase waits for its next item. When
// stderr is not a TTY it emits no ANSI redraws: it prints line as the
// phase starts, every plainInterval while it runs, and as it ends.
//
// The draws come from a goroutine of their own, so mu guards the fields
// below it. Every method must be called from the main goroutine. A nil
@@ -131,7 +134,6 @@ type progress struct {
mu sync.Mutex
bar *progressbar.ProgressBar // what the library draws; nil until then
withBar bool // bar is the library's bar, not its spinner
widest int // the widest line bar has drawn, in columns
label string
unit string
@@ -141,7 +143,7 @@ type progress struct {
stalledOn string
count int64
start time.Time // when the phase started, for the elapsed time
rateStart time.Time // when the count started, for the ETA
rateStart time.Time // when the count started, for the time remaining
lastItem time.Time // when the last item was completed, or the count started
// drawn holds the count at each draw over the last rateWindow, oldest
// first; the rate counts from the first. It keeps the draw before the
@@ -215,55 +217,34 @@ func (p *progress) draw() {
}
if p.width == nil {
fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true))
fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, false))
return
}
width := p.width()
text := p.line(now, width-spinnerMargin, true)
// A phase with an exact total gets the library's bar when its whole
// line, without the percent the bar shows, fits beside the bar: the
// bar gives way before any part of the line. A total of 0 leaves the
// bar nothing to fill.
text, withBar := "", false
if p.total > 0 && !p.estimate {
text = p.line(now, math.MaxInt, false)
withBar = len(text) <= width-barMargin
}
margin := barMargin
if !withBar {
margin = spinnerMargin
text = p.line(now, width-margin, true)
}
// The library makes a bar or a spinner, and before each draw erases
// its line with as many spaces as the widest line it has drawn. A
// display gets a new one when it turns from one to the other, as its
// line grows or the terminal is resized, and when the terminal has
// been made too narrow for that erase, which would wrap and leave a
// line behind at every draw.
if p.bar == nil || p.withBar != withBar || p.widest >= width {
p.newBar(withBar, width)
}
if withBar {
_ = p.bar.Set64(p.count)
// Before each draw the library erases its line with as many spaces as
// the widest line it has drawn. A display gets a new one when the
// terminal has been made too narrow for that erase, which would wrap
// and leave a line behind at every draw.
if p.bar == nil || p.widest >= width {
p.newBar(width)
}
p.bar.Describe(text)
// The library drew the text and its margin, less the terminal's last
// column, which the margin keeps clear.
p.widest = max(p.widest, len(text)+margin-1)
// The library drew the spinner, the text and the spaces around it,
// less the terminal's last column, which spinnerMargin keeps clear. A
// bar's cells take one column each but more than one byte.
p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1)
}
// newBar has the library make what draw draws: a bar for the phase's
// total when withBar is set, and a spinner when not. Either shows the
// text draw gives it, and nothing of its own besides the spinner, or
// the bar and its percent. p.mu must be held.
func (p *progress) newBar(withBar bool, width int) {
// newBar has the library make what draw draws: its spinner, followed by
// the text draw gives it and nothing of the library's own. p.mu must be
// held.
func (p *progress) newBar(width int) {
if p.bar != nil {
// Erase what was drawn before, short of the last column of a
// terminal that may have been made narrower since: the library's
@@ -272,29 +253,36 @@ func (p *progress) newBar(withBar bool, width int) {
fmt.Fprint(os.Stderr, "\r"+blank+"\r")
}
total := int64(-1) // the library's spinner
if withBar {
total = p.total
}
p.bar = progressbar.NewOptions64(total,
// A total of -1 gives the library's spinner.
p.bar = progressbar.NewOptions64(-1,
progressbar.OptionSetWriter(os.Stderr),
progressbar.OptionSetWidth(barWidth),
progressbar.OptionSpinnerType(spinnerType),
// The spinner turns at each draw, not on a timer of the
// library's own.
progressbar.OptionSetSpinnerChangeInterval(0),
// The text carries the elapsed time and the ETA.
// The text carries the elapsed time and the time remaining.
progressbar.OptionSetElapsedTime(false),
progressbar.OptionSetPredictTime(false),
)
p.withBar, p.widest = withBar, 0
p.widest = 0
}
// percentBar returns the percent and the bar that end the text of a
// phase with an exact total, as " 25% |██ |": the bar's
// barWidth columns fill in proportion to the items done. A count past
// the total, as when a scan adds records while trees reads them, shows
// as 100%. p.mu must be held.
func (p *progress) percentBar() string {
done := min(float64(p.count)/float64(p.total), 1)
filled := strings.Repeat(barCell, int(done*barWidth))
return fmt.Sprintf("%4.0f%% |%-*s|", done*percentScale, barWidth, filled)
}
// setTotal gives a phase that started without a total its total, once
// it is known. The rate, the ETA and the time with no item completed
// count from then on: the time spent waiting for the total says nothing
// about how fast items go.
// it is known. The rate, the time remaining and the time with no item
// completed count from then on: the time spent waiting for the total
// says nothing about how fast items go.
func (p *progress) setTotal(total int64) {
if p == nil {
return
@@ -438,22 +426,26 @@ func (p *progress) finish() {
// line formats the phase's progress as of now in at most room columns,
// as
//
// [3/4] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s
// [3/4] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, remain 13s
//
// where "files" is the phase's unit, without the percent when percent
// is false, as before the library's bar, which shows its own. A phase
// with no total shows the items done instead, or what it waits for. The
// rate is that since the first draw in drawn, and the ETA comes from
// the rate since the count started. Once no item has been completed for
// stallTime, the rate, which would read 0, gives way to the word
// "stalled" and where the phase waits, as "stalled disk/db".
// where "files" is the phase's unit. A phase with no total shows the
// items done instead, or what it waits for. The rate is that since the
// first draw in drawn, and the time remaining comes from the rate since
// the count started. Once no item has been completed for stallTime, the
// rate, which would read 0, gives way to the word "stalled" and where
// the phase waits, as "stalled disk/db".
//
// The phase and its place, the progress, the elapsed time and the ETA
// are kept whole. A line wider than room leaves out the rate, the words
// of a stalled phase, or what the phase waits for; only if it is still
// wider is it cut at room. Every part is ASCII, so its length is its
// width in columns. p.mu must be held.
func (p *progress) line(now time.Time, room int, percent bool) string {
// When bar is true, as on a terminal, a phase with an exact total ends
// the line with its percent and bar, in place of the percent after its
// count. A total of 0 leaves the bar nothing to fill.
//
// The phase and its place, the progress, the elapsed time and the time
// remaining are kept whole. A line wider than room first leaves out the
// rate, the words of a stalled phase, or what the phase waits for. Next
// it leaves out the bar. Only if it is still wider is it cut at room.
// Every part but the bar is ASCII, so its length is its width in
// columns. p.mu must be held.
func (p *progress) line(now time.Time, room int, bar bool) string {
elapsed := now.Sub(p.start).Round(time.Second)
// head and tail hold the parts kept whole; extra, between them, is
@@ -462,21 +454,41 @@ func (p *progress) line(now time.Time, room int, percent bool) string {
if p.total < 0 && p.waitingFor != "" {
head, extra = p.label+": ", p.waitingFor+", "
tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed)
tail = fmt.Sprintf("elapsed %s, remain unknown", elapsed)
} else {
since := p.drawn[0]
rate := perSecond(p.count-since.count, now.Sub(since.at))
average := perSecond(p.count, now.Sub(p.rateStart))
head = p.label + ": " + p.itemsDone(percent) + ", "
head = p.label + ": " + p.itemsDone(true) + ", "
extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit)
tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average))
tail = fmt.Sprintf("elapsed %s, remain %s", elapsed, p.remain(average))
if now.Sub(p.lastItem) >= stallTime {
extra = "stalled " + p.stalledOn + ", "
}
}
if bar && p.total > 0 && !p.estimate {
// Beside the bar, which shows the percent, the count shows none.
// The line is padded to the room the bar leaves it, so the bar
// stays at room's right edge however long the line is.
beside := room - barColumns
barHead := p.label + ": " + p.itemsDone(false) + ", "
switch {
case len(barHead+extra+tail) <= beside:
return fmt.Sprintf("%-*s%s", beside, barHead+extra+tail,
p.percentBar())
case len(barHead+tail) <= beside:
return fmt.Sprintf("%-*s%s", beside, barHead+tail, p.percentBar())
}
// The bar gives way. The rate, or the words of a stalled phase,
// gave way before it and stay out.
extra = ""
}
full, kept := head+extra+tail, head+tail
switch {
@@ -526,10 +538,10 @@ func (p *progress) itemsDone(percent bool) string {
return fmt.Sprintf("%s %.0f%%", done, pct)
}
// eta returns the time the phase has left at rate items a second, or
// remain returns the time the phase has left at rate items a second, or
// "unknown": with no total, before the first item, and once the count
// passes an estimate. p.mu must be held.
func (p *progress) eta(rate float64) string {
func (p *progress) remain(rate float64) string {
switch {
case p.total < 0, p.estimate && p.count > p.total:
return "unknown"
@@ -539,8 +551,8 @@ func (p *progress) eta(rate float64) string {
return "unknown"
}
remaining := time.Duration(
left := time.Duration(
float64(p.total-p.count) / rate * float64(time.Second))
return remaining.Round(time.Second).String()
return left.Round(time.Second).String()
}
progress_test.go
+261 -101
View File
@@ -117,24 +117,24 @@ func TestProgressLine(t *testing.T) {
label: scanPhase("load"), total: -1,
waitingFor: countingRecords,
},
want: "[1/4] load: counting the records, elapsed 10s, eta unknown",
want: "[1/4] load: counting the records, elapsed 10s, remain unknown",
}, {
p: &progress{
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
},
want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s",
}, {
p: &progress{label: scanPhase("hash"), unit: scanUnit, total: 100},
want: "[3/4] hash: [0/100] 0%, 0 files/s, elapsed 10s, eta unknown",
want: "[3/4] hash: [0/100] 0%, 0 files/s, elapsed 10s, remain unknown",
}, {
p: &progress{label: scanPhase("update"), unit: scanUnit, total: 0},
want: "[4/4] update: [0/0] 100%, 0 files/s, elapsed 10s, eta 0s",
want: "[4/4] update: [0/0] 100%, 0 files/s, elapsed 10s, remain 0s",
}, {
p: &progress{
label: scanPhase("walk"), unit: scanUnit, total: 100,
estimate: true, count: 20,
},
want: "[2/4] walk: [20/~100] 20%, 2 files/s, elapsed 10s, eta 40s",
want: "[2/4] walk: [20/~100] 20%, 2 files/s, elapsed 10s, remain 40s",
}, {
// Past the estimate the walk cannot tell how long it has left.
p: &progress{
@@ -142,13 +142,13 @@ func TestProgressLine(t *testing.T) {
estimate: true, count: 120,
},
want: "[2/4] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " +
"eta unknown",
"remain unknown",
}, {
// A first scan's walk has no estimate.
p: &progress{
label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20,
},
want: "[2/4] walk: 20 files, 2 files/s, elapsed 10s, eta unknown",
want: "[2/4] walk: 20 files, 2 files/s, elapsed 10s, remain unknown",
}, {
// The rate is in what the phase counts.
p: &progress{
@@ -156,14 +156,14 @@ func TestProgressLine(t *testing.T) {
count: 20,
},
want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " +
"eta 40s",
"remain 40s",
}}
for _, tc := range cases {
tc.p.start, tc.p.rateStart, tc.p.lastItem = ago, ago, now
tc.p.drawn = []drawnCount{{at: ago}}
if got := tc.p.line(now, math.MaxInt, true); got != tc.want {
if got := tc.p.line(now, math.MaxInt, false); got != tc.want {
t.Errorf("line = %q, want %q", got, tc.want)
}
}
@@ -198,48 +198,49 @@ func TestLineFitsRoom(t *testing.T) {
}
cases := []struct {
p *progress
room int
percent bool
want string
p *progress
room int
bar bool
want string
}{{
p: hash, room: 57, percent: true,
want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
p: hash, room: 60,
want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s",
}, {
p: hash, room: 56, percent: true,
want: "[3/4] hash: [20/100] 20%, elapsed 10s, eta 40s",
p: hash, room: 59,
want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40s",
}, {
p: hash, room: 45, percent: true,
want: "[3/4] hash: [20/100] 20%, elapsed 10s, eta 40",
p: hash, room: 48,
want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40",
}, {
// Before the library's bar, which shows the percent.
p: hash, room: 57, percent: false,
want: "[3/4] hash: [20/100], 2 files/s, elapsed 10s, eta 40s",
// The bar shows the percent.
p: hash, room: 74, bar: true,
want: "[3/4] hash: [20/100], 2 files/s, elapsed 10s, remain 40s" +
" 20% |██ |",
}, {
p: load, room: 40, percent: true,
want: "[1/4] load: elapsed 10s, eta unknown",
p: load, room: 43,
want: "[1/4] load: elapsed 10s, remain unknown",
}, {
p: stalled, room: 65, percent: true,
p: stalled, room: 68,
want: "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " +
"eta 3m20s",
"remain 3m20s",
}, {
p: stalled, room: 64, percent: true,
want: "[3/4] hash: [20/100] 20%, elapsed 50s, eta 3m20s",
p: stalled, room: 67,
want: "[3/4] hash: [20/100] 20%, elapsed 50s, remain 3m20s",
}}
for _, tc := range cases {
if got := tc.p.line(now, tc.room, tc.percent); got != tc.want {
t.Errorf("line(%d, %t) = %q, want %q",
tc.room, tc.percent, got, tc.want)
if got := tc.p.line(now, tc.room, tc.bar); got != tc.want {
t.Errorf("line(%d, %t) = %q, want %q", tc.room, tc.bar, got,
tc.want)
}
}
}
// TestSetTotalStartsRate checks that setTotal starts a phase's rate,
// its ETA and its time with no record read from when the total becomes
// known, not from when the phase started waiting for it: after 20
// seconds of waiting for the total, the phase has not stalled 5 seconds
// after it, and has 10 seconds after it with no record read.
// its time remaining and its time with no record read from when the
// total becomes known, not from when the phase started waiting for it:
// after 20 seconds of waiting for the total, the phase has not stalled 5
// seconds after it, and has 10 seconds after it with no record read.
func TestSetTotalStartsRate(t *testing.T) {
t.Parallel()
@@ -255,15 +256,15 @@ func TestSetTotalStartsRate(t *testing.T) {
clock.Add(5 * time.Second)
want := "[1/4] load: [0/100] 0%, 0 files/s, elapsed 25s, eta unknown"
if got := p.line(clock.Now(), math.MaxInt, true); got != want {
want := "[1/4] load: [0/100] 0%, 0 files/s, elapsed 25s, remain unknown"
if got := p.line(clock.Now(), math.MaxInt, false); got != want {
t.Errorf("line = %q, want %q", got, want)
}
clock.Add(5 * time.Second)
want = "[1/4] load: [0/100] 0%, stalled db, elapsed 30s, eta unknown"
if got := p.line(clock.Now(), math.MaxInt, true); got != want {
want = "[1/4] load: [0/100] 0%, stalled db, elapsed 30s, remain unknown"
if got := p.line(clock.Now(), math.MaxInt, false); got != want {
t.Errorf("line = %q, want %q", got, want)
}
@@ -272,8 +273,8 @@ func TestSetTotalStartsRate(t *testing.T) {
p.increment()
}
want = "[1/4] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s"
if got := p.line(clock.Now(), math.MaxInt, true); got != want {
want = "[1/4] load: [20/100] 20%, 2 files/s, elapsed 30s, remain 40s"
if got := p.line(clock.Now(), math.MaxInt, false); got != want {
t.Errorf("line = %q, want %q", got, want)
}
}
@@ -316,7 +317,7 @@ func TestPlainProgressPrintsAtStartAndEnd(t *testing.T) {
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
want := "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown"
want := "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown"
if len(lines) != 2 || lines[0] != want ||
!strings.HasPrefix(lines[1], "[3/4] hash: [1000/1000] 100%, ") {
t.Errorf("stderr lines = %q, want %q, then the line at 1000",
@@ -336,7 +337,7 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) {
}, nil, testRedraw, time.Now)
awaitOutput(t, stderr,
"[1/4] load: counting the records, elapsed 0s, eta unknown\n", 3)
"[1/4] load: counting the records, elapsed 0s, remain unknown\n", 3)
p.finish()
}
@@ -366,23 +367,23 @@ func TestPlainProgressStalledProducer(t *testing.T) {
clock.Add(5 * time.Second)
awaitOutput(t, stderr,
"[3/4] hash: [20/100] 20%, 4 files/s, elapsed 5s, eta 20s\n", 3)
"[3/4] hash: [20/100] 20%, 4 files/s, elapsed 5s, remain 20s\n", 3)
clock.Add(time.Second)
awaitOutput(t, stderr,
"[3/4] hash: [20/100] 20%, 0 files/s, elapsed 6s, eta 24s\n", 3)
"[3/4] hash: [20/100] 20%, 0 files/s, elapsed 6s, remain 24s\n", 3)
clock.Add(4 * time.Second)
awaitOutput(t, stderr,
"[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 10s, eta 40s\n",
"[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 10s, remain 40s\n",
3)
}
// TestPlainProgressRateAfterStall draws a phase's plain line by hand, a
// little over plainInterval after the line before, as a late tick
// does, through a stall and a burst of files after it: the rate covers
// the files read since the line before, while the ETA comes from the
// phase's average.
// the files read since the line before, while the time remaining comes
// from the phase's average.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestPlainProgressRateAfterStall(t *testing.T) {
@@ -410,10 +411,10 @@ func TestPlainProgressRateAfterStall(t *testing.T) {
p.finish()
want := []string{
"[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown",
"[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 5s, eta unknown",
"[3/4] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, eta unknown",
"[3/4] hash: [100/1000] 10%, 20 files/s, elapsed 15s, eta 2m15s",
"[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown",
"[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 5s, remain unknown",
"[3/4] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, remain unknown",
"[3/4] hash: [100/1000] 10%, 20 files/s, elapsed 15s, remain 2m15s",
}
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
@@ -426,8 +427,8 @@ func TestPlainProgressRateAfterStall(t *testing.T) {
// redrawInterval, as a terminal does, through a stall of stallTime and
// a burst of files after it: the rate covers the files read over the
// last rateWindow, and falls to 0 once the burst is older than that,
// while the ETA comes from the phase's average. Plain lines show each
// draw.
// while the time remaining comes from the phase's average. Plain lines
// show each draw.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressRateAfterStallRedrawn(t *testing.T) {
@@ -457,8 +458,8 @@ func TestProgressRateAfterStallRedrawn(t *testing.T) {
redraw(rateWindow + redrawInterval)
p.finish()
want := "[3/4] hash: [50/1000] 5%, 10 files/s, elapsed 15s, eta 4m45s\n" +
"[3/4] hash: [50/1000] 5%, 0 files/s, elapsed 15s, eta 4m47s\n"
want := "[3/4] hash: [50/1000] 5%, 10 files/s, elapsed 15s, remain 4m45s\n" +
"[3/4] hash: [50/1000] 5%, 0 files/s, elapsed 15s, remain 4m47s\n"
if !strings.Contains(stderr(), want) {
t.Errorf("stderr = %q, want the lines %q", stderr(), want)
}
@@ -480,10 +481,10 @@ func lastFrame(out string) string {
}
// TestTerminalRedrawsWhileWaiting checks that the terminal display of a
// phase, spinner or bar, keeps being drawn while the phase waits for its
// next item, with the elapsed time as of now, and shows every item
// completed before the wait. stderr is not a terminal here, so the
// displays take it to be fallbackWidth wide.
// phase, with a bar or without, keeps being drawn while the phase waits
// for its next item, with the elapsed time as of now, and shows every
// item completed before the wait. The terminal is 120 columns wide, so
// the hash phase's display has room for its bar.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalRedrawsWhileWaiting(t *testing.T) {
@@ -506,7 +507,8 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) {
for _, tc := range cases {
before := len(stderr())
since := func() string { return stderr()[before:] }
p := startProgress(tc.params, terminalWidth, testRedraw, time.Now)
p := startProgress(tc.params, func() int { return 120 }, testRedraw,
time.Now)
for range 50 {
p.increment()
@@ -552,13 +554,13 @@ func drawLate(stderr func() string, params progressParams, width int,
return stderr()[before:]
}
// TestTerminalLinesKeepETA checks the line each display shows on an
// 80-column terminal at a large scan's counts and times, and the line
// of trees' read phase at the same counts: the rate is left out, so
// that the elapsed time and the ETA stay whole.
// TestTerminalLinesKeepTimeRemaining checks the line each display shows
// on an 80-column terminal at a large scan's counts and times, and the
// line of trees' read phase at the same counts: the rate is left out,
// so that the elapsed time and the time remaining stay whole.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalLinesKeepETA(t *testing.T) {
func TestTerminalLinesKeepTimeRemaining(t *testing.T) {
stderr := captureStderr(t)
cases := []struct {
@@ -571,19 +573,19 @@ func TestTerminalLinesKeepETA(t *testing.T) {
estimate: true,
},
want: "[2/4] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " +
"eta unknown",
"remain unknown",
}, {
params: progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
},
want: "[3/4] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s",
"remain 17h30m0s",
}, {
params: progressParams{
label: treesPhase("read"), unit: "records", total: 9876543,
},
want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s",
"remain 17h30m0s",
}}
for _, tc := range cases {
@@ -594,30 +596,32 @@ func TestTerminalLinesKeepETA(t *testing.T) {
}
}
// TestTerminalStalledLines draws every phase's line on an 80-column
// terminal once the phase has stalled, its last item done 42 seconds
// before. At an ordinary scan's counts and times, 12345 of 45678 items
// done four and a half minutes in, the line shows where the phase waits
// whole, beside the phase, progress, elapsed time and ETA. At a large
// scan's, 1234567 of 9876543 two and a half hours in, those four still
// show whole, as the words give way first where they do not fit.
// TestTerminalStalledLines draws every phase's line once the phase has
// stalled, its last item done 42 seconds before. At an ordinary scan's
// counts and times, 12345 of 45678 items done four and a half minutes
// in, on a terminal 100 columns wide, the line shows where the phase
// waits whole, beside the phase, progress, elapsed time and time
// remaining. At a large scan's, 1234567 of 9876543 two and a half hours
// in, on an 80-column terminal, those four still show whole, as the
// words give way first where they do not fit.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalStalledLines(t *testing.T) {
stderr := captureStderr(t)
// drawStalled draws the display params describes as it starts and
// once more as it ends, with count of total items done elapsed into
// the phase, and returns what the terminal shows.
drawStalled := func(params progressParams, total, count int64,
elapsed time.Duration,
// drawStalled draws the display params describes on a terminal width
// columns wide as it starts and once more as it ends, with count of
// total items done elapsed into the phase, and returns what the
// terminal shows.
drawStalled := func(params progressParams, width int, total,
count int64, elapsed time.Duration,
) string {
before := len(stderr())
clock := &testClock{now: time.Now()}
params.total = total
// No timed draws come between the two.
p := startProgress(params, func() int { return 80 }, time.Hour,
p := startProgress(params, func() int { return width }, time.Hour,
clock.Now)
clock.Add(elapsed - 42*time.Second)
@@ -649,19 +653,22 @@ func TestTerminalStalledLines(t *testing.T) {
approx = "~"
}
shown := drawStalled(params, 45678, 12345,
// The percent is in the line, or beside the bar.
shown := drawStalled(params, 100, 45678, 12345,
4*time.Minute+30*time.Second)
want := params.label + ": [12345/" + approx + "45678] 27%, stalled " +
params.stalledOn + ", elapsed 4m30s, eta 12m9s"
if !strings.Contains(shown, want) {
t.Errorf("terminal shows %q, want %q", shown, want)
for _, want := range []string{
params.label + ": [12345/" + approx + "45678]",
", stalled " + params.stalledOn + ", elapsed 4m30s, remain 12m9s",
} {
if !strings.Contains(shown, want) {
t.Errorf("terminal shows %q, want %q", shown, want)
}
}
shown = drawStalled(params, 9876543, 1234567, 150*time.Minute)
shown = drawStalled(params, 80, 9876543, 1234567, 150*time.Minute)
for _, want := range []string{
params.label + ": [1234567/" + approx + "9876543] 12%, ",
", elapsed 2h30m0s, eta 17h30m0s",
", elapsed 2h30m0s, remain 17h30m0s",
} {
if !strings.Contains(shown, want) {
t.Errorf("terminal shows %q, want %q", shown, want)
@@ -706,13 +713,164 @@ func TestTerminalLinesFitWidth(t *testing.T) {
}
}
// TestTerminalLinesStartWithSpinner draws a display with no total and
// one with a bar, two and a half hours into the phase, on a terminal 120
// columns wide. Each line must start with the library's spinner in the
// first column, then a space and the phase's label.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalLinesStartWithSpinner(t *testing.T) {
stderr := captureStderr(t)
for _, params := range []progressParams{
{label: scanPhase("walk"), unit: scanUnit, total: -1},
{label: scanPhase("hash"), unit: scanUnit, total: 9876543},
} {
shown := lastFrame(drawLate(stderr, params, 120, 1234567))
spinner, rest, _ := strings.Cut(shown, " ")
if utf8.RuneCountInString(spinner) != 1 ||
!strings.HasPrefix(rest, params.label+": ") {
t.Errorf("terminal shows %q, want the spinner, a space and %q",
shown, params.label)
}
if params.total > 0 && !strings.Contains(shown, "% |") {
t.Errorf("terminal shows %q, want a bar", shown)
}
}
}
// TestTerminalBarStaysAtRightEdge draws the hash phase's display two
// and a half hours in, at two counts whose lines differ in length, on
// terminals 110 and 120 columns wide. The bar is partly filled at the
// first count and full at the second. It must start the same number of
// columns from the terminal's right edge each time, and end the same
// number of columns from it: a line that grows or shrinks, as its time
// remaining does, never moves the bar, and nor do the bar's filled
// cells.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalBarStaysAtRightEdge(t *testing.T) {
stderr := captureStderr(t)
params := progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
}
widths := []int{110, 120}
counts := []int64{1234567, 9876543}
startFromEdge := make([]int, 0, len(widths)*len(counts))
endFromEdge := make([]int, 0, len(widths)*len(counts))
for _, width := range widths {
for _, count := range counts {
shown := lastFrame(drawLate(stderr, params, width, count))
beforeBar, _, found := strings.Cut(shown, "% |")
if !found {
t.Fatalf("terminal %d wide shows %q, want a bar", width, shown)
}
// The bar ends with the last "|".
throughBar := shown[:strings.LastIndex(shown, "|")+1]
startFromEdge = append(startFromEdge,
width-utf8.RuneCountInString(beforeBar))
endFromEdge = append(endFromEdge,
width-utf8.RuneCountInString(throughBar))
}
}
if slices.Min(startFromEdge) != slices.Max(startFromEdge) {
t.Errorf("bar starts %v columns from the right edge, want the same "+
"every time", startFromEdge)
}
if slices.Min(endFromEdge) != slices.Max(endFromEdge) {
t.Errorf("bar ends %v columns from the right edge, want the same "+
"every time", endFromEdge)
}
}
// TestTerminalRateGivesWayBeforeBar draws the hash phase's display two
// and a half hours in. On a terminal 90 columns wide the line fits
// beside the bar without its rate, but not with it: the bar is drawn
// and the rate is not. On a terminal 85 columns wide the line does not
// fit beside the bar even without its rate: neither is drawn, though
// the rate would fit where the bar was.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalRateGivesWayBeforeBar(t *testing.T) {
stderr := captureStderr(t)
params := progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
}
shown := lastFrame(drawLate(stderr, params, 90, 1234567))
if !strings.Contains(shown, "% |") || strings.Contains(shown, "files/s") {
t.Errorf("terminal 90 wide shows %q, want the bar and no rate", shown)
}
shown = lastFrame(drawLate(stderr, params, 85, 1234567))
if strings.Contains(shown, "% |") || strings.Contains(shown, "files/s") {
t.Errorf("terminal 85 wide shows %q, want neither bar nor rate", shown)
}
}
// TestTerminalSpinnerTurnsBesideBar draws the hash phase's display, its
// bar partly filled, several times on a terminal 120 columns wide. The
// spinner must turn at every draw. A filled cell of the bar takes one
// column but three bytes. A line measured in bytes would seem too wide
// for the terminal, and every draw would start the display over, its
// spinner included.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalSpinnerTurnsBesideBar(t *testing.T) {
stderr := captureStderr(t)
params := progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
}
// No timed draws: the test draws.
p := startProgress(params, func() int { return 120 }, time.Hour,
time.Now)
p.mu.Lock()
p.count = 1234567
p.mu.Unlock()
const draws = 4
spinners := make([]string, 0, draws)
for range draws {
before := len(stderr())
p.draw()
spinner, _, _ := strings.Cut(lastFrame(stderr()[before:]), " ")
spinners = append(spinners, spinner)
}
p.finish()
for i := 1; i < len(spinners); i++ {
if spinners[i] == spinners[i-1] {
t.Errorf("spinners at successive draws are %q, want each to "+
"differ from the one before", spinners)
break
}
}
}
// TestTerminalNarrowedRedrawsInPlace draws a large scan's display, two
// and a half hours into the phase, on a terminal 120 columns wide, then
// makes the terminal narrower than that line and draws it again a few
// times. Nothing written after that, the erase before each redraw
// included, may reach the narrower terminal's last column: it would
// wrap, and every redraw would leave a line behind. At these widths the
// spinner stays a spinner and the bar gives way to the spinner.
// wrap, and every redraw would leave a line behind. At the narrower
// width the hash phase's bar gives way.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
@@ -730,7 +888,7 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
},
90,
80,
},
}
@@ -770,10 +928,10 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
}
}
// TestProgressWarningsOnOwnLines drives the terminal displays, spinner
// and bar, through a run of warnings with no items between them, as when
// the walk meets many unreadable paths, for many redraws: every warning
// must land on a line of its own, never inside a redraw.
// TestProgressWarningsOnOwnLines drives the terminal displays, with a
// bar and without, through a run of warnings with no items between
// them, as when the walk meets many unreadable paths, for many redraws:
// every warning must land on a line of its own, never inside a redraw.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressWarningsOnOwnLines(t *testing.T) {
@@ -819,11 +977,12 @@ func TestProgressWarningsOnOwnLines(t *testing.T) {
}
}
// TestProgressWriterRowsOnOwnLines drives the terminal displays,
// spinner and bar, through a run of rows written through the display's
// TestProgressWriterRowsOnOwnLines drives the terminal displays, with a
// bar and without, through a run of rows written through the display's
// writer to the same terminal, as stdout is in a plain "sfdupes trees",
// each written out at once, for many redraws: the terminal must show
// every row whole, in order, on a line of its own.
// every row whole, in order, on a line of its own. The terminal is 120
// columns wide, so the write phase's display has room for its bar.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressWriterRowsOnOwnLines(t *testing.T) {
@@ -840,7 +999,8 @@ func TestProgressWriterRowsOnOwnLines(t *testing.T) {
var want []string
for _, params := range cases {
p := startProgress(params, terminalWidth, testRedraw, time.Now)
p := startProgress(params, func() int { return 120 }, testRedraw,
time.Now)
out := p.writer(os.Stderr)
// No pause between rows: one is garbled only if a redraw lands
report_test.go
+2 -2
View File
@@ -187,7 +187,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
// reportOpenLine is the plain line of report's open phase when it opens
// the database at once; it prints it as the phase starts and as it ends.
const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown"
"remain unknown"
// smokeReport is report's stdout for the smoke-test records.
const smokeReport = "first\tdupe\tsize\n" +
@@ -225,7 +225,7 @@ func TestRunReportProgressNamesEachPhase(t *testing.T) {
"report: database " + path + " (from " + databaseEnv + ")",
reportOpenLine,
reportOpenLine,
"[2/2] read: counting the records, elapsed 0s, eta unknown",
"[2/2] read: counting the records, elapsed 0s, remain unknown",
"[2/2] read: [6/6] 100%, ",
"report: 6 records read, 2 duplicate groups, 4 dupe files, " +
"6.1 KiB reclaimable",
scan.go
+57 -18
View File
@@ -418,7 +418,8 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
// roots and are not deleted as unverified. An operand on a filesystem
// in rules that is not persistent is dropped the same way, with a line
// saying so that is not a warning, and is not counted as skipped. An
// operand that fails lstat here is kept, and the walk warns about it.
// operand that fails lstat here is kept, and the walk warns about it
// unless it no longer exists.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
kept := make([]string, 0, len(roots))
@@ -629,8 +630,9 @@ func isDatabaseFile(f fileRec, rules walkRules) bool {
// (an interrupted scan resumes cheaply: the next run skips everything
// already recorded). A run that fails to hash is warned about and
// counted as skipped; stale records for its paths, if any, are deleted
// by the update phase. The total counts actual reads, so the display
// shows a real ETA.
// by the update phase. A run whose file no longer exists gets no record
// and no warning, and is not counted as skipped. The total counts actual
// reads, so the display's time remaining is meaningful.
//
// Returning early — a failed database write, or a cancelled scan — must
// not strand the pool: the feeder would park forever on a full jobs
@@ -662,6 +664,11 @@ func (s *scanState) hashPhase(ctx context.Context, workers int) error {
prog.increment()
// A file removed since the walk is no error: it gets no record.
if errors.Is(r.err, fs.ErrNotExist) {
continue
}
if r.err != nil {
s.st.skipped += len(r.run)
@@ -829,10 +836,12 @@ type walkEvent struct {
// startWalk seeds every root into the shared walk worker pool and
// returns the event stream: one record per regular file, one warning
// event per per-path error, and one event per path -x or rules kept the
// walk from entering or recording. The channel is closed when the walk
// completes, and also when ctx is cancelled — every goroutine in the
// pool abandons its blocking send in that case, so the consumer sees a
// truncated but properly terminated stream instead of a stalled one.
// walk from entering or recording. A path that no longer exists when
// the walk reaches it is no error, and gets no event. The channel is
// closed when the walk completes, and also when ctx is cancelled —
// every goroutine in the pool abandons its blocking send in that case,
// so the consumer sees a truncated but properly terminated stream
// instead of a stalled one.
func startWalk(ctx context.Context, roots []string, oneFS bool,
rules walkRules, workers int,
) <-chan walkEvent {
@@ -899,13 +908,19 @@ func operandWarning(root string, fi fs.FileInfo) string {
// one of the scan's own database files in rules, and a directory
// operand becomes an initial job. walkableRoots has already dropped
// every other operand. One that has changed into something else since
// is warned about and skipped here; it is still a root, so the records
// stored beneath it, other than those under another root, are deleted
// as unverified.
// is warned about and skipped here, and one removed since is skipped
// without a warning. Either is still a root, so the records stored
// beneath it, other than those under another root, are deleted as
// unverified.
func seedRoot(ctx context.Context, root string, rules walkRules,
events chan<- walkEvent,
) []dirJob {
fi, err := os.Lstat(root)
if errors.Is(err, fs.ErrNotExist) {
// Removed since the scan started: no error, and no files.
return nil
}
if err != nil {
sendEvent(ctx, events, walkEvent{
warn: fmt.Sprintf("walk %s: %v", root, err),
@@ -1023,11 +1038,17 @@ func dispatchDirs(ctx context.Context, initial []dirJob,
// walkOneDir reads one directory, emitting an event per regular-file
// entry and a warning event per unreadable one, and returns the
// subdirectories to descend into.
// subdirectories to descend into. A directory or entry removed since
// the walk found it gets no event.
func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
rules walkRules, events chan<- walkEvent,
) []dirJob {
entries, err := os.ReadDir(job.path)
if errors.Is(err, fs.ErrNotExist) {
// Removed since its parent was read: no error, and no files.
return nil
}
if err != nil {
sendEvent(ctx, events, walkEvent{
warn: fmt.Sprintf("walk %s: %v", job.path, err),
@@ -1070,16 +1091,22 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
// emitFile stats one regular directory entry and emits its record.
// The lstat happens here in the walk worker, while the directory's
// metadata is still hot; a path that fails to stat (or stops being a
// regular file) between the directory read and the lstat is warned
// about and skipped. A file on a filesystem in rules that is not
// persistent, as through a bind mount of one file, is not recorded: it
// gets an event naming it instead. One of the scan's own database files
// in rules gets no event at all.
// metadata is still hot; a path that fails to stat between the
// directory read and the lstat is warned about and skipped, unless it
// no longer exists, and one that has stopped being a regular file is
// skipped. A file on a filesystem in rules that is not persistent, as
// through a bind mount of one file, is not recorded: it gets an event
// naming it instead. One of the scan's own database files in rules gets
// no event at all.
func emitFile(ctx context.Context, p string, e fs.DirEntry,
rules walkRules, events chan<- walkEvent,
) {
info, err := e.Info()
if errors.Is(err, fs.ErrNotExist) {
// Removed since its directory was read: no error.
return
}
if err != nil {
sendEvent(ctx, events, walkEvent{
warn: fmt.Sprintf("stat %s: %v", p, err),
@@ -1131,6 +1158,11 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
}
info, err := e.Info()
if errors.Is(err, fs.ErrNotExist) {
// Removed since its parent was read: no error, and no files.
return dirJob{}, false
}
if err != nil {
sendEvent(ctx, events, walkEvent{
warn: fmt.Sprintf("walk %s: %v", p, err),
@@ -1195,7 +1227,7 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
// hashResult carries the hashes computed for one inode run (or the
// error that prevented computing them) from the pool's workers to the
// hash phase.
// hash phase. Its run leaves out the paths found removed.
type hashResult struct {
run []fileRec
head [sha256.Size]byte
@@ -1296,6 +1328,13 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
head, tail, content, err := hash(run[0].path, run[0].size)
// A path removed since the walk is dropped from its run. The
// next path is a hard link to the same file, and is read instead.
for errors.Is(err, fs.ErrNotExist) && len(run) > 1 {
run = run[1:]
head, tail, content, err = hash(run[0].path, run[0].size)
}
select {
case results <- hashResult{
run: run, head: head, tail: tail, content: content, err: err,
scan_test.go
+155 -4
View File
@@ -683,6 +683,108 @@ func TestSubdirJobStatError(t *testing.T) {
}
}
// TestWalkPassesOverRemovedPaths removes a file and a directory after
// their parent was read, as when they are deleted while a scan runs.
// The walk then reaches them, the directory also as one to read and as
// an operand. None of that is an error: the walk sends no event.
func TestWalkPassesOverRemovedPaths(t *testing.T) {
t.Parallel()
dir := t.TempDir()
file := writeFile(t, dir, "file", []byte("x"))
writeFile(t, dir, "sub/f", []byte("y"))
sub := filepath.Join(dir, "sub")
fileEntry := dirEntryFor(t, dir, "file")
subEntry := dirEntryFor(t, dir, "sub")
for _, p := range []string{file, sub} {
err := os.RemoveAll(p)
if err != nil {
t.Fatal(err)
}
}
ctx := t.Context()
events := make(chan walkEvent, 4)
emitFile(ctx, file, fileEntry, walkRules{}, events)
_, descend := subdirJob(ctx, sub, subEntry, dirJob{path: dir}, false,
walkRules{}, events)
subs := walkOneDir(ctx, dirJob{path: sub}, false, walkRules{}, events)
jobs := seedRoot(ctx, sub, walkRules{}, events)
close(events)
for ev := range events {
t.Errorf("event %+v for a removed path, want none", ev)
}
if descend || len(subs) != 0 || len(jobs) != 0 {
t.Errorf("descend = %t, subdirectories %+v, jobs %+v, want none",
descend, subs, jobs)
}
}
// TestScanWarnsAboutUnreadablePaths checks that a permission error is
// still warned about and counted as skipped, unlike a removed path. The
// walk reaches an unreadable directory, and a file in a directory that
// can be read but not searched. Another such file is an operand.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores file permissions")
}
stderr := captureStderr(t)
dir := t.TempDir()
tree := filepath.Join(dir, "tree")
walked := writeFile(t, tree, "unsearchable/f", []byte("x"))
operand := writeFile(t, dir, "unsearchable/f", []byte("y"))
unreadable := filepath.Join(tree, "unreadable")
err := os.Mkdir(unreadable, 0o750)
if err != nil {
t.Fatal(err)
}
modes := map[string]fs.FileMode{
unreadable: 0,
filepath.Dir(walked): 0o400,
filepath.Dir(operand): 0o400,
}
for p, mode := range modes {
err = os.Chmod(p, mode)
if err != nil {
t.Fatal(err)
}
// Runs before t.TempDir's own cleanup, which must delete the files.
t.Cleanup(func() {
//nolint:gosec // removing the files needs the search bit back
_ = os.Chmod(p, 0o700)
})
}
st := syncTree(t, openTestDB(t), tree, operand)
if st != (scanStats{skipped: 3}) {
t.Errorf("stats = %+v, want 3 skipped and nothing else", st)
}
got := stderr()
for _, warning := range []string{
"walk " + unreadable + ": open " + unreadable + ": permission denied\n",
"stat " + walked + ": lstat " + walked + ": permission denied\n",
"walk " + operand + ": lstat " + operand + ": permission denied\n",
} {
if !strings.Contains(got, warning) {
t.Errorf("stderr = %q, want %q", got, warning)
}
}
}
// syncOneFilesystem scans roots into db with -x and returns the scan
// stats. Every device pruneRoots looks up, and the device each
// operand's walk compares its subdirectories with, comes from devices
@@ -1128,7 +1230,7 @@ func TestScanPipeline(t *testing.T) {
// database's records. Each phase's first line names it with its place
// among the four: load counts the records before reading them, and the
// walk estimates its total from the records under its operand, or says
// its ETA is unknown when there are none.
// its time remaining is unknown when there are none.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanProgressNamesEachPhase(t *testing.T) {
@@ -1143,14 +1245,14 @@ func TestScanProgressNamesEachPhase(t *testing.T) {
syncTree(t, db, filepath.Join(dir, "a"))
rescan := stderr()[len(first):]
load := "[1/4] load: counting the records, elapsed 0s, eta unknown"
load := "[1/4] load: counting the records, elapsed 0s, remain unknown"
cases := map[string][]string{
first: {
load, "[2/4] walk: 0 files, 0 files/s, elapsed 0s, eta unknown",
load, "[2/4] walk: 0 files, 0 files/s, elapsed 0s, remain unknown",
"[3/4] hash: [0/", "[4/4] update: [0/",
},
rescan: {
load, "[2/4] walk: [0/~2] 0%, 0 files/s, elapsed 0s, eta unknown",
load, "[2/4] walk: [0/~2] 0%, 0 files/s, elapsed 0s, remain unknown",
"[3/4] hash: [0/", "[4/4] update: [0/",
},
}
@@ -1685,6 +1787,55 @@ func TestHashPhaseSkipsFileReplacedByFIFO(t *testing.T) {
}
}
// TestHashPhasePassesOverRemovedFiles removes two walked files before
// the hash phase reads them: one with no other path, and the first of
// two hard links to one file. Neither is an error. The phase prints
// nothing but its progress, counts nothing as skipped, and records only
// the hard link still there.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestHashPhasePassesOverRemovedFiles(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
alone := writeFile(t, dir, "alone.bin", pattern(1, 300))
a := writeFile(t, dir, "a.bin", pattern(2, 300))
b := filepath.Join(dir, "b.bin")
err := os.Link(a, b)
if err != nil {
t.Fatal(err)
}
walked, _ := collectWalk(t, []string{dir}, false, 1)
// a.bin sorts before b.bin, so the run of the two reads it first.
for _, p := range []string{alone, a} {
err = os.Remove(p)
if err != nil {
t.Fatal(err)
}
}
s := &scanState{db: openTestDB(t), toHash: walked}
err = s.hashPhase(t.Context(), 1)
if err != nil {
t.Fatalf("hashPhase = %v, want nil", err)
}
if s.st.skipped != 0 || len(s.batch) != 1 || s.batch[0].path != b {
t.Errorf("skipped = %d, records %+v, want none skipped and a "+
"record for %s alone", s.st.skipped, s.batch, b)
}
for line := range strings.Lines(stderr()) {
if !strings.HasPrefix(line, scanPhase("hash")+": ") {
t.Errorf("stderr holds %q, want the hash phase's progress alone",
line)
}
}
}
// injectedWriteFailure is the message the injected database trigger
// aborts with, so the test can recognize its own failure in the error
// the scan reports.
trees_test.go
+4 -4
View File
@@ -13,7 +13,7 @@ import (
// treesOpenLine is the plain line of trees' open phase when it opens
// the database at once; it prints it as the phase starts and as it ends.
const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
"eta unknown"
"remain unknown"
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
// /d/t2 are identical, /d/t3 differs from them only by one filename.
@@ -236,11 +236,11 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
"trees: database " + path + " (from " + databaseEnv + ")",
treesOpenLine,
treesOpenLine,
"[2/4] read: counting the records, elapsed 0s, eta unknown",
"[2/4] read: counting the records, elapsed 0s, remain unknown",
"[2/4] read: [6/6] 100%, ",
"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown",
"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, remain unknown",
"[3/4] group: [8/8] 100%, ",
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown",
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown",
"[4/4] write: [1/1] 100%, ",
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
}