Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3f4ff92dae |
@@ -799,7 +799,8 @@ A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
|
||||
`head`, and `tail`, without fetching the records. It then says it is searching
|
||||
the database until SQLite returns the first record; the checks, and their rate
|
||||
and ETA, start there. The files to read then get a display of their own, with
|
||||
their count as its total.
|
||||
their count as its total, and so does the writing of the last of their records
|
||||
to the database, with the count of those records as its total.
|
||||
|
||||
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
|
||||
`[4/4] write`, see "`trees` mode"):
|
||||
@@ -837,7 +838,7 @@ Each display is one line holding:
|
||||
or, when stderr is not a terminal, since the line before, so it falls to 0
|
||||
while nothing completes, and a burst of items after a wait shows as a burst
|
||||
rather than as a steady rate. Once no item has completed for 10 seconds, the
|
||||
word `stalled` and what the phase is doing take the rate's place (see below).
|
||||
word `stalled` and where the phase waits take the rate's place (see below).
|
||||
- elapsed time
|
||||
- estimated time remaining, or `unknown`. It comes from the phase's average rate
|
||||
since it started counting, so that it does not swing with each burst.
|
||||
@@ -847,18 +848,20 @@ Example shape (exact layout is flexible, content is not):
|
||||
```
|
||||
[3/5] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
|
||||
[2/5] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
|
||||
[3/5] hash: [12345/45678] 27%, stalled reading, elapsed 4m30s, eta 12m9s
|
||||
[3/5] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
|
||||
```
|
||||
|
||||
When no item has completed for 10 seconds, as while one large file is read, a
|
||||
disk is slow or the database is busy, the line says `stalled` and what the phase
|
||||
is doing in place of the rate, which would read 0: `walk` is `listing`
|
||||
directories; `hash` and the content phase's reads are `reading` files; `update`
|
||||
is `writing` to the database; `load` and the `read` phase of `trees` are
|
||||
`reading` the database; the content phase's checks are `searching` the database;
|
||||
`trees`' `group` is `grouping` directories, and its `write` is `writing` the
|
||||
report. The `read` phase of `report` writes the report while it reads the
|
||||
records and may be waiting on either, so it says `read/write`. A phase still
|
||||
disk is slow or the database is busy, the line says `stalled` and where the
|
||||
phase waits in place of the rate, which would read 0: `disk` for files, `db` for
|
||||
the database, `out` for stdout, and two of them, as `disk/db`, for a phase that
|
||||
may wait on either. `walk` says `disk`. `hash` and the content phase's reads say
|
||||
`disk/db`, as they read files and write their records to the database between
|
||||
them, and so do the content phase's checks, which `lstat` each record's file as
|
||||
the database search returns it. `load`, `update`, the content phase's last write
|
||||
and the `read` phase of `trees` say `db`. `trees`' `write` says `out`, and its
|
||||
`group`, which works in memory, says `cpu`. The `read` phase of `report` writes
|
||||
the report while it reads the records, so it says `db/out`. A phase still
|
||||
waiting for its total already says what it waits for, and says nothing more.
|
||||
|
||||
Additional requirements:
|
||||
@@ -872,18 +875,18 @@ Additional requirements:
|
||||
times a second, so its elapsed time keeps moving while a phase waits for its
|
||||
next item.
|
||||
- On a terminal no line reaches the terminal's last column, however narrow the
|
||||
terminal is. The phase and its place, the progress, what a stalled phase is
|
||||
doing, the elapsed time and the ETA are kept whole: a line too wide for the
|
||||
terminal leaves out its rate (or what the phase is waiting for), and a bar
|
||||
with no room beside the parts kept whole gives way to the spinner. The words a
|
||||
stalled phase shows are short, so that a line like the stalled one in the
|
||||
example above, with tens of thousands of files and times in minutes, fits on
|
||||
an 80-column terminal. Only when the parts kept whole are too wide, as on a
|
||||
terminal narrower than about 70 columns hours into a scan of millions of
|
||||
files, or about 90 once the phase has stalled, is the line cut short at the
|
||||
terminal's edge, so the end of the ETA is lost first. A terminal made narrower
|
||||
while a phase runs gets the same: the next draw erases the line within the new
|
||||
width and fits the line to it, and the display goes on redrawing in place.
|
||||
terminal is. A line too wide for the terminal first loses its bar, which gives
|
||||
way to the spinner, then its rate, the words of a stalled phase, or what the
|
||||
phase is waiting for. The phase and its place, the progress, the elapsed time
|
||||
and the ETA are kept whole. On an 80-column terminal, a stalled line at counts
|
||||
and times like those of the stalled line in the example above keeps its words
|
||||
whole in every phase; where the words and the ETA do not both fit, as at
|
||||
millions of files and times in hours, the words give way. Only when those four
|
||||
alone are too wide, as on a terminal narrower than about 70 columns hours into
|
||||
a scan of millions of files, is the line cut short at the terminal's edge, so
|
||||
the end of the ETA is lost first. A terminal made narrower while a phase runs
|
||||
gets the same: the next draw erases the line within the new width and fits the
|
||||
line to it, and the display goes on redrawing in place.
|
||||
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
|
||||
redraws: print the line the moment each phase starts, every 5 seconds while it
|
||||
runs (whether or not an item has completed, and never more often), and once
|
||||
|
||||
@@ -34,12 +34,14 @@
|
||||
to 0 while nothing completes, and a burst after a wait no longer reads as a
|
||||
steady rate of over 1,000 files a second; the ETA still comes from the phase's
|
||||
average. Once no item has completed for 10 seconds, the word `stalled` and
|
||||
what the phase is doing take the rate's place, as `stalled reading`, each
|
||||
phase naming its own work where it is defined; `report`'s `read` says
|
||||
`read/write`, as it writes the report while it reads. Those words are kept
|
||||
whole with the phase, progress, elapsed time and ETA, so only the bar gives
|
||||
way before them, and they fit on an 80-column terminal in an ordinary scan
|
||||
(2026-10-16, https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||
where the phase waits take the rate's place, as `stalled disk/db`: `disk`,
|
||||
`db`, `out` for stdout or `cpu`, or two of them for a phase that may wait on
|
||||
either, each phase giving its word where it is defined. A line too wide for
|
||||
the terminal loses its bar first, then its rate or those words, so the phase,
|
||||
progress, elapsed time and ETA stay whole; on an 80-column terminal the words
|
||||
fit beside them in an ordinary scan. The content phase's last partial batch is
|
||||
now written under a display of its own (2026-10-16,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/125)
|
||||
|
||||
- new subcommand `info` prints the database's details on stdout, one per line as
|
||||
a name, a tab and the value: its path and where it came from, the sizes of the
|
||||
|
||||
+23
-30
@@ -28,7 +28,7 @@ const redrawInterval = 100 * time.Millisecond
|
||||
const rateWindow = 5 * time.Second
|
||||
|
||||
// stallTime is how long a phase goes with no item completed before its
|
||||
// line says it has stalled, and what it is doing, in place of the rate.
|
||||
// line says it has stalled, and where it waits, in place of the rate.
|
||||
const stallTime = 10 * time.Second
|
||||
|
||||
// spinnerType selects the progressbar library's spinner style.
|
||||
@@ -68,10 +68,6 @@ const (
|
||||
openingDatabase = "opening the database"
|
||||
)
|
||||
|
||||
// reading is what the display of a phase that reads files or records
|
||||
// says it is doing once it has stalled; see stallTime.
|
||||
const reading = "reading"
|
||||
|
||||
// stderrIsTTY reports whether stderr is attached to a terminal.
|
||||
func stderrIsTTY() bool {
|
||||
return term.IsTerminal(int(os.Stderr.Fd()))
|
||||
@@ -102,7 +98,7 @@ type progressParams struct {
|
||||
total int64 // the items the phase will complete; -1 when not known
|
||||
estimate bool // total is an estimate, which the count may pass
|
||||
waitingFor string // what the phase waits for until setTotal is called
|
||||
doing string // what the phase is doing, as "reading"; see stallTime
|
||||
stalledOn string // where the phase waits, as "disk/db"; see stallTime
|
||||
}
|
||||
|
||||
// drawnCount is the count of items completed as of one draw, and when
|
||||
@@ -142,7 +138,7 @@ type progress struct {
|
||||
total int64
|
||||
estimate bool
|
||||
waitingFor string
|
||||
doing string
|
||||
stalledOn string
|
||||
count int64
|
||||
start time.Time // when the phase started, for the elapsed time
|
||||
rateStart time.Time // when the count started, for the ETA
|
||||
@@ -176,7 +172,7 @@ func startProgress(params progressParams, width func() int,
|
||||
stop: make(chan struct{}), done: make(chan struct{}),
|
||||
label: params.label, unit: params.unit, total: params.total,
|
||||
estimate: params.estimate, waitingFor: params.waitingFor,
|
||||
doing: params.doing, start: start, rateStart: start,
|
||||
stalledOn: params.stalledOn, start: start, rateStart: start,
|
||||
lastItem: start, drawn: []drawnCount{{at: start}},
|
||||
}
|
||||
|
||||
@@ -219,26 +215,27 @@ func (p *progress) draw() {
|
||||
}
|
||||
|
||||
if p.width == nil {
|
||||
text, _ := p.line(now, math.MaxInt, true)
|
||||
fmt.Fprintln(os.Stderr, text)
|
||||
fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true))
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
width := p.width()
|
||||
|
||||
// A phase with an exact total gets the library's bar when its line,
|
||||
// without the percent the bar shows, fits whole beside the bar. A
|
||||
// total of 0 leaves the bar nothing to fill.
|
||||
// A phase with an exact total gets the library's bar when its whole
|
||||
// line, without the percent the bar shows, fits beside the bar: the
|
||||
// bar gives way before any part of the line. A total of 0 leaves the
|
||||
// bar nothing to fill.
|
||||
text, withBar := "", false
|
||||
if p.total > 0 && !p.estimate {
|
||||
text, withBar = p.line(now, width-barMargin, false)
|
||||
text = p.line(now, math.MaxInt, false)
|
||||
withBar = len(text) <= width-barMargin
|
||||
}
|
||||
|
||||
margin := barMargin
|
||||
if !withBar {
|
||||
margin = spinnerMargin
|
||||
text, _ = p.line(now, width-margin, true)
|
||||
text = p.line(now, width-margin, true)
|
||||
}
|
||||
|
||||
// The library makes a bar or a spinner, and before each draw erases
|
||||
@@ -449,17 +446,14 @@ func (p *progress) finish() {
|
||||
// rate is that since the first draw in drawn, and the ETA comes from
|
||||
// the rate since the count started. Once no item has been completed for
|
||||
// stallTime, the rate, which would read 0, gives way to the word
|
||||
// "stalled" and what the phase is doing, as "stalled reading".
|
||||
// "stalled" and where the phase waits, as "stalled disk/db".
|
||||
//
|
||||
// The phase and its place, the progress, what a stalled phase is doing,
|
||||
// the elapsed time and the ETA are kept whole. A line wider than room
|
||||
// leaves out the rate, or what the phase waits for; only if it is still
|
||||
// wider is it cut at room, and line reports that it does not fit. Every
|
||||
// part is ASCII, so its length is its width in columns. p.mu must be
|
||||
// held.
|
||||
func (p *progress) line(now time.Time, room int,
|
||||
percent bool,
|
||||
) (string, bool) {
|
||||
// The phase and its place, the progress, the elapsed time and the ETA
|
||||
// are kept whole. A line wider than room leaves out the rate, the words
|
||||
// of a stalled phase, or what the phase waits for; only if it is still
|
||||
// wider is it cut at room. Every part is ASCII, so its length is its
|
||||
// width in columns. p.mu must be held.
|
||||
func (p *progress) line(now time.Time, room int, percent bool) string {
|
||||
elapsed := now.Sub(p.start).Round(time.Second)
|
||||
|
||||
// head and tail hold the parts kept whole; extra, between them, is
|
||||
@@ -479,8 +473,7 @@ func (p *progress) line(now time.Time, room int,
|
||||
tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average))
|
||||
|
||||
if now.Sub(p.lastItem) >= stallTime {
|
||||
head += "stalled " + p.doing + ", "
|
||||
extra = ""
|
||||
extra = "stalled " + p.stalledOn + ", "
|
||||
}
|
||||
}
|
||||
|
||||
@@ -488,12 +481,12 @@ func (p *progress) line(now time.Time, room int,
|
||||
|
||||
switch {
|
||||
case len(full) <= room:
|
||||
return full, true
|
||||
return full
|
||||
case len(kept) <= room:
|
||||
return kept, true
|
||||
return kept
|
||||
}
|
||||
|
||||
return kept[:max(room, 0)], false
|
||||
return kept[:max(room, 0)]
|
||||
}
|
||||
|
||||
// perSecond returns count items over d as items a second, or 0 when no
|
||||
|
||||
+97
-71
@@ -22,6 +22,10 @@ const testRedraw = time.Millisecond
|
||||
// it expects.
|
||||
const testDeadline = 5 * time.Second
|
||||
|
||||
// hashStalledOn is where the hash phase says it waits once it has
|
||||
// stalled, as readRuns gives it.
|
||||
const hashStalledOn = "disk/db"
|
||||
|
||||
// testClock is a clock a test moves by hand, for a display that reads
|
||||
// it from its own goroutine too.
|
||||
type testClock struct {
|
||||
@@ -159,7 +163,7 @@ func TestProgressLine(t *testing.T) {
|
||||
tc.p.start, tc.p.rateStart, tc.p.lastItem = ago, ago, now
|
||||
tc.p.drawn = []drawnCount{{at: ago}}
|
||||
|
||||
if got, _ := tc.p.line(now, math.MaxInt, true); got != tc.want {
|
||||
if got := tc.p.line(now, math.MaxInt, true); got != tc.want {
|
||||
t.Errorf("line = %q, want %q", got, tc.want)
|
||||
}
|
||||
}
|
||||
@@ -167,9 +171,9 @@ func TestProgressLine(t *testing.T) {
|
||||
|
||||
// TestLineFitsRoom checks how a line is fitted to the room it has, ten
|
||||
// seconds into the phase, and fifty into a phase with no file read for
|
||||
// the last 42: it leaves out the rate, or what the phase waits for, but
|
||||
// not what a stalled phase is doing, and is cut only when what is left
|
||||
// is still too wide.
|
||||
// the last 42: it leaves out the rate, what the phase waits for, or the
|
||||
// words of a stalled phase, and is cut only when what is left is still
|
||||
// too wide.
|
||||
func TestLineFitsRoom(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -188,7 +192,7 @@ func TestLineFitsRoom(t *testing.T) {
|
||||
started := now.Add(-50 * time.Second)
|
||||
stalled := &progress{
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
|
||||
doing: reading, start: started, rateStart: started,
|
||||
stalledOn: hashStalledOn, start: started, rateStart: started,
|
||||
lastItem: now.Add(-42 * time.Second),
|
||||
drawn: []drawnCount{{at: now.Add(-rateWindow), count: 20}},
|
||||
}
|
||||
@@ -198,52 +202,44 @@ func TestLineFitsRoom(t *testing.T) {
|
||||
room int
|
||||
percent bool
|
||||
want string
|
||||
fits bool
|
||||
}{{
|
||||
p: hash, room: 57, percent: true,
|
||||
want: "[3/5] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s",
|
||||
fits: true,
|
||||
}, {
|
||||
p: hash, room: 56, percent: true,
|
||||
want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40s",
|
||||
fits: true,
|
||||
}, {
|
||||
p: hash, room: 45, percent: true,
|
||||
want: "[3/5] hash: [20/100] 20%, elapsed 10s, eta 40",
|
||||
fits: false,
|
||||
}, {
|
||||
// Before the library's bar, which shows the percent.
|
||||
p: hash, room: 57, percent: false,
|
||||
want: "[3/5] hash: [20/100], 2 files/s, elapsed 10s, eta 40s",
|
||||
fits: true,
|
||||
}, {
|
||||
p: content, room: 40, percent: true,
|
||||
want: "[5/5] content: elapsed 10s, eta unknown",
|
||||
fits: true,
|
||||
}, {
|
||||
p: stalled, room: 65, percent: true,
|
||||
want: "[3/5] hash: [20/100] 20%, stalled reading, elapsed 50s, " +
|
||||
want: "[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " +
|
||||
"eta 3m20s",
|
||||
fits: true,
|
||||
}, {
|
||||
p: stalled, room: 64, percent: true,
|
||||
want: "[3/5] hash: [20/100] 20%, stalled reading, elapsed 50s, " +
|
||||
"eta 3m20",
|
||||
fits: false,
|
||||
want: "[3/5] hash: [20/100] 20%, elapsed 50s, eta 3m20s",
|
||||
}}
|
||||
|
||||
for _, tc := range cases {
|
||||
got, fits := tc.p.line(now, tc.room, tc.percent)
|
||||
if got != tc.want || fits != tc.fits {
|
||||
t.Errorf("line(%d, %t) = %q, %t, want %q, %t",
|
||||
tc.room, tc.percent, got, fits, tc.want, tc.fits)
|
||||
if got := tc.p.line(now, tc.room, tc.percent); got != tc.want {
|
||||
t.Errorf("line(%d, %t) = %q, want %q",
|
||||
tc.room, tc.percent, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSetTotalStartsRate checks that setTotal starts a phase's rate,
|
||||
// its ETA and its time with no record read from when the total becomes
|
||||
// known, not from when the phase started waiting for it.
|
||||
// known, not from when the phase started waiting for it: after 20
|
||||
// seconds of waiting for the total, the phase has not stalled 5 seconds
|
||||
// after it, and has 10 seconds after it with no record read.
|
||||
func TestSetTotalStartsRate(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -251,28 +247,33 @@ func TestSetTotalStartsRate(t *testing.T) {
|
||||
waited := clock.Now().Add(-20 * time.Second)
|
||||
p := &progress{
|
||||
now: clock.Now, label: scanPhase("load"), unit: scanUnit, total: -1,
|
||||
doing: reading, start: waited, rateStart: waited,
|
||||
stalledOn: "db", start: waited, rateStart: waited,
|
||||
lastItem: waited, drawn: []drawnCount{{at: waited}},
|
||||
}
|
||||
|
||||
p.setTotal(100)
|
||||
|
||||
// Ten seconds after the total came, no record has been read yet.
|
||||
clock.Add(10 * time.Second)
|
||||
clock.Add(5 * time.Second)
|
||||
|
||||
want := "[1/5] load: [0/100] 0%, stalled reading, elapsed 30s, " +
|
||||
"eta unknown"
|
||||
if got, _ := p.line(clock.Now(), math.MaxInt, true); got != want {
|
||||
want := "[1/5] load: [0/100] 0%, 0 files/s, elapsed 25s, eta unknown"
|
||||
if got := p.line(clock.Now(), math.MaxInt, true); got != want {
|
||||
t.Errorf("line = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
// Then 20 are.
|
||||
clock.Add(5 * time.Second)
|
||||
|
||||
want = "[1/5] load: [0/100] 0%, stalled db, elapsed 30s, eta unknown"
|
||||
if got := p.line(clock.Now(), math.MaxInt, true); got != want {
|
||||
t.Errorf("line = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
// Then 20 records are read.
|
||||
for range 20 {
|
||||
p.increment()
|
||||
}
|
||||
|
||||
want = "[1/5] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s"
|
||||
if got, _ := p.line(clock.Now(), math.MaxInt, true); got != want {
|
||||
if got := p.line(clock.Now(), math.MaxInt, true); got != want {
|
||||
t.Errorf("line = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
@@ -343,8 +344,8 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) {
|
||||
// coming, as when one large file is being read or a disk is slow, goes
|
||||
// on printing its line every interval: the rate falls to 0 once no file
|
||||
// has been read for rateWindow, and once none has for stallTime the
|
||||
// line says the phase has stalled, and what it is doing, in place of
|
||||
// the rate. The test moves the display's clock by hand.
|
||||
// line says the phase has stalled, and where it waits, in place of the
|
||||
// rate. The test moves the display's clock by hand.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestPlainProgressStalledProducer(t *testing.T) {
|
||||
@@ -352,7 +353,7 @@ func TestPlainProgressStalledProducer(t *testing.T) {
|
||||
clock := &testClock{now: time.Now()}
|
||||
p := startProgress(progressParams{
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 100,
|
||||
doing: reading,
|
||||
stalledOn: hashStalledOn,
|
||||
}, nil, testRedraw, clock.Now)
|
||||
|
||||
// Pass or fail, the display ends before stderr is put back.
|
||||
@@ -373,7 +374,7 @@ func TestPlainProgressStalledProducer(t *testing.T) {
|
||||
|
||||
clock.Add(4 * time.Second)
|
||||
awaitOutput(t, stderr,
|
||||
"[3/5] hash: [20/100] 20%, stalled reading, elapsed 10s, eta 40s\n",
|
||||
"[3/5] hash: [20/100] 20%, stalled disk/db, elapsed 10s, eta 40s\n",
|
||||
3)
|
||||
}
|
||||
|
||||
@@ -391,7 +392,7 @@ func TestPlainProgressRateAfterStall(t *testing.T) {
|
||||
// No timed draws: the test draws.
|
||||
p := startProgress(progressParams{
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 1000,
|
||||
doing: reading,
|
||||
stalledOn: hashStalledOn,
|
||||
}, nil, time.Hour, clock.Now)
|
||||
|
||||
late := plainInterval + time.Millisecond
|
||||
@@ -411,7 +412,7 @@ func TestPlainProgressRateAfterStall(t *testing.T) {
|
||||
want := []string{
|
||||
"[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown",
|
||||
"[3/5] hash: [0/1000] 0%, 0 files/s, elapsed 5s, eta unknown",
|
||||
"[3/5] hash: [0/1000] 0%, stalled reading, elapsed 10s, eta unknown",
|
||||
"[3/5] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, eta unknown",
|
||||
"[3/5] hash: [100/1000] 10%, 20 files/s, elapsed 15s, eta 2m15s",
|
||||
}
|
||||
|
||||
@@ -436,7 +437,7 @@ func TestProgressRateAfterStallRedrawn(t *testing.T) {
|
||||
// No timed draws: the test draws.
|
||||
p := startProgress(progressParams{
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 1000,
|
||||
doing: reading,
|
||||
stalledOn: hashStalledOn,
|
||||
}, nil, time.Hour, clock.Now)
|
||||
|
||||
// redraw draws the display every redrawInterval for d.
|
||||
@@ -599,56 +600,81 @@ func TestTerminalLinesKeepETA(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestTerminalStalledLineKeepsWords draws a stalled phase on an
|
||||
// 80-column terminal at an ordinary scan's counts and times, 12345 of
|
||||
// 45678 files read four and a half minutes in, the last 42 seconds ago,
|
||||
// and report's read phase at the same counts: the bar gives way, and
|
||||
// what the phase is doing shows whole in place of the rate, beside the
|
||||
// phase, progress, elapsed time and ETA.
|
||||
// TestTerminalStalledLines draws every phase's line on an 80-column
|
||||
// terminal once the phase has stalled, its last item done 42 seconds
|
||||
// before. At an ordinary scan's counts and times, 12345 of 45678 items
|
||||
// done four and a half minutes in, the line shows where the phase waits
|
||||
// whole, beside the phase, progress, elapsed time and ETA. At a large
|
||||
// scan's, 1234567 of 9876543 two and a half hours in, those four still
|
||||
// show whole, as the words give way first where they do not fit.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestTerminalStalledLineKeepsWords(t *testing.T) {
|
||||
func TestTerminalStalledLines(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
|
||||
cases := []struct {
|
||||
params progressParams
|
||||
want string
|
||||
}{{
|
||||
params: progressParams{
|
||||
label: scanPhase("hash"), unit: scanUnit, total: 45678,
|
||||
doing: reading,
|
||||
},
|
||||
want: "[3/5] hash: [12345/45678] 27%, stalled reading, " +
|
||||
"elapsed 4m30s, eta 12m9s",
|
||||
}, {
|
||||
params: progressParams{
|
||||
label: reportPhase("read"), total: 45678, doing: "read/write",
|
||||
},
|
||||
want: "[2/2] read: [12345/45678] 27%, stalled read/write, " +
|
||||
"elapsed 4m30s, eta 12m9s",
|
||||
}}
|
||||
|
||||
for _, tc := range cases {
|
||||
// drawStalled draws the display params describes as it starts and
|
||||
// once more as it ends, with count of total items done elapsed into
|
||||
// the phase, and returns what the terminal shows.
|
||||
drawStalled := func(params progressParams, total, count int64,
|
||||
elapsed time.Duration,
|
||||
) string {
|
||||
before := len(stderr())
|
||||
clock := &testClock{now: time.Now()}
|
||||
params.total = total
|
||||
|
||||
// No timed draws: the display is drawn as it starts and as it
|
||||
// ends.
|
||||
p := startProgress(tc.params, func() int { return 80 }, time.Hour,
|
||||
// No timed draws come between the two.
|
||||
p := startProgress(params, func() int { return 80 }, time.Hour,
|
||||
clock.Now)
|
||||
|
||||
clock.Add(4*time.Minute + 30*time.Second - 42*time.Second)
|
||||
clock.Add(elapsed - 42*time.Second)
|
||||
|
||||
p.mu.Lock()
|
||||
p.count, p.lastItem = 12345, clock.Now()
|
||||
p.count, p.lastItem = count, clock.Now()
|
||||
p.mu.Unlock()
|
||||
|
||||
clock.Add(42 * time.Second)
|
||||
p.finish()
|
||||
|
||||
shown := lastFrame(stderr()[before:])
|
||||
if !strings.Contains(shown, tc.want) {
|
||||
t.Errorf("terminal shows %q, want %q", shown, tc.want)
|
||||
return lastFrame(stderr()[before:])
|
||||
}
|
||||
|
||||
phases := []progressParams{
|
||||
{label: scanPhase("load"), stalledOn: "db"},
|
||||
{label: scanPhase("walk"), estimate: true, stalledOn: "disk"},
|
||||
{label: scanPhase("hash"), stalledOn: hashStalledOn},
|
||||
{label: scanPhase("update"), stalledOn: "db"},
|
||||
// The content phase's checks and reads, then its last write.
|
||||
{label: scanPhase("content"), stalledOn: "disk/db"},
|
||||
{label: scanPhase("content"), stalledOn: "db"},
|
||||
{label: treesPhase("read"), stalledOn: "db"},
|
||||
{label: treesPhase("group"), stalledOn: "cpu"},
|
||||
{label: treesPhase("write"), stalledOn: "out"},
|
||||
{label: reportPhase("read"), stalledOn: "db/out"},
|
||||
}
|
||||
|
||||
for _, params := range phases {
|
||||
approx := ""
|
||||
if params.estimate {
|
||||
approx = "~"
|
||||
}
|
||||
|
||||
shown := drawStalled(params, 45678, 12345,
|
||||
4*time.Minute+30*time.Second)
|
||||
want := params.label + ": [12345/" + approx + "45678] 27%, stalled " +
|
||||
params.stalledOn + ", elapsed 4m30s, eta 12m9s"
|
||||
|
||||
if !strings.Contains(shown, want) {
|
||||
t.Errorf("terminal shows %q, want %q", shown, want)
|
||||
}
|
||||
|
||||
shown = drawStalled(params, 9876543, 1234567, 150*time.Minute)
|
||||
for _, want := range []string{
|
||||
params.label + ": [1234567/" + approx + "9876543] 12%, ",
|
||||
", elapsed 2h30m0s, eta 17h30m0s",
|
||||
} {
|
||||
if !strings.Contains(shown, want) {
|
||||
t.Errorf("terminal shows %q, want %q", shown, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -695,7 +721,7 @@ func TestTerminalLinesFitWidth(t *testing.T) {
|
||||
// times. Nothing written after that, the erase before each redraw
|
||||
// included, may reach the narrower terminal's last column: it would
|
||||
// wrap, and every redraw would leave a line behind. At these widths the
|
||||
// spinner stays a spinner and the bar stays a bar.
|
||||
// spinner stays a spinner and the bar gives way to the spinner.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
|
||||
|
||||
@@ -74,11 +74,11 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
// The read phase writes the report while it reads the records, so a
|
||||
// stall may be in either.
|
||||
// The read phase writes the report to stdout while it reads the
|
||||
// records, so it may wait on either.
|
||||
prog = newProgress(progressParams{
|
||||
label: reportPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, doing: "read/write",
|
||||
waitingFor: countingRecords, stalledOn: "db/out",
|
||||
})
|
||||
report, err := readReport(ctx, db, dbPath, stdout, prog)
|
||||
|
||||
|
||||
@@ -446,7 +446,7 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||
// first gives the display its total.
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("load"), unit: scanUnit, total: -1,
|
||||
waitingFor: countingRecords, doing: reading,
|
||||
waitingFor: countingRecords, stalledOn: "db",
|
||||
})
|
||||
defer prog.finish()
|
||||
|
||||
@@ -497,7 +497,7 @@ func (s *scanState) walkPhase(
|
||||
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("walk"), unit: scanUnit, total: estimate,
|
||||
estimate: true, doing: "listing",
|
||||
estimate: true, stalledOn: "disk",
|
||||
})
|
||||
|
||||
var (
|
||||
@@ -687,9 +687,11 @@ func (s *scanState) readRuns(ctx context.Context, workers int,
|
||||
pool := startHashPool(ctx, runs, workers, hash)
|
||||
defer pool.stop()
|
||||
|
||||
// Between results the phase may wait on a file being read or on a
|
||||
// batch of records being written.
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase(label), unit: scanUnit, total: int64(len(runs)),
|
||||
doing: reading,
|
||||
stalledOn: "disk/db",
|
||||
})
|
||||
defer prog.finish()
|
||||
|
||||
@@ -779,7 +781,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
||||
total := len(s.batch) + len(s.sentinels) + len(deletes)
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("update"), unit: scanUnit, total: int64(total),
|
||||
doing: "writing",
|
||||
stalledOn: "db",
|
||||
})
|
||||
|
||||
defer prog.finish()
|
||||
@@ -842,7 +844,15 @@ func (s *scanState) contentPhase(ctx context.Context, workers int) error {
|
||||
return err
|
||||
}
|
||||
|
||||
return applyChanges(ctx, s.db, s.batch, nil, nil)
|
||||
// The last partial batch is written under a display of its own, as
|
||||
// the update phase writes the hash phase's.
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("content"), unit: scanUnit,
|
||||
total: int64(len(s.batch)), stalledOn: "db",
|
||||
})
|
||||
defer prog.finish()
|
||||
|
||||
return applyChanges(ctx, s.db, s.batch, nil, prog)
|
||||
}
|
||||
|
||||
// contentCandidates returns the files the content phase reads, and
|
||||
@@ -865,10 +875,11 @@ func (s *scanState) contentCandidates(
|
||||
// without a display the scan looks hung before the reads begin.
|
||||
// The display says it is searching until SQLite returns the first
|
||||
// record. The checks start there, against a count of the records
|
||||
// the search returns, and so do their rate and ETA.
|
||||
// the search returns, and so do their rate and ETA. Between records
|
||||
// they may wait on a file's lstat or on the search.
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("content"), unit: scanUnit, total: -1,
|
||||
waitingFor: "searching the database", doing: "searching",
|
||||
waitingFor: "searching the database", stalledOn: "disk/db",
|
||||
})
|
||||
defer prog.finish()
|
||||
|
||||
|
||||
@@ -72,7 +72,7 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
|
||||
prog = newProgress(progressParams{
|
||||
label: treesPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, doing: reading,
|
||||
waitingFor: countingRecords, stalledOn: "db",
|
||||
})
|
||||
records, super, allDirs, err := readTree(ctx, db, prog)
|
||||
|
||||
@@ -82,9 +82,11 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
return fmt.Errorf("database %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
// Grouping works in memory, so it waits on nothing but the
|
||||
// processor.
|
||||
prog = newProgress(progressParams{
|
||||
label: treesPhase("group"), unit: "directories",
|
||||
total: int64(len(allDirs)), doing: "grouping",
|
||||
total: int64(len(allDirs)), stalledOn: "cpu",
|
||||
})
|
||||
dupes := collectTreeGroups(allDirs, super, prog)
|
||||
|
||||
@@ -92,7 +94,7 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
|
||||
prog = newProgress(progressParams{
|
||||
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
|
||||
doing: "writing",
|
||||
stalledOn: "out",
|
||||
})
|
||||
dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user