diff --git a/README.md b/README.md index 53af2c3..ad7ff32 100644 --- a/README.md +++ b/README.md @@ -525,11 +525,11 @@ hash phase: Files are hashed in **inode order**, which minimizes seeks on spinning disks. Paths that are hard links to the same inode are **read once**, and all share the one result. A hard-link backup farm therefore costs one read - per inode, not per path. The phase total counts actual reads, so progress - and ETA are meaningful. Completed records are committed in batched - transactions **while hashing runs**. A scan interrupted after hours - therefore keeps everything hashed so far, and the next scan resumes cheaply, - skipping records already written. + per inode, not per path. The phase total counts actual reads, so its + progress and its time remaining are meaningful. Completed records are + committed in batched transactions **while hashing runs**. A scan interrupted + after hours therefore keeps everything hashed so far, and the next scan + resumes cheaply, skipping records already written. 4. **update** — commit the final partial batch of hashed records. Then delete the records the scan did not verify: those of vanished files, and of paths that failed to stat or hash. @@ -625,14 +625,19 @@ Rules for the walk: as described in "`scan` mode" above: they are not deleted. That does not hold below an operand at or inside it. -- On any per-path error (permission denied, file vanished between passes, - unreadable): print a one-line warning to stderr, skip the path, and continue. - Per-file errors never abort the run; the final summary reports how many were - skipped. As specified above, a skipped path that has a database record from an - earlier scan loses that record. An unreadable directory subtree likewise loses - its records. This is accepted: the database mirrors what the latest scan could - actually verify. The exception is an operand dropped before the database was - read, whose record counts as outside the scanned operands. +- On a per-path error, such as permission denied or an unreadable file: print a + one-line warning to stderr, skip the path, and continue. Per-file errors never + abort the run; the final summary reports how many were skipped. As specified + above, a skipped path that has a database record from an earlier scan loses + that record. An unreadable directory subtree likewise loses its records. This + is accepted: the database mirrors what the latest scan could actually verify. + The exception is an operand dropped before the database was read, whose record + counts as outside the scanned operands. +- A file or directory removed before the scan reaches it is not an error. The + scan prints no line for it and does not count it as skipped. That holds in the + walk, and in the hash phase for a file removed since the walk. Such a path + gets no record, and the records an earlier scan left at or below it are + deleted. Concurrency: the walk phase, which also stats files, and the hash phase each use a worker pool of `--workers` workers (default `runtime.NumCPU()`). The walk @@ -822,9 +827,9 @@ Use the progress-bar library for all progress of `scan`, `report`, `trees` and stderr. `scan`, `report`, `trees` and `info` must always show that they are working and -not hung. Each phase gets its own display from the moment it starts, naming the -phase and its place among the command's phases and showing its progress, its -elapsed time and its estimated time remaining (ETA). +not hung. Each phase gets its own display from the moment it starts. The display +names the phase and its place among the command's phases. It shows the phase's +progress, its elapsed time and its time remaining. A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`, `[4/4] update`): @@ -834,9 +839,9 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`, - `walk` has no exact total. Its estimate is the number of records the last scan left under the operands, known once `load` is done and shown with a `~`. Once the walk passes the estimate, or when there are no such records, as on a first - scan, the ETA is unknown. + scan, the time remaining is unknown. - `hash` and `update` have exact totals. Only files that actually need hashing - appear in the hash total, so its ETA is meaningful. + appear in the hash total, so its time remaining is meaningful. `trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`, `[4/4] write`, see "`trees` mode"): @@ -861,8 +866,8 @@ A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`, - `open` has no total. While it opens the database, waiting on a busy one included, the display says it is opening the database. - `count` has no total either, because one query counts everything: the display - says it is counting the records until the query returns, and its ETA is - unknown. + says it is counting the records until the query returns, and its time + remaining is unknown. Each display is one line holding: @@ -875,16 +880,17 @@ Each display is one line holding: while nothing completes, and a burst of items after a wait shows as a burst rather than as a steady rate. Once no item has completed for 10 seconds, the word `stalled` and where the phase waits take the rate's place (see below). -- elapsed time -- estimated time remaining, or `unknown`. It comes from the phase's average rate - since it started counting, so that it does not swing with each burst. +- elapsed time, labelled `elapsed` +- time remaining, labelled `remain`: a duration such as `12m9s`, or `unknown`. + It comes from the phase's average rate since it started counting, so that it + does not swing with each burst. -Example shape (exact layout is flexible, content is not): +Example lines, as printed when stderr is not a terminal: ``` -[3/4] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54 -[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown -[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s +[3/4] hash: [12345/98765] 12%, 92 files/s, elapsed 2m32s, remain 17m44s +[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, remain unknown +[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, remain 12m9s ``` When no item has completed for 10 seconds, as while one large file is read, a @@ -900,27 +906,44 @@ for its total already says what it waits for, and says nothing more. Additional requirements: -- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`, - the `open` phase of `report` and `trees`, and both phases of `info`, once it - has counted what it will do) is drawn as its line followed by the progress-bar - library's bar, 10 columns wide, which shows the percent in place of the line's - own. Any other phase, and a phase whose bar has no room, is drawn as the - library's spinner followed by its line. Each display is drawn again several - times a second, so its elapsed time keeps moving while a phase waits for its - next item. +- On a terminal, every display starts with the progress-bar library's spinner in + its first column, followed by its line. Each display is drawn again several + times a second, so its spinner turns and its elapsed time keeps moving while a + phase waits for its next item. +- On a terminal, a phase with an exact total ends its line with its percent and + a bar 10 columns wide. This percent takes the place of the line's own. The bar + stays at the terminal's right edge, so it does not move as the line grows or + shrinks. Every phase has an exact total once it has counted what it will do, + except these: + - `scan`'s `walk`; + - the `open` phase of `report` and `trees`; + - both phases of `info`. + + On a terminal 100 columns wide: + + ``` + ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ | + ``` + - On a terminal no line reaches the terminal's last column, however narrow the - terminal is. A line too wide for the terminal first loses its bar, which gives - way to the spinner, then its rate, the words of a stalled phase, or what the - phase is waiting for. The phase and its place, the progress, the elapsed time - and the ETA are kept whole. On an 80-column terminal, at counts like those in - the example above, a stalled line keeps its words whenever the elapsed time - and the ETA are both in minutes. Where the words and the ETA do not both fit, - as at millions of files and times in hours, the words give way. Only when - those four alone are too wide, as on a terminal narrower than about 70 columns - hours into a scan of millions of files, is the line cut short at the - terminal's edge, so the end of the ETA is lost first. A terminal made narrower - while a phase runs gets the same: the next draw erases the line within the new - width and fits the line to it, and the display goes on redrawing in place. + terminal is. A line too wide for the terminal first loses its bar. Next it + loses whichever of these it shows: its rate, the words of a stalled phase, or + what the phase is waiting for. The other four parts are kept whole: + - the phase and its place; + - the progress; + - the elapsed time; + - the time remaining. + + The line is cut short at the terminal's edge only when those four alone are + too wide, and then the end of the time remaining is lost first. That happens + hours into a scan of millions of files, on a terminal narrower than about 75 + columns. On an 80-column terminal, the stalled line of the example above + keeps its words. Where the words and the time remaining do not both fit, as + at millions of files and times in hours, the words give way. A terminal made + narrower while a phase runs gets the same: the next draw erases the line + within the new width and fits the line to it, and the display goes on + redrawing in place. + - When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI redraws: print the line the moment each phase starts, every 5 seconds while it runs (whether or not an item has completed, and never more often), and once diff --git a/TODO.md b/TODO.md index 9b2ead0..dc5990f 100644 --- a/TODO.md +++ b/TODO.md @@ -29,6 +29,16 @@ # Completed Steps +- a file or directory removed before the scan reaches it is not an error, in the + walk or in the hash phase. The scan prints no line for it, does not count it + as skipped, and gives it no record. When the first of several hard links to a + file is gone, the hash phase reads the next one instead. On a terminal every + progress display starts with the spinner, the hash phase's included. A phase + with an exact total ends its line with its percent and a 10-column bar at the + terminal's right edge, which no longer moves as the line's length changes. The + time remaining is labelled `remain`, not `eta` (2026-10-26, + https://git.eeqj.de/sneak/sfdupes/issues/133) + - the `head`, `tail` and `content` columns are BLOBs, each holding the raw 32 bytes of a SHA-256, half the size of the hex text they held. The hashes stay raw from the hash phase through the database to `report` and `trees`. Reading diff --git a/info_test.go b/info_test.go index c927f41..5d6a58e 100644 --- a/info_test.go +++ b/info_test.go @@ -12,13 +12,13 @@ import ( // infoOpenLine is the plain line of info's open phase when it opens the // database at once; it prints it as the phase starts and as it ends. const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " + - "eta unknown" + "remain unknown" // infoCountLine is the plain line of info's count phase when the count // takes under a second; it prints it as the phase starts and as it // ends. const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " + - "eta unknown" + "remain unknown" // infoRecs are four records: a duplicate pair, a file of 20 MiB and // another small file. diff --git a/progress.go b/progress.go index 0c61bea..3eb9f4f 100644 --- a/progress.go +++ b/progress.go @@ -9,6 +9,7 @@ import ( "strings" "sync" "time" + "unicode/utf8" "github.com/schollz/progressbar/v3" "golang.org/x/term" @@ -34,10 +35,17 @@ const stallTime = 10 * time.Second // spinnerType selects the progressbar library's spinner style. const spinnerType = 14 -// barWidth is the width of the progressbar library's bar between its -// two ends, as in the example line in README "Progress". +// barWidth is the width of a phase's bar between its two ends, as in +// the example line in README "Progress". const barWidth = 10 +// barCell is one column of the filled part of a bar. +const barCell = "█" + +// barColumns is the width of what percentBar returns: the percent in +// four columns and its sign, a space, and the bar with its two ends. +const barColumns = barWidth + 8 + // percentScale converts a fraction to a percentage. const percentScale = 100 @@ -45,17 +53,12 @@ const percentScale = 100 // read, as the progressbar library takes it. const fallbackWidth = 80 -// spinnerMargin is the columns of a spinner's line besides its text: -// the spinner and the three spaces the library draws around the text, -// and the terminal's last column, which every line stays clear of so -// that no terminal wraps it. +// spinnerMargin is the columns of a line besides its text: the spinner +// and the three spaces the library draws around the text, and the +// terminal's last column, which every line stays clear of so that no +// terminal wraps it. const spinnerMargin = 5 -// barMargin is the columns of a bar's line besides its text: the -// percent and the bar the library draws after the text, the spaces -// around them, and the terminal's last column. -const barMargin = barWidth + 11 - // countingRecords is what the display of a phase that reads the records // says while it counts them, before it has its total. const countingRecords = "counting the records" @@ -109,13 +112,13 @@ type drawnCount struct { } // progress renders one phase's progress on stderr. On a TTY the -// progressbar library draws line, fitted to the terminal's width: after -// the library's spinner, or, for a phase with an exact total, before the -// library's bar when the terminal has room for the bar beside it. It is -// drawn again every redrawInterval, so its elapsed time keeps moving -// while the phase waits for its next item. When stderr is not a TTY it -// emits no ANSI redraws: it prints line as the phase starts, every -// plainInterval while it runs, and as it ends. +// progressbar library draws its spinner and then line, fitted to the +// terminal's width. A phase with an exact total ends the line with its +// percent and bar, at the terminal's right edge, when the terminal has +// room for them. It is drawn again every redrawInterval, so its elapsed +// time keeps moving while the phase waits for its next item. When +// stderr is not a TTY it emits no ANSI redraws: it prints line as the +// phase starts, every plainInterval while it runs, and as it ends. // // The draws come from a goroutine of their own, so mu guards the fields // below it. Every method must be called from the main goroutine. A nil @@ -131,7 +134,6 @@ type progress struct { mu sync.Mutex bar *progressbar.ProgressBar // what the library draws; nil until then - withBar bool // bar is the library's bar, not its spinner widest int // the widest line bar has drawn, in columns label string unit string @@ -141,7 +143,7 @@ type progress struct { stalledOn string count int64 start time.Time // when the phase started, for the elapsed time - rateStart time.Time // when the count started, for the ETA + rateStart time.Time // when the count started, for the time remaining lastItem time.Time // when the last item was completed, or the count started // drawn holds the count at each draw over the last rateWindow, oldest // first; the rate counts from the first. It keeps the draw before the @@ -221,49 +223,42 @@ func (p *progress) draw() { } width := p.width() + room := width - spinnerMargin + text := p.line(now, room, true) - // A phase with an exact total gets the library's bar when its whole - // line, without the percent the bar shows, fits beside the bar: the - // bar gives way before any part of the line. A total of 0 leaves the - // bar nothing to fill. - text, withBar := "", false + // A phase with an exact total ends its text with its percent and bar + // when its whole line, without the percent the bar shows, fits beside + // them: the bar gives way before any part of the line. The line is + // padded to the room the bar leaves it, so the bar stays at the + // terminal's right edge however long the line is. A total of 0 leaves + // the bar nothing to fill. if p.total > 0 && !p.estimate { - text = p.line(now, math.MaxInt, false) - withBar = len(text) <= width-barMargin + line := p.line(now, math.MaxInt, false) + if len(line) <= room-barColumns { + text = fmt.Sprintf("%-*s%s", room-barColumns, line, p.percentBar()) + } } - margin := barMargin - if !withBar { - margin = spinnerMargin - text = p.line(now, width-margin, true) - } - - // The library makes a bar or a spinner, and before each draw erases - // its line with as many spaces as the widest line it has drawn. A - // display gets a new one when it turns from one to the other, as its - // line grows or the terminal is resized, and when the terminal has - // been made too narrow for that erase, which would wrap and leave a - // line behind at every draw. - if p.bar == nil || p.withBar != withBar || p.widest >= width { - p.newBar(withBar, width) - } - - if withBar { - _ = p.bar.Set64(p.count) + // Before each draw the library erases its line with as many spaces as + // the widest line it has drawn. A display gets a new one when the + // terminal has been made too narrow for that erase, which would wrap + // and leave a line behind at every draw. + if p.bar == nil || p.widest >= width { + p.newBar(width) } p.bar.Describe(text) - // The library drew the text and its margin, less the terminal's last - // column, which the margin keeps clear. - p.widest = max(p.widest, len(text)+margin-1) + // The library drew the spinner, the text and the spaces around it, + // less the terminal's last column, which spinnerMargin keeps clear. A + // bar's cells take one column each but more than one byte. + p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1) } -// newBar has the library make what draw draws: a bar for the phase's -// total when withBar is set, and a spinner when not. Either shows the -// text draw gives it, and nothing of its own besides the spinner, or -// the bar and its percent. p.mu must be held. -func (p *progress) newBar(withBar bool, width int) { +// newBar has the library make what draw draws: its spinner, followed by +// the text draw gives it and nothing of the library's own. p.mu must be +// held. +func (p *progress) newBar(width int) { if p.bar != nil { // Erase what was drawn before, short of the last column of a // terminal that may have been made narrower since: the library's @@ -272,29 +267,36 @@ func (p *progress) newBar(withBar bool, width int) { fmt.Fprint(os.Stderr, "\r"+blank+"\r") } - total := int64(-1) // the library's spinner - if withBar { - total = p.total - } - - p.bar = progressbar.NewOptions64(total, + // A total of -1 gives the library's spinner. + p.bar = progressbar.NewOptions64(-1, progressbar.OptionSetWriter(os.Stderr), - progressbar.OptionSetWidth(barWidth), progressbar.OptionSpinnerType(spinnerType), // The spinner turns at each draw, not on a timer of the // library's own. progressbar.OptionSetSpinnerChangeInterval(0), - // The text carries the elapsed time and the ETA. + // The text carries the elapsed time and the time remaining. progressbar.OptionSetElapsedTime(false), progressbar.OptionSetPredictTime(false), ) - p.withBar, p.widest = withBar, 0 + p.widest = 0 +} + +// percentBar returns the percent and the bar that end the text of a +// phase with an exact total, as " 25% |██ |": the bar's +// barWidth columns fill in proportion to the items done. A count past +// the total, as when a scan adds records while trees reads them, shows +// as 100%. p.mu must be held. +func (p *progress) percentBar() string { + done := min(float64(p.count)/float64(p.total), 1) + filled := strings.Repeat(barCell, int(done*barWidth)) + + return fmt.Sprintf("%4.0f%% |%-*s|", done*percentScale, barWidth, filled) } // setTotal gives a phase that started without a total its total, once -// it is known. The rate, the ETA and the time with no item completed -// count from then on: the time spent waiting for the total says nothing -// about how fast items go. +// it is known. The rate, the time remaining and the time with no item +// completed count from then on: the time spent waiting for the total +// says nothing about how fast items go. func (p *progress) setTotal(total int64) { if p == nil { return @@ -438,21 +440,21 @@ func (p *progress) finish() { // line formats the phase's progress as of now in at most room columns, // as // -// [3/4] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s +// [3/4] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, remain 13s // // where "files" is the phase's unit, without the percent when percent -// is false, as before the library's bar, which shows its own. A phase -// with no total shows the items done instead, or what it waits for. The -// rate is that since the first draw in drawn, and the ETA comes from +// is false, as before the bar, which shows its own. A phase with no +// total shows the items done instead, or what it waits for. The rate is +// that since the first draw in drawn, and the time remaining comes from // the rate since the count started. Once no item has been completed for // stallTime, the rate, which would read 0, gives way to the word // "stalled" and where the phase waits, as "stalled disk/db". // -// The phase and its place, the progress, the elapsed time and the ETA -// are kept whole. A line wider than room leaves out the rate, the words -// of a stalled phase, or what the phase waits for; only if it is still -// wider is it cut at room. Every part is ASCII, so its length is its -// width in columns. p.mu must be held. +// The phase and its place, the progress, the elapsed time and the time +// remaining are kept whole. A line wider than room leaves out the rate, +// the words of a stalled phase, or what the phase waits for; only if it +// is still wider is it cut at room. Every part is ASCII, so its length +// is its width in columns. p.mu must be held. func (p *progress) line(now time.Time, room int, percent bool) string { elapsed := now.Sub(p.start).Round(time.Second) @@ -462,7 +464,7 @@ func (p *progress) line(now time.Time, room int, percent bool) string { if p.total < 0 && p.waitingFor != "" { head, extra = p.label+": ", p.waitingFor+", " - tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed) + tail = fmt.Sprintf("elapsed %s, remain unknown", elapsed) } else { since := p.drawn[0] rate := perSecond(p.count-since.count, now.Sub(since.at)) @@ -470,7 +472,7 @@ func (p *progress) line(now time.Time, room int, percent bool) string { head = p.label + ": " + p.itemsDone(percent) + ", " extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit) - tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average)) + tail = fmt.Sprintf("elapsed %s, remain %s", elapsed, p.remain(average)) if now.Sub(p.lastItem) >= stallTime { extra = "stalled " + p.stalledOn + ", " @@ -526,10 +528,10 @@ func (p *progress) itemsDone(percent bool) string { return fmt.Sprintf("%s %.0f%%", done, pct) } -// eta returns the time the phase has left at rate items a second, or +// remain returns the time the phase has left at rate items a second, or // "unknown": with no total, before the first item, and once the count // passes an estimate. p.mu must be held. -func (p *progress) eta(rate float64) string { +func (p *progress) remain(rate float64) string { switch { case p.total < 0, p.estimate && p.count > p.total: return "unknown" @@ -539,8 +541,8 @@ func (p *progress) eta(rate float64) string { return "unknown" } - remaining := time.Duration( + left := time.Duration( float64(p.total-p.count) / rate * float64(time.Second)) - return remaining.Round(time.Second).String() + return left.Round(time.Second).String() } diff --git a/progress_test.go b/progress_test.go index b5a5132..222d1fd 100644 --- a/progress_test.go +++ b/progress_test.go @@ -117,24 +117,24 @@ func TestProgressLine(t *testing.T) { label: scanPhase("load"), total: -1, waitingFor: countingRecords, }, - want: "[1/4] load: counting the records, elapsed 10s, eta unknown", + want: "[1/4] load: counting the records, elapsed 10s, remain unknown", }, { p: &progress{ label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20, }, - want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s", + want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s", }, { p: &progress{label: scanPhase("hash"), unit: scanUnit, total: 100}, - want: "[3/4] hash: [0/100] 0%, 0 files/s, elapsed 10s, eta unknown", + want: "[3/4] hash: [0/100] 0%, 0 files/s, elapsed 10s, remain unknown", }, { p: &progress{label: scanPhase("update"), unit: scanUnit, total: 0}, - want: "[4/4] update: [0/0] 100%, 0 files/s, elapsed 10s, eta 0s", + want: "[4/4] update: [0/0] 100%, 0 files/s, elapsed 10s, remain 0s", }, { p: &progress{ label: scanPhase("walk"), unit: scanUnit, total: 100, estimate: true, count: 20, }, - want: "[2/4] walk: [20/~100] 20%, 2 files/s, elapsed 10s, eta 40s", + want: "[2/4] walk: [20/~100] 20%, 2 files/s, elapsed 10s, remain 40s", }, { // Past the estimate the walk cannot tell how long it has left. p: &progress{ @@ -142,13 +142,13 @@ func TestProgressLine(t *testing.T) { estimate: true, count: 120, }, want: "[2/4] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " + - "eta unknown", + "remain unknown", }, { // A first scan's walk has no estimate. p: &progress{ label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20, }, - want: "[2/4] walk: 20 files, 2 files/s, elapsed 10s, eta unknown", + want: "[2/4] walk: 20 files, 2 files/s, elapsed 10s, remain unknown", }, { // The rate is in what the phase counts. p: &progress{ @@ -156,7 +156,7 @@ func TestProgressLine(t *testing.T) { count: 20, }, want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " + - "eta 40s", + "remain 40s", }} for _, tc := range cases { @@ -203,28 +203,28 @@ func TestLineFitsRoom(t *testing.T) { percent bool want string }{{ - p: hash, room: 57, percent: true, - want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s", + p: hash, room: 60, percent: true, + want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s", }, { - p: hash, room: 56, percent: true, - want: "[3/4] hash: [20/100] 20%, elapsed 10s, eta 40s", + p: hash, room: 59, percent: true, + want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40s", }, { - p: hash, room: 45, percent: true, - want: "[3/4] hash: [20/100] 20%, elapsed 10s, eta 40", + p: hash, room: 48, percent: true, + want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40", }, { - // Before the library's bar, which shows the percent. - p: hash, room: 57, percent: false, - want: "[3/4] hash: [20/100], 2 files/s, elapsed 10s, eta 40s", + // Before the bar, which shows the percent. + p: hash, room: 60, percent: false, + want: "[3/4] hash: [20/100], 2 files/s, elapsed 10s, remain 40s", }, { - p: load, room: 40, percent: true, - want: "[1/4] load: elapsed 10s, eta unknown", + p: load, room: 43, percent: true, + want: "[1/4] load: elapsed 10s, remain unknown", }, { - p: stalled, room: 65, percent: true, + p: stalled, room: 68, percent: true, want: "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " + - "eta 3m20s", + "remain 3m20s", }, { - p: stalled, room: 64, percent: true, - want: "[3/4] hash: [20/100] 20%, elapsed 50s, eta 3m20s", + p: stalled, room: 67, percent: true, + want: "[3/4] hash: [20/100] 20%, elapsed 50s, remain 3m20s", }} for _, tc := range cases { @@ -236,10 +236,10 @@ func TestLineFitsRoom(t *testing.T) { } // TestSetTotalStartsRate checks that setTotal starts a phase's rate, -// its ETA and its time with no record read from when the total becomes -// known, not from when the phase started waiting for it: after 20 -// seconds of waiting for the total, the phase has not stalled 5 seconds -// after it, and has 10 seconds after it with no record read. +// its time remaining and its time with no record read from when the +// total becomes known, not from when the phase started waiting for it: +// after 20 seconds of waiting for the total, the phase has not stalled 5 +// seconds after it, and has 10 seconds after it with no record read. func TestSetTotalStartsRate(t *testing.T) { t.Parallel() @@ -255,14 +255,14 @@ func TestSetTotalStartsRate(t *testing.T) { clock.Add(5 * time.Second) - want := "[1/4] load: [0/100] 0%, 0 files/s, elapsed 25s, eta unknown" + want := "[1/4] load: [0/100] 0%, 0 files/s, elapsed 25s, remain unknown" if got := p.line(clock.Now(), math.MaxInt, true); got != want { t.Errorf("line = %q, want %q", got, want) } clock.Add(5 * time.Second) - want = "[1/4] load: [0/100] 0%, stalled db, elapsed 30s, eta unknown" + want = "[1/4] load: [0/100] 0%, stalled db, elapsed 30s, remain unknown" if got := p.line(clock.Now(), math.MaxInt, true); got != want { t.Errorf("line = %q, want %q", got, want) } @@ -272,7 +272,7 @@ func TestSetTotalStartsRate(t *testing.T) { p.increment() } - want = "[1/4] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s" + want = "[1/4] load: [20/100] 20%, 2 files/s, elapsed 30s, remain 40s" if got := p.line(clock.Now(), math.MaxInt, true); got != want { t.Errorf("line = %q, want %q", got, want) } @@ -316,7 +316,7 @@ func TestPlainProgressPrintsAtStartAndEnd(t *testing.T) { lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n") - want := "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown" + want := "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown" if len(lines) != 2 || lines[0] != want || !strings.HasPrefix(lines[1], "[3/4] hash: [1000/1000] 100%, ") { t.Errorf("stderr lines = %q, want %q, then the line at 1000", @@ -336,7 +336,7 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) { }, nil, testRedraw, time.Now) awaitOutput(t, stderr, - "[1/4] load: counting the records, elapsed 0s, eta unknown\n", 3) + "[1/4] load: counting the records, elapsed 0s, remain unknown\n", 3) p.finish() } @@ -366,23 +366,23 @@ func TestPlainProgressStalledProducer(t *testing.T) { clock.Add(5 * time.Second) awaitOutput(t, stderr, - "[3/4] hash: [20/100] 20%, 4 files/s, elapsed 5s, eta 20s\n", 3) + "[3/4] hash: [20/100] 20%, 4 files/s, elapsed 5s, remain 20s\n", 3) clock.Add(time.Second) awaitOutput(t, stderr, - "[3/4] hash: [20/100] 20%, 0 files/s, elapsed 6s, eta 24s\n", 3) + "[3/4] hash: [20/100] 20%, 0 files/s, elapsed 6s, remain 24s\n", 3) clock.Add(4 * time.Second) awaitOutput(t, stderr, - "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 10s, eta 40s\n", + "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 10s, remain 40s\n", 3) } // TestPlainProgressRateAfterStall draws a phase's plain line by hand, a // little over plainInterval after the line before, as a late tick // does, through a stall and a burst of files after it: the rate covers -// the files read since the line before, while the ETA comes from the -// phase's average. +// the files read since the line before, while the time remaining comes +// from the phase's average. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestPlainProgressRateAfterStall(t *testing.T) { @@ -410,10 +410,10 @@ func TestPlainProgressRateAfterStall(t *testing.T) { p.finish() want := []string{ - "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown", - "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 5s, eta unknown", - "[3/4] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, eta unknown", - "[3/4] hash: [100/1000] 10%, 20 files/s, elapsed 15s, eta 2m15s", + "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown", + "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 5s, remain unknown", + "[3/4] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, remain unknown", + "[3/4] hash: [100/1000] 10%, 20 files/s, elapsed 15s, remain 2m15s", } lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n") @@ -426,8 +426,8 @@ func TestPlainProgressRateAfterStall(t *testing.T) { // redrawInterval, as a terminal does, through a stall of stallTime and // a burst of files after it: the rate covers the files read over the // last rateWindow, and falls to 0 once the burst is older than that, -// while the ETA comes from the phase's average. Plain lines show each -// draw. +// while the time remaining comes from the phase's average. Plain lines +// show each draw. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestProgressRateAfterStallRedrawn(t *testing.T) { @@ -457,8 +457,8 @@ func TestProgressRateAfterStallRedrawn(t *testing.T) { redraw(rateWindow + redrawInterval) p.finish() - want := "[3/4] hash: [50/1000] 5%, 10 files/s, elapsed 15s, eta 4m45s\n" + - "[3/4] hash: [50/1000] 5%, 0 files/s, elapsed 15s, eta 4m47s\n" + want := "[3/4] hash: [50/1000] 5%, 10 files/s, elapsed 15s, remain 4m45s\n" + + "[3/4] hash: [50/1000] 5%, 0 files/s, elapsed 15s, remain 4m47s\n" if !strings.Contains(stderr(), want) { t.Errorf("stderr = %q, want the lines %q", stderr(), want) } @@ -480,10 +480,10 @@ func lastFrame(out string) string { } // TestTerminalRedrawsWhileWaiting checks that the terminal display of a -// phase, spinner or bar, keeps being drawn while the phase waits for its -// next item, with the elapsed time as of now, and shows every item -// completed before the wait. stderr is not a terminal here, so the -// displays take it to be fallbackWidth wide. +// phase, with a bar or without, keeps being drawn while the phase waits +// for its next item, with the elapsed time as of now, and shows every +// item completed before the wait. The terminal is 120 columns wide, so +// the hash phase's display has room for its bar. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestTerminalRedrawsWhileWaiting(t *testing.T) { @@ -506,7 +506,8 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) { for _, tc := range cases { before := len(stderr()) since := func() string { return stderr()[before:] } - p := startProgress(tc.params, terminalWidth, testRedraw, time.Now) + p := startProgress(tc.params, func() int { return 120 }, testRedraw, + time.Now) for range 50 { p.increment() @@ -552,13 +553,13 @@ func drawLate(stderr func() string, params progressParams, width int, return stderr()[before:] } -// TestTerminalLinesKeepETA checks the line each display shows on an -// 80-column terminal at a large scan's counts and times, and the line -// of trees' read phase at the same counts: the rate is left out, so -// that the elapsed time and the ETA stay whole. +// TestTerminalLinesKeepTimeRemaining checks the line each display shows +// on an 80-column terminal at a large scan's counts and times, and the +// line of trees' read phase at the same counts: the rate is left out, +// so that the elapsed time and the time remaining stay whole. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr -func TestTerminalLinesKeepETA(t *testing.T) { +func TestTerminalLinesKeepTimeRemaining(t *testing.T) { stderr := captureStderr(t) cases := []struct { @@ -571,19 +572,19 @@ func TestTerminalLinesKeepETA(t *testing.T) { estimate: true, }, want: "[2/4] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " + - "eta unknown", + "remain unknown", }, { params: progressParams{ label: scanPhase("hash"), unit: scanUnit, total: 9876543, }, want: "[3/4] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " + - "eta 17h30m0s", + "remain 17h30m0s", }, { params: progressParams{ label: treesPhase("read"), unit: "records", total: 9876543, }, want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " + - "eta 17h30m0s", + "remain 17h30m0s", }} for _, tc := range cases { @@ -598,9 +599,10 @@ func TestTerminalLinesKeepETA(t *testing.T) { // terminal once the phase has stalled, its last item done 42 seconds // before. At an ordinary scan's counts and times, 12345 of 45678 items // done four and a half minutes in, the line shows where the phase waits -// whole, beside the phase, progress, elapsed time and ETA. At a large -// scan's, 1234567 of 9876543 two and a half hours in, those four still -// show whole, as the words give way first where they do not fit. +// whole, beside the phase, progress, elapsed time and time remaining. +// At a large scan's, 1234567 of 9876543 two and a half hours in, those +// four still show whole, as the words give way first where they do not +// fit. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestTerminalStalledLines(t *testing.T) { @@ -652,7 +654,7 @@ func TestTerminalStalledLines(t *testing.T) { shown := drawStalled(params, 45678, 12345, 4*time.Minute+30*time.Second) want := params.label + ": [12345/" + approx + "45678] 27%, stalled " + - params.stalledOn + ", elapsed 4m30s, eta 12m9s" + params.stalledOn + ", elapsed 4m30s, remain 12m9s" if !strings.Contains(shown, want) { t.Errorf("terminal shows %q, want %q", shown, want) @@ -661,7 +663,7 @@ func TestTerminalStalledLines(t *testing.T) { shown = drawStalled(params, 9876543, 1234567, 150*time.Minute) for _, want := range []string{ params.label + ": [1234567/" + approx + "9876543] 12%, ", - ", elapsed 2h30m0s, eta 17h30m0s", + ", elapsed 2h30m0s, remain 17h30m0s", } { if !strings.Contains(shown, want) { t.Errorf("terminal shows %q, want %q", shown, want) @@ -706,13 +708,78 @@ func TestTerminalLinesFitWidth(t *testing.T) { } } +// TestTerminalLinesStartWithSpinner draws a display with no total and +// one with a bar, two and a half hours into the phase, on a terminal 120 +// columns wide. Each line must start with the library's spinner in the +// first column, then a space and the phase's label. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalLinesStartWithSpinner(t *testing.T) { + stderr := captureStderr(t) + + for _, params := range []progressParams{ + {label: scanPhase("walk"), unit: scanUnit, total: -1}, + {label: scanPhase("hash"), unit: scanUnit, total: 9876543}, + } { + shown := lastFrame(drawLate(stderr, params, 120, 1234567)) + spinner, rest, _ := strings.Cut(shown, " ") + + if utf8.RuneCountInString(spinner) != 1 || + !strings.HasPrefix(rest, params.label+": ") { + t.Errorf("terminal shows %q, want the spinner, a space and %q", + shown, params.label) + } + + if params.total > 0 && !strings.Contains(shown, "% |") { + t.Errorf("terminal shows %q, want a bar", shown) + } + } +} + +// TestTerminalBarStaysAtRightEdge draws the hash phase's display two +// and a half hours in, at two counts whose lines differ in length, on +// terminals 110 and 120 columns wide. Its bar must start the same number +// of columns from the terminal's right edge each time: a line that grows +// or shrinks, as its time remaining does, never moves the bar. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestTerminalBarStaysAtRightEdge(t *testing.T) { + stderr := captureStderr(t) + params := progressParams{ + label: scanPhase("hash"), unit: scanUnit, total: 9876543, + } + + widths := []int{110, 120} + counts := []int64{1234567, 9876543} + fromEdge := make([]int, 0, len(widths)*len(counts)) + + for _, width := range widths { + for _, count := range counts { + shown := lastFrame(drawLate(stderr, params, width, count)) + + beforeBar, _, found := strings.Cut(shown, "% |") + if !found { + t.Fatalf("terminal %d wide shows %q, want a bar", width, shown) + } + + fromEdge = append(fromEdge, + width-utf8.RuneCountInString(beforeBar)) + } + } + + if slices.Min(fromEdge) != slices.Max(fromEdge) { + t.Errorf("bar starts %v columns from the right edge, want the same "+ + "every time", fromEdge) + } +} + // TestTerminalNarrowedRedrawsInPlace draws a large scan's display, two // and a half hours into the phase, on a terminal 120 columns wide, then // makes the terminal narrower than that line and draws it again a few // times. Nothing written after that, the erase before each redraw // included, may reach the narrower terminal's last column: it would -// wrap, and every redraw would leave a line behind. At these widths the -// spinner stays a spinner and the bar gives way to the spinner. +// wrap, and every redraw would leave a line behind. At the narrower +// width the hash phase's bar gives way. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestTerminalNarrowedRedrawsInPlace(t *testing.T) { @@ -770,10 +837,10 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) { } } -// TestProgressWarningsOnOwnLines drives the terminal displays, spinner -// and bar, through a run of warnings with no items between them, as when -// the walk meets many unreadable paths, for many redraws: every warning -// must land on a line of its own, never inside a redraw. +// TestProgressWarningsOnOwnLines drives the terminal displays, with a +// bar and without, through a run of warnings with no items between +// them, as when the walk meets many unreadable paths, for many redraws: +// every warning must land on a line of its own, never inside a redraw. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestProgressWarningsOnOwnLines(t *testing.T) { @@ -819,11 +886,12 @@ func TestProgressWarningsOnOwnLines(t *testing.T) { } } -// TestProgressWriterRowsOnOwnLines drives the terminal displays, -// spinner and bar, through a run of rows written through the display's +// TestProgressWriterRowsOnOwnLines drives the terminal displays, with a +// bar and without, through a run of rows written through the display's // writer to the same terminal, as stdout is in a plain "sfdupes trees", // each written out at once, for many redraws: the terminal must show -// every row whole, in order, on a line of its own. +// every row whole, in order, on a line of its own. The terminal is 120 +// columns wide, so the write phase's display has room for its bar. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestProgressWriterRowsOnOwnLines(t *testing.T) { @@ -840,7 +908,8 @@ func TestProgressWriterRowsOnOwnLines(t *testing.T) { var want []string for _, params := range cases { - p := startProgress(params, terminalWidth, testRedraw, time.Now) + p := startProgress(params, func() int { return 120 }, testRedraw, + time.Now) out := p.writer(os.Stderr) // No pause between rows: one is garbled only if a redraw lands diff --git a/report_test.go b/report_test.go index df6d50f..9b47034 100644 --- a/report_test.go +++ b/report_test.go @@ -187,7 +187,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) { // reportOpenLine is the plain line of report's open phase when it opens // the database at once; it prints it as the phase starts and as it ends. const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " + - "eta unknown" + "remain unknown" // smokeReport is report's stdout for the smoke-test records. const smokeReport = "first\tdupe\tsize\n" + @@ -225,7 +225,7 @@ func TestRunReportProgressNamesEachPhase(t *testing.T) { "report: database " + path + " (from " + databaseEnv + ")", reportOpenLine, reportOpenLine, - "[2/2] read: counting the records, elapsed 0s, eta unknown", + "[2/2] read: counting the records, elapsed 0s, remain unknown", "[2/2] read: [6/6] 100%, ", "report: 6 records read, 2 duplicate groups, 4 dupe files, " + "6.1 KiB reclaimable", diff --git a/scan.go b/scan.go index 1162e6e..2aed86b 100644 --- a/scan.go +++ b/scan.go @@ -418,7 +418,8 @@ func (s *scanState) runPhases(ctx context.Context, roots []string, // roots and are not deleted as unverified. An operand on a filesystem // in rules that is not persistent is dropped the same way, with a line // saying so that is not a warning, and is not counted as skipped. An -// operand that fails lstat here is kept, and the walk warns about it. +// operand that fails lstat here is kept, and the walk warns about it +// unless it no longer exists. func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { kept := make([]string, 0, len(roots)) @@ -629,8 +630,9 @@ func isDatabaseFile(f fileRec, rules walkRules) bool { // (an interrupted scan resumes cheaply: the next run skips everything // already recorded). A run that fails to hash is warned about and // counted as skipped; stale records for its paths, if any, are deleted -// by the update phase. The total counts actual reads, so the display -// shows a real ETA. +// by the update phase. A run whose file no longer exists gets no record +// and no warning, and is not counted as skipped. The total counts actual +// reads, so the display's time remaining is meaningful. // // Returning early — a failed database write, or a cancelled scan — must // not strand the pool: the feeder would park forever on a full jobs @@ -662,6 +664,11 @@ func (s *scanState) hashPhase(ctx context.Context, workers int) error { prog.increment() + // A file removed since the walk is no error: it gets no record. + if errors.Is(r.err, fs.ErrNotExist) { + continue + } + if r.err != nil { s.st.skipped += len(r.run) @@ -829,10 +836,12 @@ type walkEvent struct { // startWalk seeds every root into the shared walk worker pool and // returns the event stream: one record per regular file, one warning // event per per-path error, and one event per path -x or rules kept the -// walk from entering or recording. The channel is closed when the walk -// completes, and also when ctx is cancelled — every goroutine in the -// pool abandons its blocking send in that case, so the consumer sees a -// truncated but properly terminated stream instead of a stalled one. +// walk from entering or recording. A path that no longer exists when +// the walk reaches it is no error, and gets no event. The channel is +// closed when the walk completes, and also when ctx is cancelled — +// every goroutine in the pool abandons its blocking send in that case, +// so the consumer sees a truncated but properly terminated stream +// instead of a stalled one. func startWalk(ctx context.Context, roots []string, oneFS bool, rules walkRules, workers int, ) <-chan walkEvent { @@ -899,13 +908,19 @@ func operandWarning(root string, fi fs.FileInfo) string { // one of the scan's own database files in rules, and a directory // operand becomes an initial job. walkableRoots has already dropped // every other operand. One that has changed into something else since -// is warned about and skipped here; it is still a root, so the records -// stored beneath it, other than those under another root, are deleted -// as unverified. +// is warned about and skipped here, and one removed since is skipped +// without a warning. Either is still a root, so the records stored +// beneath it, other than those under another root, are deleted as +// unverified. func seedRoot(ctx context.Context, root string, rules walkRules, events chan<- walkEvent, ) []dirJob { fi, err := os.Lstat(root) + if errors.Is(err, fs.ErrNotExist) { + // Removed since the scan started: no error, and no files. + return nil + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("walk %s: %v", root, err), @@ -1023,11 +1038,17 @@ func dispatchDirs(ctx context.Context, initial []dirJob, // walkOneDir reads one directory, emitting an event per regular-file // entry and a warning event per unreadable one, and returns the -// subdirectories to descend into. +// subdirectories to descend into. A directory or entry removed since +// the walk found it gets no event. func walkOneDir(ctx context.Context, job dirJob, oneFS bool, rules walkRules, events chan<- walkEvent, ) []dirJob { entries, err := os.ReadDir(job.path) + if errors.Is(err, fs.ErrNotExist) { + // Removed since its parent was read: no error, and no files. + return nil + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("walk %s: %v", job.path, err), @@ -1070,16 +1091,22 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool, // emitFile stats one regular directory entry and emits its record. // The lstat happens here in the walk worker, while the directory's -// metadata is still hot; a path that fails to stat (or stops being a -// regular file) between the directory read and the lstat is warned -// about and skipped. A file on a filesystem in rules that is not -// persistent, as through a bind mount of one file, is not recorded: it -// gets an event naming it instead. One of the scan's own database files -// in rules gets no event at all. +// metadata is still hot; a path that fails to stat between the +// directory read and the lstat is warned about and skipped, unless it +// no longer exists, and one that has stopped being a regular file is +// skipped. A file on a filesystem in rules that is not persistent, as +// through a bind mount of one file, is not recorded: it gets an event +// naming it instead. One of the scan's own database files in rules gets +// no event at all. func emitFile(ctx context.Context, p string, e fs.DirEntry, rules walkRules, events chan<- walkEvent, ) { info, err := e.Info() + if errors.Is(err, fs.ErrNotExist) { + // Removed since its directory was read: no error. + return + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("stat %s: %v", p, err), @@ -1131,6 +1158,11 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry, } info, err := e.Info() + if errors.Is(err, fs.ErrNotExist) { + // Removed since its parent was read: no error, and no files. + return dirJob{}, false + } + if err != nil { sendEvent(ctx, events, walkEvent{ warn: fmt.Sprintf("walk %s: %v", p, err), @@ -1195,7 +1227,7 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) { // hashResult carries the hashes computed for one inode run (or the // error that prevented computing them) from the pool's workers to the -// hash phase. +// hash phase. Its run leaves out the paths found removed. type hashResult struct { run []fileRec head [sha256.Size]byte @@ -1296,6 +1328,13 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec, head, tail, content, err := hash(run[0].path, run[0].size) + // A path removed since the walk is dropped from its run. The + // next path is a hard link to the same file, and is read instead. + for errors.Is(err, fs.ErrNotExist) && len(run) > 1 { + run = run[1:] + head, tail, content, err = hash(run[0].path, run[0].size) + } + select { case results <- hashResult{ run: run, head: head, tail: tail, content: content, err: err, diff --git a/scan_test.go b/scan_test.go index 5f14906..09ece06 100644 --- a/scan_test.go +++ b/scan_test.go @@ -683,6 +683,49 @@ func TestSubdirJobStatError(t *testing.T) { } } +// TestWalkPassesOverRemovedPaths removes a file and a directory after +// their parent was read, as when they are deleted while a scan runs. +// The walk then reaches them, the directory also as one to read and as +// an operand. None of that is an error: the walk sends no event. +func TestWalkPassesOverRemovedPaths(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + file := writeFile(t, dir, "file", []byte("x")) + writeFile(t, dir, "sub/f", []byte("y")) + sub := filepath.Join(dir, "sub") + fileEntry := dirEntryFor(t, dir, "file") + subEntry := dirEntryFor(t, dir, "sub") + + for _, p := range []string{file, sub} { + err := os.RemoveAll(p) + if err != nil { + t.Fatal(err) + } + } + + ctx := t.Context() + events := make(chan walkEvent, 4) + + emitFile(ctx, file, fileEntry, walkRules{}, events) + + _, descend := subdirJob(ctx, sub, subEntry, dirJob{path: dir}, false, + walkRules{}, events) + subs := walkOneDir(ctx, dirJob{path: sub}, false, walkRules{}, events) + jobs := seedRoot(ctx, sub, walkRules{}, events) + + close(events) + + for ev := range events { + t.Errorf("event %+v for a removed path, want none", ev) + } + + if descend || len(subs) != 0 || len(jobs) != 0 { + t.Errorf("descend = %t, subdirectories %+v, jobs %+v, want none", + descend, subs, jobs) + } +} + // syncOneFilesystem scans roots into db with -x and returns the scan // stats. Every device pruneRoots looks up, and the device each // operand's walk compares its subdirectories with, comes from devices @@ -1128,7 +1171,7 @@ func TestScanPipeline(t *testing.T) { // database's records. Each phase's first line names it with its place // among the four: load counts the records before reading them, and the // walk estimates its total from the records under its operand, or says -// its ETA is unknown when there are none. +// its time remaining is unknown when there are none. // //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr func TestScanProgressNamesEachPhase(t *testing.T) { @@ -1143,14 +1186,14 @@ func TestScanProgressNamesEachPhase(t *testing.T) { syncTree(t, db, filepath.Join(dir, "a")) rescan := stderr()[len(first):] - load := "[1/4] load: counting the records, elapsed 0s, eta unknown" + load := "[1/4] load: counting the records, elapsed 0s, remain unknown" cases := map[string][]string{ first: { - load, "[2/4] walk: 0 files, 0 files/s, elapsed 0s, eta unknown", + load, "[2/4] walk: 0 files, 0 files/s, elapsed 0s, remain unknown", "[3/4] hash: [0/", "[4/4] update: [0/", }, rescan: { - load, "[2/4] walk: [0/~2] 0%, 0 files/s, elapsed 0s, eta unknown", + load, "[2/4] walk: [0/~2] 0%, 0 files/s, elapsed 0s, remain unknown", "[3/4] hash: [0/", "[4/4] update: [0/", }, } @@ -1685,6 +1728,55 @@ func TestHashPhaseSkipsFileReplacedByFIFO(t *testing.T) { } } +// TestHashPhasePassesOverRemovedFiles removes two walked files before +// the hash phase reads them: one with no other path, and the first of +// two hard links to one file. Neither is an error. The phase prints +// nothing but its progress, counts nothing as skipped, and records only +// the hard link still there. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestHashPhasePassesOverRemovedFiles(t *testing.T) { + stderr := captureStderr(t) + dir := t.TempDir() + alone := writeFile(t, dir, "alone.bin", pattern(1, 300)) + a := writeFile(t, dir, "a.bin", pattern(2, 300)) + b := filepath.Join(dir, "b.bin") + + err := os.Link(a, b) + if err != nil { + t.Fatal(err) + } + + walked, _ := collectWalk(t, []string{dir}, false, 1) + + // a.bin sorts before b.bin, so the run of the two reads it first. + for _, p := range []string{alone, a} { + err = os.Remove(p) + if err != nil { + t.Fatal(err) + } + } + + s := &scanState{db: openTestDB(t), toHash: walked} + + err = s.hashPhase(t.Context(), 1) + if err != nil { + t.Fatalf("hashPhase = %v, want nil", err) + } + + if s.st.skipped != 0 || len(s.batch) != 1 || s.batch[0].path != b { + t.Errorf("skipped = %d, records %+v, want none skipped and a "+ + "record for %s alone", s.st.skipped, s.batch, b) + } + + for line := range strings.Lines(stderr()) { + if !strings.HasPrefix(line, scanPhase("hash")+": ") { + t.Errorf("stderr holds %q, want the hash phase's progress alone", + line) + } + } +} + // injectedWriteFailure is the message the injected database trigger // aborts with, so the test can recognize its own failure in the error // the scan reports. diff --git a/trees_test.go b/trees_test.go index e800d26..540c117 100644 --- a/trees_test.go +++ b/trees_test.go @@ -13,7 +13,7 @@ import ( // treesOpenLine is the plain line of trees' open phase when it opens // the database at once; it prints it as the phase starts and as it ends. const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " + - "eta unknown" + "remain unknown" // smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and // /d/t2 are identical, /d/t3 differs from them only by one filename. @@ -236,11 +236,11 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) { "trees: database " + path + " (from " + databaseEnv + ")", treesOpenLine, treesOpenLine, - "[2/4] read: counting the records, elapsed 0s, eta unknown", + "[2/4] read: counting the records, elapsed 0s, remain unknown", "[2/4] read: [6/6] 100%, ", - "[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown", + "[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, remain unknown", "[3/4] group: [8/8] 100%, ", - "[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown", + "[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown", "[4/4] write: [1/1] 100%, ", "trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ", }