Author SHA1 Message Date
sneak 2b46d374b1 Rewrite README.md sentence by sentence for a newcomer
check / check (push) Canceled after 0s
Rewrites `README.md` from scratch for
#135. Every statement was checked
against the code on `next` and against what the built binary prints on
small trees under `/var/tmp`. The headings that the code comments cite
are kept.

Where the old text and the repository disagreed, the new text states
what the code does. #140 lists
every such point. Among them: `@latest` installs the newest tagged
release, ten million files in both "Rationale" and "Design", no Windows
excludes, a relative operand is joined to the working directory as `pwd`
prints it, and an operand reached through a symlinked directory below
another operand is dropped (#144).

Dropped from "Constraints": the permission to add `github.com/spf13/viper`
if configuration-file support is ever needed.

Model: fable-5-1
2026-10-27 00:39:18 +00:00
clawbot ee09cda355 Never scan an operand inside a .zfs directory (closes #142)
check / check (push) Canceled after 0s
An operand inside a directory named .zfs was walked, because the
operand check looked only at the operand's own name. The check now
also looks at every directory above the operand, both as given and
with its symlinks resolved. Such an operand gets a warning, counts as
skipped and is not walked, whether given alone or under another
operand. The check runs before overlapping operands are pruned, so an
operand under another one still gets its warning.

The test for an operand named .zfs now writes its record beneath .zfs
straight into the database, since no scan stores one there any more.

Judgement call: a path component named .zfs counts whatever it is, a
symlink included.

Model: opus-5-5
2026-10-26 15:15:37 +01:00
clawbot aa99b37b99 Skip removed paths silently and fix the progress line (closes #133)
check / check (push) Canceled after 0s
A path removed while the scan runs gets no warning and no skipped
count, in the walk or the hash phase. If the first of several hard
links is gone, the next is read. Other errors are still warned about.

On a terminal every display is the library's spinner, then the line. A
phase with an exact total ends the padded line with its percent and a
bar at a fixed column, drawn by sfdupes, as the library shows no
spinner beside its own bar. A too-wide line loses its rate first.

The time remaining is labelled remain, not eta.

Judgement call: an operand removed mid-scan is skipped silently too.
Deviation: README's stalled-line example keeps its words on 100
columns, not 80.

Model: opus-5-5
2026-10-26 06:32:38 +01:00
11 changed files with 1820 additions and 1125 deletions
+1043 -864
View File
File diff suppressed because it is too large Load Diff
+23
View File
@@ -29,6 +29,29 @@
# Completed Steps # Completed Steps
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
it. It leads with what the program is and how to use it. A reference section
covers every setting and every subcommand. Development and testing come last.
Every statement was checked against the code on `next`. Five comments in
`main_test.go` cite "Settings", where their rules now live. One comment each
in `scan_test.go`, `trees_test.go` and `exclude_test.go` cites what the README
now says (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
- an operand at or inside a directory named `.zfs` is never scanned, whether
given alone or under another operand. `scan` warns about it, counts it as
skipped and does not walk it (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/142)
- a file or directory removed while the scan runs is not an error, in the walk
or in the hash phase. The scan prints no line for it and does not count it as
skipped. When the first of several hard links to a file is gone, the hash
phase reads the next one instead. On a terminal every progress display starts
with the spinner, the hash phase's included. A phase with an exact total ends
its line with its percent and a 10-column bar at the terminal's right edge,
which no longer moves as the line's length changes. A line too wide for the
terminal now loses its rate before its bar. The time remaining is labelled
`remain`, not `eta` (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/133)
- the `head`, `tail` and `content` columns are BLOBs, each holding the raw 32 - the `head`, `tail` and `content` columns are BLOBs, each holding the raw 32
bytes of a SHA-256, half the size of the hex text they held. The hashes stay bytes of a SHA-256, half the size of the hex text they held. The hashes stay
raw from the hash phase through the database to `report` and `trees`. Reading raw from the hash phase through the database to `report` and `trees`. Reading
+3 -1
View File
@@ -17,7 +17,9 @@ import (
func TestDefaultExcludes(t *testing.T) { func TestDefaultExcludes(t *testing.T) {
t.Parallel() t.Parallel()
// README "Rules for the walk": each system's list. // README "Rules for the walk": the Linux and macOS lists, and none
// on any other system. The Windows list exists in the code only: the
// binary does not build for Windows, so the README leaves it out.
cases := map[string][]string{ cases := map[string][]string{
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}, goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
goosDarwin: { goosDarwin: {
+2 -2
View File
@@ -12,13 +12,13 @@ import (
// infoOpenLine is the plain line of info's open phase when it opens the // infoOpenLine is the plain line of info's open phase when it opens the
// database at once; it prints it as the phase starts and as it ends. // database at once; it prints it as the phase starts and as it ends.
const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " + const infoOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown" "remain unknown"
// infoCountLine is the plain line of info's count phase when the count // infoCountLine is the plain line of info's count phase when the count
// takes under a second; it prints it as the phase starts and as it // takes under a second; it prints it as the phase starts and as it
// ends. // ends.
const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " + const infoCountLine = "[2/2] count: counting the records, elapsed 0s, " +
"eta unknown" "remain unknown"
// infoRecs are four records: a duplicate pair, a file of 20 MiB and // infoRecs are four records: a duplicate pair, a file of 20 MiB and
// another small file. // another small file.
+136 -25
View File
@@ -393,7 +393,7 @@ func TestRunNamesRelativeDatabase(t *testing.T) {
} }
func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) { func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) {
// README §Database: the path names the file exactly. With link a // README §Settings: the path names the file exactly. With link a
// symlink to the directory real/sub, link/../db.sqlite names // symlink to the directory real/sub, link/../db.sqlite names
// real/db.sqlite, not db.sqlite beside link. Every subcommand must // real/db.sqlite, not db.sqlite beside link. Every subcommand must
// use that file, relative or absolute, and its first line must name // use that file, relative or absolute, and its first line must name
@@ -648,7 +648,7 @@ func TestRunUsageErrors(t *testing.T) {
} }
func TestRunScanRejectsWorkersBelowOne(t *testing.T) { func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
// README §scan mode: --workers below 1 is a usage error reported in // README §Settings: --workers below 1 is a usage error reported in
// one line on stderr, before the scan opens the database. // one line on stderr, before the scan opens the database.
for _, workers := range []string{"0", "-1"} { for _, workers := range []string{"0", "-1"} {
t.Run(workers, func(t *testing.T) { t.Run(workers, func(t *testing.T) {
@@ -686,8 +686,8 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
func TestRunHelp(t *testing.T) { func TestRunHelp(t *testing.T) {
t.Parallel() t.Parallel()
// README §Subcommands: help goes to stderr, exits 0, and leaves // README §Settings: help goes to stderr, exits 0, and leaves stdout
// stdout empty. // empty.
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}} cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
for _, args := range cases { for _, args := range cases {
@@ -713,8 +713,8 @@ func TestRunHelp(t *testing.T) {
func TestRunVersion(t *testing.T) { func TestRunVersion(t *testing.T) {
t.Parallel() t.Parallel()
// README §Subcommands: the version is one line on stdout, with // README §Settings: the version is one line on stdout, with nothing
// nothing on stderr, and exits 0. // on stderr, and exits 0.
info, _ := debug.ReadBuildInfo() info, _ := debug.ReadBuildInfo()
for _, arg := range []string{"--version", "-v"} { for _, arg := range []string{"--version", "-v"} {
@@ -773,8 +773,8 @@ func TestBuildVersion(t *testing.T) {
func TestRunVersionWriteFailureIsFatal(t *testing.T) { func TestRunVersionWriteFailureIsFatal(t *testing.T) {
t.Parallel() t.Parallel()
// README §Error handling: a stdout write failure exits 1, reported // README §Error handling: a failed write to stdout is a fatal error,
// in one line on stderr. // exit 1, reported on stderr starting with "sfdupes:".
var stderr bytes.Buffer var stderr bytes.Buffer
code := run([]string{"--version"}, failingWriter{}, &stderr) code := run([]string{"--version"}, failingWriter{}, &stderr)
@@ -928,24 +928,21 @@ func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
} }
func TestRunScanSkipsZFSOperand(t *testing.T) { func TestRunScanSkipsZFSOperand(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
zfs := filepath.Join(t.TempDir(), ".zfs") zfs := filepath.Join(t.TempDir(), ".zfs")
snapshot := filepath.Join(zfs, "snapshot", "hourly") f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
f := writeFile(t, snapshot, "f", pattern(1, 10))
// An operand beneath a .zfs directory is walked, because it is not // No scan records a file inside a .zfs directory, so the record
// itself named .zfs. // beneath the operand is written straight into the database.
scanOK(t, snapshot) path := seedDatabase(t, []scanRec{{size: 10, path: f}})
t.Setenv(databaseEnv, path)
assertOperandSkipped(t, path, zfs, ".zfs directory", f) assertOperandSkipped(t, path, zfs, ".zfs directory", f)
} }
// assertOperandSkipped scans operand alone and checks that it is skipped // assertOperandSkipped scans operand alone and checks that it is skipped
// as kind: a warning naming it, one skip in the summary, exit 0, and the // as kind: a warning naming it, one skip in the summary, exit 0, and the
// record for kept, which an earlier scan stored beneath operand, still // record for kept, stored beneath operand before this scan, still in the
// in the database at dbPath. // database at dbPath.
func assertOperandSkipped(t *testing.T, dbPath, operand, kind, func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
kept string, kept string,
) { ) {
@@ -974,6 +971,121 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
recordByPath(t, dbRecords(t, db), kept) recordByPath(t, dbRecords(t, db), kept)
} }
// TestRunScanSkipsOperandInsideZFS scans a directory inside a .zfs
// directory, first alone and then under another operand. Each time it
// is warned about and counted as skipped, and nothing in it is walked.
// The other operand is scanned as usual.
func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
writeFile(t, hourly, "f", pattern(1, 10))
f := writeFile(t, dir, "f", pattern(2, 10))
warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n"
for _, tc := range []struct {
operands []string
summary string
}{
{
[]string{hourly},
"scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n",
},
{
[]string{dir, hourly},
"scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n",
},
} {
stderr := scanOK(t, tc.operands...)
if !strings.Contains(stderr, warning) ||
!strings.Contains(stderr, tc.summary) {
t.Errorf("scan %q: stderr = %q, want %q and %q", tc.operands,
stderr, warning, tc.summary)
}
}
db, err := openDB(path, reportParams)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
if got := recordPaths(dbRecords(t, db)); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f})
}
}
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
// by the path given, counted as skipped, and not walked.
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
dir := t.TempDir()
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
snaps := filepath.Join(dir, "snaps")
err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
if err != nil {
t.Fatal(err)
}
daily := filepath.Join(snaps, "daily")
stderr := scanOK(t, daily)
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n"
if !strings.Contains(stderr, warning) ||
!strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q and %q", stderr, warning, summary)
}
}
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
// a symlink to a directory outside any .zfs directory. The operand is
// scanned as usual, and its file is recorded by the path given.
func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
data := filepath.Join(dir, "data")
err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
if err != nil {
t.Fatal(err)
}
daily := filepath.Join(data, "daily")
stderr := scanOK(t, daily)
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
db, err := openDB(path, reportParams)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
}
func TestRunReportSucceeds(t *testing.T) { func TestRunReportSucceeds(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
@@ -1122,7 +1234,7 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
} }
func TestRunDatabasePathUsedAsGiven(t *testing.T) { func TestRunDatabasePathUsedAsGiven(t *testing.T) {
// README §Database: the path names the database file exactly. In // README §Settings: the path names the database file exactly. In
// SQLite's connection string an unescaped ? or # would end the file // SQLite's connection string an unescaped ? or # would end the file
// name and % would start an escape, and a path starting with // // name and % would start an escape, and a path starting with //
// could be read as a host name. %25 is a valid escape, so unescaped // could be read as a host name. %25 is a valid escape, so unescaped
@@ -1167,9 +1279,8 @@ func holdScanLock(t *testing.T, path string) {
} }
func TestRunSecondScanFails(t *testing.T) { func TestRunSecondScanFails(t *testing.T) {
// README §Database: while one scan holds the lock, a second scan // README §Database: a second scan that finds the lock held fails
// fails at once, naming the lock file, without opening the // at once, naming the lock file, without opening the database.
// database.
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
@@ -1228,7 +1339,7 @@ func assertSecondScanFails(t *testing.T, path string) {
} }
func TestRunSecondScanThroughSymlinkFails(t *testing.T) { func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
// README §Database: every path to one database names the same lock // README §Database: a path through a symlink names the same lock
// file, so while a scan holds it through one path, a scan through // file, so while a scan holds it through one path, a scan through
// another fails, naming the lock file beside the database's real // another fails, naming the lock file beside the database's real
// path. // path.
@@ -1333,8 +1444,8 @@ func TestRunReportsDuringScan(t *testing.T) {
} }
func TestRunStdoutClosedIsFatal(t *testing.T) { func TestRunStdoutClosedIsFatal(t *testing.T) {
// README §Error handling: a stdout write failure exits 1, reported // README §Error handling: a failed write to stdout is a fatal error,
// in one line on stderr. // exit 1, reported on stderr starting with "sfdupes:".
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} { for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t)) t.Setenv(databaseEnv, testDBPath(t))
+102 -90
View File
@@ -9,6 +9,7 @@ import (
"strings" "strings"
"sync" "sync"
"time" "time"
"unicode/utf8"
"github.com/schollz/progressbar/v3" "github.com/schollz/progressbar/v3"
"golang.org/x/term" "golang.org/x/term"
@@ -34,10 +35,17 @@ const stallTime = 10 * time.Second
// spinnerType selects the progressbar library's spinner style. // spinnerType selects the progressbar library's spinner style.
const spinnerType = 14 const spinnerType = 14
// barWidth is the width of the progressbar library's bar between its // barWidth is the width of a phase's bar between its two ends, as in
// two ends, as in the example line in README "Progress". // the example line in README "Progress".
const barWidth = 10 const barWidth = 10
// barCell is one column of the filled part of a bar.
const barCell = "█"
// barColumns is the width of what percentBar returns: the percent in
// four columns and its sign, a space, and the bar with its two ends.
const barColumns = barWidth + 8
// percentScale converts a fraction to a percentage. // percentScale converts a fraction to a percentage.
const percentScale = 100 const percentScale = 100
@@ -45,17 +53,12 @@ const percentScale = 100
// read, as the progressbar library takes it. // read, as the progressbar library takes it.
const fallbackWidth = 80 const fallbackWidth = 80
// spinnerMargin is the columns of a spinner's line besides its text: // spinnerMargin is the columns of a line besides its text: the spinner
// the spinner and the three spaces the library draws around the text, // and the three spaces the library draws around the text, and the
// and the terminal's last column, which every line stays clear of so // terminal's last column, which every line stays clear of so that no
// that no terminal wraps it. // terminal wraps it.
const spinnerMargin = 5 const spinnerMargin = 5
// barMargin is the columns of a bar's line besides its text: the
// percent and the bar the library draws after the text, the spaces
// around them, and the terminal's last column.
const barMargin = barWidth + 11
// countingRecords is what the display of a phase that reads the records // countingRecords is what the display of a phase that reads the records
// says while it counts them, before it has its total. // says while it counts them, before it has its total.
const countingRecords = "counting the records" const countingRecords = "counting the records"
@@ -109,13 +112,13 @@ type drawnCount struct {
} }
// progress renders one phase's progress on stderr. On a TTY the // progress renders one phase's progress on stderr. On a TTY the
// progressbar library draws line, fitted to the terminal's width: after // progressbar library draws its spinner and then line, fitted to the
// the library's spinner, or, for a phase with an exact total, before the // terminal's width. A phase with an exact total ends the line with its
// library's bar when the terminal has room for the bar beside it. It is // percent and bar, at the terminal's right edge, when the terminal has
// drawn again every redrawInterval, so its elapsed time keeps moving // room for them. It is drawn again every redrawInterval, so its elapsed
// while the phase waits for its next item. When stderr is not a TTY it // time keeps moving while the phase waits for its next item. When
// emits no ANSI redraws: it prints line as the phase starts, every // stderr is not a TTY it emits no ANSI redraws: it prints line as the
// plainInterval while it runs, and as it ends. // phase starts, every plainInterval while it runs, and as it ends.
// //
// The draws come from a goroutine of their own, so mu guards the fields // The draws come from a goroutine of their own, so mu guards the fields
// below it. Every method must be called from the main goroutine. A nil // below it. Every method must be called from the main goroutine. A nil
@@ -131,7 +134,6 @@ type progress struct {
mu sync.Mutex mu sync.Mutex
bar *progressbar.ProgressBar // what the library draws; nil until then bar *progressbar.ProgressBar // what the library draws; nil until then
withBar bool // bar is the library's bar, not its spinner
widest int // the widest line bar has drawn, in columns widest int // the widest line bar has drawn, in columns
label string label string
unit string unit string
@@ -141,7 +143,7 @@ type progress struct {
stalledOn string stalledOn string
count int64 count int64
start time.Time // when the phase started, for the elapsed time start time.Time // when the phase started, for the elapsed time
rateStart time.Time // when the count started, for the ETA rateStart time.Time // when the count started, for the time remaining
lastItem time.Time // when the last item was completed, or the count started lastItem time.Time // when the last item was completed, or the count started
// drawn holds the count at each draw over the last rateWindow, oldest // drawn holds the count at each draw over the last rateWindow, oldest
// first; the rate counts from the first. It keeps the draw before the // first; the rate counts from the first. It keeps the draw before the
@@ -215,55 +217,34 @@ func (p *progress) draw() {
} }
if p.width == nil { if p.width == nil {
fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, true)) fmt.Fprintln(os.Stderr, p.line(now, math.MaxInt, false))
return return
} }
width := p.width() width := p.width()
text := p.line(now, width-spinnerMargin, true)
// A phase with an exact total gets the library's bar when its whole // Before each draw the library erases its line with as many spaces as
// line, without the percent the bar shows, fits beside the bar: the // the widest line it has drawn. A display gets a new one when the
// bar gives way before any part of the line. A total of 0 leaves the // terminal has been made too narrow for that erase, which would wrap
// bar nothing to fill. // and leave a line behind at every draw.
text, withBar := "", false if p.bar == nil || p.widest >= width {
if p.total > 0 && !p.estimate { p.newBar(width)
text = p.line(now, math.MaxInt, false)
withBar = len(text) <= width-barMargin
}
margin := barMargin
if !withBar {
margin = spinnerMargin
text = p.line(now, width-margin, true)
}
// The library makes a bar or a spinner, and before each draw erases
// its line with as many spaces as the widest line it has drawn. A
// display gets a new one when it turns from one to the other, as its
// line grows or the terminal is resized, and when the terminal has
// been made too narrow for that erase, which would wrap and leave a
// line behind at every draw.
if p.bar == nil || p.withBar != withBar || p.widest >= width {
p.newBar(withBar, width)
}
if withBar {
_ = p.bar.Set64(p.count)
} }
p.bar.Describe(text) p.bar.Describe(text)
// The library drew the text and its margin, less the terminal's last // The library drew the spinner, the text and the spaces around it,
// column, which the margin keeps clear. // less the terminal's last column, which spinnerMargin keeps clear. A
p.widest = max(p.widest, len(text)+margin-1) // bar's cells take one column each but more than one byte.
p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1)
} }
// newBar has the library make what draw draws: a bar for the phase's // newBar has the library make what draw draws: its spinner, followed by
// total when withBar is set, and a spinner when not. Either shows the // the text draw gives it and nothing of the library's own. p.mu must be
// text draw gives it, and nothing of its own besides the spinner, or // held.
// the bar and its percent. p.mu must be held. func (p *progress) newBar(width int) {
func (p *progress) newBar(withBar bool, width int) {
if p.bar != nil { if p.bar != nil {
// Erase what was drawn before, short of the last column of a // Erase what was drawn before, short of the last column of a
// terminal that may have been made narrower since: the library's // terminal that may have been made narrower since: the library's
@@ -272,29 +253,36 @@ func (p *progress) newBar(withBar bool, width int) {
fmt.Fprint(os.Stderr, "\r"+blank+"\r") fmt.Fprint(os.Stderr, "\r"+blank+"\r")
} }
total := int64(-1) // the library's spinner // A total of -1 gives the library's spinner.
if withBar { p.bar = progressbar.NewOptions64(-1,
total = p.total
}
p.bar = progressbar.NewOptions64(total,
progressbar.OptionSetWriter(os.Stderr), progressbar.OptionSetWriter(os.Stderr),
progressbar.OptionSetWidth(barWidth),
progressbar.OptionSpinnerType(spinnerType), progressbar.OptionSpinnerType(spinnerType),
// The spinner turns at each draw, not on a timer of the // The spinner turns at each draw, not on a timer of the
// library's own. // library's own.
progressbar.OptionSetSpinnerChangeInterval(0), progressbar.OptionSetSpinnerChangeInterval(0),
// The text carries the elapsed time and the ETA. // The text carries the elapsed time and the time remaining.
progressbar.OptionSetElapsedTime(false), progressbar.OptionSetElapsedTime(false),
progressbar.OptionSetPredictTime(false), progressbar.OptionSetPredictTime(false),
) )
p.withBar, p.widest = withBar, 0 p.widest = 0
}
// percentBar returns the percent and the bar that end the text of a
// phase with an exact total, as " 25% |██ |": the bar's
// barWidth columns fill in proportion to the items done. A count past
// the total, as when a scan adds records while trees reads them, shows
// as 100%. p.mu must be held.
func (p *progress) percentBar() string {
done := min(float64(p.count)/float64(p.total), 1)
filled := strings.Repeat(barCell, int(done*barWidth))
return fmt.Sprintf("%4.0f%% |%-*s|", done*percentScale, barWidth, filled)
} }
// setTotal gives a phase that started without a total its total, once // setTotal gives a phase that started without a total its total, once
// it is known. The rate, the ETA and the time with no item completed // it is known. The rate, the time remaining and the time with no item
// count from then on: the time spent waiting for the total says nothing // completed count from then on: the time spent waiting for the total
// about how fast items go. // says nothing about how fast items go.
func (p *progress) setTotal(total int64) { func (p *progress) setTotal(total int64) {
if p == nil { if p == nil {
return return
@@ -438,22 +426,26 @@ func (p *progress) finish() {
// line formats the phase's progress as of now in at most room columns, // line formats the phase's progress as of now in at most room columns,
// as // as
// //
// [3/4] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, eta 13s // [3/4] hash: [1234/5000] 25%, 300 files/s, elapsed 4s, remain 13s
// //
// where "files" is the phase's unit, without the percent when percent // where "files" is the phase's unit. A phase with no total shows the
// is false, as before the library's bar, which shows its own. A phase // items done instead, or what it waits for. The rate is that since the
// with no total shows the items done instead, or what it waits for. The // first draw in drawn, and the time remaining comes from the rate since
// rate is that since the first draw in drawn, and the ETA comes from // the count started. Once no item has been completed for stallTime, the
// the rate since the count started. Once no item has been completed for // rate, which would read 0, gives way to the word "stalled" and where
// stallTime, the rate, which would read 0, gives way to the word // the phase waits, as "stalled disk/db".
// "stalled" and where the phase waits, as "stalled disk/db".
// //
// The phase and its place, the progress, the elapsed time and the ETA // When bar is true, as on a terminal, a phase with an exact total ends
// are kept whole. A line wider than room leaves out the rate, the words // the line with its percent and bar, in place of the percent after its
// of a stalled phase, or what the phase waits for; only if it is still // count. A total of 0 leaves the bar nothing to fill.
// wider is it cut at room. Every part is ASCII, so its length is its //
// width in columns. p.mu must be held. // The phase and its place, the progress, the elapsed time and the time
func (p *progress) line(now time.Time, room int, percent bool) string { // remaining are kept whole. A line wider than room first leaves out the
// rate, the words of a stalled phase, or what the phase waits for. Next
// it leaves out the bar. Only if it is still wider is it cut at room.
// Every part but the bar is ASCII, so its length is its width in
// columns. p.mu must be held.
func (p *progress) line(now time.Time, room int, bar bool) string {
elapsed := now.Sub(p.start).Round(time.Second) elapsed := now.Sub(p.start).Round(time.Second)
// head and tail hold the parts kept whole; extra, between them, is // head and tail hold the parts kept whole; extra, between them, is
@@ -462,21 +454,41 @@ func (p *progress) line(now time.Time, room int, percent bool) string {
if p.total < 0 && p.waitingFor != "" { if p.total < 0 && p.waitingFor != "" {
head, extra = p.label+": ", p.waitingFor+", " head, extra = p.label+": ", p.waitingFor+", "
tail = fmt.Sprintf("elapsed %s, eta unknown", elapsed) tail = fmt.Sprintf("elapsed %s, remain unknown", elapsed)
} else { } else {
since := p.drawn[0] since := p.drawn[0]
rate := perSecond(p.count-since.count, now.Sub(since.at)) rate := perSecond(p.count-since.count, now.Sub(since.at))
average := perSecond(p.count, now.Sub(p.rateStart)) average := perSecond(p.count, now.Sub(p.rateStart))
head = p.label + ": " + p.itemsDone(percent) + ", " head = p.label + ": " + p.itemsDone(true) + ", "
extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit) extra = fmt.Sprintf("%.0f %s/s, ", rate, p.unit)
tail = fmt.Sprintf("elapsed %s, eta %s", elapsed, p.eta(average)) tail = fmt.Sprintf("elapsed %s, remain %s", elapsed, p.remain(average))
if now.Sub(p.lastItem) >= stallTime { if now.Sub(p.lastItem) >= stallTime {
extra = "stalled " + p.stalledOn + ", " extra = "stalled " + p.stalledOn + ", "
} }
} }
if bar && p.total > 0 && !p.estimate {
// Beside the bar, which shows the percent, the count shows none.
// The line is padded to the room the bar leaves it, so the bar
// stays at room's right edge however long the line is.
beside := room - barColumns
barHead := p.label + ": " + p.itemsDone(false) + ", "
switch {
case len(barHead+extra+tail) <= beside:
return fmt.Sprintf("%-*s%s", beside, barHead+extra+tail,
p.percentBar())
case len(barHead+tail) <= beside:
return fmt.Sprintf("%-*s%s", beside, barHead+tail, p.percentBar())
}
// The bar gives way. The rate, or the words of a stalled phase,
// gave way before it and stay out.
extra = ""
}
full, kept := head+extra+tail, head+tail full, kept := head+extra+tail, head+tail
switch { switch {
@@ -526,10 +538,10 @@ func (p *progress) itemsDone(percent bool) string {
return fmt.Sprintf("%s %.0f%%", done, pct) return fmt.Sprintf("%s %.0f%%", done, pct)
} }
// eta returns the time the phase has left at rate items a second, or // remain returns the time the phase has left at rate items a second, or
// "unknown": with no total, before the first item, and once the count // "unknown": with no total, before the first item, and once the count
// passes an estimate. p.mu must be held. // passes an estimate. p.mu must be held.
func (p *progress) eta(rate float64) string { func (p *progress) remain(rate float64) string {
switch { switch {
case p.total < 0, p.estimate && p.count > p.total: case p.total < 0, p.estimate && p.count > p.total:
return "unknown" return "unknown"
@@ -539,8 +551,8 @@ func (p *progress) eta(rate float64) string {
return "unknown" return "unknown"
} }
remaining := time.Duration( left := time.Duration(
float64(p.total-p.count) / rate * float64(time.Second)) float64(p.total-p.count) / rate * float64(time.Second))
return remaining.Round(time.Second).String() return left.Round(time.Second).String()
} }
+261 -101
View File
@@ -117,24 +117,24 @@ func TestProgressLine(t *testing.T) {
label: scanPhase("load"), total: -1, label: scanPhase("load"), total: -1,
waitingFor: countingRecords, waitingFor: countingRecords,
}, },
want: "[1/4] load: counting the records, elapsed 10s, eta unknown", want: "[1/4] load: counting the records, elapsed 10s, remain unknown",
}, { }, {
p: &progress{ p: &progress{
label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20, label: scanPhase("hash"), unit: scanUnit, total: 100, count: 20,
}, },
want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s", want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s",
}, { }, {
p: &progress{label: scanPhase("hash"), unit: scanUnit, total: 100}, p: &progress{label: scanPhase("hash"), unit: scanUnit, total: 100},
want: "[3/4] hash: [0/100] 0%, 0 files/s, elapsed 10s, eta unknown", want: "[3/4] hash: [0/100] 0%, 0 files/s, elapsed 10s, remain unknown",
}, { }, {
p: &progress{label: scanPhase("update"), unit: scanUnit, total: 0}, p: &progress{label: scanPhase("update"), unit: scanUnit, total: 0},
want: "[4/4] update: [0/0] 100%, 0 files/s, elapsed 10s, eta 0s", want: "[4/4] update: [0/0] 100%, 0 files/s, elapsed 10s, remain 0s",
}, { }, {
p: &progress{ p: &progress{
label: scanPhase("walk"), unit: scanUnit, total: 100, label: scanPhase("walk"), unit: scanUnit, total: 100,
estimate: true, count: 20, estimate: true, count: 20,
}, },
want: "[2/4] walk: [20/~100] 20%, 2 files/s, elapsed 10s, eta 40s", want: "[2/4] walk: [20/~100] 20%, 2 files/s, elapsed 10s, remain 40s",
}, { }, {
// Past the estimate the walk cannot tell how long it has left. // Past the estimate the walk cannot tell how long it has left.
p: &progress{ p: &progress{
@@ -142,13 +142,13 @@ func TestProgressLine(t *testing.T) {
estimate: true, count: 120, estimate: true, count: 120,
}, },
want: "[2/4] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " + want: "[2/4] walk: [120/~100] 120%, 12 files/s, elapsed 10s, " +
"eta unknown", "remain unknown",
}, { }, {
// A first scan's walk has no estimate. // A first scan's walk has no estimate.
p: &progress{ p: &progress{
label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20, label: scanPhase("walk"), unit: scanUnit, total: -1, count: 20,
}, },
want: "[2/4] walk: 20 files, 2 files/s, elapsed 10s, eta unknown", want: "[2/4] walk: 20 files, 2 files/s, elapsed 10s, remain unknown",
}, { }, {
// The rate is in what the phase counts. // The rate is in what the phase counts.
p: &progress{ p: &progress{
@@ -156,14 +156,14 @@ func TestProgressLine(t *testing.T) {
count: 20, count: 20,
}, },
want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " + want: "[3/4] group: [20/100] 20%, 2 directories/s, elapsed 10s, " +
"eta 40s", "remain 40s",
}} }}
for _, tc := range cases { for _, tc := range cases {
tc.p.start, tc.p.rateStart, tc.p.lastItem = ago, ago, now tc.p.start, tc.p.rateStart, tc.p.lastItem = ago, ago, now
tc.p.drawn = []drawnCount{{at: ago}} tc.p.drawn = []drawnCount{{at: ago}}
if got := tc.p.line(now, math.MaxInt, true); got != tc.want { if got := tc.p.line(now, math.MaxInt, false); got != tc.want {
t.Errorf("line = %q, want %q", got, tc.want) t.Errorf("line = %q, want %q", got, tc.want)
} }
} }
@@ -198,48 +198,49 @@ func TestLineFitsRoom(t *testing.T) {
} }
cases := []struct { cases := []struct {
p *progress p *progress
room int room int
percent bool bar bool
want string want string
}{{ }{{
p: hash, room: 57, percent: true, p: hash, room: 60,
want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, eta 40s", want: "[3/4] hash: [20/100] 20%, 2 files/s, elapsed 10s, remain 40s",
}, { }, {
p: hash, room: 56, percent: true, p: hash, room: 59,
want: "[3/4] hash: [20/100] 20%, elapsed 10s, eta 40s", want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40s",
}, { }, {
p: hash, room: 45, percent: true, p: hash, room: 48,
want: "[3/4] hash: [20/100] 20%, elapsed 10s, eta 40", want: "[3/4] hash: [20/100] 20%, elapsed 10s, remain 40",
}, { }, {
// Before the library's bar, which shows the percent. // The bar shows the percent.
p: hash, room: 57, percent: false, p: hash, room: 74, bar: true,
want: "[3/4] hash: [20/100], 2 files/s, elapsed 10s, eta 40s", want: "[3/4] hash: [20/100], 2 files/s, elapsed 10s, remain 40s" +
" 20% |██ |",
}, { }, {
p: load, room: 40, percent: true, p: load, room: 43,
want: "[1/4] load: elapsed 10s, eta unknown", want: "[1/4] load: elapsed 10s, remain unknown",
}, { }, {
p: stalled, room: 65, percent: true, p: stalled, room: 68,
want: "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " + want: "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 50s, " +
"eta 3m20s", "remain 3m20s",
}, { }, {
p: stalled, room: 64, percent: true, p: stalled, room: 67,
want: "[3/4] hash: [20/100] 20%, elapsed 50s, eta 3m20s", want: "[3/4] hash: [20/100] 20%, elapsed 50s, remain 3m20s",
}} }}
for _, tc := range cases { for _, tc := range cases {
if got := tc.p.line(now, tc.room, tc.percent); got != tc.want { if got := tc.p.line(now, tc.room, tc.bar); got != tc.want {
t.Errorf("line(%d, %t) = %q, want %q", t.Errorf("line(%d, %t) = %q, want %q", tc.room, tc.bar, got,
tc.room, tc.percent, got, tc.want) tc.want)
} }
} }
} }
// TestSetTotalStartsRate checks that setTotal starts a phase's rate, // TestSetTotalStartsRate checks that setTotal starts a phase's rate,
// its ETA and its time with no record read from when the total becomes // its time remaining and its time with no record read from when the
// known, not from when the phase started waiting for it: after 20 // total becomes known, not from when the phase started waiting for it:
// seconds of waiting for the total, the phase has not stalled 5 seconds // after 20 seconds of waiting for the total, the phase has not stalled 5
// after it, and has 10 seconds after it with no record read. // seconds after it, and has 10 seconds after it with no record read.
func TestSetTotalStartsRate(t *testing.T) { func TestSetTotalStartsRate(t *testing.T) {
t.Parallel() t.Parallel()
@@ -255,15 +256,15 @@ func TestSetTotalStartsRate(t *testing.T) {
clock.Add(5 * time.Second) clock.Add(5 * time.Second)
want := "[1/4] load: [0/100] 0%, 0 files/s, elapsed 25s, eta unknown" want := "[1/4] load: [0/100] 0%, 0 files/s, elapsed 25s, remain unknown"
if got := p.line(clock.Now(), math.MaxInt, true); got != want { if got := p.line(clock.Now(), math.MaxInt, false); got != want {
t.Errorf("line = %q, want %q", got, want) t.Errorf("line = %q, want %q", got, want)
} }
clock.Add(5 * time.Second) clock.Add(5 * time.Second)
want = "[1/4] load: [0/100] 0%, stalled db, elapsed 30s, eta unknown" want = "[1/4] load: [0/100] 0%, stalled db, elapsed 30s, remain unknown"
if got := p.line(clock.Now(), math.MaxInt, true); got != want { if got := p.line(clock.Now(), math.MaxInt, false); got != want {
t.Errorf("line = %q, want %q", got, want) t.Errorf("line = %q, want %q", got, want)
} }
@@ -272,8 +273,8 @@ func TestSetTotalStartsRate(t *testing.T) {
p.increment() p.increment()
} }
want = "[1/4] load: [20/100] 20%, 2 files/s, elapsed 30s, eta 40s" want = "[1/4] load: [20/100] 20%, 2 files/s, elapsed 30s, remain 40s"
if got := p.line(clock.Now(), math.MaxInt, true); got != want { if got := p.line(clock.Now(), math.MaxInt, false); got != want {
t.Errorf("line = %q, want %q", got, want) t.Errorf("line = %q, want %q", got, want)
} }
} }
@@ -316,7 +317,7 @@ func TestPlainProgressPrintsAtStartAndEnd(t *testing.T) {
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n") lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
want := "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown" want := "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown"
if len(lines) != 2 || lines[0] != want || if len(lines) != 2 || lines[0] != want ||
!strings.HasPrefix(lines[1], "[3/4] hash: [1000/1000] 100%, ") { !strings.HasPrefix(lines[1], "[3/4] hash: [1000/1000] 100%, ") {
t.Errorf("stderr lines = %q, want %q, then the line at 1000", t.Errorf("stderr lines = %q, want %q, then the line at 1000",
@@ -336,7 +337,7 @@ func TestPlainProgressPrintsWhileWaiting(t *testing.T) {
}, nil, testRedraw, time.Now) }, nil, testRedraw, time.Now)
awaitOutput(t, stderr, awaitOutput(t, stderr,
"[1/4] load: counting the records, elapsed 0s, eta unknown\n", 3) "[1/4] load: counting the records, elapsed 0s, remain unknown\n", 3)
p.finish() p.finish()
} }
@@ -366,23 +367,23 @@ func TestPlainProgressStalledProducer(t *testing.T) {
clock.Add(5 * time.Second) clock.Add(5 * time.Second)
awaitOutput(t, stderr, awaitOutput(t, stderr,
"[3/4] hash: [20/100] 20%, 4 files/s, elapsed 5s, eta 20s\n", 3) "[3/4] hash: [20/100] 20%, 4 files/s, elapsed 5s, remain 20s\n", 3)
clock.Add(time.Second) clock.Add(time.Second)
awaitOutput(t, stderr, awaitOutput(t, stderr,
"[3/4] hash: [20/100] 20%, 0 files/s, elapsed 6s, eta 24s\n", 3) "[3/4] hash: [20/100] 20%, 0 files/s, elapsed 6s, remain 24s\n", 3)
clock.Add(4 * time.Second) clock.Add(4 * time.Second)
awaitOutput(t, stderr, awaitOutput(t, stderr,
"[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 10s, eta 40s\n", "[3/4] hash: [20/100] 20%, stalled disk/db, elapsed 10s, remain 40s\n",
3) 3)
} }
// TestPlainProgressRateAfterStall draws a phase's plain line by hand, a // TestPlainProgressRateAfterStall draws a phase's plain line by hand, a
// little over plainInterval after the line before, as a late tick // little over plainInterval after the line before, as a late tick
// does, through a stall and a burst of files after it: the rate covers // does, through a stall and a burst of files after it: the rate covers
// the files read since the line before, while the ETA comes from the // the files read since the line before, while the time remaining comes
// phase's average. // from the phase's average.
// //
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestPlainProgressRateAfterStall(t *testing.T) { func TestPlainProgressRateAfterStall(t *testing.T) {
@@ -410,10 +411,10 @@ func TestPlainProgressRateAfterStall(t *testing.T) {
p.finish() p.finish()
want := []string{ want := []string{
"[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, eta unknown", "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 0s, remain unknown",
"[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 5s, eta unknown", "[3/4] hash: [0/1000] 0%, 0 files/s, elapsed 5s, remain unknown",
"[3/4] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, eta unknown", "[3/4] hash: [0/1000] 0%, stalled disk/db, elapsed 10s, remain unknown",
"[3/4] hash: [100/1000] 10%, 20 files/s, elapsed 15s, eta 2m15s", "[3/4] hash: [100/1000] 10%, 20 files/s, elapsed 15s, remain 2m15s",
} }
lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n") lines := strings.Split(strings.TrimSuffix(stderr(), "\n"), "\n")
@@ -426,8 +427,8 @@ func TestPlainProgressRateAfterStall(t *testing.T) {
// redrawInterval, as a terminal does, through a stall of stallTime and // redrawInterval, as a terminal does, through a stall of stallTime and
// a burst of files after it: the rate covers the files read over the // a burst of files after it: the rate covers the files read over the
// last rateWindow, and falls to 0 once the burst is older than that, // last rateWindow, and falls to 0 once the burst is older than that,
// while the ETA comes from the phase's average. Plain lines show each // while the time remaining comes from the phase's average. Plain lines
// draw. // show each draw.
// //
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressRateAfterStallRedrawn(t *testing.T) { func TestProgressRateAfterStallRedrawn(t *testing.T) {
@@ -457,8 +458,8 @@ func TestProgressRateAfterStallRedrawn(t *testing.T) {
redraw(rateWindow + redrawInterval) redraw(rateWindow + redrawInterval)
p.finish() p.finish()
want := "[3/4] hash: [50/1000] 5%, 10 files/s, elapsed 15s, eta 4m45s\n" + want := "[3/4] hash: [50/1000] 5%, 10 files/s, elapsed 15s, remain 4m45s\n" +
"[3/4] hash: [50/1000] 5%, 0 files/s, elapsed 15s, eta 4m47s\n" "[3/4] hash: [50/1000] 5%, 0 files/s, elapsed 15s, remain 4m47s\n"
if !strings.Contains(stderr(), want) { if !strings.Contains(stderr(), want) {
t.Errorf("stderr = %q, want the lines %q", stderr(), want) t.Errorf("stderr = %q, want the lines %q", stderr(), want)
} }
@@ -480,10 +481,10 @@ func lastFrame(out string) string {
} }
// TestTerminalRedrawsWhileWaiting checks that the terminal display of a // TestTerminalRedrawsWhileWaiting checks that the terminal display of a
// phase, spinner or bar, keeps being drawn while the phase waits for its // phase, with a bar or without, keeps being drawn while the phase waits
// next item, with the elapsed time as of now, and shows every item // for its next item, with the elapsed time as of now, and shows every
// completed before the wait. stderr is not a terminal here, so the // item completed before the wait. The terminal is 120 columns wide, so
// displays take it to be fallbackWidth wide. // the hash phase's display has room for its bar.
// //
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalRedrawsWhileWaiting(t *testing.T) { func TestTerminalRedrawsWhileWaiting(t *testing.T) {
@@ -506,7 +507,8 @@ func TestTerminalRedrawsWhileWaiting(t *testing.T) {
for _, tc := range cases { for _, tc := range cases {
before := len(stderr()) before := len(stderr())
since := func() string { return stderr()[before:] } since := func() string { return stderr()[before:] }
p := startProgress(tc.params, terminalWidth, testRedraw, time.Now) p := startProgress(tc.params, func() int { return 120 }, testRedraw,
time.Now)
for range 50 { for range 50 {
p.increment() p.increment()
@@ -552,13 +554,13 @@ func drawLate(stderr func() string, params progressParams, width int,
return stderr()[before:] return stderr()[before:]
} }
// TestTerminalLinesKeepETA checks the line each display shows on an // TestTerminalLinesKeepTimeRemaining checks the line each display shows
// 80-column terminal at a large scan's counts and times, and the line // on an 80-column terminal at a large scan's counts and times, and the
// of trees' read phase at the same counts: the rate is left out, so // line of trees' read phase at the same counts: the rate is left out,
// that the elapsed time and the ETA stay whole. // so that the elapsed time and the time remaining stay whole.
// //
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalLinesKeepETA(t *testing.T) { func TestTerminalLinesKeepTimeRemaining(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
cases := []struct { cases := []struct {
@@ -571,19 +573,19 @@ func TestTerminalLinesKeepETA(t *testing.T) {
estimate: true, estimate: true,
}, },
want: "[2/4] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " + want: "[2/4] walk: [1234567/~1200000] 103%, elapsed 2h30m0s, " +
"eta unknown", "remain unknown",
}, { }, {
params: progressParams{ params: progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543, label: scanPhase("hash"), unit: scanUnit, total: 9876543,
}, },
want: "[3/4] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " + want: "[3/4] hash: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s", "remain 17h30m0s",
}, { }, {
params: progressParams{ params: progressParams{
label: treesPhase("read"), unit: "records", total: 9876543, label: treesPhase("read"), unit: "records", total: 9876543,
}, },
want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " + want: "[2/4] read: [1234567/9876543] 12%, elapsed 2h30m0s, " +
"eta 17h30m0s", "remain 17h30m0s",
}} }}
for _, tc := range cases { for _, tc := range cases {
@@ -594,30 +596,32 @@ func TestTerminalLinesKeepETA(t *testing.T) {
} }
} }
// TestTerminalStalledLines draws every phase's line on an 80-column // TestTerminalStalledLines draws every phase's line once the phase has
// terminal once the phase has stalled, its last item done 42 seconds // stalled, its last item done 42 seconds before. At an ordinary scan's
// before. At an ordinary scan's counts and times, 12345 of 45678 items // counts and times, 12345 of 45678 items done four and a half minutes
// done four and a half minutes in, the line shows where the phase waits // in, on a terminal 100 columns wide, the line shows where the phase
// whole, beside the phase, progress, elapsed time and ETA. At a large // waits whole, beside the phase, progress, elapsed time and time
// scan's, 1234567 of 9876543 two and a half hours in, those four still // remaining. At a large scan's, 1234567 of 9876543 two and a half hours
// show whole, as the words give way first where they do not fit. // in, on an 80-column terminal, those four still show whole, as the
// words give way first where they do not fit.
// //
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalStalledLines(t *testing.T) { func TestTerminalStalledLines(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
// drawStalled draws the display params describes as it starts and // drawStalled draws the display params describes on a terminal width
// once more as it ends, with count of total items done elapsed into // columns wide as it starts and once more as it ends, with count of
// the phase, and returns what the terminal shows. // total items done elapsed into the phase, and returns what the
drawStalled := func(params progressParams, total, count int64, // terminal shows.
elapsed time.Duration, drawStalled := func(params progressParams, width int, total,
count int64, elapsed time.Duration,
) string { ) string {
before := len(stderr()) before := len(stderr())
clock := &testClock{now: time.Now()} clock := &testClock{now: time.Now()}
params.total = total params.total = total
// No timed draws come between the two. // No timed draws come between the two.
p := startProgress(params, func() int { return 80 }, time.Hour, p := startProgress(params, func() int { return width }, time.Hour,
clock.Now) clock.Now)
clock.Add(elapsed - 42*time.Second) clock.Add(elapsed - 42*time.Second)
@@ -649,19 +653,22 @@ func TestTerminalStalledLines(t *testing.T) {
approx = "~" approx = "~"
} }
shown := drawStalled(params, 45678, 12345, // The percent is in the line, or beside the bar.
shown := drawStalled(params, 100, 45678, 12345,
4*time.Minute+30*time.Second) 4*time.Minute+30*time.Second)
want := params.label + ": [12345/" + approx + "45678] 27%, stalled " + for _, want := range []string{
params.stalledOn + ", elapsed 4m30s, eta 12m9s" params.label + ": [12345/" + approx + "45678]",
", stalled " + params.stalledOn + ", elapsed 4m30s, remain 12m9s",
if !strings.Contains(shown, want) { } {
t.Errorf("terminal shows %q, want %q", shown, want) if !strings.Contains(shown, want) {
t.Errorf("terminal shows %q, want %q", shown, want)
}
} }
shown = drawStalled(params, 9876543, 1234567, 150*time.Minute) shown = drawStalled(params, 80, 9876543, 1234567, 150*time.Minute)
for _, want := range []string{ for _, want := range []string{
params.label + ": [1234567/" + approx + "9876543] 12%, ", params.label + ": [1234567/" + approx + "9876543] 12%, ",
", elapsed 2h30m0s, eta 17h30m0s", ", elapsed 2h30m0s, remain 17h30m0s",
} { } {
if !strings.Contains(shown, want) { if !strings.Contains(shown, want) {
t.Errorf("terminal shows %q, want %q", shown, want) t.Errorf("terminal shows %q, want %q", shown, want)
@@ -706,13 +713,164 @@ func TestTerminalLinesFitWidth(t *testing.T) {
} }
} }
// TestTerminalLinesStartWithSpinner draws a display with no total and
// one with a bar, two and a half hours into the phase, on a terminal 120
// columns wide. Each line must start with the library's spinner in the
// first column, then a space and the phase's label.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalLinesStartWithSpinner(t *testing.T) {
stderr := captureStderr(t)
for _, params := range []progressParams{
{label: scanPhase("walk"), unit: scanUnit, total: -1},
{label: scanPhase("hash"), unit: scanUnit, total: 9876543},
} {
shown := lastFrame(drawLate(stderr, params, 120, 1234567))
spinner, rest, _ := strings.Cut(shown, " ")
if utf8.RuneCountInString(spinner) != 1 ||
!strings.HasPrefix(rest, params.label+": ") {
t.Errorf("terminal shows %q, want the spinner, a space and %q",
shown, params.label)
}
if params.total > 0 && !strings.Contains(shown, "% |") {
t.Errorf("terminal shows %q, want a bar", shown)
}
}
}
// TestTerminalBarStaysAtRightEdge draws the hash phase's display two
// and a half hours in, at two counts whose lines differ in length, on
// terminals 110 and 120 columns wide. The bar is partly filled at the
// first count and full at the second. It must start the same number of
// columns from the terminal's right edge each time, and end the same
// number of columns from it: a line that grows or shrinks, as its time
// remaining does, never moves the bar, and nor do the bar's filled
// cells.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalBarStaysAtRightEdge(t *testing.T) {
stderr := captureStderr(t)
params := progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
}
widths := []int{110, 120}
counts := []int64{1234567, 9876543}
startFromEdge := make([]int, 0, len(widths)*len(counts))
endFromEdge := make([]int, 0, len(widths)*len(counts))
for _, width := range widths {
for _, count := range counts {
shown := lastFrame(drawLate(stderr, params, width, count))
beforeBar, _, found := strings.Cut(shown, "% |")
if !found {
t.Fatalf("terminal %d wide shows %q, want a bar", width, shown)
}
// The bar ends with the last "|".
throughBar := shown[:strings.LastIndex(shown, "|")+1]
startFromEdge = append(startFromEdge,
width-utf8.RuneCountInString(beforeBar))
endFromEdge = append(endFromEdge,
width-utf8.RuneCountInString(throughBar))
}
}
if slices.Min(startFromEdge) != slices.Max(startFromEdge) {
t.Errorf("bar starts %v columns from the right edge, want the same "+
"every time", startFromEdge)
}
if slices.Min(endFromEdge) != slices.Max(endFromEdge) {
t.Errorf("bar ends %v columns from the right edge, want the same "+
"every time", endFromEdge)
}
}
// TestTerminalRateGivesWayBeforeBar draws the hash phase's display two
// and a half hours in. On a terminal 90 columns wide the line fits
// beside the bar without its rate, but not with it: the bar is drawn
// and the rate is not. On a terminal 85 columns wide the line does not
// fit beside the bar even without its rate: neither is drawn, though
// the rate would fit where the bar was.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalRateGivesWayBeforeBar(t *testing.T) {
stderr := captureStderr(t)
params := progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
}
shown := lastFrame(drawLate(stderr, params, 90, 1234567))
if !strings.Contains(shown, "% |") || strings.Contains(shown, "files/s") {
t.Errorf("terminal 90 wide shows %q, want the bar and no rate", shown)
}
shown = lastFrame(drawLate(stderr, params, 85, 1234567))
if strings.Contains(shown, "% |") || strings.Contains(shown, "files/s") {
t.Errorf("terminal 85 wide shows %q, want neither bar nor rate", shown)
}
}
// TestTerminalSpinnerTurnsBesideBar draws the hash phase's display, its
// bar partly filled, several times on a terminal 120 columns wide. The
// spinner must turn at every draw. A filled cell of the bar takes one
// column but three bytes. A line measured in bytes would seem too wide
// for the terminal, and every draw would start the display over, its
// spinner included.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalSpinnerTurnsBesideBar(t *testing.T) {
stderr := captureStderr(t)
params := progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543,
}
// No timed draws: the test draws.
p := startProgress(params, func() int { return 120 }, time.Hour,
time.Now)
p.mu.Lock()
p.count = 1234567
p.mu.Unlock()
const draws = 4
spinners := make([]string, 0, draws)
for range draws {
before := len(stderr())
p.draw()
spinner, _, _ := strings.Cut(lastFrame(stderr()[before:]), " ")
spinners = append(spinners, spinner)
}
p.finish()
for i := 1; i < len(spinners); i++ {
if spinners[i] == spinners[i-1] {
t.Errorf("spinners at successive draws are %q, want each to "+
"differ from the one before", spinners)
break
}
}
}
// TestTerminalNarrowedRedrawsInPlace draws a large scan's display, two // TestTerminalNarrowedRedrawsInPlace draws a large scan's display, two
// and a half hours into the phase, on a terminal 120 columns wide, then // and a half hours into the phase, on a terminal 120 columns wide, then
// makes the terminal narrower than that line and draws it again a few // makes the terminal narrower than that line and draws it again a few
// times. Nothing written after that, the erase before each redraw // times. Nothing written after that, the erase before each redraw
// included, may reach the narrower terminal's last column: it would // included, may reach the narrower terminal's last column: it would
// wrap, and every redraw would leave a line behind. At these widths the // wrap, and every redraw would leave a line behind. At the narrower
// spinner stays a spinner and the bar gives way to the spinner. // width the hash phase's bar gives way.
// //
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalNarrowedRedrawsInPlace(t *testing.T) { func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
@@ -730,7 +888,7 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
progressParams{ progressParams{
label: scanPhase("hash"), unit: scanUnit, total: 9876543, label: scanPhase("hash"), unit: scanUnit, total: 9876543,
}, },
90, 80,
}, },
} }
@@ -770,10 +928,10 @@ func TestTerminalNarrowedRedrawsInPlace(t *testing.T) {
} }
} }
// TestProgressWarningsOnOwnLines drives the terminal displays, spinner // TestProgressWarningsOnOwnLines drives the terminal displays, with a
// and bar, through a run of warnings with no items between them, as when // bar and without, through a run of warnings with no items between
// the walk meets many unreadable paths, for many redraws: every warning // them, as when the walk meets many unreadable paths, for many redraws:
// must land on a line of its own, never inside a redraw. // every warning must land on a line of its own, never inside a redraw.
// //
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressWarningsOnOwnLines(t *testing.T) { func TestProgressWarningsOnOwnLines(t *testing.T) {
@@ -819,11 +977,12 @@ func TestProgressWarningsOnOwnLines(t *testing.T) {
} }
} }
// TestProgressWriterRowsOnOwnLines drives the terminal displays, // TestProgressWriterRowsOnOwnLines drives the terminal displays, with a
// spinner and bar, through a run of rows written through the display's // bar and without, through a run of rows written through the display's
// writer to the same terminal, as stdout is in a plain "sfdupes trees", // writer to the same terminal, as stdout is in a plain "sfdupes trees",
// each written out at once, for many redraws: the terminal must show // each written out at once, for many redraws: the terminal must show
// every row whole, in order, on a line of its own. // every row whole, in order, on a line of its own. The terminal is 120
// columns wide, so the write phase's display has room for its bar.
// //
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestProgressWriterRowsOnOwnLines(t *testing.T) { func TestProgressWriterRowsOnOwnLines(t *testing.T) {
@@ -840,7 +999,8 @@ func TestProgressWriterRowsOnOwnLines(t *testing.T) {
var want []string var want []string
for _, params := range cases { for _, params := range cases {
p := startProgress(params, terminalWidth, testRedraw, time.Now) p := startProgress(params, func() int { return 120 }, testRedraw,
time.Now)
out := p.writer(os.Stderr) out := p.writer(os.Stderr)
// No pause between rows: one is garbled only if a redraw lands // No pause between rows: one is garbled only if a redraw lands
+2 -2
View File
@@ -187,7 +187,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
// reportOpenLine is the plain line of report's open phase when it opens // reportOpenLine is the plain line of report's open phase when it opens
// the database at once; it prints it as the phase starts and as it ends. // the database at once; it prints it as the phase starts and as it ends.
const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " + const reportOpenLine = "[1/2] open: opening the database, elapsed 0s, " +
"eta unknown" "remain unknown"
// smokeReport is report's stdout for the smoke-test records. // smokeReport is report's stdout for the smoke-test records.
const smokeReport = "first\tdupe\tsize\n" + const smokeReport = "first\tdupe\tsize\n" +
@@ -225,7 +225,7 @@ func TestRunReportProgressNamesEachPhase(t *testing.T) {
"report: database " + path + " (from " + databaseEnv + ")", "report: database " + path + " (from " + databaseEnv + ")",
reportOpenLine, reportOpenLine,
reportOpenLine, reportOpenLine,
"[2/2] read: counting the records, elapsed 0s, eta unknown", "[2/2] read: counting the records, elapsed 0s, remain unknown",
"[2/2] read: [6/6] 100%, ", "[2/2] read: [6/6] 100%, ",
"report: 6 records read, 2 duplicate groups, 4 dupe files, " + "report: 6 records read, 2 duplicate groups, 4 dupe files, " +
"6.1 KiB reclaimable", "6.1 KiB reclaimable",
+83 -28
View File
@@ -383,7 +383,8 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
) error { ) error {
// Types and filesystems are checked before pruning so that an // Types and filesystems are checked before pruning so that an
// operand under a dropped one is still scanned, not dropped as lying // operand under a dropped one is still scanned, not dropped as lying
// under it. // under it, and so that an operand inside a .zfs directory gets its
// warning even when it lies under another operand.
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules, roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
deviceOfPath) deviceOfPath)
@@ -411,14 +412,15 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
} }
// walkableRoots returns the operands the walk can start from: regular // walkableRoots returns the operands the walk can start from: regular
// files, and directories not named .zfs. Every other operand is warned // files, and directories not named .zfs, none of them inside a
// about, counted as skipped, and dropped. A dropped operand is no // directory named .zfs. Every other operand is warned about, counted as
// longer a root, so the records stored beneath it are treated as if it // skipped, and dropped. A dropped operand is no longer a root, so the
// had not been given: those under no other root count as outside the // records stored beneath it are treated as if it had not been given:
// roots and are not deleted as unverified. An operand on a filesystem // those under no other root count as outside the roots and are not
// in rules that is not persistent is dropped the same way, with a line // deleted as unverified. An operand on a filesystem in rules that is not
// saying so that is not a warning, and is not counted as skipped. An // persistent is dropped the same way, with a line saying so that is not
// operand that fails lstat here is kept, and the walk warns about it. // a warning, and is not counted as skipped. An operand that fails lstat
// here is kept, and the walk warns about it unless it no longer exists.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
kept := make([]string, 0, len(roots)) kept := make([]string, 0, len(roots))
@@ -629,8 +631,9 @@ func isDatabaseFile(f fileRec, rules walkRules) bool {
// (an interrupted scan resumes cheaply: the next run skips everything // (an interrupted scan resumes cheaply: the next run skips everything
// already recorded). A run that fails to hash is warned about and // already recorded). A run that fails to hash is warned about and
// counted as skipped; stale records for its paths, if any, are deleted // counted as skipped; stale records for its paths, if any, are deleted
// by the update phase. The total counts actual reads, so the display // by the update phase. A run whose file no longer exists gets no record
// shows a real ETA. // and no warning, and is not counted as skipped. The total counts actual
// reads, so the display's time remaining is meaningful.
// //
// Returning early — a failed database write, or a cancelled scan — must // Returning early — a failed database write, or a cancelled scan — must
// not strand the pool: the feeder would park forever on a full jobs // not strand the pool: the feeder would park forever on a full jobs
@@ -662,6 +665,11 @@ func (s *scanState) hashPhase(ctx context.Context, workers int) error {
prog.increment() prog.increment()
// A file removed since the walk is no error: it gets no record.
if errors.Is(r.err, fs.ErrNotExist) {
continue
}
if r.err != nil { if r.err != nil {
s.st.skipped += len(r.run) s.st.skipped += len(r.run)
@@ -829,10 +837,12 @@ type walkEvent struct {
// startWalk seeds every root into the shared walk worker pool and // startWalk seeds every root into the shared walk worker pool and
// returns the event stream: one record per regular file, one warning // returns the event stream: one record per regular file, one warning
// event per per-path error, and one event per path -x or rules kept the // event per per-path error, and one event per path -x or rules kept the
// walk from entering or recording. The channel is closed when the walk // walk from entering or recording. A path that no longer exists when
// completes, and also when ctx is cancelled — every goroutine in the // the walk reaches it is no error, and gets no event. The channel is
// pool abandons its blocking send in that case, so the consumer sees a // closed when the walk completes, and also when ctx is cancelled —
// truncated but properly terminated stream instead of a stalled one. // every goroutine in the pool abandons its blocking send in that case,
// so the consumer sees a truncated but properly terminated stream
// instead of a stalled one.
func startWalk(ctx context.Context, roots []string, oneFS bool, func startWalk(ctx context.Context, roots []string, oneFS bool,
rules walkRules, workers int, rules walkRules, workers int,
) <-chan walkEvent { ) <-chan walkEvent {
@@ -865,9 +875,24 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
// operandWarning returns the one-line warning for an operand the walk // operandWarning returns the one-line warning for an operand the walk
// does not start from, naming the path and what it is, or "" for one it // does not start from, naming the path and what it is, or "" for one it
// does: a regular file, or a directory not named .zfs. Symlinks are // does: a regular file, or a directory not named .zfs, that is not
// never followed, including as operands. // inside a directory named .zfs, neither as given nor with its symlinks
// resolved. Symlinks are never followed, including as operands.
func operandWarning(root string, fi fs.FileInfo) string { func operandWarning(root string, fi fs.FileInfo) string {
// The walk never enters a .zfs directory, so it starts from nothing
// inside one either, not even through a symlink. An operand whose
// symlinks cannot be resolved is checked as given.
resolved, err := filepath.EvalSymlinks(root)
if err != nil {
resolved = root
}
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
root)
}
var kind string var kind string
switch mode := fi.Mode(); { switch mode := fi.Mode(); {
@@ -899,13 +924,19 @@ func operandWarning(root string, fi fs.FileInfo) string {
// one of the scan's own database files in rules, and a directory // one of the scan's own database files in rules, and a directory
// operand becomes an initial job. walkableRoots has already dropped // operand becomes an initial job. walkableRoots has already dropped
// every other operand. One that has changed into something else since // every other operand. One that has changed into something else since
// is warned about and skipped here; it is still a root, so the records // is warned about and skipped here, and one removed since is skipped
// stored beneath it, other than those under another root, are deleted // without a warning. Either is still a root, so the records stored
// as unverified. // beneath it, other than those under another root, are deleted as
// unverified.
func seedRoot(ctx context.Context, root string, rules walkRules, func seedRoot(ctx context.Context, root string, rules walkRules,
events chan<- walkEvent, events chan<- walkEvent,
) []dirJob { ) []dirJob {
fi, err := os.Lstat(root) fi, err := os.Lstat(root)
if errors.Is(err, fs.ErrNotExist) {
// Removed since the scan started: no error, and no files.
return nil
}
if err != nil { if err != nil {
sendEvent(ctx, events, walkEvent{ sendEvent(ctx, events, walkEvent{
warn: fmt.Sprintf("walk %s: %v", root, err), warn: fmt.Sprintf("walk %s: %v", root, err),
@@ -1023,11 +1054,17 @@ func dispatchDirs(ctx context.Context, initial []dirJob,
// walkOneDir reads one directory, emitting an event per regular-file // walkOneDir reads one directory, emitting an event per regular-file
// entry and a warning event per unreadable one, and returns the // entry and a warning event per unreadable one, and returns the
// subdirectories to descend into. // subdirectories to descend into. A directory or entry removed since
// the walk found it gets no event.
func walkOneDir(ctx context.Context, job dirJob, oneFS bool, func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
rules walkRules, events chan<- walkEvent, rules walkRules, events chan<- walkEvent,
) []dirJob { ) []dirJob {
entries, err := os.ReadDir(job.path) entries, err := os.ReadDir(job.path)
if errors.Is(err, fs.ErrNotExist) {
// Removed since its parent was read: no error, and no files.
return nil
}
if err != nil { if err != nil {
sendEvent(ctx, events, walkEvent{ sendEvent(ctx, events, walkEvent{
warn: fmt.Sprintf("walk %s: %v", job.path, err), warn: fmt.Sprintf("walk %s: %v", job.path, err),
@@ -1070,16 +1107,22 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
// emitFile stats one regular directory entry and emits its record. // emitFile stats one regular directory entry and emits its record.
// The lstat happens here in the walk worker, while the directory's // The lstat happens here in the walk worker, while the directory's
// metadata is still hot; a path that fails to stat (or stops being a // metadata is still hot; a path that fails to stat between the
// regular file) between the directory read and the lstat is warned // directory read and the lstat is warned about and skipped, unless it
// about and skipped. A file on a filesystem in rules that is not // no longer exists, and one that has stopped being a regular file is
// persistent, as through a bind mount of one file, is not recorded: it // skipped. A file on a filesystem in rules that is not persistent, as
// gets an event naming it instead. One of the scan's own database files // through a bind mount of one file, is not recorded: it gets an event
// in rules gets no event at all. // naming it instead. One of the scan's own database files in rules gets
// no event at all.
func emitFile(ctx context.Context, p string, e fs.DirEntry, func emitFile(ctx context.Context, p string, e fs.DirEntry,
rules walkRules, events chan<- walkEvent, rules walkRules, events chan<- walkEvent,
) { ) {
info, err := e.Info() info, err := e.Info()
if errors.Is(err, fs.ErrNotExist) {
// Removed since its directory was read: no error.
return
}
if err != nil { if err != nil {
sendEvent(ctx, events, walkEvent{ sendEvent(ctx, events, walkEvent{
warn: fmt.Sprintf("stat %s: %v", p, err), warn: fmt.Sprintf("stat %s: %v", p, err),
@@ -1131,6 +1174,11 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
} }
info, err := e.Info() info, err := e.Info()
if errors.Is(err, fs.ErrNotExist) {
// Removed since its parent was read: no error, and no files.
return dirJob{}, false
}
if err != nil { if err != nil {
sendEvent(ctx, events, walkEvent{ sendEvent(ctx, events, walkEvent{
warn: fmt.Sprintf("walk %s: %v", p, err), warn: fmt.Sprintf("walk %s: %v", p, err),
@@ -1195,7 +1243,7 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
// hashResult carries the hashes computed for one inode run (or the // hashResult carries the hashes computed for one inode run (or the
// error that prevented computing them) from the pool's workers to the // error that prevented computing them) from the pool's workers to the
// hash phase. // hash phase. Its run leaves out the paths found removed.
type hashResult struct { type hashResult struct {
run []fileRec run []fileRec
head [sha256.Size]byte head [sha256.Size]byte
@@ -1296,6 +1344,13 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
head, tail, content, err := hash(run[0].path, run[0].size) head, tail, content, err := hash(run[0].path, run[0].size)
// A path removed since the walk is dropped from its run. The
// next path is a hard link to the same file, and is read instead.
for errors.Is(err, fs.ErrNotExist) && len(run) > 1 {
run = run[1:]
head, tail, content, err = hash(run[0].path, run[0].size)
}
select { select {
case results <- hashResult{ case results <- hashResult{
run: run, head: head, tail: tail, content: content, err: err, run: run, head: head, tail: tail, content: content, err: err,
+158 -6
View File
@@ -683,6 +683,108 @@ func TestSubdirJobStatError(t *testing.T) {
} }
} }
// TestWalkPassesOverRemovedPaths removes a file and a directory after
// their parent was read, as when they are deleted while a scan runs.
// The walk then reaches them, the directory also as one to read and as
// an operand. None of that is an error: the walk sends no event.
func TestWalkPassesOverRemovedPaths(t *testing.T) {
t.Parallel()
dir := t.TempDir()
file := writeFile(t, dir, "file", []byte("x"))
writeFile(t, dir, "sub/f", []byte("y"))
sub := filepath.Join(dir, "sub")
fileEntry := dirEntryFor(t, dir, "file")
subEntry := dirEntryFor(t, dir, "sub")
for _, p := range []string{file, sub} {
err := os.RemoveAll(p)
if err != nil {
t.Fatal(err)
}
}
ctx := t.Context()
events := make(chan walkEvent, 4)
emitFile(ctx, file, fileEntry, walkRules{}, events)
_, descend := subdirJob(ctx, sub, subEntry, dirJob{path: dir}, false,
walkRules{}, events)
subs := walkOneDir(ctx, dirJob{path: sub}, false, walkRules{}, events)
jobs := seedRoot(ctx, sub, walkRules{}, events)
close(events)
for ev := range events {
t.Errorf("event %+v for a removed path, want none", ev)
}
if descend || len(subs) != 0 || len(jobs) != 0 {
t.Errorf("descend = %t, subdirectories %+v, jobs %+v, want none",
descend, subs, jobs)
}
}
// TestScanWarnsAboutUnreadablePaths checks that a permission error is
// still warned about and counted as skipped, unlike a removed path. The
// walk reaches an unreadable directory, and a file in a directory that
// can be read but not searched. Another such file is an operand.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
if os.Geteuid() == 0 {
t.Skip("root ignores file permissions")
}
stderr := captureStderr(t)
dir := t.TempDir()
tree := filepath.Join(dir, "tree")
walked := writeFile(t, tree, "unsearchable/f", []byte("x"))
operand := writeFile(t, dir, "unsearchable/f", []byte("y"))
unreadable := filepath.Join(tree, "unreadable")
err := os.Mkdir(unreadable, 0o750)
if err != nil {
t.Fatal(err)
}
modes := map[string]fs.FileMode{
unreadable: 0,
filepath.Dir(walked): 0o400,
filepath.Dir(operand): 0o400,
}
for p, mode := range modes {
err = os.Chmod(p, mode)
if err != nil {
t.Fatal(err)
}
// Runs before t.TempDir's own cleanup, which must delete the files.
t.Cleanup(func() {
//nolint:gosec // removing the files needs the search bit back
_ = os.Chmod(p, 0o700)
})
}
st := syncTree(t, openTestDB(t), tree, operand)
if st != (scanStats{skipped: 3}) {
t.Errorf("stats = %+v, want 3 skipped and nothing else", st)
}
got := stderr()
for _, warning := range []string{
"walk " + unreadable + ": open " + unreadable + ": permission denied\n",
"stat " + walked + ": lstat " + walked + ": permission denied\n",
"walk " + operand + ": lstat " + operand + ": permission denied\n",
} {
if !strings.Contains(got, warning) {
t.Errorf("stderr = %q, want %q", got, warning)
}
}
}
// syncOneFilesystem scans roots into db with -x and returns the scan // syncOneFilesystem scans roots into db with -x and returns the scan
// stats. Every device pruneRoots looks up, and the device each // stats. Every device pruneRoots looks up, and the device each
// operand's walk compares its subdirectories with, comes from devices // operand's walk compares its subdirectories with, comes from devices
@@ -962,8 +1064,9 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
} }
} }
// buildSmokeTree recreates the README smoke-test filesystem layout // buildSmokeTree recreates the tree of the test under README
// with deterministic content and returns the tree root. // "Definition of done" with deterministic content and returns the tree
// root.
func buildSmokeTree(t *testing.T) string { func buildSmokeTree(t *testing.T) string {
t.Helper() t.Helper()
@@ -1128,7 +1231,7 @@ func TestScanPipeline(t *testing.T) {
// database's records. Each phase's first line names it with its place // database's records. Each phase's first line names it with its place
// among the four: load counts the records before reading them, and the // among the four: load counts the records before reading them, and the
// walk estimates its total from the records under its operand, or says // walk estimates its total from the records under its operand, or says
// its ETA is unknown when there are none. // its time remaining is unknown when there are none.
// //
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr //nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestScanProgressNamesEachPhase(t *testing.T) { func TestScanProgressNamesEachPhase(t *testing.T) {
@@ -1143,14 +1246,14 @@ func TestScanProgressNamesEachPhase(t *testing.T) {
syncTree(t, db, filepath.Join(dir, "a")) syncTree(t, db, filepath.Join(dir, "a"))
rescan := stderr()[len(first):] rescan := stderr()[len(first):]
load := "[1/4] load: counting the records, elapsed 0s, eta unknown" load := "[1/4] load: counting the records, elapsed 0s, remain unknown"
cases := map[string][]string{ cases := map[string][]string{
first: { first: {
load, "[2/4] walk: 0 files, 0 files/s, elapsed 0s, eta unknown", load, "[2/4] walk: 0 files, 0 files/s, elapsed 0s, remain unknown",
"[3/4] hash: [0/", "[4/4] update: [0/", "[3/4] hash: [0/", "[4/4] update: [0/",
}, },
rescan: { rescan: {
load, "[2/4] walk: [0/~2] 0%, 0 files/s, elapsed 0s, eta unknown", load, "[2/4] walk: [0/~2] 0%, 0 files/s, elapsed 0s, remain unknown",
"[3/4] hash: [0/", "[4/4] update: [0/", "[3/4] hash: [0/", "[4/4] update: [0/",
}, },
} }
@@ -1685,6 +1788,55 @@ func TestHashPhaseSkipsFileReplacedByFIFO(t *testing.T) {
} }
} }
// TestHashPhasePassesOverRemovedFiles removes two walked files before
// the hash phase reads them: one with no other path, and the first of
// two hard links to one file. Neither is an error. The phase prints
// nothing but its progress, counts nothing as skipped, and records only
// the hard link still there.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestHashPhasePassesOverRemovedFiles(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
alone := writeFile(t, dir, "alone.bin", pattern(1, 300))
a := writeFile(t, dir, "a.bin", pattern(2, 300))
b := filepath.Join(dir, "b.bin")
err := os.Link(a, b)
if err != nil {
t.Fatal(err)
}
walked, _ := collectWalk(t, []string{dir}, false, 1)
// a.bin sorts before b.bin, so the run of the two reads it first.
for _, p := range []string{alone, a} {
err = os.Remove(p)
if err != nil {
t.Fatal(err)
}
}
s := &scanState{db: openTestDB(t), toHash: walked}
err = s.hashPhase(t.Context(), 1)
if err != nil {
t.Fatalf("hashPhase = %v, want nil", err)
}
if s.st.skipped != 0 || len(s.batch) != 1 || s.batch[0].path != b {
t.Errorf("skipped = %d, records %+v, want none skipped and a "+
"record for %s alone", s.st.skipped, s.batch, b)
}
for line := range strings.Lines(stderr()) {
if !strings.HasPrefix(line, scanPhase("hash")+": ") {
t.Errorf("stderr holds %q, want the hash phase's progress alone",
line)
}
}
}
// injectedWriteFailure is the message the injected database trigger // injectedWriteFailure is the message the injected database trigger
// aborts with, so the test can recognize its own failure in the error // aborts with, so the test can recognize its own failure in the error
// the scan reports. // the scan reports.
+7 -6
View File
@@ -13,10 +13,11 @@ import (
// treesOpenLine is the plain line of trees' open phase when it opens // treesOpenLine is the plain line of trees' open phase when it opens
// the database at once; it prints it as the phase starts and as it ends. // the database at once; it prints it as the phase starts and as it ends.
const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " + const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
"eta unknown" "remain unknown"
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and // smokeTreeRecs mirrors the tree of the test under README "Definition
// /d/t2 are identical, /d/t3 differs from them only by one filename. // of done": /d/t1 and /d/t2 are identical, /d/t3 differs from them only
// by one filename.
func smokeTreeRecs() []scanRec { func smokeTreeRecs() []scanRec {
f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c") f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c") f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")
@@ -236,11 +237,11 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
"trees: database " + path + " (from " + databaseEnv + ")", "trees: database " + path + " (from " + databaseEnv + ")",
treesOpenLine, treesOpenLine,
treesOpenLine, treesOpenLine,
"[2/4] read: counting the records, elapsed 0s, eta unknown", "[2/4] read: counting the records, elapsed 0s, remain unknown",
"[2/4] read: [6/6] 100%, ", "[2/4] read: [6/6] 100%, ",
"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, eta unknown", "[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, remain unknown",
"[3/4] group: [8/8] 100%, ", "[3/4] group: [8/8] 100%, ",
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, eta unknown", "[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown",
"[4/4] write: [1/1] 100%, ", "[4/4] write: [1/1] 100%, ",
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ", "trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
} }