Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
91dcb19953 |
@@ -113,7 +113,8 @@ Goals, in order:
|
|||||||
`sfdupes`.
|
`sfdupes`.
|
||||||
- Dependencies: standard library, `github.com/spf13/cobra` for the
|
- Dependencies: standard library, `github.com/spf13/cobra` for the
|
||||||
CLI, **one progress-bar library**
|
CLI, **one progress-bar library**
|
||||||
(`github.com/schollz/progressbar/v3`), and **one SQLite driver**
|
(`github.com/schollz/progressbar/v3`), `golang.org/x/term` to tell
|
||||||
|
whether stderr is a terminal, and **one SQLite driver**
|
||||||
(`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
|
(`modernc.org/sqlite`, pure Go, so builds keep cgo disabled).
|
||||||
`github.com/spf13/viper` is permitted if configuration-file support
|
`github.com/spf13/viper` is permitted if configuration-file support
|
||||||
is ever needed, but is not currently used. No other third-party
|
is ever needed, but is not currently used. No other third-party
|
||||||
@@ -263,18 +264,6 @@ duplicates another or lies under another is dropped before walking,
|
|||||||
so every file is reached exactly once and produces one database
|
so every file is reached exactly once and produces one database
|
||||||
record.
|
record.
|
||||||
|
|
||||||
An operand that is a symlink (never followed, not even as an operand),
|
|
||||||
socket, FIFO, or device node, or a directory named `.zfs`, is not
|
|
||||||
scanned. `scan` prints a one-line warning naming the path and what it
|
|
||||||
is, counts it as skipped, and drops it from the scanned operands before
|
|
||||||
reading the database. Another operand beneath it is still scanned. The
|
|
||||||
records stored beneath it are not deleted: they are treated like any
|
|
||||||
other record outside the scanned operands, including the content-phase
|
|
||||||
exception below. If it lies under another operand, they are under that
|
|
||||||
operand instead, and are deleted like any other record there that this
|
|
||||||
scan did not verify. This is not an error: a scan whose every operand
|
|
||||||
is dropped walks nothing and exits 0.
|
|
||||||
|
|
||||||
`scan` synchronizes the database with the filesystem state under the
|
`scan` synchronizes the database with the filesystem state under the
|
||||||
scanned operands:
|
scanned operands:
|
||||||
|
|
||||||
@@ -380,12 +369,9 @@ during the hash phase:
|
|||||||
Rules for the walk:
|
Rules for the walk:
|
||||||
|
|
||||||
- Only regular files. Skip directories, symlinks (do not follow,
|
- Only regular files. Skip directories, symlinks (do not follow,
|
||||||
including symlink operands), sockets, FIFOs, and device nodes. An
|
including symlink operands), sockets, FIFOs, and device nodes.
|
||||||
operand that is a symlink, socket, FIFO, or device node is dropped
|
|
||||||
as described in "`scan` mode" above.
|
|
||||||
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs;
|
- Never descend into a directory named `.zfs` (ZFS snapshot pseudo-dirs;
|
||||||
walking them would list every file once per snapshot), not even
|
walking them would list every file once per snapshot).
|
||||||
when it is an operand; such an operand is dropped the same way.
|
|
||||||
- Filesystem boundaries are crossed by default. With `-x`
|
- Filesystem boundaries are crossed by default. With `-x`
|
||||||
(long form `--one-file-system`, following the GNU `du`/`rsync`
|
(long form `--one-file-system`, following the GNU `du`/`rsync`
|
||||||
convention), never descend into a directory on a different
|
convention), never descend into a directory on a different
|
||||||
@@ -396,11 +382,9 @@ Rules for the walk:
|
|||||||
path, and continue. Per-file errors never abort the run; the final
|
path, and continue. Per-file errors never abort the run; the final
|
||||||
summary reports how many were skipped. As specified above, a
|
summary reports how many were skipped. As specified above, a
|
||||||
skipped path that has a database record from an earlier scan loses
|
skipped path that has a database record from an earlier scan loses
|
||||||
that record, unless it failed only in the content phase, or is an
|
that record, unless it failed only in the content phase; an
|
||||||
operand dropped before the database was read that lies under no
|
unreadable directory subtree likewise loses its records (accepted:
|
||||||
other operand; an unreadable directory subtree likewise loses its
|
the database mirrors what the latest scan could actually verify).
|
||||||
records (accepted: the database mirrors what the latest scan could
|
|
||||||
actually verify).
|
|
||||||
|
|
||||||
Concurrency: the walk phase (which also stats files), the hash phase,
|
Concurrency: the walk phase (which also stats files), the hash phase,
|
||||||
and the content phase each use a worker pool of `--workers` workers
|
and the content phase each use a worker pool of `--workers` workers
|
||||||
@@ -574,10 +558,16 @@ hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
|
|||||||
|
|
||||||
Additional requirements:
|
Additional requirements:
|
||||||
|
|
||||||
- When stderr is not a TTY, do not emit ANSI redraws: print a plain
|
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not
|
||||||
one-line progress update no more often than every 5 seconds instead.
|
emit ANSI redraws: print a plain one-line progress update the moment
|
||||||
|
each phase starts, then no more often than every 5 seconds.
|
||||||
- Progress updates are driven from the main goroutine and must be
|
- Progress updates are driven from the main goroutine and must be
|
||||||
non-blocking with respect to the worker pool.
|
non-blocking with respect to the worker pool. On a terminal the
|
||||||
|
spinner-style displays also redraw on their own several times a
|
||||||
|
second, so their count and elapsed time stay current while a phase
|
||||||
|
waits for its next item.
|
||||||
|
- A warning printed during a phase always lands on a line of its own,
|
||||||
|
never inside the progress display.
|
||||||
- `report` and `trees` modes need no progress display, only their
|
- `report` and `trees` modes need no progress display, only their
|
||||||
stderr summaries.
|
stderr summaries.
|
||||||
|
|
||||||
|
|||||||
@@ -29,9 +29,9 @@
|
|||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
- warn about and skip symlink, socket, FIFO, device and `.zfs`
|
- progress prints at once on a non-terminal, uses a real terminal test,
|
||||||
operands, keeping the records beneath them (2026-10-03,
|
and prints warnings through a spinner instead of racing its redraw
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/9)
|
(2026-10-03, https://git.eeqj.de/sneak/sfdupes/issues/13)
|
||||||
|
|
||||||
- `report` and `trees` open the database read-only, and `scan` leaves it
|
- `report` and `trees` open the database read-only, and `scan` leaves it
|
||||||
out of WAL mode, so reading needs only read access (2026-10-03, closes
|
out of WAL mode, so reading needs only read access (2026-10-03, closes
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ go 1.25.7
|
|||||||
require (
|
require (
|
||||||
github.com/schollz/progressbar/v3 v3.19.1
|
github.com/schollz/progressbar/v3 v3.19.1
|
||||||
github.com/spf13/cobra v1.10.2
|
github.com/spf13/cobra v1.10.2
|
||||||
|
golang.org/x/term v0.44.0
|
||||||
modernc.org/sqlite v1.54.0
|
modernc.org/sqlite v1.54.0
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -19,7 +20,6 @@ require (
|
|||||||
github.com/rivo/uniseg v0.4.7 // indirect
|
github.com/rivo/uniseg v0.4.7 // indirect
|
||||||
github.com/spf13/pflag v1.0.9 // indirect
|
github.com/spf13/pflag v1.0.9 // indirect
|
||||||
golang.org/x/sys v0.46.0 // indirect
|
golang.org/x/sys v0.46.0 // indirect
|
||||||
golang.org/x/term v0.44.0 // indirect
|
|
||||||
modernc.org/libc v1.74.1 // indirect
|
modernc.org/libc v1.74.1 // indirect
|
||||||
modernc.org/mathutil v1.7.1 // indirect
|
modernc.org/mathutil v1.7.1 // indirect
|
||||||
modernc.org/memory v1.11.0 // indirect
|
modernc.org/memory v1.11.0 // indirect
|
||||||
|
|||||||
+9
-124
@@ -89,25 +89,16 @@ func makeReadOnly(t *testing.T, path string) {
|
|||||||
func captureStdout(t *testing.T) func() string {
|
func captureStdout(t *testing.T) func() string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
return captureStream(t, &os.Stdout)
|
f, err := os.Create(filepath.Join(t.TempDir(), "stdout"))
|
||||||
}
|
|
||||||
|
|
||||||
// captureStream redirects *stream (os.Stdout or os.Stderr) to a file
|
|
||||||
// for the rest of the test and returns a function reading back
|
|
||||||
// everything written to it.
|
|
||||||
func captureStream(t *testing.T, stream **os.File) func() string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
f, err := os.Create(filepath.Join(t.TempDir(), "capture"))
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
saved := *stream
|
saved := os.Stdout
|
||||||
*stream = f
|
os.Stdout = f
|
||||||
|
|
||||||
t.Cleanup(func() {
|
t.Cleanup(func() {
|
||||||
*stream = saved
|
os.Stdout = saved
|
||||||
|
|
||||||
_ = f.Close()
|
_ = f.Close()
|
||||||
})
|
})
|
||||||
@@ -367,30 +358,21 @@ func scanFixture(t *testing.T) []string {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
scanOK(t, dir)
|
var stderr bytes.Buffer
|
||||||
|
|
||||||
return dupes
|
|
||||||
}
|
|
||||||
|
|
||||||
// scanOK runs scan over operands, fails the test unless it exits 0 with
|
|
||||||
// nothing on stdout, and returns everything it printed to stderr.
|
|
||||||
func scanOK(t *testing.T, operands ...string) string {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
stdout := captureStdout(t)
|
stdout := captureStdout(t)
|
||||||
stderr := captureStream(t, &os.Stderr)
|
|
||||||
|
|
||||||
code := run(append([]string{cmdScan}, operands...), os.Stderr)
|
code := run([]string{cmdScan, dir}, &stderr)
|
||||||
if code != exitOK {
|
if code != exitOK {
|
||||||
t.Fatalf("run(scan %q) = %d, want %d; stderr: %s",
|
t.Fatalf("run(scan) = %d, want %d; stderr: %s",
|
||||||
operands, code, exitOK, stderr())
|
code, exitOK, stderr.String())
|
||||||
}
|
}
|
||||||
|
|
||||||
if got := stdout(); got != "" {
|
if got := stdout(); got != "" {
|
||||||
t.Errorf("scan stdout = %q, want nothing (data only)", got)
|
t.Errorf("scan stdout = %q, want nothing (data only)", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
return stderr()
|
return dupes
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
||||||
@@ -401,103 +383,6 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
|||||||
assertNoSidecars(t, path)
|
assertNoSidecars(t, path)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunScanSkipsSymlinkOperand(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
|
||||||
|
|
||||||
link := filepath.Join(dir, "link")
|
|
||||||
|
|
||||||
err := os.Symlink(filepath.Join(dir, "target"), link)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Scanning a directory through the symlink stores a record beneath
|
|
||||||
// the symlink's own path for a file beneath its target.
|
|
||||||
scanOK(t, filepath.Join(link, "sub"))
|
|
||||||
|
|
||||||
assertOperandSkipped(t, path, link, "symlink",
|
|
||||||
filepath.Join(link, "sub", "f"))
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
|
||||||
|
|
||||||
link := filepath.Join(dir, "link")
|
|
||||||
|
|
||||||
err := os.Symlink(filepath.Join(dir, "target"), link)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
// link is dropped as a symlink, but link/sub must still be scanned,
|
|
||||||
// not dropped as lying under link.
|
|
||||||
scanOK(t, link, filepath.Join(link, "sub"))
|
|
||||||
|
|
||||||
db, err := openDB(path, reportParams)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
recordByPath(t, dbRecords(t, db), filepath.Join(link, "sub", "f"))
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRunScanSkipsZFSOperand(t *testing.T) {
|
|
||||||
path := testDBPath(t)
|
|
||||||
t.Setenv(databaseEnv, path)
|
|
||||||
|
|
||||||
zfs := filepath.Join(t.TempDir(), ".zfs")
|
|
||||||
snapshot := filepath.Join(zfs, "snapshot", "hourly")
|
|
||||||
f := writeFile(t, snapshot, "f", pattern(1, 10))
|
|
||||||
|
|
||||||
// An operand beneath a .zfs directory is walked, because it is not
|
|
||||||
// itself named .zfs.
|
|
||||||
scanOK(t, snapshot)
|
|
||||||
|
|
||||||
assertOperandSkipped(t, path, zfs, ".zfs directory", f)
|
|
||||||
}
|
|
||||||
|
|
||||||
// assertOperandSkipped scans operand alone and checks that it is skipped
|
|
||||||
// as kind: a warning naming it, one skip in the summary, exit 0, and the
|
|
||||||
// record for kept, which an earlier scan stored beneath operand, still
|
|
||||||
// in the database at dbPath.
|
|
||||||
func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
|
|
||||||
kept string,
|
|
||||||
) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
stderr := scanOK(t, operand)
|
|
||||||
|
|
||||||
warning := "walk " + operand + ": skipping " + kind + " operand\n"
|
|
||||||
if !strings.Contains(stderr, warning) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
|
||||||
}
|
|
||||||
|
|
||||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
|
||||||
"0 unchanged), 1 skipped\n"
|
|
||||||
if !strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
|
||||||
}
|
|
||||||
|
|
||||||
db, err := openDB(dbPath, reportParams)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
recordByPath(t, dbRecords(t, db), kept)
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRunReportSucceeds(t *testing.T) {
|
func TestRunReportSucceeds(t *testing.T) {
|
||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|||||||
+37
-16
@@ -6,6 +6,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"github.com/schollz/progressbar/v3"
|
"github.com/schollz/progressbar/v3"
|
||||||
|
"golang.org/x/term"
|
||||||
)
|
)
|
||||||
|
|
||||||
// plainInterval is the minimum time between progress lines when stderr
|
// plainInterval is the minimum time between progress lines when stderr
|
||||||
@@ -24,24 +25,23 @@ const percentScale = 100
|
|||||||
|
|
||||||
// stderrIsTTY reports whether stderr is attached to a terminal.
|
// stderrIsTTY reports whether stderr is attached to a terminal.
|
||||||
func stderrIsTTY() bool {
|
func stderrIsTTY() bool {
|
||||||
fi, err := os.Stderr.Stat()
|
return term.IsTerminal(int(os.Stderr.Fd()))
|
||||||
if err != nil {
|
|
||||||
return false
|
|
||||||
}
|
|
||||||
|
|
||||||
return fi.Mode()&os.ModeCharDevice != 0
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// progress renders one scan pass's progress on stderr. On a TTY it
|
// progress renders one scan pass's progress on stderr. On a TTY it
|
||||||
// delegates to the progressbar library (spinner style when the total is
|
// delegates to the progressbar library (spinner style when the total is
|
||||||
// unknown, full bar with count/percent/rate/elapsed/ETA otherwise). When
|
// unknown, full bar with count/percent/rate/elapsed/ETA otherwise). When
|
||||||
// stderr is not a TTY it emits no ANSI redraws: it prints a plain
|
// stderr is not a TTY it emits no ANSI redraws: it prints a plain
|
||||||
// one-line update no more often than every plainInterval.
|
// one-line update as the pass starts, then no more often than every
|
||||||
|
// plainInterval.
|
||||||
//
|
//
|
||||||
// All methods must be called from the main goroutine only. A nil
|
// All methods must be called from the main goroutine only. On a TTY
|
||||||
// *progress is a valid no-display receiver: every method is a no-op,
|
// the library also redraws a spinner from its own goroutine, several
|
||||||
// so batched database flushes during the streaming pass can reuse the
|
// times a second, so its count and elapsed time stay current while a
|
||||||
// update-pass helpers without rendering anything.
|
// pass waits for its next item. A nil *progress is a valid
|
||||||
|
// no-display receiver: every method is a no-op, so batched database
|
||||||
|
// flushes during the streaming pass can reuse the update-pass helpers
|
||||||
|
// without rendering anything.
|
||||||
type progress struct {
|
type progress struct {
|
||||||
label string
|
label string
|
||||||
total int64 // -1 when unknown (walk pass)
|
total int64 // -1 when unknown (walk pass)
|
||||||
@@ -53,10 +53,22 @@ type progress struct {
|
|||||||
|
|
||||||
func newProgress(label string, total int64) *progress {
|
func newProgress(label string, total int64) *progress {
|
||||||
p := &progress{label: label, total: total, start: time.Now()}
|
p := &progress{label: label, total: total, start: time.Now()}
|
||||||
if !stderrIsTTY() {
|
if stderrIsTTY() {
|
||||||
|
p.bar = newBar(label, total)
|
||||||
|
|
||||||
return p
|
return p
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Print the zero state at once: the first item may take minutes,
|
||||||
|
// and a pass must never look hung.
|
||||||
|
p.last = p.start
|
||||||
|
fmt.Fprintln(os.Stderr, p.plainLine())
|
||||||
|
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
// newBar builds the TTY display for newProgress.
|
||||||
|
func newBar(label string, total int64) *progressbar.ProgressBar {
|
||||||
opts := []progressbar.Option{
|
opts := []progressbar.Option{
|
||||||
progressbar.OptionSetWriter(os.Stderr),
|
progressbar.OptionSetWriter(os.Stderr),
|
||||||
progressbar.OptionSetDescription(label),
|
progressbar.OptionSetDescription(label),
|
||||||
@@ -81,9 +93,7 @@ func newProgress(label string, total int64) *progress {
|
|||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
p.bar = progressbar.NewOptions64(total, opts...)
|
return progressbar.NewOptions64(total, opts...)
|
||||||
|
|
||||||
return p
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// increment records one completed item and refreshes the display.
|
// increment records one completed item and refreshes the display.
|
||||||
@@ -113,11 +123,22 @@ func (p *progress) warnf(format string, args ...any) {
|
|||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
msg := escapePath(fmt.Sprintf(format, args...))
|
||||||
|
|
||||||
|
if p.bar != nil && p.total < 0 {
|
||||||
|
// The library also redraws a spinner from its own goroutine, so
|
||||||
|
// a direct write could land inside a redraw. The bar prints the
|
||||||
|
// warning itself, just before its next redraw.
|
||||||
|
_, _ = progressbar.Bprintln(p.bar, msg)
|
||||||
|
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
if p.bar != nil {
|
if p.bar != nil {
|
||||||
_ = p.bar.Clear()
|
_ = p.bar.Clear()
|
||||||
}
|
}
|
||||||
|
|
||||||
fmt.Fprintln(os.Stderr, escapePath(fmt.Sprintf(format, args...)))
|
fmt.Fprintln(os.Stderr, msg)
|
||||||
}
|
}
|
||||||
|
|
||||||
// finish terminates the pass's display.
|
// finish terminates the pass's display.
|
||||||
|
|||||||
@@ -0,0 +1,186 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
// spinnerIdle comfortably outlasts the 100ms interval at which the
|
||||||
|
// progressbar library redraws a spinner from its own goroutine.
|
||||||
|
const spinnerIdle = 500 * time.Millisecond
|
||||||
|
|
||||||
|
// captureStderr points os.Stderr at a file for the rest of the test and
|
||||||
|
// returns a function reading back everything written to it.
|
||||||
|
func captureStderr(t *testing.T) func() string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
path := filepath.Join(t.TempDir(), "stderr")
|
||||||
|
|
||||||
|
f, err := os.Create(path) //nolint:gosec // test-controlled path
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
saved := os.Stderr
|
||||||
|
os.Stderr = f
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
os.Stderr = saved
|
||||||
|
|
||||||
|
_ = f.Close()
|
||||||
|
})
|
||||||
|
|
||||||
|
return func() string {
|
||||||
|
b, err := os.ReadFile(path) //nolint:gosec // test-controlled path
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return string(b)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // replaces the process-wide os.Stderr
|
||||||
|
func TestStderrIsTTYFalseForNonTerminals(t *testing.T) {
|
||||||
|
r, pipe, err := os.Pipe()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
regular, err := os.Create(filepath.Join(t.TempDir(), "stderr"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
devNull, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
saved := os.Stderr
|
||||||
|
|
||||||
|
t.Cleanup(func() {
|
||||||
|
os.Stderr = saved
|
||||||
|
|
||||||
|
for _, f := range []*os.File{r, pipe, regular, devNull} {
|
||||||
|
_ = f.Close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
cases := map[string]*os.File{
|
||||||
|
"a pipe": pipe,
|
||||||
|
"a regular file": regular,
|
||||||
|
os.DevNull: devNull,
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, f := range cases {
|
||||||
|
os.Stderr = f
|
||||||
|
|
||||||
|
if stderrIsTTY() {
|
||||||
|
t.Errorf("stderrIsTTY() = true with stderr on %s", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestNewProgressPrintsBeforeFirstItem checks that each pass shows its
|
||||||
|
// zero state the moment it starts when stderr is not a terminal, and
|
||||||
|
// that the next line still waits for plainInterval.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
|
func TestNewProgressPrintsBeforeFirstItem(t *testing.T) {
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
|
newProgress("walk", -1).increment()
|
||||||
|
newProgress("hash", 10).increment()
|
||||||
|
|
||||||
|
want := "walk: 0 files, elapsed 0s\n" +
|
||||||
|
"hash: [0/10] 0% 0 files/s elapsed 0s eta ?\n"
|
||||||
|
if got := stderr(); got != want {
|
||||||
|
t.Errorf("stderr = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// newWalkSpinner returns the walk pass's terminal display, writing to
|
||||||
|
// os.Stderr whether or not it is a terminal, and stops the library's
|
||||||
|
// redraws when the test ends.
|
||||||
|
func newWalkSpinner(t *testing.T) *progress {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
p := &progress{
|
||||||
|
label: "walk", total: -1, start: time.Now(),
|
||||||
|
bar: newBar("walk", -1),
|
||||||
|
}
|
||||||
|
t.Cleanup(p.finish)
|
||||||
|
|
||||||
|
return p
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestProgressWarningsOnOwnLines drives the terminal display of the walk
|
||||||
|
// pass: every warning must land on a line of its own, never inside a
|
||||||
|
// redraw of the spinner.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
|
func TestProgressWarningsOnOwnLines(t *testing.T) {
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
p := newWalkSpinner(t)
|
||||||
|
|
||||||
|
for _, w := range []string{"first", "second", "third"} {
|
||||||
|
p.increment()
|
||||||
|
p.warnf("warning %s", w)
|
||||||
|
}
|
||||||
|
|
||||||
|
p.increment()
|
||||||
|
|
||||||
|
// The spinner prints the warnings at its next redraw.
|
||||||
|
time.Sleep(spinnerIdle)
|
||||||
|
|
||||||
|
// A terminal shows each line as the text after its last carriage
|
||||||
|
// return.
|
||||||
|
var warnings []string
|
||||||
|
|
||||||
|
for line := range strings.SplitSeq(stderr(), "\n") {
|
||||||
|
if strings.Contains(line, "warning") {
|
||||||
|
warnings = append(warnings, line[strings.LastIndex(line, "\r")+1:])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{"warning first", "warning second", "warning third"}
|
||||||
|
if !slices.Equal(warnings, want) {
|
||||||
|
t.Errorf("warning lines = %q, want %q", warnings, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestSpinnerShowsCountAfterBurst checks that once a burst of items
|
||||||
|
// faster than the redraw limit is over, the walk display shows every
|
||||||
|
// item completed while it waits for the next one.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
|
func TestSpinnerShowsCountAfterBurst(t *testing.T) {
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
p := newWalkSpinner(t)
|
||||||
|
|
||||||
|
for range 50 {
|
||||||
|
p.increment()
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(spinnerIdle)
|
||||||
|
|
||||||
|
// A terminal shows the last frame drawn. The library starts each
|
||||||
|
// frame with a carriage return and erases the previous one with
|
||||||
|
// spaces first.
|
||||||
|
var shown string
|
||||||
|
|
||||||
|
for frame := range strings.SplitSeq(stderr(), "\r") {
|
||||||
|
if strings.TrimSpace(frame) != "" {
|
||||||
|
shown = frame
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if !strings.Contains(shown, "(50/-,") {
|
||||||
|
t.Errorf("terminal shows %q, want a count of 50", shown)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -211,17 +211,13 @@ type scanState struct {
|
|||||||
// in the content hash of every record of headTailMin or more whose
|
// in the content hash of every record of headTailMin or more whose
|
||||||
// size, head, and tail match another record's). Records outside the
|
// size, head, and tail match another record's). Records outside the
|
||||||
// roots are never touched, except that the content phase fills in
|
// roots are never touched, except that the content phase fills in
|
||||||
// their content hash. Operands the walk cannot start from are dropped
|
// their content hash.
|
||||||
// first, so the records beneath them count as outside the roots unless
|
|
||||||
// they lie under another root.
|
|
||||||
func syncScan(ctx context.Context, db *sql.DB, roots []string,
|
func syncScan(ctx context.Context, db *sql.DB, roots []string,
|
||||||
workers int, oneFS bool,
|
workers int, oneFS bool,
|
||||||
) (scanStats, error) {
|
) (scanStats, error) {
|
||||||
s := &scanState{db: db}
|
roots = pruneRoots(roots)
|
||||||
|
|
||||||
// Types are checked before pruning so that an operand under a
|
s := &scanState{db: db}
|
||||||
// dropped one is still scanned, not dropped as lying under it.
|
|
||||||
roots = pruneRoots(s.walkableRoots(roots))
|
|
||||||
|
|
||||||
err := s.loadIndex(ctx, roots)
|
err := s.loadIndex(ctx, roots)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -258,35 +254,6 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
|
|||||||
return s.st, s.contentPhase(ctx, workers)
|
return s.st, s.contentPhase(ctx, workers)
|
||||||
}
|
}
|
||||||
|
|
||||||
// walkableRoots returns the operands the walk can start from: regular
|
|
||||||
// files, and directories not named .zfs. Every other operand is warned
|
|
||||||
// about, counted as skipped, and dropped. A dropped operand is no
|
|
||||||
// longer a root, so the records stored beneath it count as outside the
|
|
||||||
// roots and are not deleted as unverified, unless it lies under another
|
|
||||||
// root. An operand that fails lstat here is kept, and the walk warns
|
|
||||||
// about it.
|
|
||||||
func (s *scanState) walkableRoots(roots []string) []string {
|
|
||||||
kept := make([]string, 0, len(roots))
|
|
||||||
|
|
||||||
for _, root := range roots {
|
|
||||||
fi, err := os.Lstat(root)
|
|
||||||
if err == nil {
|
|
||||||
warn := operandWarning(root, fi)
|
|
||||||
if warn != "" {
|
|
||||||
s.st.skipped++
|
|
||||||
|
|
||||||
fmt.Fprintln(os.Stderr, escapePath(warn))
|
|
||||||
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
kept = append(kept, root)
|
|
||||||
}
|
|
||||||
|
|
||||||
return kept
|
|
||||||
}
|
|
||||||
|
|
||||||
// loadIndex indexes the database records under the scan roots for
|
// loadIndex indexes the database records under the scan roots for
|
||||||
// change detection and collects the sizes of every record outside
|
// change detection and collects the sizes of every record outside
|
||||||
// them: out-of-scope records join the size census so a scanned file
|
// them: out-of-scope records join the size census so a scanned file
|
||||||
@@ -823,43 +790,11 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// operandWarning returns the one-line warning for an operand the walk
|
|
||||||
// does not start from, naming the path and what it is, or "" for one it
|
|
||||||
// does: a regular file, or a directory not named .zfs. Symlinks are
|
|
||||||
// never followed, including as operands.
|
|
||||||
func operandWarning(root string, fi fs.FileInfo) string {
|
|
||||||
var kind string
|
|
||||||
|
|
||||||
switch mode := fi.Mode(); {
|
|
||||||
case mode.IsRegular():
|
|
||||||
return ""
|
|
||||||
case mode.IsDir():
|
|
||||||
if filepath.Base(root) != ".zfs" {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
kind = ".zfs directory"
|
|
||||||
case mode&fs.ModeSymlink != 0:
|
|
||||||
kind = "symlink"
|
|
||||||
case mode&fs.ModeSocket != 0:
|
|
||||||
kind = "socket"
|
|
||||||
case mode&fs.ModeNamedPipe != 0:
|
|
||||||
kind = "FIFO"
|
|
||||||
case mode&fs.ModeDevice != 0:
|
|
||||||
kind = "device node"
|
|
||||||
default:
|
|
||||||
kind = "non-regular file"
|
|
||||||
}
|
|
||||||
|
|
||||||
return fmt.Sprintf("walk %s: skipping %s operand", root, kind)
|
|
||||||
}
|
|
||||||
|
|
||||||
// seedRoot turns one PATH operand into the walk's starting state: a
|
// seedRoot turns one PATH operand into the walk's starting state: a
|
||||||
// regular-file operand is statted and emitted directly, and a directory
|
// regular-file operand is statted and emitted directly, a directory
|
||||||
// operand becomes an initial job. walkableRoots has already dropped
|
// operand becomes an initial job, and a symlink or other non-regular
|
||||||
// every other operand. One that has changed into something else since
|
// operand yields nothing (symlinks are never followed, including as
|
||||||
// is warned about and skipped here; it is still a root, so the records
|
// operands).
|
||||||
// stored beneath it are deleted as unverified.
|
|
||||||
func seedRoot(ctx context.Context, root string,
|
func seedRoot(ctx context.Context, root string,
|
||||||
events chan<- walkEvent,
|
events chan<- walkEvent,
|
||||||
) []dirJob {
|
) []dirJob {
|
||||||
@@ -873,19 +808,16 @@ func seedRoot(ctx context.Context, root string,
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
warn := operandWarning(root, fi)
|
switch {
|
||||||
if warn != "" {
|
case fi.IsDir():
|
||||||
sendEvent(ctx, events, walkEvent{warn: warn, fail: true})
|
if filepath.Base(root) == ".zfs" {
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
if fi.IsDir() {
|
|
||||||
dev, ok := deviceOfInfo(fi)
|
dev, ok := deviceOfInfo(fi)
|
||||||
|
|
||||||
return []dirJob{{path: root, rootDev: dev, rootDevOK: ok}}
|
return []dirJob{{path: root, rootDev: dev, rootDevOK: ok}}
|
||||||
}
|
case fi.Mode().IsRegular():
|
||||||
|
|
||||||
dev, ino := inodeOfInfo(fi)
|
dev, ino := inodeOfInfo(fi)
|
||||||
|
|
||||||
sendEvent(ctx, events, walkEvent{rec: fileRec{
|
sendEvent(ctx, events, walkEvent{rec: fileRec{
|
||||||
@@ -897,6 +829,9 @@ func seedRoot(ctx context.Context, root string,
|
|||||||
}})
|
}})
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
|
default:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// startWalkWorkers starts the walk worker pool. Each worker processes
|
// startWalkWorkers starts the walk worker pool. Each worker processes
|
||||||
|
|||||||
+2
-4
@@ -856,11 +856,9 @@ func TestWalkFileAndSymlinkOperands(t *testing.T) {
|
|||||||
t.Fatalf("file operand: recs = %+v, errs = %d", recs, errs)
|
t.Fatalf("file operand: recs = %+v, errs = %d", recs, errs)
|
||||||
}
|
}
|
||||||
|
|
||||||
// A symlink operand that reaches the walk (it became one after
|
// A symlink operand is not followed and yields nothing.
|
||||||
// walkableRoots checked it) is not followed: it yields a warning and
|
|
||||||
// no records.
|
|
||||||
recs, errs = collectWalk(t, []string{link}, false, 2)
|
recs, errs = collectWalk(t, []string{link}, false, 2)
|
||||||
if errs != 1 || len(recs) != 0 {
|
if errs != 0 || len(recs) != 0 {
|
||||||
t.Fatalf("symlink operand: recs = %+v, errs = %d", recs, errs)
|
t.Fatalf("symlink operand: recs = %+v, errs = %d", recs, errs)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user