Show the walk's path on the line below its display (closes #160)

On a terminal, each draw of the walk shows the path of the last file
it found on the line below, cut short before the last column with
wide characters counted as two columns. The progressbar library's
detail row draws it. Control characters in the path are written out,
as \t or \x1b, so that the terminal does not act on them. The line is
erased before a warning and when the walk ends, so the next phase is
drawn on it. Nothing changes when stderr is not a terminal.

github.com/rivo/uniseg, already in the module graph through the
progressbar library, becomes a direct dependency for the column count.

Model: opus-5-5

README.md

@@ -128,8 +128,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
     - one line naming the database;
     - progress lines, as described under "Progress";
     - a warning for each path it skips;
-    - a line for each system, temporary or cache directory the walk leaves out
+    - a line for each system or temporary directory the walk leaves out (see
-      (see "Rules for the walk");
+      "Rules for the walk");
     - a line for each path left out because its filesystem is one the kernel,
       the core of the operating system, makes up or keeps in memory, such as
       `tmpfs`;
@@ -300,9 +300,8 @@ The reference below is also the program's specification.
   of directories on other filesystems, as the same flag does for `du` and
   `rsync`. By default the walk also enters directories on other filesystems.
 - `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
-  excludes. Those are the system, temporary and cache directories the walk
+  excludes. Those are the system and temporary directories the walk otherwise
-  otherwise leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules
+  leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
-  for the walk").
 - `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
   below it. `PATH` may be a directory or a file. The flag may be given more than
   once. The walk knows `PATH` by its device ID and inode number, as it knows the
@@ -572,17 +571,12 @@ walk practical on large or busy storage pools.
     `--no-default-excludes` does not change this rule.
 
 - Below an operand, the walk does not enter a default exclude. On Linux, the
-  default excludes hold the kernel's view of the running system, temporary
+  default excludes hold temporary files and the kernel's view of the running
-  files, and caches. A cache here is data that programs and package managers
+  system. They depend on the system, and `sfdupes scan --help` lists those of
-  download or build again on their own. The default excludes depend on the
+  the system it runs on. On Linux they are:
-  system, and `sfdupes scan --help` lists those of the system it runs on. In the
-  lists, `*` stands for any one directory name, so `/home/*/.cache` matches
-  `/home/alice/.cache`. On Linux they are:
 
     ```
-    /proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache
+    /proc, /sys, /dev, /run, /tmp, /var/tmp
-    /root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
-    /home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
     ```
 
     On macOS they are:
@@ -590,21 +584,13 @@ walk practical on large or busy storage pools.
     ```
     /dev, /private/tmp, /private/var/tmp, /private/var/vm
     /System/Volumes/VM, /System/Volumes/Data
-    /private/var/folders, /Library/Caches
-    /private/var/root/Library/Caches, /private/var/root/.cache
-    /private/var/root/.npm, /private/var/root/.cargo/registry
-    /private/var/root/go/pkg/mod
-    /Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
-    /Users/*/.cargo/registry, /Users/*/go/pkg/mod
     ```
 
-    `/private/var/folders` holds each user's temporary files and caches on
+    A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already,
-    macOS. Root's home on macOS is `/var/root`, which the walk reaches as
+    through firmlinks. Firmlinks are the links macOS uses to show the data
-    `/private/var/root`. A walk of `/` on macOS reaches everything in
+    volume inside the system volume. Walking `/System/Volumes/Data` as well
-    `/System/Volumes/Data` already, through firmlinks. Firmlinks are the links
+    would record each of those files twice. Any other system has no default
-    macOS uses to show the data volume inside the system volume. Walking
+    excludes.
-    `/System/Volumes/Data` as well would record each of those files twice. Any
-    other system has no default excludes.
 
     An operand at or inside a default exclude is walked, so
     `sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem.
@@ -1215,6 +1201,14 @@ On a terminal:
     ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s            12% |█         |
     ```
 
+- During `scan`'s `walk`, the line below the display shows the path of the last
+  file the walk found. Each draw shows the path of that moment, cut short before
+  the terminal's last column. The path is escaped as in the reports (see "Report
+  output format"). Each control character left after that is written as `\x` and
+  its code in two hexadecimal digits, such as `\x1b` for the escape character. A
+  control character is one that a terminal acts on instead of showing. The line
+  is erased when the walk ends, and the next phase is drawn on it.
+
 - No line reaches the terminal's last column. A line too wide for the terminal
   first leaves out its rate, the words of a stalled phase, or what the phase is
   waiting for. It leaves out its bar next. It is cut short at the edge only when

go.mod

@@ -3,6 +3,7 @@ module sneak.berlin/go/sfdupes
 go 1.25.7
 
 require (
+	github.com/rivo/uniseg v0.4.7
 	github.com/schollz/progressbar/v3 v3.19.1
 	github.com/spf13/cobra v1.10.2
 	golang.org/x/sys v0.46.0
@@ -18,7 +19,6 @@ require (
 	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
 	github.com/ncruces/go-strftime v1.0.0 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
-	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	modernc.org/libc v1.74.1 // indirect
 	modernc.org/mathutil v1.7.1 // indirect

internal/sfdupes/exclude.go

@@ -72,31 +72,14 @@ type walkRules struct {
 func defaultExcludes(goos string) []string {
 	switch goos {
 	case goosLinux:
-		// The kernel's view of the running system, temporary files, and
+		return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
-		// caches that programs and package managers fill again on their
-		// own: the system's, root's and those in each home in /home.
-		return []string{
-			"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
-			"/root/.cache", "/root/.npm", "/root/.cargo/registry",
-			"/root/go/pkg/mod",
-			"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
-			"/home/*/go/pkg/mod",
-		}
 	case goosDarwin:
 		// A walk of / reaches everything in /System/Volumes/Data through
 		// the firmlinks, so walking it as well would record each of
-		// those files twice. /private/var/folders holds each user's
+		// those files twice.
-		// temporary files and caches. The walk reaches root's home,
-		// /var/root, as /private/var/root.
 		return []string{
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
-			"/private/var/folders", "/Library/Caches",
-			"/private/var/root/Library/Caches", "/private/var/root/.cache",
-			"/private/var/root/.npm", "/private/var/root/.cargo/registry",
-			"/private/var/root/go/pkg/mod",
-			"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
-			"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
 		}
 	case goosWindows:
 		// Relative to the root of the drive an operand lies on. The

internal/sfdupes/exclude_test.go

@@ -24,22 +24,10 @@ func TestDefaultExcludes(t *testing.T) {
 	// on any other system. The Windows list exists in the code only: the
 	// binary does not build for Windows, so the README leaves it out.
 	cases := map[string][]string{
-		goosLinux: {
+		goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
-			"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
-			"/root/.cache", "/root/.npm", "/root/.cargo/registry",
-			"/root/go/pkg/mod",
-			"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
-			"/home/*/go/pkg/mod",
-		},
 		goosDarwin: {
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
-			"/private/var/folders", "/Library/Caches",
-			"/private/var/root/Library/Caches", "/private/var/root/.cache",
-			"/private/var/root/.npm", "/private/var/root/.cargo/registry",
-			"/private/var/root/go/pkg/mod",
-			"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
-			"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
 		},
 		goosWindows: {
 			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
@@ -75,29 +63,6 @@ func TestIsDefaultExclude(t *testing.T) {
 			t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
 		}
 	}
-
-	// The Linux list matches the cache directories in /root and in each
-	// home in /home, but nothing below them and no .cache directory
-	// elsewhere.
-	linux := map[string]bool{
-		"/home/u/.cache":           true,
-		"/home/u/go/pkg/mod":       true,
-		"/root/.cache":             true,
-		"/var/cache":               true,
-		"/home/u/.cache/pip":       false,
-		"/home/u/src/.cache":       false,
-		"/srv/home/u/.cache":       false,
-		"/home/u/.cargo/bin":       false,
-		"/home/u/go/pkg/mod/cache": false,
-	}
-
-	for path, want := range linux {
-		got := isDefaultExclude(path, defaultExcludes(goosLinux))
-		if got != want {
-			t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
-				got, want)
-		}
-	}
 }
 
 func TestReadMountInfo(t *testing.T) {
@@ -325,7 +290,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
 	t.Setenv(databaseEnv, path)
 
 	dir := realTempDir(t)
-	if !insideDefaultExclude(dir, defaultExcludes(runtime.GOOS)) {
+	if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
 		t.Skip(dir + " is not inside a default exclude")
 	}
 
@@ -746,20 +711,6 @@ func shmTempDir(t *testing.T) (string, string, walkRules) {
 	return dir, fsType, walkRules{nonPersistent: nonPersistent}
 }
 
-// insideDefaultExclude reports whether path, which is absolute and
-// clean, or a directory above it is one of excludes.
-func insideDefaultExclude(path string, excludes []string) bool {
-	for dir := path; ; dir = filepath.Dir(dir) {
-		if isDefaultExclude(dir, excludes) {
-			return true
-		}
-
-		if dir == "/" {
-			return false
-		}
-	}
-}
-
 // assertRecorded checks that the database at dbPath holds a record for
 // path when want is set, and none when it is not.
 func assertRecorded(t *testing.T, dbPath, path string, want bool) {

internal/sfdupes/progress.go

@@ -9,8 +9,10 @@ import (
 	"strings"
 	"sync"
 	"time"
+	"unicode"
 	"unicode/utf8"
 
+	"github.com/rivo/uniseg"
 	"github.com/schollz/progressbar/v3"
 	"golang.org/x/term"
 )
@@ -116,7 +118,8 @@ type drawnCount struct {
 // terminal's width. A phase with an exact total ends the line with its
 // percent and bar, at the terminal's right edge, when the terminal has
 // room for them. It is drawn again every redrawInterval, so its elapsed
-// time keeps moving while the phase waits for its next item. When
+// time keeps moving while the phase waits for its next item. Each draw
+// of the walk shows the path it reached last on the line below. When
 // stderr is not a TTY it emits no ANSI redraws: it prints line as the
 // phase starts, every plainInterval while it runs, and as it ends.
 //
@@ -141,6 +144,7 @@ type progress struct {
 	estimate   bool
 	waitingFor string
 	stalledOn  string
+	path       string // the path the walk reached last; "" for none
 	count      int64
 	start      time.Time // when the phase started, for the elapsed time
 	rateStart  time.Time // when the count started, for the time remaining
@@ -239,6 +243,82 @@ func (p *progress) draw() {
 	// less the terminal's last column, which spinnerMargin keeps clear. A
 	// bar's cells take one column each but more than one byte.
 	p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1)
+
+	if p.path != "" {
+		// The library draws the path on the line below and returns to
+		// the phase's line. The path is escaped as warnings are, and its
+		// other control characters are written out, so that the terminal
+		// shows them instead of acting on them. It stays clear of the
+		// last column, so that no terminal wraps it.
+		shown := escapeControlChars(escapePath(p.path))
+		_ = p.bar.AddDetail(cutToColumns(shown, width-1))
+	}
+}
+
+// escapeControlChars returns s with each control character written as
+// \x and its code in two hex digits, as \x1b for the escape character.
+// The control characters are U+0000 to U+001F, U+007F and U+0080 to
+// U+009F, as unicode.IsControl reports them. A byte from 0x80 to 0x9F
+// that is not part of a valid UTF-8 character is written the same way,
+// as \x9b, since a terminal not in UTF-8 mode acts on it. Every other
+// byte is kept as it is.
+func escapeControlChars(s string) string {
+	var b strings.Builder
+
+	for rest := s; rest != ""; {
+		char, size := utf8.DecodeRuneInString(rest)
+		if char == utf8.RuneError && size == 1 {
+			// Not part of a valid UTF-8 character. A terminal not in
+			// UTF-8 mode reads the byte as the character with its code,
+			// so that 0x9b is a control character there and 0xe9 is not.
+			char = rune(rest[0])
+		}
+
+		if unicode.IsControl(char) {
+			fmt.Fprintf(&b, `\x%02x`, char)
+		} else {
+			b.WriteString(rest[:size])
+		}
+
+		rest = rest[size:]
+	}
+
+	return b.String()
+}
+
+// cutToColumns returns s cut to at most columns terminal columns, at a
+// character boundary. A wide character, as in Chinese, takes two
+// columns.
+func cutToColumns(s string, columns int) string {
+	used, end, state := 0, 0, -1
+
+	for rest := s; rest != ""; {
+		char, next, width, nextState := uniseg.FirstGraphemeClusterInString(
+			rest, state)
+		if used+width > columns {
+			break
+		}
+
+		used += width
+		end += len(char)
+		rest, state = next, nextState
+	}
+
+	return s[:end]
+}
+
+// clear erases the display from the terminal: its line, and the path on
+// the line below when one is shown. p.mu must be held.
+func (p *progress) clear() {
+	if p.bar == nil {
+		return
+	}
+
+	if p.path != "" {
+		_ = p.bar.AddDetail("")
+	}
+
+	_ = p.bar.Clear()
 }
 
 // newBar has the library make what draw draws: its spinner, followed by
@@ -263,6 +343,8 @@ func (p *progress) newBar(width int) {
 		// The text carries the elapsed time and the time remaining.
 		progressbar.OptionSetElapsedTime(false),
 		progressbar.OptionSetPredictTime(false),
+		// The line below the phase's line, for the walk's path.
+		progressbar.OptionSetMaxDetailRow(1),
 	)
 	p.widest = 0
 }
@@ -314,6 +396,19 @@ func (p *progress) increment() {
 	p.lastItem = now
 }
 
+// setPath records the path the walk reached last, which a display on a
+// terminal shows at its next draw.
+func (p *progress) setPath(path string) {
+	if p == nil {
+		return
+	}
+
+	p.mu.Lock()
+	defer p.mu.Unlock()
+
+	p.path = path
+}
+
 // warnf prints a one-line warning to stderr without corrupting the
 // display. The whole message is escaped like a report's path columns,
 // so a path holding a newline cannot split the warning.
@@ -327,11 +422,9 @@ func (p *progress) warnf(format string, args ...any) {
 	p.mu.Lock()
 	defer p.mu.Unlock()
 
-	if p.bar != nil {
+	// The warning takes the display's line; the next draw shows the
-		// The warning takes the display's line; the next draw shows
+	// display again, on the line below.
-		// the display again, on the line below.
+	p.clear()
-		_ = p.bar.Clear()
-	}
 
 	fmt.Fprintln(os.Stderr, msg)
 }
@@ -392,9 +485,7 @@ func (w *progressWriter) Flush() error {
 		w.p.mu.Lock()
 		defer w.p.mu.Unlock()
 
-		if w.p.bar != nil {
+		w.p.clear()
-			_ = w.p.bar.Clear()
-		}
 	}
 
 	_, err := w.out.Write(w.buf)
@@ -414,6 +505,13 @@ func (p *progress) finish() {
 	close(p.stop)
 	<-p.done
 
+	// The walk's path is only for someone watching the walk. Its line is
+	// erased, and the next phase is drawn on it.
+	p.mu.Lock()
+	p.clear()
+	p.path = ""
+	p.mu.Unlock()
+
 	p.draw()
 
 	if p.width != nil {

internal/sfdupes/progress_test.go

@@ -6,6 +6,7 @@ import (
 	"math"
 	"os"
 	"path/filepath"
+	"regexp"
 	"slices"
 	"strings"
 	"sync"
@@ -1138,3 +1139,164 @@ func TestWarningAfterLastItemKeepsBar(t *testing.T) {
 		t.Errorf("terminal shows %q, want a count of 2 of 2", shown)
 	}
 }
+
+func TestCutToColumns(t *testing.T) {
+	t.Parallel()
+
+	cases := []struct {
+		s       string
+		columns int
+		want    string
+	}{
+		{"/srv/a", 10, "/srv/a"},
+		{"/srv/photos", 8, "/srv/pho"},
+		// Each of these two fullwidth letters takes two columns.
+		{"/ＡＢ/a", 4, "/Ａ"},
+		{"/ＡＢ/a", 5, "/ＡＢ"},
+		// An accent that combines with the letter before stays with it.
+		{"/café/a", 5, "/café"},
+		{"/srv", 0, ""},
+	}
+
+	for _, tc := range cases {
+		if got := cutToColumns(tc.s, tc.columns); got != tc.want {
+			t.Errorf("cutToColumns(%q, %d) = %q, want %q",
+				tc.s, tc.columns, got, tc.want)
+		}
+	}
+}
+
+func TestEscapeControlChars(t *testing.T) {
+	t.Parallel()
+
+	cases := map[string]string{
+		"/srv/music":    "/srv/music",
+		"/a\x1b[2Jb":    `/a\x1b[2Jb`,
+		"/a\x00\a\x7fb": `/a\x00\x07\x7fb`,
+		// U+009B takes two bytes in UTF-8.
+		"/a\u009b2Jb": `/a\x9b2Jb`,
+		// Byte 0x9b alone is not valid UTF-8. A terminal not in UTF-8
+		// mode acts on it, as on every byte from 0x80 to 0x9f.
+		"/a\x9b2Jb":       `/a\x9b2Jb`,
+		"/a\x80\x9f\xa0b": `/a\x80\x9f` + "\xa0b",
+		// In ě, which is c4 9b in UTF-8, byte 0x9b is part of a valid
+		// character.
+		"/aě2Jb":          "/aě2Jb",
+		"/café/ＡＢ":        "/café/ＡＢ",
+		"/latin1-caf\xe9": "/latin1-caf\xe9",
+	}
+	for in, want := range cases {
+		if got := escapeControlChars(in); got != want {
+			t.Errorf("escapeControlChars(%q) = %q, want %q", in, got, want)
+		}
+	}
+}
+
+// pathLines returns the text of each line the library wrote below a
+// display's line. It writes one as an erase to the end of the line, a
+// carriage return, the text and a newline.
+func pathLines(out string) []string {
+	matches := regexp.MustCompile("\x1b\\[K\r([^\n]*)\n").
+		FindAllStringSubmatch(out, -1)
+	lines := make([]string, 0, len(matches))
+
+	for _, m := range matches {
+		lines = append(lines, m[1])
+	}
+
+	return lines
+}
+
+// TestTerminalWalkShowsPathBelow checks that the walk's display on a
+// terminal 30 columns wide shows the path it reached last on the line
+// below, cut to 29 columns. That line is erased before a warning is
+// printed, and again when the walk ends, so the next phase is drawn on
+// an empty line.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestTerminalWalkShowsPathBelow(t *testing.T) {
+	stderr := captureStderr(t)
+	p := startProgress(progressParams{
+		label: scanPhase("walk"), unit: scanUnit, total: -1,
+	}, func() int { return 30 }, testRedraw, time.Now)
+
+	const shown = "/srv/photos/2026/holiday/beac"
+
+	p.setPath("/srv/photos/2026/holiday/beach.jpg")
+	p.increment()
+	awaitOutput(t, stderr, shown, 1)
+
+	p.warnf("warning")
+
+	before, _, _ := strings.Cut(stderr(), "warning")
+	if lines := pathLines(before); lines[len(lines)-1] != "" {
+		t.Errorf("line below shows %q as the warning is printed, want it "+
+			"empty", lines[len(lines)-1])
+	}
+
+	// Drawn again below the warning.
+	awaitOutput(t, stderr, shown, strings.Count(before, shown)+1)
+	p.finish()
+
+	lines := pathLines(stderr())
+	if !slices.Contains(lines, shown) {
+		t.Errorf("lines below the display = %q, want %q among them",
+			lines, shown)
+	}
+
+	if last := lines[len(lines)-1]; last != "" {
+		t.Errorf("line below the display shows %q after the walk, want it "+
+			"empty", last)
+	}
+}
+
+// TestTerminalWalkEscapesControlChars checks that a walked path holding
+// escape characters, which would clear the screen and set the window
+// title, puts none on the line below the walk's display. The line shows
+// each control character written out, as \x1b.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestTerminalWalkEscapesControlChars(t *testing.T) {
+	stderr := captureStderr(t)
+	p := startProgress(progressParams{
+		label: scanPhase("walk"), unit: scanUnit, total: -1,
+	}, func() int { return 80 }, testRedraw, time.Now)
+
+	p.setPath("/srv/\x1b[2J\x1b]0;title\a/beach.jpg")
+	p.increment()
+	awaitOutput(t, stderr, "beach.jpg", 1)
+	p.finish()
+
+	lines := pathLines(stderr())
+	for _, line := range lines {
+		if strings.Contains(line, "\x1b") {
+			t.Errorf("line below the display = %q, want no escape "+
+				"character", line)
+		}
+	}
+
+	const shown = `/srv/\x1b[2J\x1b]0;title\x07/beach.jpg`
+	if !slices.Contains(lines, shown) {
+		t.Errorf("lines below the display = %q, want %q among them",
+			lines, shown)
+	}
+}
+
+// TestPlainWalkShowsNoPath checks that the walk's display prints no path
+// when stderr is not a terminal.
+//
+//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
+func TestPlainWalkShowsNoPath(t *testing.T) {
+	stderr := captureStderr(t)
+	p := startProgress(progressParams{
+		label: scanPhase("walk"), unit: scanUnit, total: -1,
+	}, nil, testRedraw, time.Now)
+
+	p.setPath("/srv/a")
+	p.increment()
+	p.finish()
+
+	if strings.Contains(stderr(), "/srv/a") {
+		t.Errorf("stderr = %q, want no path", stderr())
+	}
+}

internal/sfdupes/scan.go

@@ -562,6 +562,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
 
 		s.st.walked++
 
+		prog.setPath(ev.rec.path)
 		prog.increment()
 
 		old, ok := s.existing[ev.rec.path]