Leave caches out of the walk by default (closes #181)

The default excludes now cover caches that programs and package
managers fill again on their own. On Linux: /var/cache and, in /root
and in each home in /home, .cache, .npm, .cargo/registry and
go/pkg/mod. On macOS: /private/var/folders, /Library/Caches and, in
/private/var/root and in each home in /Users, Library/Caches plus the
same four. README and scan --help list them; --no-default-excludes
still turns them all off.

Model: opus-5-5

README.md

@@ -128,8 +128,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
     - one line naming the database;
     - progress lines, as described under "Progress";
     - a warning for each path it skips;
-    - a line for each system or temporary directory the walk leaves out (see
-      "Rules for the walk");
+    - a line for each system, temporary or cache directory the walk leaves out
+      (see "Rules for the walk");
     - a line for each path left out because its filesystem is one the kernel,
       the core of the operating system, makes up or keeps in memory, such as
       `tmpfs`;
@@ -300,8 +300,9 @@ The reference below is also the program's specification.
   of directories on other filesystems, as the same flag does for `du` and
   `rsync`. By default the walk also enters directories on other filesystems.
 - `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
-  excludes. Those are the system and temporary directories the walk otherwise
-  leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
+  excludes. Those are the system, temporary and cache directories the walk
+  otherwise leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules
+  for the walk").
 - `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
   below it. `PATH` may be a directory or a file. The flag may be given more than
   once. The walk knows `PATH` by its device ID and inode number, as it knows the
@@ -571,12 +572,17 @@ walk practical on large or busy storage pools.
     `--no-default-excludes` does not change this rule.
 
 - Below an operand, the walk does not enter a default exclude. On Linux, the
-  default excludes hold temporary files and the kernel's view of the running
-  system. They depend on the system, and `sfdupes scan --help` lists those of
-  the system it runs on. On Linux they are:
+  default excludes hold the kernel's view of the running system, temporary
+  files, and caches. A cache here is data that programs and package managers
+  download or build again on their own. The default excludes depend on the
+  system, and `sfdupes scan --help` lists those of the system it runs on. In the
+  lists, `*` stands for any one directory name, so `/home/*/.cache` matches
+  `/home/alice/.cache`. On Linux they are:
 
     ```
-    /proc, /sys, /dev, /run, /tmp, /var/tmp
+    /proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache
+    /root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
+    /home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
     ```
 
     On macOS they are:
@@ -584,13 +590,21 @@ walk practical on large or busy storage pools.
     ```
     /dev, /private/tmp, /private/var/tmp, /private/var/vm
     /System/Volumes/VM, /System/Volumes/Data
+    /private/var/folders, /Library/Caches
+    /private/var/root/Library/Caches, /private/var/root/.cache
+    /private/var/root/.npm, /private/var/root/.cargo/registry
+    /private/var/root/go/pkg/mod
+    /Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
+    /Users/*/.cargo/registry, /Users/*/go/pkg/mod
     ```
 
-    A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already,
-    through firmlinks. Firmlinks are the links macOS uses to show the data
-    volume inside the system volume. Walking `/System/Volumes/Data` as well
-    would record each of those files twice. Any other system has no default
-    excludes.
+    `/private/var/folders` holds each user's temporary files and caches on
+    macOS. Root's home on macOS is `/var/root`, which the walk reaches as
+    `/private/var/root`. A walk of `/` on macOS reaches everything in
+    `/System/Volumes/Data` already, through firmlinks. Firmlinks are the links
+    macOS uses to show the data volume inside the system volume. Walking
+    `/System/Volumes/Data` as well would record each of those files twice. Any
+    other system has no default excludes.
 
     An operand at or inside a default exclude is walked, so
     `sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem.
@@ -1201,14 +1215,6 @@ On a terminal:
     ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s            12% |█         |
     ```
 
-- During `scan`'s `walk`, the line below the display shows the path of the last
-  file the walk found. Each draw shows the path of that moment, cut short before
-  the terminal's last column. The path is escaped as in the reports (see "Report
-  output format"). Each control character left after that is written as `\x` and
-  its code in two hexadecimal digits, such as `\x1b` for the escape character. A
-  control character is one that a terminal acts on instead of showing. The line
-  is erased when the walk ends, and the next phase is drawn on it.
-
 - No line reaches the terminal's last column. A line too wide for the terminal
   first leaves out its rate, the words of a stalled phase, or what the phase is
   waiting for. It leaves out its bar next. It is cut short at the edge only when

go.mod

@@ -3,7 +3,6 @@ module sneak.berlin/go/sfdupes
 go 1.25.7
 
 require (
-	github.com/rivo/uniseg v0.4.7
 	github.com/schollz/progressbar/v3 v3.19.1
 	github.com/spf13/cobra v1.10.2
 	golang.org/x/sys v0.46.0
@@ -19,6 +18,7 @@ require (
 	github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
 	github.com/ncruces/go-strftime v1.0.0 // indirect
 	github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
+	github.com/rivo/uniseg v0.4.7 // indirect
 	github.com/spf13/pflag v1.0.9 // indirect
 	modernc.org/libc v1.74.1 // indirect
 	modernc.org/mathutil v1.7.1 // indirect

internal/sfdupes/exclude.go

@@ -72,14 +72,31 @@ type walkRules struct {
 func defaultExcludes(goos string) []string {
 	switch goos {
 	case goosLinux:
-		return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
+		// The kernel's view of the running system, temporary files, and
+		// caches that programs and package managers fill again on their
+		// own: the system's, root's and those in each home in /home.
+		return []string{
+			"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
+			"/root/.cache", "/root/.npm", "/root/.cargo/registry",
+			"/root/go/pkg/mod",
+			"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
+			"/home/*/go/pkg/mod",
+		}
 	case goosDarwin:
 		// A walk of / reaches everything in /System/Volumes/Data through
 		// the firmlinks, so walking it as well would record each of
-		// those files twice.
+		// those files twice. /private/var/folders holds each user's
+		// temporary files and caches. The walk reaches root's home,
+		// /var/root, as /private/var/root.
 		return []string{
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
+			"/private/var/folders", "/Library/Caches",
+			"/private/var/root/Library/Caches", "/private/var/root/.cache",
+			"/private/var/root/.npm", "/private/var/root/.cargo/registry",
+			"/private/var/root/go/pkg/mod",
+			"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
+			"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
 		}
 	case goosWindows:
 		// Relative to the root of the drive an operand lies on. The

internal/sfdupes/exclude_test.go

@@ -24,10 +24,22 @@ func TestDefaultExcludes(t *testing.T) {
 	// on any other system. The Windows list exists in the code only: the
 	// binary does not build for Windows, so the README leaves it out.
 	cases := map[string][]string{
-		goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
+		goosLinux: {
+			"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
+			"/root/.cache", "/root/.npm", "/root/.cargo/registry",
+			"/root/go/pkg/mod",
+			"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
+			"/home/*/go/pkg/mod",
+		},
 		goosDarwin: {
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
+			"/private/var/folders", "/Library/Caches",
+			"/private/var/root/Library/Caches", "/private/var/root/.cache",
+			"/private/var/root/.npm", "/private/var/root/.cargo/registry",
+			"/private/var/root/go/pkg/mod",
+			"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
+			"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
 		},
 		goosWindows: {
 			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
@@ -63,6 +75,29 @@ func TestIsDefaultExclude(t *testing.T) {
 			t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
 		}
 	}
+
+	// The Linux list matches the cache directories in /root and in each
+	// home in /home, but nothing below them and no .cache directory
+	// elsewhere.
+	linux := map[string]bool{
+		"/home/u/.cache":           true,
+		"/home/u/go/pkg/mod":       true,
+		"/root/.cache":             true,
+		"/var/cache":               true,
+		"/home/u/.cache/pip":       false,
+		"/home/u/src/.cache":       false,
+		"/srv/home/u/.cache":       false,
+		"/home/u/.cargo/bin":       false,
+		"/home/u/go/pkg/mod/cache": false,
+	}
+
+	for path, want := range linux {
+		got := isDefaultExclude(path, defaultExcludes(goosLinux))
+		if got != want {
+			t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
+				got, want)
+		}
+	}
 }
 
 func TestReadMountInfo(t *testing.T) {
@@ -290,7 +325,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
 	t.Setenv(databaseEnv, path)
 
 	dir := realTempDir(t)
-	if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
+	if !insideDefaultExclude(dir, defaultExcludes(runtime.GOOS)) {
 		t.Skip(dir + " is not inside a default exclude")
 	}
 
@@ -711,6 +746,20 @@ func shmTempDir(t *testing.T) (string, string, walkRules) {
 	return dir, fsType, walkRules{nonPersistent: nonPersistent}
 }
 
+// insideDefaultExclude reports whether path, which is absolute and
+// clean, or a directory above it is one of excludes.
+func insideDefaultExclude(path string, excludes []string) bool {
+	for dir := path; ; dir = filepath.Dir(dir) {
+		if isDefaultExclude(dir, excludes) {
+			return true
+		}
+
+		if dir == "/" {
+			return false
+		}
+	}
+}
+
 // assertRecorded checks that the database at dbPath holds a record for
 // path when want is set, and none when it is not.
 func assertRecorded(t *testing.T, dbPath, path string, want bool) {

internal/sfdupes/progress.go

@@ -9,10 +9,8 @@ import (
 	"strings"
 	"sync"
 	"time"
-	"unicode"
 	"unicode/utf8"
 
-	"github.com/rivo/uniseg"
 	"github.com/schollz/progressbar/v3"
 	"golang.org/x/term"
 )
@@ -118,8 +116,7 @@ type drawnCount struct {
 // terminal's width. A phase with an exact total ends the line with its
 // percent and bar, at the terminal's right edge, when the terminal has
 // room for them. It is drawn again every redrawInterval, so its elapsed
-// time keeps moving while the phase waits for its next item. Each draw
-// of the walk shows the path it reached last on the line below. When
+// time keeps moving while the phase waits for its next item. When
 // stderr is not a TTY it emits no ANSI redraws: it prints line as the
 // phase starts, every plainInterval while it runs, and as it ends.
 //
@@ -144,7 +141,6 @@ type progress struct {
 	estimate   bool
 	waitingFor string
 	stalledOn  string
-	path       string // the path the walk reached last; "" for none
 	count      int64
 	start      time.Time // when the phase started, for the elapsed time
 	rateStart  time.Time // when the count started, for the time remaining
@@ -243,82 +239,6 @@ func (p *progress) draw() {
 	// less the terminal's last column, which spinnerMargin keeps clear. A
 	// bar's cells take one column each but more than one byte.
 	p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1)
-
-	if p.path != "" {
-		// The library draws the path on the line below and returns to
-		// the phase's line. The path is escaped as warnings are, and its
-		// other control characters are written out, so that the terminal
-		// shows them instead of acting on them. It stays clear of the
-		// last column, so that no terminal wraps it.
-		shown := escapeControlChars(escapePath(p.path))
-		_ = p.bar.AddDetail(cutToColumns(shown, width-1))
-	}
-}
-
-// escapeControlChars returns s with each control character written as
-// \x and its code in two hex digits, as \x1b for the escape character.
-// The control characters are U+0000 to U+001F, U+007F and U+0080 to
-// U+009F, as unicode.IsControl reports them. A byte from 0x80 to 0x9F
-// that is not part of a valid UTF-8 character is written the same way,
-// as \x9b, since a terminal not in UTF-8 mode acts on it. Every other
-// byte is kept as it is.
-func escapeControlChars(s string) string {
-	var b strings.Builder
-
-	for rest := s; rest != ""; {
-		char, size := utf8.DecodeRuneInString(rest)
-		if char == utf8.RuneError && size == 1 {
-			// Not part of a valid UTF-8 character. A terminal not in
-			// UTF-8 mode reads the byte as the character with its code,
-			// so that 0x9b is a control character there and 0xe9 is not.
-			char = rune(rest[0])
-		}
-
-		if unicode.IsControl(char) {
-			fmt.Fprintf(&b, `\x%02x`, char)
-		} else {
-			b.WriteString(rest[:size])
-		}
-
-		rest = rest[size:]
-	}
-
-	return b.String()
-}
-
-// cutToColumns returns s cut to at most columns terminal columns, at a
-// character boundary. A wide character, as in Chinese, takes two
-// columns.
-func cutToColumns(s string, columns int) string {
-	used, end, state := 0, 0, -1
-
-	for rest := s; rest != ""; {
-		char, next, width, nextState := uniseg.FirstGraphemeClusterInString(
-			rest, state)
-		if used+width > columns {
-			break
-		}
-
-		used += width
-		end += len(char)
-		rest, state = next, nextState
-	}
-
-	return s[:end]
-}
-
-// clear erases the display from the terminal: its line, and the path on
-// the line below when one is shown. p.mu must be held.
-func (p *progress) clear() {
-	if p.bar == nil {
-		return
-	}
-
-	if p.path != "" {
-		_ = p.bar.AddDetail("")
-	}
-
-	_ = p.bar.Clear()
 }
 
 // newBar has the library make what draw draws: its spinner, followed by
@@ -343,8 +263,6 @@ func (p *progress) newBar(width int) {
 		// The text carries the elapsed time and the time remaining.
 		progressbar.OptionSetElapsedTime(false),
 		progressbar.OptionSetPredictTime(false),
-		// The line below the phase's line, for the walk's path.
-		progressbar.OptionSetMaxDetailRow(1),
 	)
 	p.widest = 0
 }
@@ -396,19 +314,6 @@ func (p *progress) increment() {
 	p.lastItem = now
 }
 
-// setPath records the path the walk reached last, which a display on a
-// terminal shows at its next draw.
-func (p *progress) setPath(path string) {
-	if p == nil {
-		return
-	}
-
-	p.mu.Lock()
-	defer p.mu.Unlock()
-
-	p.path = path
-}
-
 // warnf prints a one-line warning to stderr without corrupting the
 // display. The whole message is escaped like a report's path columns,
 // so a path holding a newline cannot split the warning.
@@ -422,9 +327,11 @@ func (p *progress) warnf(format string, args ...any) {
 	p.mu.Lock()
 	defer p.mu.Unlock()
 
-	// The warning takes the display's line; the next draw shows the
-	// display again, on the line below.
-	p.clear()
+	if p.bar != nil {
+		// The warning takes the display's line; the next draw shows
+		// the display again, on the line below.
+		_ = p.bar.Clear()
+	}
 
 	fmt.Fprintln(os.Stderr, msg)
 }
@@ -485,7 +392,9 @@ func (w *progressWriter) Flush() error {
 		w.p.mu.Lock()
 		defer w.p.mu.Unlock()
 
-		w.p.clear()
+		if w.p.bar != nil {
+			_ = w.p.bar.Clear()
+		}
 	}
 
 	_, err := w.out.Write(w.buf)
@@ -505,13 +414,6 @@ func (p *progress) finish() {
 	close(p.stop)
 	<-p.done
 
-	// The walk's path is only for someone watching the walk. Its line is
-	// erased, and the next phase is drawn on it.
-	p.mu.Lock()
-	p.clear()
-	p.path = ""
-	p.mu.Unlock()
-
 	p.draw()
 
 	if p.width != nil {

internal/sfdupes/progress_test.go

@@ -6,7 +6,6 @@ import (
 	"math"
 	"os"
 	"path/filepath"
-	"regexp"
 	"slices"
 	"strings"
 	"sync"
@@ -1139,164 +1138,3 @@ func TestWarningAfterLastItemKeepsBar(t *testing.T) {
 		t.Errorf("terminal shows %q, want a count of 2 of 2", shown)
 	}
 }
-
-func TestCutToColumns(t *testing.T) {
-	t.Parallel()
-
-	cases := []struct {
-		s       string
-		columns int
-		want    string
-	}{
-		{"/srv/a", 10, "/srv/a"},
-		{"/srv/photos", 8, "/srv/pho"},
-		// Each of these two fullwidth letters takes two columns.
-		{"/ＡＢ/a", 4, "/Ａ"},
-		{"/ＡＢ/a", 5, "/ＡＢ"},
-		// An accent that combines with the letter before stays with it.
-		{"/café/a", 5, "/café"},
-		{"/srv", 0, ""},
-	}
-
-	for _, tc := range cases {
-		if got := cutToColumns(tc.s, tc.columns); got != tc.want {
-			t.Errorf("cutToColumns(%q, %d) = %q, want %q",
-				tc.s, tc.columns, got, tc.want)
-		}
-	}
-}
-
-func TestEscapeControlChars(t *testing.T) {
-	t.Parallel()
-
-	cases := map[string]string{
-		"/srv/music":    "/srv/music",
-		"/a\x1b[2Jb":    `/a\x1b[2Jb`,
-		"/a\x00\a\x7fb": `/a\x00\x07\x7fb`,
-		// U+009B takes two bytes in UTF-8.
-		"/a\u009b2Jb": `/a\x9b2Jb`,
-		// Byte 0x9b alone is not valid UTF-8. A terminal not in UTF-8
-		// mode acts on it, as on every byte from 0x80 to 0x9f.
-		"/a\x9b2Jb":       `/a\x9b2Jb`,
-		"/a\x80\x9f\xa0b": `/a\x80\x9f` + "\xa0b",
-		// In ě, which is c4 9b in UTF-8, byte 0x9b is part of a valid
-		// character.
-		"/aě2Jb":          "/aě2Jb",
-		"/café/ＡＢ":        "/café/ＡＢ",
-		"/latin1-caf\xe9": "/latin1-caf\xe9",
-	}
-	for in, want := range cases {
-		if got := escapeControlChars(in); got != want {
-			t.Errorf("escapeControlChars(%q) = %q, want %q", in, got, want)
-		}
-	}
-}
-
-// pathLines returns the text of each line the library wrote below a
-// display's line. It writes one as an erase to the end of the line, a
-// carriage return, the text and a newline.
-func pathLines(out string) []string {
-	matches := regexp.MustCompile("\x1b\\[K\r([^\n]*)\n").
-		FindAllStringSubmatch(out, -1)
-	lines := make([]string, 0, len(matches))
-
-	for _, m := range matches {
-		lines = append(lines, m[1])
-	}
-
-	return lines
-}
-
-// TestTerminalWalkShowsPathBelow checks that the walk's display on a
-// terminal 30 columns wide shows the path it reached last on the line
-// below, cut to 29 columns. That line is erased before a warning is
-// printed, and again when the walk ends, so the next phase is drawn on
-// an empty line.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestTerminalWalkShowsPathBelow(t *testing.T) {
-	stderr := captureStderr(t)
-	p := startProgress(progressParams{
-		label: scanPhase("walk"), unit: scanUnit, total: -1,
-	}, func() int { return 30 }, testRedraw, time.Now)
-
-	const shown = "/srv/photos/2026/holiday/beac"
-
-	p.setPath("/srv/photos/2026/holiday/beach.jpg")
-	p.increment()
-	awaitOutput(t, stderr, shown, 1)
-
-	p.warnf("warning")
-
-	before, _, _ := strings.Cut(stderr(), "warning")
-	if lines := pathLines(before); lines[len(lines)-1] != "" {
-		t.Errorf("line below shows %q as the warning is printed, want it "+
-			"empty", lines[len(lines)-1])
-	}
-
-	// Drawn again below the warning.
-	awaitOutput(t, stderr, shown, strings.Count(before, shown)+1)
-	p.finish()
-
-	lines := pathLines(stderr())
-	if !slices.Contains(lines, shown) {
-		t.Errorf("lines below the display = %q, want %q among them",
-			lines, shown)
-	}
-
-	if last := lines[len(lines)-1]; last != "" {
-		t.Errorf("line below the display shows %q after the walk, want it "+
-			"empty", last)
-	}
-}
-
-// TestTerminalWalkEscapesControlChars checks that a walked path holding
-// escape characters, which would clear the screen and set the window
-// title, puts none on the line below the walk's display. The line shows
-// each control character written out, as \x1b.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestTerminalWalkEscapesControlChars(t *testing.T) {
-	stderr := captureStderr(t)
-	p := startProgress(progressParams{
-		label: scanPhase("walk"), unit: scanUnit, total: -1,
-	}, func() int { return 80 }, testRedraw, time.Now)
-
-	p.setPath("/srv/\x1b[2J\x1b]0;title\a/beach.jpg")
-	p.increment()
-	awaitOutput(t, stderr, "beach.jpg", 1)
-	p.finish()
-
-	lines := pathLines(stderr())
-	for _, line := range lines {
-		if strings.Contains(line, "\x1b") {
-			t.Errorf("line below the display = %q, want no escape "+
-				"character", line)
-		}
-	}
-
-	const shown = `/srv/\x1b[2J\x1b]0;title\x07/beach.jpg`
-	if !slices.Contains(lines, shown) {
-		t.Errorf("lines below the display = %q, want %q among them",
-			lines, shown)
-	}
-}
-
-// TestPlainWalkShowsNoPath checks that the walk's display prints no path
-// when stderr is not a terminal.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestPlainWalkShowsNoPath(t *testing.T) {
-	stderr := captureStderr(t)
-	p := startProgress(progressParams{
-		label: scanPhase("walk"), unit: scanUnit, total: -1,
-	}, nil, testRedraw, time.Now)
-
-	p.setPath("/srv/a")
-	p.increment()
-	p.finish()
-
-	if strings.Contains(stderr(), "/srv/a") {
-		t.Errorf("stderr = %q, want no path", stderr())
-	}
-}

internal/sfdupes/scan.go

@@ -562,7 +562,6 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
 
 		s.st.walked++
 
-		prog.setPath(ev.rec.path)
 		prog.increment()
 
 		old, ok := s.existing[ev.rec.path]