Leave caches out of the walk by default (closes #181)

The default excludes now cover caches that programs and package
managers fill again on their own. On Linux: /var/cache and, in /root
and in each home in /home, .cache, .npm, .cargo/registry and
go/pkg/mod. On macOS: /private/var/folders, /Library/Caches and, in
/private/var/root and in each home in /Users, Library/Caches plus the
same four. README and scan --help list them; --no-default-excludes
still turns them all off.

Model: opus-5-5

README.md

@@ -22,9 +22,8 @@ records the files it finds in an SQLite database. SQLite keeps a whole database
 in one file. The database is kept between runs. A rescan reads only new and
 changed files. That makes `scan` cheap enough to run every day from `cron`, the
 system's job scheduler. `report`, `trees` and `info` work from the database and
-can run at any time. `report` and `trees` leave out a copy that was deleted
+can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
-after the scan. `dwim` scans `/` and then prints the `trees` report, for a first
+first look at a new machine with one command.
-look at a new machine with one command.
 
 ## Getting Started
 
@@ -129,8 +128,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
     - one line naming the database;
     - progress lines, as described under "Progress";
     - a warning for each path it skips;
-    - a line for each system or temporary directory the walk leaves out (see
+    - a line for each system, temporary or cache directory the walk leaves out
-      "Rules for the walk");
+      (see "Rules for the walk");
     - a line for each path left out because its filesystem is one the kernel,
       the core of the operating system, makes up or keeps in memory, such as
       `tmpfs`;
@@ -302,8 +301,9 @@ The reference below is also the program's specification.
   directories on other filesystems, as the same flag does for `du` and `rsync`.
   By default the walk also enters directories on other filesystems.
 - `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
-  Those are the system and temporary directories the walk otherwise leaves out,
+  Those are the system, temporary and cache directories the walk otherwise
-  such as `/var/tmp` on Linux (see "Rules for the walk").
+  leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules for the
+  walk").
 - `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
   it. `PATH` may be a directory or a file. The flag may be given more than once.
   The walk knows `PATH` by its device ID and inode number, as it knows the
@@ -317,9 +317,11 @@ The reference below is also the program's specification.
 - `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
   exits 0. It writes nothing to stderr.
 - `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
-  and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the
+  and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
-  default excludes of the system it runs on, and the filesystem types the walk
+  names the database. The help of `report`, `trees`, `info` and `dwim` says what
-  never records anything on.
+  each prints. `sfdupes scan --help` also lists the default excludes of the
+  system it runs on, and the filesystem types the walk never records anything
+  on.
 
 ### Duplicate detection
 
@@ -570,13 +572,18 @@ walk practical on large or busy storage pools.
     it, as described under "What a scan does to the database".
     `--no-default-excludes` does not change this rule.
 
-- Below an operand, the walk does not enter a default exclude. On Linux, the
+- Below an operand, the walk does not enter a default exclude. The default
-  default excludes hold temporary files and the kernel's view of the running
+  excludes hold the kernel's view of the running system, temporary files, and
-  system. They depend on the system, and `sfdupes scan --help` lists those of
+  caches. A cache here is data that programs and package managers download or
-  the system it runs on. On Linux they are:
+  build again on their own. The default excludes depend on the system, and
+  `sfdupes scan --help` lists those of the system it runs on. In the lists, `*`
+  stands for any one directory name, so `/home/*/.cache` matches
+  `/home/alice/.cache`. On Linux they are:
 
     ```
-    /proc, /sys, /dev, /run, /tmp, /var/tmp
+    /proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache
+    /root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
+    /home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
     ```
 
     On macOS they are:
@@ -584,13 +591,21 @@ walk practical on large or busy storage pools.
     ```
     /dev, /private/tmp, /private/var/tmp, /private/var/vm
     /System/Volumes/VM, /System/Volumes/Data
+    /private/var/folders, /Library/Caches
+    /private/var/root/Library/Caches, /private/var/root/.cache
+    /private/var/root/.npm, /private/var/root/.cargo/registry
+    /private/var/root/go/pkg/mod
+    /Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
+    /Users/*/.cargo/registry, /Users/*/go/pkg/mod
     ```
 
-    A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already,
+    `/private/var/folders` holds each user's temporary files and caches on
-    through firmlinks. Firmlinks are the links macOS uses to show the data
+    macOS. Root's home on macOS is `/var/root`, which the walk reaches as
-    volume inside the system volume. Walking `/System/Volumes/Data` as well
+    `/private/var/root`. A walk of `/` on macOS reaches everything in
-    would record each of those files twice. Any other system has no default
+    `/System/Volumes/Data` already, through firmlinks. Firmlinks are the links
-    excludes.
+    macOS uses to show the data volume inside the system volume. Walking
+    `/System/Volumes/Data` as well would record each of those files twice. Any
+    other system has no default excludes.
 
     An operand at or inside a default exclude is walked, so
     `sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem.

TODO.md

@@ -29,6 +29,18 @@
 Newest first. An entry ends with its date and its issue or branch, where those
 are known.
 
+- The default excludes cover caches as well. On Linux they add `/var/cache`. In
+  `/root` and in each home in `/home` they add `.cache`, `.npm`,
+  `.cargo/registry` and `go/pkg/mod`. On macOS they add `/private/var/folders`
+  and `/Library/Caches`. In `/private/var/root` and in each home in `/Users`
+  they add `Library/Caches` and the same four. `--no-default-excludes` still
+  turns them all off (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/181)
+
+- The help of `sfdupes` and of `scan` names the database, its default path and
+  `SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
+  order of the rows and the escapes. The help of `info` lists its lines. The
+  help of `dwim` says that it scans `/`, should run as root and keeps the scan
+  in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
 - New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
   enters nor records an excluded path, and keeps the records below it. An
   operand at or below one is a fatal error (2026-11-11,

internal/sfdupes/cli.go

@@ -32,6 +32,69 @@ const (
 	cmdDwim   = "dwim"
 )
 
+// databaseHelp is the paragraph of the root and scan help that names
+// the database (README "Location").
+const databaseHelp = "scan and dwim store what they find in a database " +
+	"file. report, trees\nand info read it. The file is $" + databaseEnv +
+	" when that is set and\nnot empty, and otherwise " +
+	defaultDatabasePath + ", where creating it\nneeds root.\n"
+
+// escapesHelp is the paragraph of the report and trees help on how
+// paths are escaped (README "Report output format").
+const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
+
+  \\  a backslash
+  \t  a tab
+  \n  a newline
+  \r  a carriage return
+`
+
+// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
+// summary in the help of their subcommands.
+const (
+	reportHelp = `The report is tab-separated text on stdout: a header line, then one
+row per duplicate file. The columns are:
+
+  first   the path that sorts first, byte by byte, in its group of copies
+  dupe    another copy of that file
+  size    the file's size in bytes
+
+Rows are ordered by size, largest first, then by path.
+
+` + escapesHelp
+
+	treesHelp = `The report is tab-separated text on stdout: a header line, then one
+row per duplicate directory tree. The columns are:
+
+  first      the path that sorts first, byte by byte, in its group of copies
+  dupe       another copy of that tree
+  files      the number of regular files in one copy
+  size       their total size in bytes
+  disk_size  the bytes the dupe copy takes on disk
+
+Rows are ordered by size, largest first, then by path.
+
+` + escapesHelp
+
+	infoHelp = `The details go to stdout, one per line. Each line holds a name and its
+value, separated by a tab:
+
+  database                the path, and where it came from
+  database file size      in bytes, then rounded, as 2147483648 (2.0 GiB)
+  -wal file size          in bytes, then rounded; only if a -wal file exists
+  modified                when the database file last changed, in UTC
+  schema version          the version of the database's layout
+  records                 the number of records
+  size of recorded files  their total size, in bytes, then rounded
+`
+
+	dwimHelp = `dwim runs what "sfdupes scan /" runs, then what "sfdupes trees" runs.
+Run it as root, so that the scan can read every file. The scan is stored
+in the database, as scan stores it, so "sfdupes trees" prints the report
+again later.
+`
+)
+
 // errNoSubcommand is returned by the root command when it is invoked
 // without a subcommand. That is a usage error, and the usage text
 // cobra prints for it is the whole message.
@@ -202,15 +265,21 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 		}),
 	}
 
+	root.Long = root.Short + ".\n\n" + databaseHelp
+	reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
+	treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
+	infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
+	dwimCmd.Long = dwimCmd.Short + ".\n\n" + dwimHelp
+
 	root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
 
 	return root
 }
 
 // newScanCommand builds the scan subcommand, with its --workers, -x,
-// --no-default-excludes and --exclude flags. Its help lists the default
+// --no-default-excludes and --exclude flags. Its help names the
-// excludes of the system it runs on, and the filesystem types it never
+// database, and lists the default excludes of the system it runs on and
-// records anything on.
+// the filesystem types it never records anything on.
 func newScanCommand() *cobra.Command {
 	var (
 		scanWorkers           int
@@ -228,7 +297,8 @@ func newScanCommand() *cobra.Command {
 		// append its generic "[flags]" to it.
 		DisableFlagsInUseLine: true,
 		Short:                 short,
-		Long:                  short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
+		Long: short + ".\n\n" + databaseHelp + "\n" +
+			defaultExcludesHelp(runtime.GOOS),
 		Args: cobra.MinimumNArgs(1),
 		PreRunE: func(cmd *cobra.Command, _ []string) error {
 			err := checkScanWorkers(cmd, scanWorkers)

internal/sfdupes/cli_test.go

@@ -22,6 +22,9 @@ import (
 // error prints and a runtime failure must not.
 const usageMarker = "Usage:"
 
+// helpFlag asks for the help text.
+const helpFlag = "--help"
+
 // TestMain stops the tests at once, saying why, when their temporary
 // directory is not on a persistent filesystem: scan records nothing
 // there (README "Rules for the walk"), so most scan tests would fail
@@ -693,7 +696,7 @@ func TestRunHelp(t *testing.T) {
 
 	// README §Settings: help goes to stderr, exits 0, and leaves stdout
 	// empty.
-	cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
+	cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
 
 	for _, args := range cases {
 		var stdout, stderr bytes.Buffer
@@ -754,6 +757,73 @@ func TestRunScanUsageLine(t *testing.T) {
 	}
 }
 
+func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
+	t.Parallel()
+
+	// README §Settings: the help names the database, and says what
+	// report, trees, info and dwim print.
+	database := []string{
+		"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
+	}
+	escapes := []string{`\\`, `\t`, `\n`, `\r`}
+
+	// The subcommand whose help is printed; none for the root.
+	cases := []struct {
+		subcommand []string
+		want       []string
+	}{
+		{nil, database},
+		{[]string{cmdScan}, database},
+		{
+			[]string{cmdReport},
+			append([]string{
+				"tab-separated", "\n  first ", "\n  dupe ", "\n  size ",
+				"largest first",
+			}, escapes...),
+		},
+		{
+			[]string{cmdTrees},
+			append([]string{
+				"tab-separated", "\n  first ", "\n  dupe ", "\n  files ",
+				"\n  size ", "\n  disk_size ", "largest first",
+			}, escapes...),
+		},
+		{
+			[]string{cmdInfo},
+			[]string{
+				"\n  database ", "\n  database file size ",
+				"\n  -wal file size ", "\n  modified ",
+				"\n  schema version ", "\n  records ",
+				"\n  size of recorded files ",
+			},
+		},
+		{
+			[]string{cmdDwim},
+			[]string{
+				"sfdupes scan /", "as root", "sfdupes trees", "database",
+			},
+		},
+	}
+
+	for _, tc := range cases {
+		var stdout, stderr bytes.Buffer
+
+		args := slices.Concat(tc.subcommand, []string{helpFlag})
+
+		code := run(args, &stdout, &stderr)
+		if code != exitOK {
+			t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
+		}
+
+		for _, want := range tc.want {
+			if !strings.Contains(stderr.String(), want) {
+				t.Errorf("run(%v) help lacks %q:\n%s",
+					args, want, stderr.String())
+			}
+		}
+	}
+}
+
 func TestRunVersion(t *testing.T) {
 	t.Parallel()
 

internal/sfdupes/exclude.go

@@ -72,14 +72,31 @@ type walkRules struct {
 func defaultExcludes(goos string) []string {
 	switch goos {
 	case goosLinux:
-		return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
+		// The kernel's view of the running system, temporary files, and
+		// caches that programs and package managers fill again on their
+		// own: the system's, root's and those in each home in /home.
+		return []string{
+			"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
+			"/root/.cache", "/root/.npm", "/root/.cargo/registry",
+			"/root/go/pkg/mod",
+			"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
+			"/home/*/go/pkg/mod",
+		}
 	case goosDarwin:
 		// A walk of / reaches everything in /System/Volumes/Data through
 		// the firmlinks, so walking it as well would record each of
-		// those files twice.
+		// those files twice. /private/var/folders holds each user's
+		// temporary files and caches. The walk reaches root's home,
+		// /var/root, as /private/var/root.
 		return []string{
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
+			"/private/var/folders", "/Library/Caches",
+			"/private/var/root/Library/Caches", "/private/var/root/.cache",
+			"/private/var/root/.npm", "/private/var/root/.cargo/registry",
+			"/private/var/root/go/pkg/mod",
+			"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
+			"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
 		}
 	case goosWindows:
 		// Relative to the root of the drive an operand lies on. The

internal/sfdupes/exclude_test.go

@@ -24,10 +24,22 @@ func TestDefaultExcludes(t *testing.T) {
 	// on any other system. The Windows list exists in the code only: the
 	// binary does not build for Windows, so the README leaves it out.
 	cases := map[string][]string{
-		goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
+		goosLinux: {
+			"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
+			"/root/.cache", "/root/.npm", "/root/.cargo/registry",
+			"/root/go/pkg/mod",
+			"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
+			"/home/*/go/pkg/mod",
+		},
 		goosDarwin: {
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
+			"/private/var/folders", "/Library/Caches",
+			"/private/var/root/Library/Caches", "/private/var/root/.cache",
+			"/private/var/root/.npm", "/private/var/root/.cargo/registry",
+			"/private/var/root/go/pkg/mod",
+			"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
+			"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
 		},
 		goosWindows: {
 			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
@@ -63,6 +75,29 @@ func TestIsDefaultExclude(t *testing.T) {
 			t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
 		}
 	}
+
+	// The Linux list matches the cache directories in /root and in each
+	// home in /home, but nothing below them and no .cache directory
+	// elsewhere.
+	linux := map[string]bool{
+		"/home/u/.cache":           true,
+		"/home/u/go/pkg/mod":       true,
+		"/root/.cache":             true,
+		"/var/cache":               true,
+		"/home/u/.cache/pip":       false,
+		"/home/u/src/.cache":       false,
+		"/srv/home/u/.cache":       false,
+		"/home/u/.cargo/bin":       false,
+		"/home/u/go/pkg/mod/cache": false,
+	}
+
+	for path, want := range linux {
+		got := isDefaultExclude(path, defaultExcludes(goosLinux))
+		if got != want {
+			t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
+				got, want)
+		}
+	}
 }
 
 func TestReadMountInfo(t *testing.T) {
@@ -290,7 +325,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
 	t.Setenv(databaseEnv, path)
 
 	dir := realTempDir(t)
-	if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
+	if !insideDefaultExclude(dir, defaultExcludes(runtime.GOOS)) {
 		t.Skip(dir + " is not inside a default exclude")
 	}
 
@@ -711,6 +746,20 @@ func shmTempDir(t *testing.T) (string, string, walkRules) {
 	return dir, fsType, walkRules{nonPersistent: nonPersistent}
 }
 
+// insideDefaultExclude reports whether path, which is absolute and
+// clean, or a directory above it is one of excludes.
+func insideDefaultExclude(path string, excludes []string) bool {
+	for dir := path; ; dir = filepath.Dir(dir) {
+		if isDefaultExclude(dir, excludes) {
+			return true
+		}
+
+		if dir == "/" {
+			return false
+		}
+	}
+}
+
 // assertRecorded checks that the database at dbPath holds a record for
 // path when want is set, and none when it is not.
 func assertRecorded(t *testing.T, dbPath, path string, want bool) {