scan: skip OS-appropriate default excludes (/proc, /sys, /tmp, /run, /var/tmp on Linux) (closes #124)

Below an operand, the walk no longer enters the default excludes: on
Linux /proc, /sys, /dev, /run, /tmp, /var/tmp and every mount point of a
kernel or memory-backed filesystem, read once from /proc/self/mountinfo;
on macOS /dev, /private/tmp, /private/var/tmp, /private/var/vm,
/System/Volumes/VM and /System/Volumes/Data. The lists live in the new
exclude.go. An operand at or inside one is still walked, also under
another operand, as with -x. Each skipped directory gets one stderr line
through the warning path, is not counted as skipped, and its records
count as outside the scanned operands. --no-default-excludes turns them
off; scan --help lists them.

Judgement call: an unreadable /proc/self/mountinfo fails the scan.
Judgement call: where filesystems are mounted over one another, the last decides.

Model: opus-5-5

README.md

@@ -101,9 +101,8 @@ As a file `/etc/cron.d/sfdupes`:
   "Database"), plain progress lines naming each phase and its place among the
   five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
   runs, and one as it ends), a warning for each path it skips, a line for each
-  default exclude its walk does not enter and for each path it leaves out
+  default exclude its walk does not enter (see "Rules for the walk"), and the
-  because it is not on a persistent filesystem (see "Rules for the walk"), and
+  summary line (see "Progress" and "`scan` mode"). The log grows with every
-  the summary line (see "Progress" and "`scan` mode"). The log grows with every
   scan; rotate it like any other.
 - Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
   scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
@@ -261,7 +260,7 @@ sfdupes [command] --help
 `--workers N` sets the size of each `scan` worker pool (default: the number of
 CPUs), `-x` (`--one-file-system`) keeps the walk of each operand on that
 operand's filesystem, and `--no-default-excludes` lets the walk enter the
-default excludes, a list of paths; all three are described under "`scan` mode".
+default excludes; all three are described under "`scan` mode".
 `sfdupes --version` (or `-v`) prints one line, `sfdupes VERSION`, to stdout and
 exits 0, writing nothing to stderr. `-h` or `--help`, alone or after a
 subcommand, prints the help text to stderr and exits 0, writing nothing to
@@ -445,9 +444,9 @@ lies under another is dropped before walking, so every file is reached exactly
 once and produces one database record. An operand under another one is kept,
 though, and walked on its own when any directory from it (from its directory,
 for a regular file) up to but not including the operand covering it is a default
-exclude, is not on a persistent filesystem (see "Rules for the walk"), or, with
+exclude (see "Rules for the walk") or, with `-x`, is on a different filesystem
-`-x`, is on a different filesystem than that operand, because that operand's
+than that operand, because that operand's walk does not enter such a directory
-walk does not enter such a directory and so does not reach it.
+and so does not reach it.
 
 An operand that is a symlink (never followed, not even as an operand), socket,
 FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -457,10 +456,7 @@ beneath it is still scanned. The records stored beneath it are not deleted: they
 are treated like any other record outside the scanned operands, including the
 content-phase exception below. If it lies under another operand, they are under
 that operand instead, and are treated like any other record there. This is not
-an error: a scan whose every operand is dropped walks nothing and exits 0. An
+an error: a scan whose every operand is dropped walks nothing and exits 0.
-operand that is not on a persistent filesystem (see "Rules for the walk") is
-dropped the same way, except that the line `scan` prints for it is not a
-warning, and it is not counted as skipped.
 
 `scan` synchronizes the database with the filesystem state under the scanned
 operands:
@@ -498,13 +494,13 @@ operands:
   a `content` hash is read for one, wherever it lies, once its size, `head`, and
   `tail` match another record's. If that file is gone or has changed since its
   record was written, the record is left as it is.
-- A record at or below a path the walk left out, because that path is a default
+- A record below a directory the walk did not enter, because that directory is a
-  exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
+  default exclude or, with `-x`, because it is on another filesystem than its
-  another filesystem than its operand, counts as outside the scanned operands:
+  operand, counts as outside the scanned operands: it is not deleted, and its
-  it is not deleted, and its size joins the size census. A filesystem mounted
+  size joins the size census. A filesystem mounted under an operand can
-  under an operand can therefore be scanned on its own schedule into the same
+  therefore be scanned on its own schedule into the same database. Records below
-  database. Records below an operand at or inside such a path are not kept this
+  an operand at or inside such a directory are not kept this way, since that
-  way, since that operand's own walk verifies them.
+  operand's own walk verifies them.
 
 `scan` runs **five sequential phases over the whole scan**. Parallelism lives
 inside the walk, hash and content phases; batched database writes begin during
@@ -580,70 +576,41 @@ Rules for the walk:
   unless they lie below an operand at or inside that directory, count as outside
   the scanned operands, as described in "`scan` mode" above: they are not
   deleted, and their sizes join the size census.
-- Never record anything that is not on a persistent filesystem. A filesystem is
+- Below an operand, never descend into a default exclude: a directory that holds
-  not persistent when the kernel makes it up or keeps it in memory: one of type
+  the kernel's view of the running system, or temporary files, so that
-  `proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
-  `debugfs`, `tracefs`, `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`,
-  `mqueue`, `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`,
-  `rpc_pipefs`, `selinuxfs` or `devfs` (the macOS filesystem of `/dev`). After
-  it names the database and before it takes the lock, `scan` lists the mounted
-  filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
-  of mounted filesystems on macOS (on any other system it lists none), and notes
-  the device ID of the mount point of each one of those types, read with
-  `lstat`. Where one filesystem is mounted over another, the one mounted last
-  counts. If `scan` cannot list them, it fails (exit 1). The walk then never
-  enters a directory, and never records a file, on one of those devices, however
-  it reaches it: as an operand, below one, or through a bind mount, which keeps
-  the device ID of what it binds. No flag turns this off. For each operand,
-  directory or file it leaves out this way, `scan` prints a line on stderr
-  naming it and the type of its filesystem, as it prints a warning, so it never
-  lands inside a progress display:
-
-    ```
-    scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
-    ```
-
-    That line is not a warning, and the path is not counted as skipped. Records
-    that an earlier scan left at or below it, unless they lie below an operand
-    at or inside it, count as outside the scanned operands, as described in
-    "`scan` mode" above: they are not deleted, and their sizes join the size
-    census.
-
-- Below an operand, never descend into a default exclude: a path that holds the
-  kernel's view of the running system, or temporary files, so that
   `sfdupes scan /` walks only the files worth comparing. The default excludes
   depend on the system, and `sfdupes scan --help` lists those of the system it
   runs on:
-    - On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`.
+    - On Linux: `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`, and the
+      mount point of every filesystem of type `proc`, `sysfs`, `devtmpfs`,
+      `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`, `debugfs`, `tracefs`,
+      `securityfs`, `pstore`, `bpf`, `configfs`, `fusectl`, `mqueue`,
+      `hugetlbfs`, `efivarfs`, `binfmt_misc`, `autofs`, `nsfs`, `rpc_pipefs` or
+      `selinuxfs`, wherever it is mounted. Where one filesystem is mounted over
+      another, the one mounted last counts. `scan` reads the mount points from
+      `/proc/self/mountinfo` once, after it names the database and before it
+      takes the lock; if it cannot read them, it fails (exit 1).
     - On macOS: `/dev`, `/private/tmp`, `/private/var/tmp`, `/private/var/vm`,
       `/System/Volumes/VM` and `/System/Volumes/Data`. A walk of `/` reaches
       everything in `/System/Volumes/Data` through the firmlinks, so walking it
       as well would record each of those files twice.
-    - On Windows, relative to the root of the drive an operand lies on: the
-      page, swap and hibernation files `pagefile.sys`, `swapfile.sys` and
-      `hiberfil.sys`, and the directories `System Volume Information`,
-      `$Recycle.Bin`, `Windows\Temp` and `Users\*\AppData\Local\Temp`, where `*`
-      matches one path component, as it does in any default exclude. sfdupes
-      does not build for Windows yet, so this list is not used.
     - On any other system there are none.
 
-    `--no-default-excludes` turns them all off; what is not on a persistent
+    `--no-default-excludes` turns them all off, and `scan` then does not read
-    filesystem stays left out. An operand at or inside a default exclude is
+    `/proc/self/mountinfo`. An operand at or inside a default exclude is still
-    still walked, so `sfdupes scan /tmp/x` works, unless it is not on a
+    walked, so `sfdupes scan /tmp/x` works. For each default exclude its walk
-    persistent filesystem. For each default exclude its walk does not enter,
+    does not enter, `scan` prints a line on stderr naming it and why, as it
-    `scan` prints a line on stderr naming it, as it prints a warning:
+    prints a warning, so it never lands inside a progress display:
 
     ```
-    scan: skipped /var/tmp (default exclude)
+    scan: skipped /proc (default exclude, proc filesystem)
+    scan: skipped /tmp (default exclude)
     ```
 
-    That line is not a warning, and the directory is not counted as skipped. A
+    That line is not a warning, and the directory is not counted as skipped. The
-    default exclude that is not on a persistent filesystem, as `/proc` is, gets
+    records below it, unless they lie below an operand at or inside it, count as
-    the line for its filesystem instead, and one that is itself an operand gets
+    outside the scanned operands, as described in "`scan` mode" above: they are
-    no line, since it is walked as that operand. The records below a default
+    not deleted, and their sizes join the size census.
-    exclude the walk does not enter, unless they lie below an operand at or
-    inside it, count as outside the scanned operands, as described in "`scan`
-    mode" above: they are not deleted, and their sizes join the size census.
 
 - On any per-path error (permission denied, file vanished between passes,
   unreadable): print a one-line warning to stderr, skip the path, and continue.

TODO.md

@@ -29,20 +29,17 @@
 
 # Completed Steps
 
-- `scan` never records anything that is not on a persistent filesystem (`tmpfs`,
+- below an operand, `scan`'s walk does not enter the default excludes, so
-  `proc`, `sysfs` and the other kernel or memory-backed types), matched by the
+  `sfdupes scan /` skips the kernel's view of the running system and the
-  device ID of each such mount point, listed as the scan starts from
+  temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp`,
-  `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
+  `/var/tmp` and the mount point of every kernel or memory-backed filesystem,
-  however the walk reaches it; no flag turns that off. Below an operand, its
+  read from `/proc/self/mountinfo` as the scan starts; on macOS `/dev`,
-  walk does not enter the default excludes either, so `sfdupes scan /` skips the
-  kernel's view of the running system and the temporary directories: on Linux
-  `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`, on macOS `/dev`,
   `/private/tmp`, `/private/var/tmp`, `/private/var/vm`, `/System/Volumes/VM`
-  and `/System/Volumes/Data`, and a Windows list, not used yet. An operand at or
+  and `/System/Volumes/Data`. An operand at or inside one is still walked. The
-  inside a default exclude is still walked. The scan prints a line for each path
+  scan prints a line for each it does not enter, as it prints a warning, without
-  it leaves out, without counting it as skipped, and the records there count as
+  counting it as skipped, and the records below it count as outside the scanned
-  outside the scanned operands, as for `-x`. `--no-default-excludes` turns off
+  operands, as for `-x`. `--no-default-excludes` turns them off, and
-  the default excludes only, and `scan --help` lists them (2026-10-17,
+  `scan --help` lists them (2026-10-16,
   https://git.eeqj.de/sneak/sfdupes/issues/124)
 
 - the rate a progress display shows covers the items completed over the last 5

cancel_test.go

@@ -196,7 +196,7 @@ func TestSyncScanCancelledMidWalkKeepsRecords(t *testing.T) {
 	base := baselineGoroutines(t)
 
 	st, err := syncScan(newWalkClock(walkCancelAtDone), db,
-		[]string{dir}, walkCancelWorkers, false, walkRules{})
+		[]string{dir}, walkCancelWorkers, false, nil)
 
 	assertWalkGuardAborted(t, st, err)
 	assertRecordsIntact(t, db, before)
@@ -269,7 +269,7 @@ func TestSyncScanCancelledBeforeLoadIndex(t *testing.T) {
 	cancel()
 
 	st, err := syncScan(ctx, db, []string{dir}, walkCancelWorkers, false,
-		walkRules{})
+		nil)
 	if !errors.Is(err, context.Canceled) {
 		t.Fatalf("syncScan on a cancelled context = %v, want %v",
 			err, context.Canceled)
@@ -302,7 +302,7 @@ func TestSyncScanCancelledMidHashKeepsHashedRecords(t *testing.T) {
 	db := openTestDB(t)
 
 	st, err := syncScan(newWalkClock(hashCancelAtDone), db,
-		[]string{dir}, walkCancelWorkers, false, walkRules{})
+		[]string{dir}, walkCancelWorkers, false, nil)
 	if !errors.Is(err, context.Canceled) {
 		t.Fatalf("syncScan cancelled mid-hash = %v, want %v",
 			err, context.Canceled)
@@ -620,8 +620,8 @@ func TestWalkOneDirStopsWhenCancelled(t *testing.T) {
 	// Unbuffered and unread: on a cancelled scan every send gives up.
 	events := make(chan walkEvent)
 
-	subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false,
+	subs := walkOneDir(cancelledContext(t), dirJob{path: dir}, false, nil,
-		walkRules{}, events)
+		events)
 	if len(subs) != 0 {
 		t.Errorf("cancelled walkOneDir returned %+v to descend into, "+
 			"want none", subs)
@@ -641,8 +641,7 @@ func TestWalkWorkersDropQueuedDirs(t *testing.T) {
 
 	missing := filepath.Join(t.TempDir(), "missing")
 
-	jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false,
+	jobs, _, events := startWalkWorkers(cancelledContext(t), 2, false, nil)
-		walkRules{})
 
 	for range 64 {
 		jobs <- dirJob{path: missing}
@@ -673,7 +672,7 @@ func TestWalkWorkerAbandonsSubdirHandoff(t *testing.T) {
 	ctx, cancel := context.WithCancel(t.Context())
 	defer cancel()
 
-	jobs, subdirs, events := startWalkWorkers(ctx, 1, false, walkRules{})
+	jobs, subdirs, events := startWalkWorkers(ctx, 1, false, nil)
 
 	// Fill the hand-back channel to its capacity — one slot per worker
 	// — so the worker's own hand-back is certain to block.

exclude.go

@@ -1,33 +1,33 @@
 package main
 
-// This file holds two rules for the walk (README "Rules for the walk"):
+// This file holds the default excludes (README "Rules for the walk"):
-// the default excludes, the paths a scan's walk does not enter below an
+// the directories a scan's walk does not enter below an operand, unless
-// operand unless scan is given --no-default-excludes, and the
+// scan is given --no-default-excludes.
-// filesystems that are not persistent, on which the walk never records
-// anything.
 
 import (
 	"bufio"
 	"errors"
 	"fmt"
 	"io"
-	"io/fs"
+	"os"
-	"path/filepath"
 	"slices"
 	"strings"
 )
 
 // The systems that have default excludes, as runtime.GOOS names them.
 const (
-	goosLinux   = "linux"
+	goosLinux  = "linux"
-	goosDarwin  = "darwin"
+	goosDarwin = "darwin"
-	goosWindows = "windows"
 )
 
-// The fields of a line of mountInfoPath that readMountInfo reads,
+// mountInfoPath is the file in which Linux lists the mounts the reading
-// counted from 0 (see proc(5)): the mount point is the fifth field, and
+// process sees.
-// after the sixth come optional fields, then a field holding "-", then
+const mountInfoPath = "/proc/self/mountinfo"
-// the filesystem type.
+
+// The fields of a line of mountInfoPath that kernelMounts reads, counted
+// from 0 (see proc(5)): the mount point is the fifth field, and after
+// the sixth come optional fields, then a field holding "-", then the
+// filesystem type.
 const (
 	mountPointField     = 4
 	optionalFieldsStart = 6
@@ -38,22 +38,13 @@ const (
 const typesPerHelpLine = 6
 
 // errMountInfoLine reports a line of mountInfoPath without the fields
-// readMountInfo reads.
+// kernelMounts reads.
 var errMountInfoLine = errors.New("unexpected line")
 
-// walkRules holds the two rules for the walk that a scan sets up as it
+// defaultExcludes returns the directories that are default excludes on
-// starts. excludes are the default excludes, or none with
+// the system goos names, as runtime.GOOS does: a fixed list on Linux and
-// --no-default-excludes. nonPersistent maps the device ID of every
+// on macOS, and none on any other system. On Linux the mount points
-// mounted filesystem that is not persistent to its type; the walk
+// kernelMounts finds are default excludes as well.
-// records nothing on one of those, whatever the flags.
-type walkRules struct {
-	excludes      []string
-	nonPersistent map[uint64]string
-}
-
-// defaultExcludes returns the default excludes of the system goos
-// names, as runtime.GOOS does: a fixed list on Linux, macOS and
-// Windows, and none on any other system.
 func defaultExcludes(goos string) []string {
 	switch goos {
 	case goosLinux:
@@ -66,94 +57,65 @@ func defaultExcludes(goos string) []string {
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
 		}
-	case goosWindows:
-		// Relative to the root of the drive an operand lies on. The
-		// first three are the page, swap and hibernation files.
-		return []string{
-			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
-			"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
-			`Users\*\AppData\Local\Temp`,
-		}
 	default:
 		return nil
 	}
 }
 
-// isDefaultExclude reports whether path is one of excludes, in which a
+// kernelFilesystemTypes returns the Linux filesystem types whose mount
-// * matches one path component.
+// points are default excludes: filesystems that the kernel makes up or
-func isDefaultExclude(path string, excludes []string) bool {
+// that are kept in memory.
-	for _, pattern := range excludes {
+func kernelFilesystemTypes() []string {
-		// The patterns are fixed, so none is malformed.
-		if ok, _ := filepath.Match(pattern, path); ok {
-			return true
-		}
-	}
-
-	return false
-}
-
-// nonPersistentFilesystemTypes returns the filesystem types that are
-// not persistent: filesystems that the kernel makes up or that are kept
-// in memory. devfs is the macOS filesystem of /dev; the others are
-// Linux names.
-func nonPersistentFilesystemTypes() []string {
 	return []string{
 		"proc", "sysfs", "devtmpfs", "devpts", "tmpfs", "ramfs",
 		"cgroup", "cgroup2", "debugfs", "tracefs", "securityfs", "pstore",
 		"bpf", "configfs", "fusectl", "mqueue", "hugetlbfs", "efivarfs",
-		"binfmt_misc", "autofs", "nsfs", "rpc_pipefs", "selinuxfs", "devfs",
+		"binfmt_misc", "autofs", "nsfs", "rpc_pipefs", "selinuxfs",
 	}
 }
 
-// nonPersistentFilesystems returns the device ID of every mounted
+// findDefaultExcludes returns the default excludes of the system goos
-// filesystem that is not persistent, with its type, read with lstat on
+// names, each with what the scan says about it on the line it prints
-// its mount point. mountedFilesystems lists the mounts. A mount point
+// when its walk does not enter it. On Linux it reads the mount points
-// that lstat cannot reach is left out.
+// from mountInfoPath.
-func nonPersistentFilesystems() (map[uint64]string, error) {
+func findDefaultExcludes(goos string) (map[string]string, error) {
-	mounts, err := mountedFilesystems()
+	excludes := make(map[string]string)
+
+	for _, dir := range defaultExcludes(goos) {
+		excludes[dir] = "default exclude"
+	}
+
+	if goos != goosLinux {
+		return excludes, nil
+	}
+
+	f, err := os.Open(mountInfoPath)
 	if err != nil {
 		return nil, err
 	}
 
-	types := nonPersistentFilesystemTypes()
+	defer func() { _ = f.Close() }()
-	devices := make(map[uint64]string)
 
-	for mountPoint, fsType := range mounts {
+	mounts, err := kernelMounts(f)
-		if !slices.Contains(types, fsType) {
+	if err != nil {
-			continue
+		return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
-		}
-
-		if dev, ok := deviceOfPath(mountPoint); ok {
-			devices[dev] = fsType
-		}
 	}
 
-	return devices, nil
+	for dir, fsType := range mounts {
+		excludes[dir] = "default exclude, " + fsType + " filesystem"
+	}
+
+	return excludes, nil
 }
 
-// nonPersistentLine returns the line the scan prints for path when fi,
+// kernelMounts reads mountinfo, laid out as mountInfoPath is, and
-// its lstat, puts it on a filesystem in rules.nonPersistent, and ""
+// returns the mount point of every filesystem whose type is one of
-// when it is on any other.
+// kernelFilesystemTypes, with that type. Where one filesystem is mounted
-func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
+// over another, a later line names the one on top, so the last line for
-	dev, ok := deviceOfInfo(fi)
+// a mount point decides.
-	if !ok {
+func kernelMounts(mountinfo io.Reader) (map[string]string, error) {
-		return ""
+	types := kernelFilesystemTypes()
-	}
 
-	fsType, found := rules.nonPersistent[dev]
-	if !found {
-		return ""
-	}
-
-	return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
-		path, fsType)
-}
-
-// readMountInfo reads mountinfo, laid out as mountInfoPath is, and
-// returns every mount point with the type of its filesystem. Where one
-// filesystem is mounted over another, a later line names the one on
-// top, so the last line for a mount point decides.
-func readMountInfo(mountinfo io.Reader) (map[string]string, error) {
 	// The kernel writes a space, tab, newline or backslash in a mount
 	// point as an octal escape.
 	unescape := strings.NewReplacer(
@@ -170,36 +132,42 @@ func readMountInfo(mountinfo io.Reader) (map[string]string, error) {
 			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
 		}
 
-		mounts[unescape.Replace(fields[mountPointField])] = fields[sep+1]
+		mountPoint := unescape.Replace(fields[mountPointField])
+
+		fsType := fields[sep+1]
+		if slices.Contains(types, fsType) {
+			mounts[mountPoint] = fsType
+		} else {
+			delete(mounts, mountPoint)
+		}
 	}
 
 	return mounts, lines.Err()
 }
 
-// defaultExcludesHelp describes, for the scan command's help, the
+// defaultExcludesHelp describes the default excludes of the system goos
-// default excludes of the system goos names and, where the scan lists
+// names, for the scan command's help.
-// the mounted filesystems, the types it never records anything on.
 func defaultExcludesHelp(goos string) string {
-	var help strings.Builder
-
 	dirs := defaultExcludes(goos)
 	if len(dirs) == 0 {
-		help.WriteString("This system has no default excludes.\n")
+		return "This system has no default excludes."
-	} else {
-		help.WriteString("Below an operand, the walk does not enter these " +
-			"default excludes, unless\n--no-default-excludes is given:\n\n")
-
-		for _, dir := range dirs {
-			fmt.Fprintf(&help, "  %s\n", dir)
-		}
 	}
 
-	if goos == goosLinux || goos == goosDarwin {
+	var help strings.Builder
-		help.WriteString("\nWith or without --no-default-excludes, the scan " +
-			"records nothing on a\nfilesystem that is not persistent, " +
-			"one of these types:\n\n")
 
-		types := nonPersistentFilesystemTypes()
+	help.WriteString("Below an operand, the walk does not enter these " +
+		"directories, the default\nexcludes, unless --no-default-excludes " +
+		"is given:\n\n")
+
+	for _, dir := range dirs {
+		fmt.Fprintf(&help, "  %s\n", dir)
+	}
+
+	if goos == goosLinux {
+		help.WriteString("\nnor the mount point of any filesystem of " +
+			"these types:\n\n")
+
+		types := kernelFilesystemTypes()
 		for line := range slices.Chunk(types, typesPerHelpLine) {
 			fmt.Fprintf(&help, "  %s\n", strings.Join(line, ", "))
 		}

exclude_darwin.go

@@ -1,32 +0,0 @@
-//go:build darwin
-
-package main
-
-import "golang.org/x/sys/unix"
-
-// mountedFilesystems returns every mount point with the type of its
-// filesystem, from the system's list of mounted filesystems. Where one
-// filesystem is mounted over another, the later one in the list counts.
-func mountedFilesystems() (map[string]string, error) {
-	// The first call counts the filesystems, the second lists them.
-	n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
-	if err != nil {
-		return nil, err
-	}
-
-	list := make([]unix.Statfs_t, n)
-
-	n, err = unix.Getfsstat(list, unix.MNT_NOWAIT)
-	if err != nil {
-		return nil, err
-	}
-
-	mounts := make(map[string]string, n)
-
-	for _, st := range list[:n] {
-		mounts[unix.ByteSliceToString(st.Mntonname[:])] =
-			unix.ByteSliceToString(st.Fstypename[:])
-	}
-
-	return mounts, nil
-}

exclude_other.go

@@ -1,36 +0,0 @@
-//go:build !darwin
-
-package main
-
-import (
-	"fmt"
-	"os"
-	"runtime"
-)
-
-// mountInfoPath is the file in which Linux lists the mounts the reading
-// process sees.
-const mountInfoPath = "/proc/self/mountinfo"
-
-// mountedFilesystems returns every mount point with the type of its
-// filesystem: on Linux read from mountInfoPath (see readMountInfo), and
-// on any other system none.
-func mountedFilesystems() (map[string]string, error) {
-	if runtime.GOOS != goosLinux {
-		return map[string]string{}, nil
-	}
-
-	f, err := os.Open(mountInfoPath)
-	if err != nil {
-		return nil, err
-	}
-
-	defer func() { _ = f.Close() }()
-
-	mounts, err := readMountInfo(f)
-	if err != nil {
-		return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
-	}
-
-	return mounts, nil
-}

exclude_test.go

@@ -3,6 +3,7 @@ package main
 import (
 	"bytes"
 	"errors"
+	"fmt"
 	"maps"
 	"os"
 	"path/filepath"
@@ -12,6 +13,10 @@ import (
 	"testing"
 )
 
+// defaultExcludeWhy is what the scan says about a default exclude from
+// the fixed list when its walk does not enter it.
+const defaultExcludeWhy = "default exclude"
+
 func TestDefaultExcludes(t *testing.T) {
 	t.Parallel()
 
@@ -22,11 +27,6 @@ func TestDefaultExcludes(t *testing.T) {
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
 		},
-		goosWindows: {
-			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
-			"System Volume Information", "$Recycle.Bin", `Windows\Temp`,
-			`Users\*\AppData\Local\Temp`,
-		},
 		"freebsd": nil,
 	}
 
@@ -37,32 +37,11 @@ func TestDefaultExcludes(t *testing.T) {
 	}
 }
 
-func TestIsDefaultExclude(t *testing.T) {
+func TestKernelMounts(t *testing.T) {
-	t.Parallel()
-
-	// A * matches one path component, no more and no fewer.
-	excludes := []string{"/scratch", "/home/*/cache"}
-
-	cases := map[string]bool{
-		"/scratch":        true,
-		"/scratch/x":      false,
-		"/home/u/cache":   true,
-		"/home/u/v/cache": false,
-		"/home/cache":     false,
-	}
-
-	for path, want := range cases {
-		if got := isDefaultExclude(path, excludes); got != want {
-			t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
-		}
-	}
-}
-
-func TestReadMountInfo(t *testing.T) {
 	t.Parallel()
 
 	// The second-to-last line has no optional fields. /mnt/over has an
-	// ext4 filesystem mounted over its tmpfs.
+	// ext4 filesystem mounted over its tmpfs, so it holds files.
 	mountinfo := strings.Join([]string{
 		"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
 		"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
@@ -73,52 +52,54 @@ func TestReadMountInfo(t *testing.T) {
 		"28 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
 	}, "\n")
 
-	got, err := readMountInfo(strings.NewReader(mountinfo))
+	got, err := kernelMounts(strings.NewReader(mountinfo))
 	if err != nil {
 		t.Fatal(err)
 	}
 
 	want := map[string]string{
-		"/": "ext4", "/proc": "proc", "/sys": "sysfs",
+		"/proc": "proc", "/sys": "sysfs", "/mnt/ram disk": "tmpfs",
-		"/mnt/ram disk": "tmpfs", "/mnt/over": "ext4", "/srv/auto": "autofs",
+		"/srv/auto": "autofs",
 	}
 	if !maps.Equal(got, want) {
-		t.Errorf("readMountInfo = %q, want %q", got, want)
+		t.Errorf("kernelMounts = %q, want %q", got, want)
 	}
 
-	_, err = readMountInfo(strings.NewReader("22 1 8:1 / / rw,relatime\n"))
+	_, err = kernelMounts(strings.NewReader("22 1 8:1 / / rw,relatime\n"))
 	if !errors.Is(err, errMountInfoLine) {
-		t.Errorf("readMountInfo of a line without a type = %v, want %v",
+		t.Errorf("kernelMounts of a line without a type = %v, want %v",
 			err, errMountInfoLine)
 	}
 }
 
-func TestNonPersistentFilesystems(t *testing.T) {
+func TestFindDefaultExcludesReadsMounts(t *testing.T) {
 	t.Parallel()
 
 	if runtime.GOOS != goosLinux {
-		t.Skip("the test knows a mount only on Linux: /proc")
+		t.Skip("only Linux lists its mounts in " + mountInfoPath)
 	}
 
-	nonPersistent, err := nonPersistentFilesystems()
+	excludes, err := findDefaultExcludes(goosLinux)
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	dev, ok := deviceOfPath("/proc")
+	for _, dir := range defaultExcludes(goosLinux) {
-	if !ok {
+		if _, ok := excludes[dir]; !ok {
-		t.Fatal("no device for /proc")
+			t.Errorf("%s is not a default exclude", dir)
+		}
 	}
 
-	if got := nonPersistent[dev]; got != "proc" {
+	want := defaultExcludeWhy + ", proc filesystem"
-		t.Errorf("filesystem of /proc = %q, want %q", got, "proc")
+	if got := excludes["/proc"]; got != want {
+		t.Errorf("/proc is %q, want %q", got, want)
 	}
 }
 
 func TestDefaultExcludesHelp(t *testing.T) {
 	t.Parallel()
 
-	for _, goos := range []string{goosLinux, goosDarwin, goosWindows} {
+	for _, goos := range []string{goosLinux, goosDarwin} {
 		help := defaultExcludesHelp(goos)
 
 		for _, dir := range defaultExcludes(goos) {
@@ -127,17 +108,17 @@ func TestDefaultExcludesHelp(t *testing.T) {
 			}
 		}
 
-		// Only Linux and macOS list their mounted filesystems.
+		// Only Linux has filesystem types among its default excludes.
-		for _, fsType := range nonPersistentFilesystemTypes() {
+		for _, fsType := range kernelFilesystemTypes() {
 			listed := strings.Contains(help, " "+fsType+",") ||
 				strings.Contains(help, " "+fsType+"\n")
-			if listed != (goos != goosWindows) {
+			if listed != (goos == goosLinux) {
 				t.Errorf("help for %s lists %s: %v", goos, fsType, listed)
 			}
 		}
 	}
 
-	want := "This system has no default excludes.\n"
+	want := "This system has no default excludes."
 	if got := defaultExcludesHelp("freebsd"); got != want {
 		t.Errorf("help for freebsd = %q, want %q", got, want)
 	}
@@ -183,7 +164,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
 	added := writeFile(t, dir, "added", pattern(3, 100))
 
 	st, err := syncScan(t.Context(), db, []string{dir}, 1, false,
-		walkRules{excludes: []string{excluded}})
+		map[string]string{excluded: defaultExcludeWhy})
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -204,7 +185,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
 		t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
 	}
 
-	line := "\nscan: skipped " + excluded + " (default exclude)\n"
+	line := "\nscan: skipped " + excluded + " (" + defaultExcludeWhy + ")\n"
 	if !strings.Contains(stderr(), line) {
 		t.Errorf("stderr = %q, want %q", stderr(), line)
 	}
@@ -234,7 +215,7 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
 			db := openTestDB(t)
 
 			st, err := syncScan(t.Context(), db, roots, 1, false,
-				walkRules{excludes: []string{excluded}})
+				map[string]string{excluded: defaultExcludeWhy})
 			if err != nil {
 				t.Fatal(err)
 			}
@@ -248,40 +229,11 @@ func TestScanWalksOperandInDefaultExclude(t *testing.T) {
 	}
 }
 
-// TestScanNamesNoDefaultExcludeThatIsAnOperand checks that a default
-// exclude given as an operand under another operand, which the walk of
-// that other operand does not enter, gets no line saying it was
-// skipped: it is walked as an operand of its own.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
-	stderr := captureStderr(t)
-	dir := t.TempDir()
-	db := openTestDB(t)
-	excluded := filepath.Join(dir, "tmp")
-	f := writeFile(t, excluded, "f", pattern(1, 10))
-
-	st, err := syncScan(t.Context(), db, []string{dir, excluded}, 1, false,
-		walkRules{excludes: []string{excluded}})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.added != 1 {
-		t.Errorf("added = %d, want %s recorded once", st.added, f)
-	}
-
-	recordByPath(t, dbRecords(t, db), f)
-
-	if strings.Contains(stderr(), "scan: skipped") {
-		t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
-	}
-}
-
 // TestRunScanTempDirOperand scans a tree in the tests' temporary
 // directory, which on Linux lies inside the default exclude /tmp unless
-// TMPDIR names another place, with the system's own default excludes:
+// TMPDIR names another place, with the system's own default excludes and
-// the scan walks the whole tree and reports no default exclude.
+// then with --no-default-excludes: each scan walks the whole tree and
+// reports no default exclude.
 func TestRunScanTempDirOperand(t *testing.T) {
 	path := testDBPath(t)
 	t.Setenv(databaseEnv, path)
@@ -291,240 +243,24 @@ func TestRunScanTempDirOperand(t *testing.T) {
 		t.Skip(dir + " is not inside a default exclude")
 	}
 
-	f := writeFile(t, dir, "sub/f", pattern(1, 10))
+	for i, flags := range [][]string{nil, {"--no-default-excludes"}} {
+		// A file added before each scan shows that the scan walked the
+		// tree.
+		f := writeFile(t, dir, fmt.Sprintf("sub/%d", i), pattern(1, 10))
 
-	stderr := scanOK(t, dir)
+		stderr := scanOK(t, append(flags, dir)...)
-	if strings.Contains(stderr, "scan: skipped") {
+		if strings.Contains(stderr, defaultExcludeWhy) {
-		t.Errorf("stderr = %q, want no line saying a path was skipped",
+			t.Errorf("scan %q: stderr = %q, want no default exclude",
-			stderr)
+				flags, stderr)
-	}
-
-	assertRecorded(t, path, f, true)
-}
-
-// TestRunScanNoDefaultExcludes scans /var through the command. Its
-// /var/tmp, on a persistent filesystem, is a default exclude on Linux:
-// without --no-default-excludes the walk does not enter it and says so,
-// and with that flag the walk records a file the test puts there.
-func TestRunScanNoDefaultExcludes(t *testing.T) {
-	if runtime.GOOS != goosLinux {
-		t.Skip("only Linux has /var/tmp among its default excludes")
-	}
-
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	rules := walkRules{nonPersistent: nonPersistent}
-
-	for _, dir := range []string{"/var", "/var/tmp"} {
-		fi, lstatErr := os.Lstat(dir)
-		if lstatErr != nil || !fi.IsDir() ||
-			rules.nonPersistentLine(dir, fi) != "" {
-			t.Skip(dir + " is not a directory on a persistent filesystem here")
-		}
-	}
-
-	//nolint:usetesting // the directory must be inside /var/tmp
-	tmp, err := os.MkdirTemp("/var/tmp", "sfdupes-test-")
-	if err != nil {
-		t.Skip("cannot write to /var/tmp here: " + err.Error())
-	}
-
-	t.Cleanup(func() { _ = os.RemoveAll(tmp) })
-
-	f := writeFile(t, tmp, "f", pattern(1, 10))
-
-	path := testDBPath(t)
-	t.Setenv(databaseEnv, path)
-
-	line := "scan: skipped /var/tmp (default exclude)\n"
-
-	if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
-		t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
-	}
-
-	assertRecorded(t, path, f, false)
-
-	stderr := scanOK(t, "--no-default-excludes", "/var")
-	if strings.Contains(stderr, line) {
-		t.Errorf("scan --no-default-excludes /var: stderr = %q, want no %q",
-			stderr, line)
-	}
-
-	assertRecorded(t, path, f, true)
-}
-
-// TestRunScanSkipsNonPersistentOperand scans, through the command, a
-// directory on a filesystem that is not persistent, with and without
-// --no-default-excludes: either way it is skipped, with a line saying
-// so, and nothing in it is recorded.
-func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
-	shm, fsType, _ := shmTempDir(t)
-	writeFile(t, shm, "f", pattern(1, 10))
-
-	t.Setenv(databaseEnv, testDBPath(t))
-
-	line := "scan: skipped " + shm + " (" + fsType +
-		", not a persistent filesystem)\n"
-	summary := "scan: 0 files seen"
-
-	for _, flags := range [][]string{nil, {"--no-default-excludes"}} {
-		stderr := scanOK(t, append(flags, shm)...)
-		if !strings.Contains(stderr, line) ||
-			!strings.Contains(stderr, summary) {
-			t.Errorf("scan %q: stderr = %q, want %q and %q", flags, stderr,
-				line, summary)
-		}
-	}
-}
-
-// TestScanKeepsRecordsOnNonPersistent scans a directory on a filesystem
-// that is not persistent beside one on the tests' temporary directory.
-// The records an earlier scan left in it count as outside the operands:
-// the record of a file gone from it is kept, and its size joins the size
-// census, while a file added to it is not recorded, and its size does
-// not join the census.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
-	shm, fsType, rules := shmTempDir(t)
-	stderr := captureStderr(t)
-	dir := t.TempDir()
-	db := openTestDB(t)
-
-	// A scan that, without the list of filesystems, records gone.
-	gone := writeFile(t, shm, "gone", pattern(1, 100))
-	syncTree(t, db, shm)
-
-	err := os.Remove(gone)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	unseen := writeFile(t, shm, "new", pattern(2, 10))
-
-	// added has the size of gone, and lonely that of unseen: each is
-	// hashed only if the census holds that size.
-	added := writeFile(t, dir, "added", pattern(3, 100))
-	lonely := writeFile(t, dir, "lonely", pattern(4, 10))
-
-	st, err := syncScan(t.Context(), db, []string{shm, dir}, 1, false, rules)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.skipped != 0 {
-		t.Errorf("skipped = %d, want 0", st.skipped)
-	}
-
-	recs := dbRecords(t, db)
-	recordByPath(t, recs, gone)
-
-	if recordByPath(t, recs, added).head == "" {
-		t.Errorf("%s not hashed: the census lacks the size of %s",
-			added, gone)
-	}
-
-	if recordByPath(t, recs, lonely).head != "" {
-		t.Errorf("%s hashed: the census holds the size of %s", lonely, unseen)
-	}
-
-	if slices.Contains(recordPaths(recs), unseen) {
-		t.Errorf("%s recorded", unseen)
-	}
-
-	line := "scan: skipped " + shm + " (" + fsType +
-		", not a persistent filesystem)\n"
-	if !strings.Contains(stderr(), line) {
-		t.Errorf("stderr = %q, want %q", stderr(), line)
-	}
-}
-
-// TestWalkLeavesOutNonPersistent starts the walk itself in a directory
-// on a filesystem that is not persistent, which a scan never does: it
-// drops such an operand first. Below a directory on a persistent
-// filesystem, one on another takes a mount, which a test cannot make.
-// The walk enters no directory and records no file there, and names
-// each.
-func TestWalkLeavesOutNonPersistent(t *testing.T) {
-	t.Parallel()
-
-	shm, fsType, rules := shmTempDir(t)
-	f := writeFile(t, shm, "f", pattern(1, 10))
-	sub := filepath.Dir(writeFile(t, shm, "sub/g", pattern(1, 10)))
-
-	var leftOut []string
-
-	for ev := range startWalk(t.Context(), []string{shm}, false, rules, 1) {
-		if ev.rec.path != "" {
-			t.Errorf("walk recorded %s", ev.rec.path)
 		}
 
-		if ev.leftOut == "" {
+		db, err := openDB(path, reportParams)
-			continue
+		if err != nil {
+			t.Fatal(err)
 		}
 
-		leftOut = append(leftOut, ev.leftOut)
+		recordByPath(t, dbRecords(t, db), f)
 
-		want := "scan: skipped " + ev.leftOut + " (" + fsType +
+		_ = db.Close()
-			", not a persistent filesystem)"
-		if ev.warn != want {
-			t.Errorf("line = %q, want %q", ev.warn, want)
-		}
-	}
-
-	slices.Sort(leftOut)
-
-	if want := []string{f, sub}; !slices.Equal(leftOut, want) {
-		t.Errorf("walk left out %q, want %q", leftOut, want)
-	}
-}
-
-// shmTempDir returns a new directory in /dev/shm, removed when the test
-// ends, the type of its filesystem, and the rules holding the real
-// filesystems that are not persistent. It skips the test where /dev/shm
-// is on a persistent filesystem, or cannot be written to.
-func shmTempDir(t *testing.T) (string, string, walkRules) {
-	t.Helper()
-
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	dev, ok := deviceOfPath("/dev/shm")
-
-	fsType, found := nonPersistent[dev]
-	if !ok || !found {
-		t.Skip("/dev/shm is not a memory-backed filesystem here")
-	}
-
-	//nolint:usetesting // the directory must be inside /dev/shm
-	dir, err := os.MkdirTemp("/dev/shm", "sfdupes-test-")
-	if err != nil {
-		t.Skip("cannot write to /dev/shm here: " + err.Error())
-	}
-
-	t.Cleanup(func() { _ = os.RemoveAll(dir) })
-
-	return dir, fsType, walkRules{nonPersistent: nonPersistent}
-}
-
-// assertRecorded checks that the database at dbPath holds a record for
-// path when want is set, and none when it is not.
-func assertRecorded(t *testing.T, dbPath, path string, want bool) {
-	t.Helper()
-
-	db, err := openDB(dbPath, reportParams)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	defer func() { _ = db.Close() }()
-
-	if got := slices.Contains(recordPaths(dbRecords(t, db)), path); got != want {
-		t.Errorf("%s recorded: %v, want %v", path, got, want)
 	}
 }

main.go

@@ -184,8 +184,7 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 
 // newScanCommand builds the scan subcommand, with its --workers, -x and
 // --no-default-excludes flags. Its help lists the default excludes of
-// the system it runs on, and the filesystem types it never records
+// the system it runs on.
-// anything on.
 func newScanCommand() *cobra.Command {
 	var (
 		scanWorkers           int
@@ -216,7 +215,7 @@ func newScanCommand() *cobra.Command {
 	scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
 		"do not cross filesystem boundaries")
 	scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
-		false, "enter the default excludes, the paths listed above")
+		false, "enter the default excludes")
 
 	return scanCmd
 }

main_test.go

@@ -4,7 +4,6 @@ import (
 	"bytes"
 	"context"
 	"errors"
-	"fmt"
 	"io"
 	"io/fs"
 	"os"
@@ -19,36 +18,6 @@ import (
 // error prints and a runtime failure must not.
 const usageMarker = "Usage:"
 
-// TestMain stops the tests at once, saying why, when their temporary
-// directory is not on a persistent filesystem: scan records nothing
-// there (README "Rules for the walk"), so most scan tests would fail
-// without saying why.
-func TestMain(m *testing.M) {
-	nonPersistent, err := nonPersistentFilesystems()
-	if err != nil {
-		fmt.Fprintf(os.Stderr, "list the mounted filesystems: %v\n", err)
-		os.Exit(1)
-	}
-
-	dir := os.TempDir()
-
-	fi, err := os.Stat(dir)
-	if err != nil {
-		fmt.Fprintln(os.Stderr, err)
-		os.Exit(1)
-	}
-
-	dev, ok := deviceOfInfo(fi)
-	if fsType, found := nonPersistent[dev]; ok && found {
-		fmt.Fprintf(os.Stderr, "%s is on %s, not a persistent filesystem, "+
-			"and scan records nothing there: set TMPDIR to a directory on a "+
-			"persistent filesystem\n", dir, fsType)
-		os.Exit(1)
-	}
-
-	os.Exit(m.Run())
-}
-
 // walSuffixes are the SQLite sidecar files a WAL-mode database keeps
 // while it is open. A clean close checkpoints the WAL and removes
 // both; finding either afterwards means the database was never closed.

scan.go

@@ -115,10 +115,8 @@ func scanPhase(name string) string {
 // SIGTERM that interruptContext catches, the scan keeps what it has
 // hashed (see syncScan), prints how many files its walk reached, and
 // returns errInterrupted. workers must be at least 1; the scan command
-// rejects anything less. Before the lock is taken, the scan lists the
+// rejects anything less. Unless noDefaultExcludes is set, the walk does
-// filesystems that are not persistent, on which the walk records
+// not enter the default excludes, found before the lock is taken.
-// nothing; unless noDefaultExcludes is set, the walk does not enter the
-// default excludes either.
 func runScan(ctx context.Context, roots []string, workers int,
 	oneFS, noDefaultExcludes bool,
 ) error {
@@ -143,14 +141,14 @@ func runScan(ctx context.Context, roots []string, workers int,
 		return err
 	}
 
-	nonPersistent, err := nonPersistentFilesystems()
+	// With no excludes the walk enters every directory.
-	if err != nil {
+	var excludes map[string]string
-		return fmt.Errorf("list the mounted filesystems: %w", err)
-	}
 
-	rules := walkRules{nonPersistent: nonPersistent}
 	if !noDefaultExcludes {
-		rules.excludes = defaultExcludes(runtime.GOOS)
+		excludes, err = findDefaultExcludes(runtime.GOOS)
+		if err != nil {
+			return fmt.Errorf("find the default excludes: %w", err)
+		}
 	}
 
 	lock, err := lockScanDatabase(dbPath)
@@ -173,7 +171,7 @@ func runScan(ctx context.Context, roots []string, workers int,
 
 	defer closeScanDatabase(ctx, db, dbPath)
 
-	st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
+	st, err := syncScan(ctx, db, roots, workers, oneFS, excludes)
 	if errors.Is(err, context.Canceled) {
 		return interrupted(st.walked)
 	}
@@ -250,7 +248,7 @@ func resolveRoots(roots []string) ([]string, error) {
 // above it does not reach it (see walkReaches). Every remaining file is
 // then reachable through exactly one root, so walked paths are unique
 // without keeping a scan-wide set of every path seen.
-func pruneRoots(roots []string, oneFS bool, rules walkRules,
+func pruneRoots(roots []string, oneFS bool, excludes map[string]string,
 	device func(path string) (uint64, bool),
 ) []string {
 	// Shorter-first ordering guarantees an ancestor is kept before any
@@ -276,7 +274,7 @@ func pruneRoots(roots []string, oneFS bool, rules walkRules,
 
 		// A duplicate is dropped, and so is an operand the walk of above
 		// reaches.
-		if above != r && !walkReaches(above, r, oneFS, rules, device) {
+		if above != r && !walkReaches(above, r, oneFS, excludes, device) {
 			kept = append(kept, r)
 		}
 	}
@@ -285,16 +283,23 @@ func pruneRoots(roots []string, oneFS bool, rules walkRules,
 }
 
 // walkReaches reports whether the walk of operand above reaches r, an
-// operand under it. That walk does not enter a default exclude in
+// operand under it. That walk does not enter a directory in excludes,
-// rules, nor a directory that device puts on a filesystem in rules that
+// nor, with -x, one that device puts on a different filesystem than
-// is not persistent, nor, with -x, one that device puts on a different
+// above, when device knows above's. It reaches every regular file in a
-// filesystem than above, when device knows above's. It reaches every
+// directory it enters, whatever device the file reports, so for a
-// regular file in a directory it enters, so for a regular-file operand
+// regular-file operand the directories checked start at its directory;
-// the directories checked start at its directory; they end below above.
+// they end below above.
-func walkReaches(above, r string, oneFS bool, rules walkRules,
+func walkReaches(above, r string, oneFS bool, excludes map[string]string,
 	device func(path string) (uint64, bool),
 ) bool {
-	aboveDev, aboveDevOK := device(above)
+	var (
+		aboveDev    uint64
+		checkDevice bool
+	)
+
+	if oneFS {
+		aboveDev, checkDevice = device(above)
+	}
 
 	dir := r
 
@@ -304,20 +309,15 @@ func walkReaches(above, r string, oneFS bool, rules walkRules,
 	}
 
 	for ; dir != above; dir = filepath.Dir(dir) {
-		if isDefaultExclude(dir, rules.excludes) {
+		if _, ok := excludes[dir]; ok {
 			return false
 		}
 
-		dev, ok := device(dir)
+		if !checkDevice {
-		if !ok {
 			continue
 		}
 
-		if _, found := rules.nonPersistent[dev]; found {
+		if dev, ok := device(dir); ok && dev != aboveDev {
-			return false
-		}
-
-		if oneFS && aboveDevOK && dev != aboveDev {
 			return false
 		}
 	}
@@ -366,11 +366,11 @@ type scanState struct {
 // the hashed records still waiting in the batch, starts no other write
 // or deletion, and returns the cancellation.
 func syncScan(ctx context.Context, db *sql.DB, roots []string,
-	workers int, oneFS bool, rules walkRules,
+	workers int, oneFS bool, excludes map[string]string,
 ) (scanStats, error) {
 	s := &scanState{db: db}
 
-	err := s.runPhases(ctx, roots, workers, oneFS, rules)
+	err := s.runPhases(ctx, roots, workers, oneFS, excludes)
 	if err == nil || ctx.Err() == nil {
 		return s.st, err
 	}
@@ -394,17 +394,16 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
 // and content (fill in the content hash of every record of headTailMin
 // or more whose size, head, and tail match another record's). Records
 // outside the roots are never touched, except that the content phase
-// fills in their content hash. Operands the walk cannot start from, or
+// fills in their content hash. Operands the walk cannot start from are
-// that are not on a persistent filesystem, are dropped first, so the
+// dropped first, so the records beneath them are treated as if those
-// records beneath them are treated as if those operands had not been
+// operands had not been given. Below an operand, the walk does not
-// given. Below an operand, the walk follows rules.
+// enter the directories in excludes.
 func (s *scanState) runPhases(ctx context.Context, roots []string,
-	workers int, oneFS bool, rules walkRules,
+	workers int, oneFS bool, excludes map[string]string,
 ) error {
-	// Types and filesystems are checked before pruning so that an
+	// Types are checked before pruning so that an operand under a
-	// operand under a dropped one is still scanned, not dropped as lying
+	// dropped one is still scanned, not dropped as lying under it.
-	// under it.
+	roots = pruneRoots(s.walkableRoots(roots), oneFS, excludes,
-	roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
 		deviceOfPath)
 
 	err := s.loadIndex(ctx, roots)
@@ -413,7 +412,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 	}
 
 	changed, unhashed := s.walkPhase(
-		startWalk(ctx, roots, oneFS, rules, workers), roots)
+		startWalk(ctx, roots, oneFS, excludes, workers), roots)
 
 	// A cancelled walk stops early, so its size census covers only part
 	// of the roots, and every file it never reached would look vanished
@@ -443,11 +442,9 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 // about, counted as skipped, and dropped. A dropped operand is no
 // longer a root, so the records stored beneath it are treated as if it
 // had not been given: those under no other root count as outside the
-// roots and are not deleted as unverified. An operand on a filesystem
+// roots and are not deleted as unverified. An operand that fails lstat
-// in rules that is not persistent is dropped the same way, with a line
+// here is kept, and the walk warns about it.
-// saying so that is not a warning, and is not counted as skipped. An
+func (s *scanState) walkableRoots(roots []string) []string {
-// operand that fails lstat here is kept, and the walk warns about it.
-func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
 	kept := make([]string, 0, len(roots))
 
 	for _, root := range roots {
@@ -461,13 +458,6 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
 
 				continue
 			}
-
-			line := rules.nonPersistentLine(root, fi)
-			if line != "" {
-				fmt.Fprintln(os.Stderr, escapePath(line))
-
-				continue
-			}
 		}
 
 		kept = append(kept, root)
@@ -522,12 +512,11 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
 // (the content phase may still fill in its content hash). It returns
 // the new-or-changed files and the unchanged files whose records lack
 // hashes; both remain candidates until the census decides whether
-// their sizes are shared. A record at or below a path the walk left
+// their sizes are shared. A record below a directory the walk did not
-// out, because -x stopped at it, it is a default exclude, or it is not
+// enter, because -x stopped at it or it is a default exclude, counts as
-// on a persistent filesystem, counts as outside the roots: it is not
+// outside the roots: it is not deleted, and its size joins the census.
-// deleted, and its size joins the census. That does not hold below a
+// That does not hold below a root at or inside that directory, since
-// root at or inside that path, since that root's own walk verifies the
+// that root's own walk verifies the records below it.
-// records below it; nor is a line printed for such a root.
 func (s *scanState) walkPhase(
 	events <-chan walkEvent, roots []string,
 ) ([]fileRec, []fileRec) {
@@ -545,25 +534,22 @@ func (s *scanState) walkPhase(
 
 	var (
 		changed, unhashed []fileRec
-		leftOut           []string
+		notEntered        []string
 	)
 
 	for ev := range events {
 		if ev.otherFSDir != "" {
-			leftOut = append(leftOut, ev.otherFSDir)
+			notEntered = append(notEntered, ev.otherFSDir)
 
 			continue
 		}
 
-		// The line saying so is printed as a warning is, but what the
+		// The line saying so is printed as a warning is, but a default
-		// walk leaves out is not counted as skipped. A default exclude
+		// exclude is not counted as skipped.
-		// that is itself a root is walked as one, so it gets no line.
+		if ev.excludedDir != "" {
-		if ev.leftOut != "" {
+			notEntered = append(notEntered, ev.excludedDir)
-			leftOut = append(leftOut, ev.leftOut)
 
-			if !slices.Contains(roots, ev.leftOut) {
+			prog.warnf("%s", ev.warn)
-				prog.warnf("%s", ev.warn)
-			}
 
 			continue
 		}
@@ -598,13 +584,12 @@ func (s *scanState) walkPhase(
 	prog.finish()
 
 	for path, old := range s.existing {
-		// Kept only when a path the walk left out lies below the deepest
+		// Kept only when a directory the walk did not enter lies below
-		// root above the record, or is the record's own. A root at or
+		// the deepest root above the record. A root at or below that
-		// below that path was walked on its own and did not find the
+		// directory was walked on its own and did not find the file.
-		// file. Both contain the record, so the longer path is the
+		// Both contain the record, so the longer path is the deeper.
-		// deeper.
+		dir := deepestRoot(path, notEntered)
-		left := deepestRoot(path, leftOut)
+		if len(dir) > len(deepestRoot(path, roots)) {
-		if len(left) > len(deepestRoot(path, roots)) {
 			delete(s.existing, path)
 
 			s.sizes = append(s.sizes, old.size)
@@ -1103,31 +1088,30 @@ type dirJob struct {
 }
 
 // walkEvent is one walk result delivered to the walk phase: a regular
-// file's statted record, a warning when fail is set, or a path the walk
+// file's statted record, a warning when fail is set, or a directory the
-// left out: when otherFSDir is set, a directory -x did not enter
+// walk did not enter: when otherFSDir is set, one -x did not enter
 // because it is on another filesystem than its operand, and when
-// leftOut is set, a default exclude or a directory or file on a
+// excludedDir is set, a default exclude, with warn holding the line
-// filesystem that is not persistent, with warn holding the line that
+// that says so.
-// says so.
 type walkEvent struct {
-	rec        fileRec
+	rec         fileRec
-	warn       string
+	warn        string
-	fail       bool
+	fail        bool
-	otherFSDir string
+	otherFSDir  string
-	leftOut    string
+	excludedDir string
 }
 
 // startWalk seeds every root into the shared walk worker pool and
 // returns the event stream: one record per regular file, one warning
-// event per per-path error, and one event per path -x or rules kept the
+// event per per-path error, and one event per directory -x or excludes
-// walk from entering or recording. The channel is closed when the walk
+// kept the walk from entering. The channel is closed when the walk
 // completes, and also when ctx is cancelled — every goroutine in the
 // pool abandons its blocking send in that case, so the consumer sees a
 // truncated but properly terminated stream instead of a stalled one.
 func startWalk(ctx context.Context, roots []string, oneFS bool,
-	rules walkRules, workers int,
+	excludes map[string]string, workers int,
 ) <-chan walkEvent {
-	jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS, rules)
+	jobs, subdirs, events := startWalkWorkers(ctx, workers, oneFS, excludes)
 
 	go func() {
 		initial := make([]dirJob, 0, len(roots))
@@ -1238,7 +1222,7 @@ func seedRoot(ctx context.Context, root string,
 // workers drop the directories still queued rather than stop reading
 // jobs, so the range always runs out and the pool always tears down.
 func startWalkWorkers(ctx context.Context, workers int,
-	oneFS bool, rules walkRules,
+	oneFS bool, excludes map[string]string,
 ) (chan dirJob, chan []dirJob, chan walkEvent) {
 	jobs := make(chan dirJob, workQueueDepth)
 	subdirs := make(chan []dirJob, workers)
@@ -1254,7 +1238,7 @@ func startWalkWorkers(ctx context.Context, workers int,
 				}
 
 				select {
-				case subdirs <- walkOneDir(ctx, job, oneFS, rules, events):
+				case subdirs <- walkOneDir(ctx, job, oneFS, excludes, events):
 				case <-ctx.Done():
 				}
 			}
@@ -1312,7 +1296,7 @@ func dispatchDirs(ctx context.Context, initial []dirJob,
 // entry and a warning event per unreadable one, and returns the
 // subdirectories to descend into.
 func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
-	rules walkRules, events chan<- walkEvent,
+	excludes map[string]string, events chan<- walkEvent,
 ) []dirJob {
 	entries, err := os.ReadDir(job.path)
 	if err != nil {
@@ -1336,7 +1320,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 		p := filepath.Join(job.path, e.Name())
 
 		if e.IsDir() {
-			sub, ok := subdirJob(ctx, p, e, job, oneFS, rules, events)
+			sub, ok := subdirJob(ctx, p, e, job, oneFS, excludes, events)
 			if ok {
 				subs = append(subs, sub)
 			}
@@ -1349,7 +1333,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 			continue
 		}
 
-		emitFile(ctx, p, e, rules, events)
+		emitFile(ctx, p, e, events)
 	}
 
 	return subs
@@ -1359,11 +1343,9 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
 // The lstat happens here in the walk worker, while the directory's
 // metadata is still hot; a path that fails to stat (or stops being a
 // regular file) between the directory read and the lstat is warned
-// about and skipped. A file on a filesystem in rules that is not
+// about and skipped.
-// persistent, as through a bind mount of one file, is not recorded: it
-// gets an event naming it instead.
 func emitFile(ctx context.Context, p string, e fs.DirEntry,
-	rules walkRules, events chan<- walkEvent,
+	events chan<- walkEvent,
 ) {
 	info, err := e.Info()
 	if err != nil {
@@ -1379,13 +1361,6 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
 		return
 	}
 
-	line := rules.nonPersistentLine(p, info)
-	if line != "" {
-		sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
-
-		return
-	}
-
 	dev, ino := inodeOfInfo(info)
 
 	sendEvent(ctx, events, walkEvent{rec: fileRec{
@@ -1399,20 +1374,33 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
 
 // subdirJob applies the descent rules to directory p: never enter
 // .zfs (ZFS snapshot pseudo-dirs would list every file once per
-// snapshot), never enter a directory on a filesystem in rules that is
+// snapshot), never enter a default exclude in excludes, and with -x
-// not persistent, nor a default exclude in rules, and with -x never
+// never enter a directory on a different filesystem than its operand.
-// enter a directory on a different filesystem than its operand. For the
+// For the last two it sends an event that names the directory, so the
-// last three it sends an event that names the directory, so the scan
+// scan keeps the records below it, except those below an operand at or
-// keeps the records below it, except those below an operand at or
 // inside it.
 func subdirJob(ctx context.Context, p string, e fs.DirEntry,
-	parent dirJob, oneFS bool, rules walkRules,
+	parent dirJob, oneFS bool, excludes map[string]string,
 	events chan<- walkEvent,
 ) (dirJob, bool) {
 	if e.Name() == ".zfs" {
 		return dirJob{}, false
 	}
 
+	if why, ok := excludes[p]; ok {
+		sendEvent(ctx, events, walkEvent{
+			excludedDir: p,
+			warn:        fmt.Sprintf("scan: skipped %s (%s)", p, why),
+		})
+
+		return dirJob{}, false
+	}
+
+	job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
+	if !oneFS || !parent.rootDevOK {
+		return job, true
+	}
+
 	info, err := e.Info()
 	if err != nil {
 		sendEvent(ctx, events, walkEvent{
@@ -1423,27 +1411,6 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
 		return dirJob{}, false
 	}
 
-	line := rules.nonPersistentLine(p, info)
-	if line != "" {
-		sendEvent(ctx, events, walkEvent{leftOut: p, warn: line})
-
-		return dirJob{}, false
-	}
-
-	if isDefaultExclude(p, rules.excludes) {
-		sendEvent(ctx, events, walkEvent{
-			leftOut: p,
-			warn:    fmt.Sprintf("scan: skipped %s (default exclude)", p),
-		})
-
-		return dirJob{}, false
-	}
-
-	job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
-	if !oneFS || !parent.rootDevOK {
-		return job, true
-	}
-
 	if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
 		sendEvent(ctx, events, walkEvent{otherFSDir: p})
 

scan_test.go

@@ -782,8 +782,7 @@ func collectWalk(t *testing.T, roots []string, oneFS bool,
 		errs int
 	)
 
-	for ev := range startWalk(t.Context(), roots, oneFS, walkRules{},
+	for ev := range startWalk(t.Context(), roots, oneFS, nil, workers) {
-		workers) {
 		if ev.fail {
 			errs++
 
@@ -1005,8 +1004,7 @@ func callSubdirJob(t *testing.T, p string, e fs.DirEntry,
 	t.Helper()
 
 	events := make(chan walkEvent, 1)
-	job, ok := subdirJob(t.Context(), p, e, parent, oneFS, walkRules{},
+	job, ok := subdirJob(t.Context(), p, e, parent, oneFS, nil, events)
-		events)
 	close(events)
 
 	var evs []walkEvent
@@ -1154,7 +1152,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
 		return dev, ok
 	}
 
-	roots = pruneRoots(roots, true, walkRules{}, device)
+	roots = pruneRoots(roots, true, nil, device)
 
 	ctx := t.Context()
 	s := &scanState{db: db}
@@ -1164,7 +1162,7 @@ func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
 		t.Fatal(err)
 	}
 
-	jobs, subdirs, events := startWalkWorkers(ctx, 2, true, walkRules{})
+	jobs, subdirs, events := startWalkWorkers(ctx, 2, true, nil)
 
 	// seedRoot emits a regular-file operand itself and returns a job
 	// only for a directory operand.
@@ -1465,7 +1463,7 @@ const smokeTreeFiles = 15
 func syncTree(t *testing.T, db *sql.DB, roots ...string) scanStats {
 	t.Helper()
 
-	st, err := syncScan(t.Context(), db, roots, 4, false, walkRules{})
+	st, err := syncScan(t.Context(), db, roots, 4, false, nil)
 	if err != nil {
 		t.Fatalf("syncScan: %v", err)
 	}
@@ -2414,7 +2412,7 @@ func TestPruneRoots(t *testing.T) {
 	// Duplicates and operands under other operands are dropped; /cc is
 	// not under /c (sibling with a shared prefix).
 	got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
-		false, walkRules{}, device)
+		false, nil, device)
 
 	want := []string{"/a", "/c", "/cc"}
 	if !slices.Equal(got, want) {
@@ -2423,7 +2421,7 @@ func TestPruneRoots(t *testing.T) {
 
 	nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
 
-	got = pruneRoots(nested, false, walkRules{}, device)
+	got = pruneRoots(nested, false, nil, device)
 
 	want = []string{"/p"}
 	if !slices.Equal(got, want) {
@@ -2431,7 +2429,7 @@ func TestPruneRoots(t *testing.T) {
 	}
 
 	// With -x, /p/q, on another filesystem than /p, is kept as well.
-	got = pruneRoots(nested, true, walkRules{}, device)
+	got = pruneRoots(nested, true, nil, device)
 
 	want = []string{"/p", pqRoot}
 	if !slices.Equal(got, want) {
@@ -2440,22 +2438,11 @@ func TestPruneRoots(t *testing.T) {
 
 	// The walk of /p does not enter /p/q when it is a default exclude, so
 	// /p/q is kept, without -x too; /p/q/r lies under it and is dropped.
-	got = pruneRoots(nested, false, walkRules{excludes: []string{pqRoot}},
+	got = pruneRoots(nested, false,
-		device)
+		map[string]string{pqRoot: defaultExcludeWhy}, device)
 	if !slices.Equal(got, want) {
 		t.Fatalf("pruneRoots excluding %s = %q, want %q", pqRoot, got, want)
 	}
-
-	// Nor does it enter /p/q when the filesystem of /p/q is not
-	// persistent, so /p/q/r/m, of no known device, is kept.
-	got = pruneRoots([]string{"/p", "/p/q/r/m"}, false,
-		walkRules{nonPersistent: map[uint64]string{2: "tmpfs"}}, device)
-
-	want = []string{"/p", "/p/q/r/m"}
-	if !slices.Equal(got, want) {
-		t.Fatalf("pruneRoots with %s not persistent = %q, want %q",
-			pqRoot, got, want)
-	}
 }
 
 func TestReportsNeverTouchFilesystem(t *testing.T) {