scan: skip OS default excludes, and record nothing on filesystems that are not persistent (closes #124)

scan never records anything on a filesystem that is not persistent
(tmpfs, proc, sysfs and the other kernel or memory-backed types). As it
starts it notes the device ID of each such mount point, from
/proc/self/mountinfo on Linux and the mounted-filesystem list on macOS,
and checks every operand, directory and file the walk meets against
them; no flag turns this off. Below an operand the walk also skips the
default excludes, a path list per system (Linux, macOS, and Windows,
which is not used yet), and --no-default-excludes turns off only those.
A default exclude that is itself an operand gets no skip line. The tests
stop at once, saying why, when TMPDIR is not on a persistent filesystem.

Judgement call: an unreadable mount list fails the scan, and no flag gets round it.
Judgement call: a file on such a filesystem below an operand gets the same line as a directory.
Judgement call: an operand on such a filesystem that another operand's walk also reaches is named twice.
Unverified: the macOS mount list is built for macOS but never run; bind mounts need root to test.

Model: opus-5-5

README.md

@@ -496,9 +496,8 @@ operands:
   can be scanned on different schedules into the same database. The one
   exception is the content phase below: a stored file of 10 MiB or more without
   a `content` hash is read for one, wherever it lies, once its size, `head`, and
-  `tail` match another record's. If that file is gone, has changed since its
-  record was written, or is not on a persistent filesystem, it is not read and
-  the record is left as it is.
+  `tail` match another record's. If that file is gone or has changed since its
+  record was written, the record is left as it is.
 - A record at or below a path the walk left out, because that path is a default
   exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
   another filesystem than its operand, counts as outside the scanned operands:
@@ -551,20 +550,19 @@ the hash phase:
     be read are kept in memory, never every file's hashes. Every record sharing
     their size, `head`, and `tail`, including one that already has a `content`
     hash, has its file checked with `lstat` first. A file that is gone, is no
-    longer a regular file, has changed (a different size, or an mtime newer than
-    recorded), or is not on a persistent filesystem (see "Rules for the walk")
-    keeps its record as it is and does not count as a match for the others. Any
-    other `lstat` error is warned about and counted as skipped, with the same
-    result. If such a record has no `content` hash, it stays out of duplicate
-    groups; if it has one, it is still reported until a scan covering its own
-    tree updates or removes it. The files that pass and have no `content` hash
-    are read only if at least two of those records pass, so a file whose only
-    matches are stale costs no read; a file that already has a `content` hash is
-    never read again. They are read by a worker pool as in the hash phase, in
-    inode order and once per inode, and their content hashes are committed in
-    batches. A failed read is warned about and counted as skipped; its record
-    keeps an empty `content`, so it is not a duplicate, and a later scan tries
-    again.
+    longer a regular file, or has changed (a different size, or an mtime newer
+    than recorded) keeps its record as it is and does not count as a match for
+    the others. Any other `lstat` error is warned about and counted as skipped,
+    with the same result. If such a record has no `content` hash, it stays out
+    of duplicate groups; if it has one, it is still reported until a scan
+    covering its own tree updates or removes it. The files that pass and have no
+    `content` hash are read only if at least two of those records pass, so a
+    file whose only matches are stale costs no read; a file that already has a
+    `content` hash is never read again. They are read by a worker pool as in the
+    hash phase, in inode order and once per inode, and their content hashes are
+    committed in batches. A failed read is warned about and counted as skipped;
+    its record keeps an empty `content`, so it is not a duplicate, and a later
+    scan tries again.
 
 Rules for the walk:
 
@@ -591,18 +589,15 @@ Rules for the walk:
   it names the database and before it takes the lock, `scan` lists the mounted
   filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
   of mounted filesystems on macOS (on any other system it lists none), and notes
-  the device ID of each one of those types. On Linux that is the device ID the
-  list gives, so a filesystem that a later mount hides still gives its own. On
-  macOS it is read with `lstat` on the mount point, and where one filesystem is
-  mounted over another on the same mount point, the one mounted last counts. If
-  `scan` cannot list them, it fails (exit 1). The walk then never enters a
-  directory, and never records a file, on one of those devices, however it
-  reaches it: as an operand, below one, or through a bind mount, which keeps the
-  device ID of what it binds. The content phase never reads a file on one of
-  them either. No flag turns this off. For each operand, directory or file it
-  leaves out this way, `scan` prints a line on stderr naming it and the type of
-  its filesystem, as it prints a warning, so it never lands inside a progress
-  display:
+  the device ID of the mount point of each one of those types, read with
+  `lstat`. Where one filesystem is mounted over another, the one mounted last
+  counts. If `scan` cannot list them, it fails (exit 1). The walk then never
+  enters a directory, and never records a file, on one of those devices, however
+  it reaches it: as an operand, below one, or through a bind mount, which keeps
+  the device ID of what it binds. No flag turns this off. For each operand,
+  directory or file it leaves out this way, `scan` prints a line on stderr
+  naming it and the type of its filesystem, as it prints a warning, so it never
+  lands inside a progress display:
 
     ```
     scan: skipped /dev/shm (tmpfs, not a persistent filesystem)

TODO.md

@@ -29,21 +29,20 @@
 
 # Completed Steps
 
-- `scan` never records or reads anything that is not on a persistent filesystem
-  (`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
-  matched by the device ID of each such filesystem, listed as the scan starts
-  from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
-  however the walk or the content phase reaches it; no flag turns that off.
-  Below an operand, its walk does not enter the default excludes either, so
-  `sfdupes scan /` skips the kernel's view of the running system and the
-  temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
-  `/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
-  `/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
-  Windows list, not used yet. An operand at or inside a default exclude is still
-  walked. The scan prints a line for each path it leaves out, without counting
-  it as skipped, and the records there count as outside the scanned operands, as
-  for `-x`. `--no-default-excludes` turns off the default excludes only, and
-  `scan --help` lists them (2026-10-17,
+- `scan` never records anything that is not on a persistent filesystem (`tmpfs`,
+  `proc`, `sysfs` and the other kernel or memory-backed types), matched by the
+  device ID of each such mount point, listed as the scan starts from
+  `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
+  however the walk reaches it; no flag turns that off. Below an operand, its
+  walk does not enter the default excludes either, so `sfdupes scan /` skips the
+  kernel's view of the running system and the temporary directories: on Linux
+  `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`, on macOS `/dev`,
+  `/private/tmp`, `/private/var/tmp`, `/private/var/vm`, `/System/Volumes/VM`
+  and `/System/Volumes/Data`, and a Windows list, not used yet. An operand at or
+  inside a default exclude is still walked. The scan prints a line for each path
+  it leaves out, without counting it as skipped, and the records there count as
+  outside the scanned operands, as for `-x`. `--no-default-excludes` turns off
+  the default excludes only, and `scan --help` lists them (2026-10-17,
   https://git.eeqj.de/sneak/sfdupes/issues/124)
 
 - the rate a progress display shows covers the items completed over the last 5

exclude.go

@@ -3,8 +3,8 @@ package main
 // This file holds two rules for the walk (README "Rules for the walk"):
 // the default excludes, the paths a scan's walk does not enter below an
 // operand unless scan is given --no-default-excludes, and the
-// filesystems that are not persistent, on which the scan never records
-// or reads anything.
+// filesystems that are not persistent, on which the walk never records
+// anything.
 
 import (
 	"bufio"
@@ -15,8 +15,6 @@ import (
 	"path/filepath"
 	"slices"
 	"strings"
-
-	"golang.org/x/sys/unix"
 )
 
 // The systems that have default excludes, as runtime.GOOS names them.
@@ -27,11 +25,11 @@ const (
 )
 
 // The fields of a line of mountInfoPath that readMountInfo reads,
-// counted from 0 (see proc(5)): the device ID, as major:minor, is the
-// third field, and after the sixth come optional fields, then a field
-// holding "-", then the filesystem type.
+// counted from 0 (see proc(5)): the mount point is the fifth field, and
+// after the sixth come optional fields, then a field holding "-", then
+// the filesystem type.
 const (
-	deviceField         = 2
+	mountPointField     = 4
 	optionalFieldsStart = 6
 )
 
@@ -46,8 +44,8 @@ var errMountInfoLine = errors.New("unexpected line")
 // walkRules holds the two rules for the walk that a scan sets up as it
 // starts. excludes are the default excludes, or none with
 // --no-default-excludes. nonPersistent maps the device ID of every
-// mounted filesystem that is not persistent to its type; the scan
-// records and reads nothing on one of those, whatever the flags.
+// mounted filesystem that is not persistent to its type; the walk
+// records nothing on one of those, whatever the flags.
 type walkRules struct {
 	excludes      []string
 	nonPersistent map[uint64]string
@@ -107,6 +105,32 @@ func nonPersistentFilesystemTypes() []string {
 	}
 }
 
+// nonPersistentFilesystems returns the device ID of every mounted
+// filesystem that is not persistent, with its type, read with lstat on
+// its mount point. mountedFilesystems lists the mounts. A mount point
+// that lstat cannot reach is left out.
+func nonPersistentFilesystems() (map[uint64]string, error) {
+	mounts, err := mountedFilesystems()
+	if err != nil {
+		return nil, err
+	}
+
+	types := nonPersistentFilesystemTypes()
+	devices := make(map[uint64]string)
+
+	for mountPoint, fsType := range mounts {
+		if !slices.Contains(types, fsType) {
+			continue
+		}
+
+		if dev, ok := deviceOfPath(mountPoint); ok {
+			devices[dev] = fsType
+		}
+	}
+
+	return devices, nil
+}
+
 // nonPersistentLine returns the line the scan prints for path when fi,
 // its lstat, puts it on a filesystem in rules.nonPersistent, and ""
 // when it is on any other.
@@ -126,13 +150,16 @@ func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
 }
 
 // readMountInfo reads mountinfo, laid out as mountInfoPath is, and
-// returns the device ID of every mounted filesystem that is not
-// persistent, with its type. The device ID is the one each line gives,
-// not that of the file now at its mount point, so a mount that a later
-// mount hides never adds the device of another filesystem.
-func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
-	types := nonPersistentFilesystemTypes()
-	devices := make(map[uint64]string)
+// returns every mount point with the type of its filesystem. Where one
+// filesystem is mounted over another, a later line names the one on
+// top, so the last line for a mount point decides.
+func readMountInfo(mountinfo io.Reader) (map[string]string, error) {
+	// The kernel writes a space, tab, newline or backslash in a mount
+	// point as an octal escape.
+	unescape := strings.NewReplacer(
+		`\040`, " ", `\011`, "\t", `\012`, "\n", `\134`, `\`)
+
+	mounts := make(map[string]string)
 	lines := bufio.NewScanner(mountinfo)
 
 	for lines.Scan() {
@@ -143,20 +170,10 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
 			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
 		}
 
-		var major, minor uint32
-
-		_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
-		if err != nil {
-			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
-		}
-
-		fsType := fields[sep+1]
-		if slices.Contains(types, fsType) {
-			devices[unix.Mkdev(major, minor)] = fsType
-		}
+		mounts[unescape.Replace(fields[mountPointField])] = fields[sep+1]
 	}
 
-	return devices, lines.Err()
+	return mounts, lines.Err()
 }
 
 // defaultExcludesHelp describes, for the scan command's help, the

exclude_darwin.go

@@ -2,19 +2,12 @@
 
 package main
 
-import (
-	"slices"
+import "golang.org/x/sys/unix"
 
-	"golang.org/x/sys/unix"
-)
-
-// nonPersistentFilesystems returns the device ID of every mounted
-// filesystem that is not persistent, with its type, from the system's
-// list of mounted filesystems. The device ID is read with lstat on the
-// mount point. Where one filesystem is mounted over another on the same
-// mount point, the later one in the list counts. A mount point that
-// lstat cannot reach is left out.
-func nonPersistentFilesystems() (map[uint64]string, error) {
+// mountedFilesystems returns every mount point with the type of its
+// filesystem, from the system's list of mounted filesystems. Where one
+// filesystem is mounted over another, the later one in the list counts.
+func mountedFilesystems() (map[string]string, error) {
 	// The first call counts the filesystems, the second lists them.
 	n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
 	if err != nil {
@@ -35,18 +28,5 @@ func nonPersistentFilesystems() (map[uint64]string, error) {
 			unix.ByteSliceToString(st.Fstypename[:])
 	}
 
-	types := nonPersistentFilesystemTypes()
-	devices := make(map[uint64]string)
-
-	for mountPoint, fsType := range mounts {
-		if !slices.Contains(types, fsType) {
-			continue
-		}
-
-		if dev, ok := deviceOfPath(mountPoint); ok {
-			devices[dev] = fsType
-		}
-	}
-
-	return devices, nil
+	return mounts, nil
 }

exclude_other.go

@@ -12,12 +12,12 @@ import (
 // process sees.
 const mountInfoPath = "/proc/self/mountinfo"
 
-// nonPersistentFilesystems returns the device ID of every mounted
-// filesystem that is not persistent, with its type: on Linux read from
-// mountInfoPath (see readMountInfo), and on any other system none.
-func nonPersistentFilesystems() (map[uint64]string, error) {
+// mountedFilesystems returns every mount point with the type of its
+// filesystem: on Linux read from mountInfoPath (see readMountInfo), and
+// on any other system none.
+func mountedFilesystems() (map[string]string, error) {
 	if runtime.GOOS != goosLinux {
-		return map[uint64]string{}, nil
+		return map[string]string{}, nil
 	}
 
 	f, err := os.Open(mountInfoPath)
@@ -27,10 +27,10 @@ func nonPersistentFilesystems() (map[uint64]string, error) {
 
 	defer func() { _ = f.Close() }()
 
-	devices, err := readMountInfo(f)
+	mounts, err := readMountInfo(f)
 	if err != nil {
 		return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
 	}
 
-	return devices, nil
+	return mounts, nil
 }

exclude_test.go

@@ -10,8 +10,6 @@ import (
 	"slices"
 	"strings"
 	"testing"
-
-	"golang.org/x/sys/unix"
 )
 
 func TestDefaultExcludes(t *testing.T) {
@@ -63,17 +61,16 @@ func TestIsDefaultExclude(t *testing.T) {
 func TestReadMountInfo(t *testing.T) {
 	t.Parallel()
 
-	// The second-to-last line has no optional fields. The tmpfs on
-	// /data/a/b is hidden by the root filesystem's /srv, mounted over
-	// /data/a after it: the tmpfs still gives its own device, and the
-	// root filesystem's device is not listed.
+	// The second-to-last line has no optional fields. /mnt/over has an
+	// ext4 filesystem mounted over its tmpfs.
 	mountinfo := strings.Join([]string{
 		"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
 		"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
 		"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
-		"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw",
-		"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw",
-		"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
+		`25 22 0:23 / /mnt/ram\040disk rw shared:2 - tmpfs none rw`,
+		"26 22 0:24 / /mnt/over rw shared:3 - tmpfs none rw",
+		"27 26 8:2 / /mnt/over rw - ext4 /dev/sda2 rw",
+		"28 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
 	}, "\n")
 
 	got, err := readMountInfo(strings.NewReader(mountinfo))
@@ -81,23 +78,18 @@ func TestReadMountInfo(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	want := map[uint64]string{
-		unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs",
-		unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs",
+	want := map[string]string{
+		"/": "ext4", "/proc": "proc", "/sys": "sysfs",
+		"/mnt/ram disk": "tmpfs", "/mnt/over": "ext4", "/srv/auto": "autofs",
 	}
 	if !maps.Equal(got, want) {
-		t.Errorf("readMountInfo = %v, want %v", got, want)
+		t.Errorf("readMountInfo = %q, want %q", got, want)
 	}
 
-	for _, line := range []string{
-		"22 1 8:1 / / rw,relatime\n",
-		"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n",
-	} {
-		_, err = readMountInfo(strings.NewReader(line))
-		if !errors.Is(err, errMountInfoLine) {
-			t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
-				errMountInfoLine)
-		}
+	_, err = readMountInfo(strings.NewReader("22 1 8:1 / / rw,relatime\n"))
+	if !errors.Is(err, errMountInfoLine) {
+		t.Errorf("readMountInfo of a line without a type = %v, want %v",
+			err, errMountInfoLine)
 	}
 }
 
@@ -450,51 +442,6 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
 	}
 }
 
-// TestScanContentSkipsNonPersistent checks that the content phase does
-// not read a file on a filesystem that is not persistent, though an
-// earlier scan left a record for it: it is treated as a changed file,
-// so its record is left as it is and its identical copy in another
-// tree, having no other match, is not read either.
-func TestScanContentSkipsNonPersistent(t *testing.T) {
-	t.Parallel()
-
-	shm, _, rules := shmTempDir(t)
-	db := openTestDB(t)
-
-	// A scan that, without the list of filesystems, records a with a
-	// head and tail but no content hash.
-	a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
-
-	syncTree(t, db, shm)
-
-	before := recordByPath(t, dbRecords(t, db), a)
-	if before.head == "" || before.content != "" {
-		t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
-	}
-
-	b := sparseFile(t, t.TempDir(), "b", headTailMin)
-
-	st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
-		rules)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if st.skipped != 0 {
-		t.Errorf("skipped = %d, want 0", st.skipped)
-	}
-
-	recs := dbRecords(t, db)
-	if r := recordByPath(t, recs, a); r != before {
-		t.Errorf("record = %+v, want it left as %+v", r, before)
-	}
-
-	if r := recordByPath(t, recs, b); r.content != "" {
-		t.Errorf("%s: content = %q, want none: its only match is on %s",
-			b, r.content, shm)
-	}
-}
-
 // TestWalkLeavesOutNonPersistent starts the walk itself in a directory
 // on a filesystem that is not persistent, which a scan never does: it
 // drops such an operand first. Below a directory on a persistent

scan.go

@@ -435,7 +435,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 		return err
 	}
 
-	return s.contentPhase(ctx, workers, rules)
+	return s.contentPhase(ctx, workers)
 }
 
 // walkableRoots returns the operands the walk can start from: regular
@@ -878,12 +878,9 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 // hash phase's worker pool and their records written back in batches. A
 // failed read is warned about and counted as skipped; the record keeps
 // its empty content, so it is never grouped, and a later scan tries
-// again. No file on a filesystem in rules that is not persistent is
-// read.
-func (s *scanState) contentPhase(ctx context.Context, workers int,
-	rules walkRules,
-) error {
-	toRead, recs, err := s.contentCandidates(ctx, rules)
+// again.
+func (s *scanState) contentPhase(ctx context.Context, workers int) error {
+	toRead, recs, err := s.contentCandidates(ctx)
 	if err != nil {
 		return err
 	}
@@ -918,19 +915,18 @@ func (s *scanState) contentPhase(ctx context.Context, workers int,
 // contentCandidates returns the files the content phase reads, and
 // their records by path. Every record contentCandidatesSQL returns has
 // its file checked with lstat, whether or not it already has a content
-// hash: a file that is gone, is no longer a regular file, has changed
-// by the walk's rule, or is on a filesystem in rules that is not
-// persistent keeps its record as it is and does not count as a match
-// for the others, and any other lstat error is warned about and counted
-// as skipped, with the same result. If such a record has no content
-// hash, it stays out of duplicate groups; if it has one, it is still
-// reported until a scan covering its own tree updates or removes it.
-// The files of a group that pass and have no content hash are read only
-// if at least minGroupSize of the group's files pass, so a group whose
-// other members are all stale costs no reads. Only the records to be
-// read are kept.
+// hash: a file that is gone, is no longer a regular file, or has
+// changed by the walk's rule keeps its record as it is and does not
+// count as a match for the others, and any other lstat error is warned
+// about and counted as skipped, with the same result. If such a record
+// has no content hash, it stays out of duplicate groups; if it has one,
+// it is still reported until a scan covering its own tree updates or
+// removes it. The files of a group that pass and have no content hash
+// are read only if at least minGroupSize of the group's files pass, so
+// a group whose other members are all stale costs no reads. Only the
+// records to be read are kept.
 func (s *scanState) contentCandidates(
-	ctx context.Context, rules walkRules,
+	ctx context.Context,
 ) ([]fileRec, map[string]scanRec, error) {
 	// The search and the checks take real time on a large database;
 	// without a display the scan looks hung before the reads begin.
@@ -988,7 +984,7 @@ func (s *scanState) contentCandidates(
 			first = r
 		}
 
-		f, ok, err := unchangedFile(r, rules)
+		f, ok, err := unchangedFile(r)
 		if err != nil {
 			s.st.skipped++
 
@@ -1022,10 +1018,9 @@ func (s *scanState) contentCandidates(
 
 // unchangedFile lstats the file r names and returns it for reading if
 // it is still the regular file r records: the same size, and an mtime
-// no newer than recorded (the walk's change rule). A file that is gone,
-// has changed, or is on a filesystem in rules that is not persistent
-// reports false; any other lstat error is returned.
-func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
+// no newer than recorded (the walk's change rule). A file that is gone
+// or has changed reports false; any other lstat error is returned.
+func unchangedFile(r scanRec) (fileRec, bool, error) {
 	fi, err := os.Lstat(r.path)
 	if errors.Is(err, fs.ErrNotExist) {
 		return fileRec{}, false, nil
@@ -1042,12 +1037,6 @@ func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
 
 	dev, ino := inodeOfInfo(fi)
 
-	// The scan reads nothing on a filesystem that is not persistent,
-	// whatever record an earlier scan left for the file.
-	if _, found := rules.nonPersistent[dev]; found {
-		return fileRec{}, false, nil
-	}
-
 	return fileRec{
 		path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
 	}, true, nil