scan: skip OS default excludes, and record nothing on filesystems that are not persistent (closes #124)

scan never records or reads anything on a filesystem that is not
persistent (tmpfs, proc, sysfs and the like), however it reaches it; no
flag turns this off. It notes each such filesystem's device ID as it
starts: on Linux from /proc/self/mountinfo, on macOS by lstat of each
listed mount point. Below an operand the walk also skips the default
excludes, a path list per system (Linux, macOS, and Windows, not used
yet); --no-default-excludes turns off only those. A default exclude that
is itself an operand gets no skip line. The tests stop at once, saying
why, when TMPDIR is not on a persistent filesystem.

Judgement call: an unreadable mount list fails the scan, and no flag gets round it.
Judgement call: a file on such a filesystem below an operand gets the same line as a directory.
Judgement call: such an operand that another operand's walk also reaches is named twice.
Unverified: whether macOS's mount list gives the device lstat reports, so macOS keeps lstat, and a hidden mount there can still add another filesystem's device.
Unverified: the macOS code is never built or run here; bind mounts need root to test.

Model: opus-5-5

README.md

@@ -496,8 +496,9 @@ operands:
   can be scanned on different schedules into the same database. The one
   exception is the content phase below: a stored file of 10 MiB or more without
   a `content` hash is read for one, wherever it lies, once its size, `head`, and
-  `tail` match another record's. If that file is gone or has changed since its
+  `tail` match another record's. If that file is gone, has changed since its
-  record was written, the record is left as it is.
+  record was written, or is not on a persistent filesystem, it is not read and
+  the record is left as it is.
 - A record at or below a path the walk left out, because that path is a default
   exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
   another filesystem than its operand, counts as outside the scanned operands:
@@ -550,19 +551,20 @@ the hash phase:
     be read are kept in memory, never every file's hashes. Every record sharing
     their size, `head`, and `tail`, including one that already has a `content`
     hash, has its file checked with `lstat` first. A file that is gone, is no
-    longer a regular file, or has changed (a different size, or an mtime newer
+    longer a regular file, has changed (a different size, or an mtime newer than
-    than recorded) keeps its record as it is and does not count as a match for
+    recorded), or is not on a persistent filesystem (see "Rules for the walk")
-    the others. Any other `lstat` error is warned about and counted as skipped,
+    keeps its record as it is and does not count as a match for the others. Any
-    with the same result. If such a record has no `content` hash, it stays out
+    other `lstat` error is warned about and counted as skipped, with the same
-    of duplicate groups; if it has one, it is still reported until a scan
+    result. If such a record has no `content` hash, it stays out of duplicate
-    covering its own tree updates or removes it. The files that pass and have no
+    groups; if it has one, it is still reported until a scan covering its own
-    `content` hash are read only if at least two of those records pass, so a
+    tree updates or removes it. The files that pass and have no `content` hash
-    file whose only matches are stale costs no read; a file that already has a
+    are read only if at least two of those records pass, so a file whose only
-    `content` hash is never read again. They are read by a worker pool as in the
+    matches are stale costs no read; a file that already has a `content` hash is
-    hash phase, in inode order and once per inode, and their content hashes are
+    never read again. They are read by a worker pool as in the hash phase, in
-    committed in batches. A failed read is warned about and counted as skipped;
+    inode order and once per inode, and their content hashes are committed in
-    its record keeps an empty `content`, so it is not a duplicate, and a later
+    batches. A failed read is warned about and counted as skipped; its record
-    scan tries again.
+    keeps an empty `content`, so it is not a duplicate, and a later scan tries
+    again.
 
 Rules for the walk:
 
@@ -589,15 +591,18 @@ Rules for the walk:
   it names the database and before it takes the lock, `scan` lists the mounted
   filesystems, from `/proc/self/mountinfo` on Linux and from the system's list
   of mounted filesystems on macOS (on any other system it lists none), and notes
-  the device ID of the mount point of each one of those types, read with
+  the device ID of each one of those types. On Linux that is the device ID the
-  `lstat`. Where one filesystem is mounted over another, the one mounted last
+  list gives, so a filesystem that a later mount hides still gives its own. On
-  counts. If `scan` cannot list them, it fails (exit 1). The walk then never
+  macOS it is read with `lstat` on the mount point, and where one filesystem is
-  enters a directory, and never records a file, on one of those devices, however
+  mounted over another on the same mount point, the one mounted last counts. If
-  it reaches it: as an operand, below one, or through a bind mount, which keeps
+  `scan` cannot list them, it fails (exit 1). The walk then never enters a
-  the device ID of what it binds. No flag turns this off. For each operand,
+  directory, and never records a file, on one of those devices, however it
-  directory or file it leaves out this way, `scan` prints a line on stderr
+  reaches it: as an operand, below one, or through a bind mount, which keeps the
-  naming it and the type of its filesystem, as it prints a warning, so it never
+  device ID of what it binds. The content phase never reads a file on one of
-  lands inside a progress display:
+  them either. No flag turns this off. For each operand, directory or file it
+  leaves out this way, `scan` prints a line on stderr naming it and the type of
+  its filesystem, as it prints a warning, so it never lands inside a progress
+  display:
 
     ```
     scan: skipped /dev/shm (tmpfs, not a persistent filesystem)

TODO.md

@@ -29,20 +29,21 @@
 
 # Completed Steps
 
-- `scan` never records anything that is not on a persistent filesystem (`tmpfs`,
+- `scan` never records or reads anything that is not on a persistent filesystem
-  `proc`, `sysfs` and the other kernel or memory-backed types), matched by the
+  (`tmpfs`, `proc`, `sysfs` and the other kernel or memory-backed types),
-  device ID of each such mount point, listed as the scan starts from
+  matched by the device ID of each such filesystem, listed as the scan starts
-  `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
+  from `/proc/self/mountinfo` on Linux and the mounted-filesystem list on macOS,
-  however the walk reaches it; no flag turns that off. Below an operand, its
+  however the walk or the content phase reaches it; no flag turns that off.
-  walk does not enter the default excludes either, so `sfdupes scan /` skips the
+  Below an operand, its walk does not enter the default excludes either, so
-  kernel's view of the running system and the temporary directories: on Linux
+  `sfdupes scan /` skips the kernel's view of the running system and the
-  `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and `/var/tmp`, on macOS `/dev`,
+  temporary directories: on Linux `/proc`, `/sys`, `/dev`, `/run`, `/tmp` and
-  `/private/tmp`, `/private/var/tmp`, `/private/var/vm`, `/System/Volumes/VM`
+  `/var/tmp`, on macOS `/dev`, `/private/tmp`, `/private/var/tmp`,
-  and `/System/Volumes/Data`, and a Windows list, not used yet. An operand at or
+  `/private/var/vm`, `/System/Volumes/VM` and `/System/Volumes/Data`, and a
-  inside a default exclude is still walked. The scan prints a line for each path
+  Windows list, not used yet. An operand at or inside a default exclude is still
-  it leaves out, without counting it as skipped, and the records there count as
+  walked. The scan prints a line for each path it leaves out, without counting
-  outside the scanned operands, as for `-x`. `--no-default-excludes` turns off
+  it as skipped, and the records there count as outside the scanned operands, as
-  the default excludes only, and `scan --help` lists them (2026-10-17,
+  for `-x`. `--no-default-excludes` turns off the default excludes only, and
+  `scan --help` lists them (2026-10-17,
   https://git.eeqj.de/sneak/sfdupes/issues/124)
 
 - the rate a progress display shows covers the items completed over the last 5

exclude.go

@@ -3,8 +3,8 @@ package main
 // This file holds two rules for the walk (README "Rules for the walk"):
 // the default excludes, the paths a scan's walk does not enter below an
 // operand unless scan is given --no-default-excludes, and the
-// filesystems that are not persistent, on which the walk never records
+// filesystems that are not persistent, on which the scan never records
-// anything.
+// or reads anything.
 
 import (
 	"bufio"
@@ -15,6 +15,8 @@ import (
 	"path/filepath"
 	"slices"
 	"strings"
+
+	"golang.org/x/sys/unix"
 )
 
 // The systems that have default excludes, as runtime.GOOS names them.
@@ -25,11 +27,11 @@ const (
 )
 
 // The fields of a line of mountInfoPath that readMountInfo reads,
-// counted from 0 (see proc(5)): the mount point is the fifth field, and
+// counted from 0 (see proc(5)): the device ID, as major:minor, is the
-// after the sixth come optional fields, then a field holding "-", then
+// third field, and after the sixth come optional fields, then a field
-// the filesystem type.
+// holding "-", then the filesystem type.
 const (
-	mountPointField     = 4
+	deviceField         = 2
 	optionalFieldsStart = 6
 )
 
@@ -44,8 +46,8 @@ var errMountInfoLine = errors.New("unexpected line")
 // walkRules holds the two rules for the walk that a scan sets up as it
 // starts. excludes are the default excludes, or none with
 // --no-default-excludes. nonPersistent maps the device ID of every
-// mounted filesystem that is not persistent to its type; the walk
+// mounted filesystem that is not persistent to its type; the scan
-// records nothing on one of those, whatever the flags.
+// records and reads nothing on one of those, whatever the flags.
 type walkRules struct {
 	excludes      []string
 	nonPersistent map[uint64]string
@@ -105,32 +107,6 @@ func nonPersistentFilesystemTypes() []string {
 	}
 }
 
-// nonPersistentFilesystems returns the device ID of every mounted
-// filesystem that is not persistent, with its type, read with lstat on
-// its mount point. mountedFilesystems lists the mounts. A mount point
-// that lstat cannot reach is left out.
-func nonPersistentFilesystems() (map[uint64]string, error) {
-	mounts, err := mountedFilesystems()
-	if err != nil {
-		return nil, err
-	}
-
-	types := nonPersistentFilesystemTypes()
-	devices := make(map[uint64]string)
-
-	for mountPoint, fsType := range mounts {
-		if !slices.Contains(types, fsType) {
-			continue
-		}
-
-		if dev, ok := deviceOfPath(mountPoint); ok {
-			devices[dev] = fsType
-		}
-	}
-
-	return devices, nil
-}
-
 // nonPersistentLine returns the line the scan prints for path when fi,
 // its lstat, puts it on a filesystem in rules.nonPersistent, and ""
 // when it is on any other.
@@ -150,16 +126,13 @@ func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
 }
 
 // readMountInfo reads mountinfo, laid out as mountInfoPath is, and
-// returns every mount point with the type of its filesystem. Where one
+// returns the device ID of every mounted filesystem that is not
-// filesystem is mounted over another, a later line names the one on
+// persistent, with its type. The device ID is the one each line gives,
-// top, so the last line for a mount point decides.
+// not that of the file now at its mount point, so a mount that a later
-func readMountInfo(mountinfo io.Reader) (map[string]string, error) {
+// mount hides never adds the device of another filesystem.
-	// The kernel writes a space, tab, newline or backslash in a mount
+func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
-	// point as an octal escape.
+	types := nonPersistentFilesystemTypes()
-	unescape := strings.NewReplacer(
+	devices := make(map[uint64]string)
-		`\040`, " ", `\011`, "\t", `\012`, "\n", `\134`, `\`)
-
-	mounts := make(map[string]string)
 	lines := bufio.NewScanner(mountinfo)
 
 	for lines.Scan() {
@@ -170,10 +143,20 @@ func readMountInfo(mountinfo io.Reader) (map[string]string, error) {
 			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
 		}
 
-		mounts[unescape.Replace(fields[mountPointField])] = fields[sep+1]
+		var major, minor uint32
+
+		_, err := fmt.Sscanf(fields[deviceField], "%d:%d", &major, &minor)
+		if err != nil {
+			return nil, fmt.Errorf("%w: %q", errMountInfoLine, lines.Text())
+		}
+
+		fsType := fields[sep+1]
+		if slices.Contains(types, fsType) {
+			devices[unix.Mkdev(major, minor)] = fsType
+		}
 	}
 
-	return mounts, lines.Err()
+	return devices, lines.Err()
 }
 
 // defaultExcludesHelp describes, for the scan command's help, the

exclude_darwin.go

@@ -2,12 +2,19 @@
 
 package main
 
-import "golang.org/x/sys/unix"
+import (
+	"slices"
 
-// mountedFilesystems returns every mount point with the type of its
+	"golang.org/x/sys/unix"
-// filesystem, from the system's list of mounted filesystems. Where one
+)
-// filesystem is mounted over another, the later one in the list counts.
+
-func mountedFilesystems() (map[string]string, error) {
+// nonPersistentFilesystems returns the device ID of every mounted
+// filesystem that is not persistent, with its type, from the system's
+// list of mounted filesystems. The device ID is read with lstat on the
+// mount point. Where one filesystem is mounted over another on the same
+// mount point, the later one in the list counts. A mount point that
+// lstat cannot reach is left out.
+func nonPersistentFilesystems() (map[uint64]string, error) {
 	// The first call counts the filesystems, the second lists them.
 	n, err := unix.Getfsstat(nil, unix.MNT_NOWAIT)
 	if err != nil {
@@ -28,5 +35,18 @@ func mountedFilesystems() (map[string]string, error) {
 			unix.ByteSliceToString(st.Fstypename[:])
 	}
 
-	return mounts, nil
+	types := nonPersistentFilesystemTypes()
+	devices := make(map[uint64]string)
+
+	for mountPoint, fsType := range mounts {
+		if !slices.Contains(types, fsType) {
+			continue
+		}
+
+		if dev, ok := deviceOfPath(mountPoint); ok {
+			devices[dev] = fsType
+		}
+	}
+
+	return devices, nil
 }

exclude_other.go

@@ -12,12 +12,12 @@ import (
 // process sees.
 const mountInfoPath = "/proc/self/mountinfo"
 
-// mountedFilesystems returns every mount point with the type of its
+// nonPersistentFilesystems returns the device ID of every mounted
-// filesystem: on Linux read from mountInfoPath (see readMountInfo), and
+// filesystem that is not persistent, with its type: on Linux read from
-// on any other system none.
+// mountInfoPath (see readMountInfo), and on any other system none.
-func mountedFilesystems() (map[string]string, error) {
+func nonPersistentFilesystems() (map[uint64]string, error) {
 	if runtime.GOOS != goosLinux {
-		return map[string]string{}, nil
+		return map[uint64]string{}, nil
 	}
 
 	f, err := os.Open(mountInfoPath)
@@ -27,10 +27,10 @@ func mountedFilesystems() (map[string]string, error) {
 
 	defer func() { _ = f.Close() }()
 
-	mounts, err := readMountInfo(f)
+	devices, err := readMountInfo(f)
 	if err != nil {
 		return nil, fmt.Errorf("read %s: %w", mountInfoPath, err)
 	}
 
-	return mounts, nil
+	return devices, nil
 }

exclude_test.go

@@ -10,6 +10,8 @@ import (
 	"slices"
 	"strings"
 	"testing"
+
+	"golang.org/x/sys/unix"
 )
 
 func TestDefaultExcludes(t *testing.T) {
@@ -61,16 +63,17 @@ func TestIsDefaultExclude(t *testing.T) {
 func TestReadMountInfo(t *testing.T) {
 	t.Parallel()
 
-	// The second-to-last line has no optional fields. /mnt/over has an
+	// The second-to-last line has no optional fields. The tmpfs on
-	// ext4 filesystem mounted over its tmpfs.
+	// /data/a/b is hidden by the root filesystem's /srv, mounted over
+	// /data/a after it: the tmpfs still gives its own device, and the
+	// root filesystem's device is not listed.
 	mountinfo := strings.Join([]string{
 		"22 1 8:1 / / rw,relatime shared:1 - ext4 /dev/sda1 rw",
 		"23 22 0:21 / /proc rw,nosuid shared:12 - proc none rw",
 		"24 22 0:22 / /sys rw,nosuid shared:7 - sysfs none rw",
-		`25 22 0:23 / /mnt/ram\040disk rw shared:2 - tmpfs none rw`,
+		"25 22 0:23 / /data/a/b rw shared:2 - tmpfs none rw",
-		"26 22 0:24 / /mnt/over rw shared:3 - tmpfs none rw",
+		"26 22 8:1 /srv /data/a rw - ext4 /dev/sda1 rw",
-		"27 26 8:2 / /mnt/over rw - ext4 /dev/sda2 rw",
+		"27 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
-		"28 22 0:25 / /srv/auto rw shared:4 - autofs systemd-1 rw",
 	}, "\n")
 
 	got, err := readMountInfo(strings.NewReader(mountinfo))
@@ -78,18 +81,23 @@ func TestReadMountInfo(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	want := map[string]string{
+	want := map[uint64]string{
-		"/": "ext4", "/proc": "proc", "/sys": "sysfs",
+		unix.Mkdev(0, 21): "proc", unix.Mkdev(0, 22): "sysfs",
-		"/mnt/ram disk": "tmpfs", "/mnt/over": "ext4", "/srv/auto": "autofs",
+		unix.Mkdev(0, 23): "tmpfs", unix.Mkdev(0, 25): "autofs",
 	}
 	if !maps.Equal(got, want) {
-		t.Errorf("readMountInfo = %q, want %q", got, want)
+		t.Errorf("readMountInfo = %v, want %v", got, want)
 	}
 
-	_, err = readMountInfo(strings.NewReader("22 1 8:1 / / rw,relatime\n"))
+	for _, line := range []string{
-	if !errors.Is(err, errMountInfoLine) {
+		"22 1 8:1 / / rw,relatime\n",
-		t.Errorf("readMountInfo of a line without a type = %v, want %v",
+		"22 1 sda / / rw,relatime shared:1 - ext4 /dev/sda1 rw\n",
-			err, errMountInfoLine)
+	} {
+		_, err = readMountInfo(strings.NewReader(line))
+		if !errors.Is(err, errMountInfoLine) {
+			t.Errorf("readMountInfo(%q) = %v, want %v", line, err,
+				errMountInfoLine)
+		}
 	}
 }
 
@@ -442,6 +450,51 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
 	}
 }
 
+// TestScanContentSkipsNonPersistent checks that the content phase does
+// not read a file on a filesystem that is not persistent, though an
+// earlier scan left a record for it: it is treated as a changed file,
+// so its record is left as it is and its identical copy in another
+// tree, having no other match, is not read either.
+func TestScanContentSkipsNonPersistent(t *testing.T) {
+	t.Parallel()
+
+	shm, _, rules := shmTempDir(t)
+	db := openTestDB(t)
+
+	// A scan that, without the list of filesystems, records a with a
+	// head and tail but no content hash.
+	a := sparseFileWithoutMatch(t, shm, "a", headTailMin)
+
+	syncTree(t, db, shm)
+
+	before := recordByPath(t, dbRecords(t, db), a)
+	if before.head == "" || before.content != "" {
+		t.Fatalf("after scanning %s: %+v, want head and tail only", shm, before)
+	}
+
+	b := sparseFile(t, t.TempDir(), "b", headTailMin)
+
+	st, err := syncScan(t.Context(), db, []string{filepath.Dir(b)}, 1, false,
+		rules)
+	if err != nil {
+		t.Fatal(err)
+	}
+
+	if st.skipped != 0 {
+		t.Errorf("skipped = %d, want 0", st.skipped)
+	}
+
+	recs := dbRecords(t, db)
+	if r := recordByPath(t, recs, a); r != before {
+		t.Errorf("record = %+v, want it left as %+v", r, before)
+	}
+
+	if r := recordByPath(t, recs, b); r.content != "" {
+		t.Errorf("%s: content = %q, want none: its only match is on %s",
+			b, r.content, shm)
+	}
+}
+
 // TestWalkLeavesOutNonPersistent starts the walk itself in a directory
 // on a filesystem that is not persistent, which a scan never does: it
 // drops such an operand first. Below a directory on a persistent

scan.go

@@ -435,7 +435,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 		return err
 	}
 
-	return s.contentPhase(ctx, workers)
+	return s.contentPhase(ctx, workers, rules)
 }
 
 // walkableRoots returns the operands the walk can start from: regular
@@ -878,9 +878,12 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 // hash phase's worker pool and their records written back in batches. A
 // failed read is warned about and counted as skipped; the record keeps
 // its empty content, so it is never grouped, and a later scan tries
-// again.
+// again. No file on a filesystem in rules that is not persistent is
-func (s *scanState) contentPhase(ctx context.Context, workers int) error {
+// read.
-	toRead, recs, err := s.contentCandidates(ctx)
+func (s *scanState) contentPhase(ctx context.Context, workers int,
+	rules walkRules,
+) error {
+	toRead, recs, err := s.contentCandidates(ctx, rules)
 	if err != nil {
 		return err
 	}
@@ -915,18 +918,19 @@ func (s *scanState) contentPhase(ctx context.Context, workers int) error {
 // contentCandidates returns the files the content phase reads, and
 // their records by path. Every record contentCandidatesSQL returns has
 // its file checked with lstat, whether or not it already has a content
-// hash: a file that is gone, is no longer a regular file, or has
+// hash: a file that is gone, is no longer a regular file, has changed
-// changed by the walk's rule keeps its record as it is and does not
+// by the walk's rule, or is on a filesystem in rules that is not
-// count as a match for the others, and any other lstat error is warned
+// persistent keeps its record as it is and does not count as a match
-// about and counted as skipped, with the same result. If such a record
+// for the others, and any other lstat error is warned about and counted
-// has no content hash, it stays out of duplicate groups; if it has one,
+// as skipped, with the same result. If such a record has no content
-// it is still reported until a scan covering its own tree updates or
+// hash, it stays out of duplicate groups; if it has one, it is still
-// removes it. The files of a group that pass and have no content hash
+// reported until a scan covering its own tree updates or removes it.
-// are read only if at least minGroupSize of the group's files pass, so
+// The files of a group that pass and have no content hash are read only
-// a group whose other members are all stale costs no reads. Only the
+// if at least minGroupSize of the group's files pass, so a group whose
-// records to be read are kept.
+// other members are all stale costs no reads. Only the records to be
+// read are kept.
 func (s *scanState) contentCandidates(
-	ctx context.Context,
+	ctx context.Context, rules walkRules,
 ) ([]fileRec, map[string]scanRec, error) {
 	// The search and the checks take real time on a large database;
 	// without a display the scan looks hung before the reads begin.
@@ -984,7 +988,7 @@ func (s *scanState) contentCandidates(
 			first = r
 		}
 
-		f, ok, err := unchangedFile(r)
+		f, ok, err := unchangedFile(r, rules)
 		if err != nil {
 			s.st.skipped++
 
@@ -1018,9 +1022,10 @@ func (s *scanState) contentCandidates(
 
 // unchangedFile lstats the file r names and returns it for reading if
 // it is still the regular file r records: the same size, and an mtime
-// no newer than recorded (the walk's change rule). A file that is gone
+// no newer than recorded (the walk's change rule). A file that is gone,
-// or has changed reports false; any other lstat error is returned.
+// has changed, or is on a filesystem in rules that is not persistent
-func unchangedFile(r scanRec) (fileRec, bool, error) {
+// reports false; any other lstat error is returned.
+func unchangedFile(r scanRec, rules walkRules) (fileRec, bool, error) {
 	fi, err := os.Lstat(r.path)
 	if errors.Is(err, fs.ErrNotExist) {
 		return fileRec{}, false, nil
@@ -1037,6 +1042,12 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
 
 	dev, ino := inodeOfInfo(fi)
 
+	// The scan reads nothing on a filesystem that is not persistent,
+	// whatever record an earlier scan left for the file.
+	if _, found := rules.nonPersistent[dev]; found {
+		return fileRec{}, false, nil
+	}
+
 	return fileRec{
 		path: r.path, size: r.size, mtime: r.mtime, dev: dev, ino: ino,
 	}, true, nil