Resolve scan operands to their real paths (closes #144)

scan now resolves every operand with its symlinks followed before the
walk, and uses that real path for pruning, walking, records and
warnings. A symlink operand is scanned at its target instead of
skipped. An operand given through a symlinked directory below another
operand is walked on its own instead of dropped with its records
deleted. A dangling symlink operand is a fatal error, like a missing
path. The walk itself still follows no symlink.

Model: opus-5-5

README.md

@@ -348,17 +348,23 @@ default. An operand may be a directory or a regular file.
 
 - `scan` with no operand is a usage error. The usage text goes to stderr, and
   the exit code is 2.
-- An operand that does not exist is a fatal error. `scan` then stops before it
+- An operand that does not exist is a fatal error, and so is a symlink whose
-  walks anything, and exits 1.
+  target does not exist. `scan` then stops before it walks anything, and exits
-- Each operand is made absolute before the walk. A relative operand is joined to
+  with code 1.
-  the working directory as `pwd` prints it, with its symlinks kept. A `.` or
+- Each operand is resolved to its real path before the walk, as `realpath`
-  `..` in it is resolved as text, and symlinks are not followed. Each record is
+  prints it. The path is made absolute, and every symlink in it is followed.
-  stored under that absolute path.
+  From then on the operand is that real path. Its files are recorded under it,
+  and warnings name it.
+- An operand that is a symlink is scanned at its target's real path. After a
+  `cd` through a symlink, `scan .` records its files under the real path of the
+  working directory.
 - All operands belong to one scan, and one walk covers them all at once (see
   "Concurrency").
-- An operand that repeats another is dropped before the walk.
+- An operand that repeats another is dropped before the walk. Two operands
+  repeat each other when their real paths are the same.
 - An operand under another operand is dropped the same way, unless one of the
-  cases below applies. No path is then walked twice.
+  cases below applies. This too is judged on real paths. No path is then walked
+  twice.
 - An operand under another is kept, and walked on its own, when the inner
   operand or a directory between the two is one of these:
     - a default exclude (see "Rules for the walk");
@@ -369,25 +375,22 @@ default. An operand may be a directory or a regular file.
     memory (see "Rules for the walk"). The outer operand's walk does not enter
     such a directory, so the inner operand is walked on its own.
 
-- No other case keeps an operand under another. An operand reached through a
+- No other case keeps an operand under another.
-  symlinked directory below the outer operand is dropped too. The outer
+- An operand given through a symlinked directory below another operand, as in
-  operand's walk does not follow the symlink, so no file under the inner operand
+  `scan /srv /srv/link/sub`, is not under it. It resolves to the real path of
-  is scanned. The records that an earlier scan left under it are deleted. The
+  `sub`, which lies outside `/srv`, so it is walked on its own.
-  `-x` case above also covers an operand reached that way.
+- An operand that is neither a regular file nor a directory is not scanned. Such
-- An operand that is neither a regular file nor a directory is not scanned.
+  an operand is, for example, a symlink to a device node. `scan` prints a
-  `scan` prints a warning naming the path and what it is. It counts the operand
+  warning naming the real path and what it is. It counts the operand as skipped
-  as skipped and drops it. An operand beneath a symlink operand is scanned all
+  and drops it. The scan keeps the records beneath a dropped operand, unless
-  the same. The scan keeps the records beneath a dropped operand, unless another
+  another operand covers them.
-  operand covers them. After a `cd` through a symlink, `scan .` names the
-  symlink and is skipped as a symlink operand. `scan sub` in that directory
-  records its files under the symlink's path. `report` can then list each as a
-  duplicate of the same file recorded under its real path.
 - An operand at or inside a directory named `.zfs` is never scanned, whether
-  given alone or under another operand. The check looks at the path as given and
+  given alone or under another operand. The check looks at the real path, so a
-  at the path with its symlinks resolved. `scan` prints a warning naming the
+  symlink into a `.zfs` directory is not scanned either. `scan` prints a warning
-  path, counts the operand as skipped and drops it. The records beneath it are
+  naming the real path, counts the operand as skipped and drops it. The records
-  kept, unless it lies under another operand. Then they are deleted, because
+  beneath it are kept, unless it lies under another operand. Then they are
-  that operand's walk never enters `.zfs` (see "Rules for the walk").
+  deleted, because that operand's walk never enters `.zfs` (see "Rules for the
+  walk").
 - An operand on a filesystem that is not persistent is dropped too. `scan`
   prints a line naming it and its filesystem type, and does not count it as
   skipped.
@@ -484,12 +487,12 @@ walk practical on large or busy storage pools.
     The walk knows them by device ID and inode number. The device ID is the
     number by which the kernel tells filesystems apart. The inode number tells
     the files of one filesystem apart. So the walk passes over them whatever
-    path leads to them. The walk follows no symlink it finds. It does open each
+    path leads to them. The walk follows no symlink it finds. It opens each
-    operand at the path given, even when that path runs through a symlinked
+    operand at its real path, not the path given. An operand given through a
-    directory. Such an operand can lead to the database files, and the walk
+    symlink can lead to the database files, and the walk passes over them there
-    passes over them there as well. The walk prints no line for the database
+    as well. The walk prints no line for the database files and does not count
-    files and does not count them as skipped. A record that an earlier scan left
+    them as skipped. A record that an earlier scan left for one of them is
-    for one of them is deleted.
+    deleted.
 
 - By default the walk enters directories on other filesystems than its
   operand's. With `-x`, it never enters a directory on a different filesystem
@@ -1138,7 +1141,7 @@ showing the count it stopped at.
 - `0`: success, even when files were skipped with warnings.
 - `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal
   error is one of these:
-    - an operand that does not exist, or whose details `scan` cannot read;
+    - an operand that does not exist, or whose real path `scan` cannot find;
     - a list of mounted filesystems that `scan` cannot read;
     - another `scan` holding the lock (see "The scan lock");
     - a database, its directory or its lock file that cannot be created or

TODO.md

@@ -29,6 +29,12 @@
 Newest first. An entry ends with its date and its issue or branch, where those
 are known.
 
+- `scan` resolves each operand to its real path, symlinks followed, before the
+  walk. A symlink operand is scanned at its target instead of skipped. An
+  operand given through a symlinked directory below another operand is walked on
+  its own instead of dropped with its records deleted (2026-11-06,
+  https://git.eeqj.de/sneak/sfdupes/issues/144)
+
 - `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's
   blocks only seconds after the write. The count can land in under a second.
   Both now say up to seconds after the write, as the comment on `diskSizeOf` in

main_test.go

@@ -895,54 +895,192 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
 	assertNoSidecars(t, path)
 }
 
-func TestRunScanSkipsSymlinkOperand(t *testing.T) {
+// realTempDir returns a new temporary directory by its real path, so
-	path := testDBPath(t)
+// tests hold where the temporary directory is reached through a
-	t.Setenv(databaseEnv, path)
+// symlink.
+func realTempDir(t *testing.T) string {
+	t.Helper()
 
-	dir := t.TempDir()
+	dir, err := filepath.EvalSymlinks(t.TempDir())
-	writeFile(t, dir, "target/sub/f", pattern(1, 10))
-
-	link := filepath.Join(dir, "link")
-
-	err := os.Symlink(filepath.Join(dir, "target"), link)
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	// Scanning a directory through the symlink stores a record beneath
+	return dir
-	// the symlink's own path for a file beneath its target.
-	scanOK(t, filepath.Join(link, "sub"))
-
-	assertOperandSkipped(t, path, link, "symlink",
-		filepath.Join(link, "sub", "f"))
 }
 
+// symlink creates link pointing at target.
+func symlink(t *testing.T, target, link string) {
+	t.Helper()
+
+	err := os.Symlink(target, link)
+	if err != nil {
+		t.Fatal(err)
+	}
+}
+
+// assertSummary checks that stderr holds the scan summary line want.
+func assertSummary(t *testing.T, stderr, want string) {
+	t.Helper()
+
+	if !strings.Contains(stderr, want) {
+		t.Errorf("stderr = %q, want %q", stderr, want)
+	}
+}
+
+// TestRunScanFollowsSymlinkOperand scans link, a symlink to target.
+// The operand is scanned at its real path, and its file is recorded
+// there.
+func TestRunScanFollowsSymlinkOperand(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := realTempDir(t)
+	f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
+
+	link := filepath.Join(dir, "link")
+	symlink(t, filepath.Join(dir, "target"), link)
+
+	assertSummary(t, scanOK(t, link), "scan: 1 files seen (1 added, "+
+		"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
+
+	if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
+		t.Errorf("records = %q, want %q", got, []string{f})
+	}
+}
+
+// TestRunScanWalksOperandUnderSymlinkOperand scans link and link/sub,
+// with link a symlink to target. Both resolve under target, so
+// target/sub is walked once, as part of target.
 func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
 	path := testDBPath(t)
 	t.Setenv(databaseEnv, path)
 
-	dir := t.TempDir()
+	dir := realTempDir(t)
-	writeFile(t, dir, "target/sub/f", pattern(1, 10))
+	f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
 
 	link := filepath.Join(dir, "link")
+	symlink(t, filepath.Join(dir, "target"), link)
 
-	err := os.Symlink(filepath.Join(dir, "target"), link)
+	assertSummary(t, scanOK(t, link, filepath.Join(link, "sub")),
+		"scan: 1 files seen (1 added, 0 updated, 0 removed, "+
+			"0 unchanged), 0 skipped\n")
+
+	if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
+		t.Errorf("records = %q, want %q", got, []string{f})
+	}
+}
+
+// TestRunScanWalksSymlinkedOperandBelowOperand is the case of
+// https://git.eeqj.de/sneak/sfdupes/issues/144: scan dir dir/link/sub,
+// with dir/link a symlink to a directory outside dir. The walk of dir
+// does not follow link, so dir/link/sub is walked on its own at its
+// real path, and the record an earlier scan stored there survives.
+func TestRunScanWalksSymlinkedOperandBelowOperand(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := realTempDir(t)
+	a := writeFile(t, dir, "a", pattern(1, 10))
+
+	outside := realTempDir(t)
+	f := writeFile(t, outside, "sub/f", pattern(2, 10))
+
+	symlink(t, outside, filepath.Join(dir, "link"))
+
+	scanOK(t, filepath.Join(outside, "sub"))
+
+	stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub"))
+	assertSummary(t, stderr, "scan: 2 files seen (1 added, 0 updated, "+
+		"0 removed, 1 unchanged), 0 skipped\n")
+
+	want := []string{a, f}
+	slices.Sort(want)
+
+	if got := storedPaths(t, path); !slices.Equal(got, want) {
+		t.Errorf("records = %q, want %q", got, want)
+	}
+}
+
+// TestRunScanDropsSymlinkOperandInsideOperand scans dir and link, with
+// link, outside dir, a symlink to dir/real. link resolves inside dir,
+// so it is dropped as covered by dir, and its file is recorded once.
+func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := realTempDir(t)
+	f := writeFile(t, dir, "real/f", pattern(1, 10))
+
+	link := filepath.Join(realTempDir(t), "link")
+	symlink(t, filepath.Join(dir, "real"), link)
+
+	assertSummary(t, scanOK(t, dir, link), "scan: 1 files seen (1 added, "+
+		"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
+
+	if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
+		t.Errorf("records = %q, want %q", got, []string{f})
+	}
+}
+
+// TestRunScanDanglingSymlinkOperandIsFatal scans a valid directory and
+// a symlink whose target does not exist. The scan fails with exit 1
+// before it walks anything, so no database is created.
+func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := realTempDir(t)
+	writeFile(t, dir, "f", pattern(1, 10))
+
+	dangling := filepath.Join(realTempDir(t), "dangling")
+	symlink(t, filepath.Join(dir, "missing"), dangling)
+
+	stdout := captureStdout(t)
+
+	var stderr bytes.Buffer
+
+	code := run([]string{cmdScan, dir, dangling}, os.Stdout, &stderr)
+	if code != exitFatal {
+		t.Errorf("run(scan %s %s) = %d, want %d", dir, dangling, code,
+			exitFatal)
+	}
+
+	assertFatalOutput(t, stderr.String(), stdout())
+
+	_, err := os.Stat(path)
+	if !errors.Is(err, fs.ErrNotExist) {
+		t.Errorf("database: stat error %v, want it missing", err)
+	}
+}
+
+// TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath runs scan .
+// after a cd through home/proj, a symlink to data/proj. The file is
+// recorded under data/proj, its real path.
+func TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath(t *testing.T) {
+	path := testDBPath(t)
+	t.Setenv(databaseEnv, path)
+
+	dir := realTempDir(t)
+	f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
+
+	err := os.Mkdir(filepath.Join(dir, "home"), 0o750)
 	if err != nil {
 		t.Fatal(err)
 	}
 
-	// link is dropped as a symlink, but link/sub must still be scanned,
+	symlink(t, filepath.Join(dir, "data", "proj"),
-	// not dropped as lying under link.
+		filepath.Join(dir, "home", "proj"))
-	scanOK(t, link, filepath.Join(link, "sub"))
 
-	db, err := openDB(path, reportParams)
+	// t.Chdir also sets PWD to the path through the symlink, as a
-	if err != nil {
+	// shell does after cd.
-		t.Fatal(err)
+	t.Chdir(filepath.Join(dir, "home", "proj"))
+
+	scanOK(t, ".")
+
+	if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
+		t.Errorf("records = %q, want %q", got, []string{f})
 	}
-
-	t.Cleanup(func() { _ = db.Close() })
-
-	recordByPath(t, dbRecords(t, db), filepath.Join(link, "sub", "f"))
 }
 
 func TestRunScanSkipsZFSOperand(t *testing.T) {
@@ -1041,23 +1179,19 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
 
 // TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
 // snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
-// by the path given, counted as skipped, and not walked.
+// by its real path, counted as skipped, and not walked.
 func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
 	t.Setenv(databaseEnv, testDBPath(t))
 
-	dir := t.TempDir()
+	dir := realTempDir(t)
 	writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
 
 	snaps := filepath.Join(dir, "snaps")
+	symlink(t, filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
 
-	err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
+	stderr := scanOK(t, filepath.Join(snaps, "daily"))
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	daily := filepath.Join(snaps, "daily")
-	stderr := scanOK(t, daily)
 
+	daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily")
 	warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
 	summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
 		"0 unchanged), 1 skipped\n"
@@ -1069,39 +1203,25 @@ func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
 }
 
 // TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
-// a symlink to a directory outside any .zfs directory. The operand is
+// a symlink to pool/data. The operand is scanned as usual, and its file
-// scanned as usual, and its file is recorded by the path given.
+// is recorded by its real path.
 func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
 	path := testDBPath(t)
 	t.Setenv(databaseEnv, path)
 
-	dir := t.TempDir()
+	dir := realTempDir(t)
-	writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
+	f := writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
 
 	data := filepath.Join(dir, "data")
+	symlink(t, filepath.Join(dir, "pool", "data"), data)
 
-	err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
+	assertSummary(t, scanOK(t, filepath.Join(data, "daily")),
-	if err != nil {
+		"scan: 1 files seen (1 added, 0 updated, 0 removed, "+
-		t.Fatal(err)
+			"0 unchanged), 0 skipped\n")
+
+	if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
+		t.Errorf("records = %q, want %q", got, []string{f})
 	}
-
-	daily := filepath.Join(data, "daily")
-	stderr := scanOK(t, daily)
-
-	summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
-		"0 unchanged), 0 skipped\n"
-	if !strings.Contains(stderr, summary) {
-		t.Errorf("stderr = %q, want %q", stderr, summary)
-	}
-
-	db, err := openDB(path, reportParams)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	t.Cleanup(func() { _ = db.Close() })
-
-	recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
 }
 
 func TestRunReportSucceeds(t *testing.T) {

scan.go

@@ -209,12 +209,14 @@ func interrupted(walked int) error {
 	return errInterrupted
 }
 
-// resolveRoots converts each PATH operand to an absolute, lexically
+// resolveRoots converts each PATH operand to its real path: absolute,
-// cleaned path (symlinks are not resolved) and verifies that it
+// with every symlink in it followed. Database records are keyed by
-// exists. Database records are keyed by absolute path, so scan results
+// that path, so scan results depend neither on the working directory
-// must not depend on the working directory.
+// nor on the symlinks an operand was given through. An operand that
+// cannot be resolved, because it or a symlink's target does not exist
+// or cannot be read, is a fatal error before any scanning.
 func resolveRoots(roots []string) ([]string, error) {
-	abs := make([]string, 0, len(roots))
+	resolved := make([]string, 0, len(roots))
 
 	for _, root := range roots {
 		a, err := filepath.Abs(root)
@@ -222,16 +224,15 @@ func resolveRoots(roots []string) ([]string, error) {
 			return nil, fmt.Errorf("resolve %s: %w", root, err)
 		}
 
-		// A nonexistent operand is a fatal error before any scanning.
+		r, err := filepath.EvalSymlinks(a)
-		_, err = os.Lstat(a)
 		if err != nil {
 			return nil, err
 		}
 
-		abs = append(abs, a)
+		resolved = append(resolved, r)
 	}
 
-	return abs, nil
+	return resolved, nil
 }
 
 // pruneRoots drops operands already covered by another operand:
@@ -417,7 +418,9 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
 
 // walkableRoots returns the operands the walk can start from: regular
 // files, and directories not named .zfs, none of them inside a
-// directory named .zfs. Every other operand is warned about, counted as
+// directory named .zfs. The operands are real paths from resolveRoots,
+// so these checks apply to what a symlink operand points at. Every
+// other operand is warned about, counted as
 // skipped, and dropped. A dropped operand is no longer a root, so the
 // records stored beneath it are treated as if it had not been given:
 // those under no other root count as outside the roots and are not
@@ -894,19 +897,13 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
 // operandWarning returns the one-line warning for an operand the walk
 // does not start from, naming the path and what it is, or "" for one it
 // does: a regular file, or a directory not named .zfs, that is not
-// inside a directory named .zfs, neither as given nor with its symlinks
+// inside a directory named .zfs. resolveRoots has already followed the
-// resolved. Symlinks are never followed, including as operands.
+// operand's symlinks, so a symlink here is an operand replaced by one
+// after it was resolved, and is not followed.
 func operandWarning(root string, fi fs.FileInfo) string {
 	// The walk never enters a .zfs directory, so it starts from nothing
-	// inside one either, not even through a symlink. An operand whose
+	// inside one either.
-	// symlinks cannot be resolved is checked as given.
+	if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") {
-	resolved, err := filepath.EvalSymlinks(root)
-	if err != nil {
-		resolved = root
-	}
-
-	if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
-		slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
 		return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
 			root)
 	}

scan_test.go

@@ -2310,15 +2310,16 @@ func TestUnderRoot(t *testing.T) {
 }
 
 // TestRunScanPassesOverDatabase scans a tree that holds the scan's own
-// database, reached directly and through a symlinked directory. The
+// database, given directly and through a symlinked directory, which
-// walk passes over the database file, its -wal and -shm files and its
+// the scan resolves to the same real path. The walk passes over the
-// .lock file with no line and no count, and the record an earlier scan
+// database file, its -wal and -shm files and its .lock file with no
-// left for the .lock file is deleted. A rescan of the unchanged tree
+// line and no count, and the record an earlier scan left for the .lock
+// file is deleted. A rescan of the unchanged tree
 // updates nothing, and report lists none of the database's files.
 func TestRunScanPassesOverDatabase(t *testing.T) {
 	for _, viaSymlink := range []bool{false, true} {
 		t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
-			base := t.TempDir()
+			base := realTempDir(t)
 			tree := filepath.Join(base, "real", "tree")
 			operand := tree
 
@@ -2346,7 +2347,7 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
 			}
 
 			err = applyChanges(t.Context(), db, []scanRec{
-				{path: filepath.Join(operand, "db.sqlite.lock")},
+				{path: filepath.Join(tree, "db.sqlite.lock")},
 			}, nil, nil)
 			if err != nil {
 				t.Fatal(err)
@@ -2363,7 +2364,7 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
 
 			report := runStdout(t, cmdReport)
 			if strings.Contains(report, "db.sqlite") ||
-				!strings.Contains(report, filepath.Join(operand, "empty2")) {
+				!strings.Contains(report, filepath.Join(tree, "empty2")) {
 				t.Errorf("report = %q, want the empty files and none of "+
 					"the database's files", report)
 			}