Author SHA1 Message Date
sneak c6066afa6e Resolve scan operands to their real paths (closes #144)
check / check (push) Waiting to run
scan now resolves every operand with its symlinks followed before the
walk, and uses that real path for pruning, walking, records and
warnings. A relative operand is joined to the working directory as the
kernel reports it, so a ".." after a symlink counts from its target, as
with realpath. A symlink operand is scanned at its target instead of
skipped. An operand given through a symlink below another operand, to a
directory outside it, is walked on its own instead of dropped with its
records deleted. An operand that cannot be resolved is a fatal error
naming the operand as given. The walk itself still follows no symlink.

Model: opus-5-5
2026-11-06 18:27:42 +00:00
clawbot f52bbe5d95 Say ZFS counts a new file's blocks up to seconds after the write (closes #152)
check / check (push) Canceled after 0s
`README.md` and a comment in `scan.go` said ZFS counts a new file's blocks only seconds after the write. The count can land in under a second, so both now say up to seconds after the write, as the comment on `diskSizeOf` already does.

Model: opus-5-5
2026-11-03 05:00:51 +01:00
6 changed files with 381 additions and 165 deletions
+36 -33
View File
@@ -348,17 +348,24 @@ default. An operand may be a directory or a regular file.
- `scan` with no operand is a usage error. The usage text goes to stderr, and - `scan` with no operand is a usage error. The usage text goes to stderr, and
the exit code is 2. the exit code is 2.
- An operand that does not exist is a fatal error. `scan` then stops before it - An operand that does not exist is a fatal error, and so is a symlink whose
walks anything, and exits 1. target does not exist. `scan` then stops before it walks anything, and exits
- Each operand is made absolute before the walk. A relative operand is joined to with code 1.
the working directory as `pwd` prints it, with its symlinks kept. A `.` or - Each operand is resolved to its real path before the walk, as `realpath`
`..` in it is resolved as text, and symlinks are not followed. Each record is prints it. A relative operand is joined to the working directory as `pwd -P`
stored under that absolute path. prints it. Every symlink in the path is then followed. A `..` after a symlink
counts from the directory the symlink leads to. From then on the operand is
that real path. Its files are recorded under it, and warnings name it.
- An operand that is a symlink is scanned at its target's real path. After a
`cd` through a symlink, `scan .` records its files under the real path of the
working directory.
- All operands belong to one scan, and one walk covers them all at once (see - All operands belong to one scan, and one walk covers them all at once (see
"Concurrency"). "Concurrency").
- An operand that repeats another is dropped before the walk. - An operand that repeats another is dropped before the walk. Two operands
repeat each other when their real paths are the same.
- An operand under another operand is dropped the same way, unless one of the - An operand under another operand is dropped the same way, unless one of the
cases below applies. No path is then walked twice. cases below applies. This too is judged on real paths. No path is then walked
twice.
- An operand under another is kept, and walked on its own, when the inner - An operand under another is kept, and walked on its own, when the inner
operand or a directory between the two is one of these: operand or a directory between the two is one of these:
- a default exclude (see "Rules for the walk"); - a default exclude (see "Rules for the walk");
@@ -369,25 +376,21 @@ default. An operand may be a directory or a regular file.
memory (see "Rules for the walk"). The outer operand's walk does not enter memory (see "Rules for the walk"). The outer operand's walk does not enter
such a directory, so the inner operand is walked on its own. such a directory, so the inner operand is walked on its own.
- No other case keeps an operand under another. An operand reached through a - No other case keeps an operand under another.
symlinked directory below the outer operand is dropped too. The outer - With `/srv/link` a symlink to a directory outside `/srv`,
operand's walk does not follow the symlink, so no file under the inner operand `scan /srv /srv/link/sub` walks `sub` on its own, at its real path.
is scanned. The records that an earlier scan left under it are deleted. The - An operand that is neither a regular file nor a directory is not scanned. Such
`-x` case above also covers an operand reached that way. an operand is, for example, a symlink to a device node. `scan` prints a
- An operand that is neither a regular file nor a directory is not scanned. warning naming the real path and what it is. It counts the operand as skipped
`scan` prints a warning naming the path and what it is. It counts the operand and drops it. The scan keeps the records beneath a dropped operand, unless
as skipped and drops it. An operand beneath a symlink operand is scanned all another operand covers them.
the same. The scan keeps the records beneath a dropped operand, unless another
operand covers them. After a `cd` through a symlink, `scan .` names the
symlink and is skipped as a symlink operand. `scan sub` in that directory
records its files under the symlink's path. `report` can then list each as a
duplicate of the same file recorded under its real path.
- An operand at or inside a directory named `.zfs` is never scanned, whether - An operand at or inside a directory named `.zfs` is never scanned, whether
given alone or under another operand. The check looks at the path as given and given alone or under another operand. The check looks at the real path, so a
at the path with its symlinks resolved. `scan` prints a warning naming the symlink into a `.zfs` directory is not scanned either. `scan` prints a warning
path, counts the operand as skipped and drops it. The records beneath it are naming the real path, counts the operand as skipped and drops it. The records
kept, unless it lies under another operand. Then they are deleted, because beneath it are kept, unless it lies under another operand. Then they are
that operand's walk never enters `.zfs` (see "Rules for the walk"). deleted, because that operand's walk never enters `.zfs` (see "Rules for the
walk").
- An operand on a filesystem that is not persistent is dropped too. `scan` - An operand on a filesystem that is not persistent is dropped too. `scan`
prints a line naming it and its filesystem type, and does not count it as prints a line naming it and its filesystem type, and does not count it as
skipped. skipped.
@@ -484,12 +487,12 @@ walk practical on large or busy storage pools.
The walk knows them by device ID and inode number. The device ID is the The walk knows them by device ID and inode number. The device ID is the
number by which the kernel tells filesystems apart. The inode number tells number by which the kernel tells filesystems apart. The inode number tells
the files of one filesystem apart. So the walk passes over them whatever the files of one filesystem apart. So the walk passes over them whatever
path leads to them. The walk follows no symlink it finds. It does open each path leads to them. The walk follows no symlink it finds. It opens each
operand at the path given, even when that path runs through a symlinked operand at its real path, not the path given. An operand given through a
directory. Such an operand can lead to the database files, and the walk symlink can lead to the database files, and the walk passes over them there
passes over them there as well. The walk prints no line for the database as well. The walk prints no line for the database files and does not count
files and does not count them as skipped. A record that an earlier scan left them as skipped. A record that an earlier scan left for one of them is
for one of them is deleted. deleted.
- By default the walk enters directories on other filesystems than its - By default the walk enters directories on other filesystems than its
operand's. With `-x`, it never enters a directory on a different filesystem operand's. With `-x`, it never enters a directory on a different filesystem
@@ -1138,7 +1141,7 @@ showing the count it stopped at.
- `0`: success, even when files were skipped with warnings. - `0`: success, even when files were skipped with warnings.
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal - `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal
error is one of these: error is one of these:
- an operand that does not exist, or whose details `scan` cannot read; - an operand that does not exist, or whose real path `scan` cannot find;
- a list of mounted filesystems that `scan` cannot read; - a list of mounted filesystems that `scan` cannot read;
- another `scan` holding the lock (see "The scan lock"); - another `scan` holding the lock (see "The scan lock");
- a database, its directory or its lock file that cannot be created or - a database, its directory or its lock file that cannot be created or
+6
View File
@@ -29,6 +29,12 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- `scan` resolves each operand to its real path, symlinks followed, before the
walk. A symlink operand is scanned at its target instead of skipped. With
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
walks `sub` on its own instead of dropping it and deleting its records
(2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144)
- `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's - `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's
blocks only seconds after the write. The count can land in under a second. blocks only seconds after the write. The count can land in under a second.
Both now say up to seconds after the write, as the comment on `diskSizeOf` in Both now say up to seconds after the write, as the comment on `diskSizeOf` in
+27 -16
View File
@@ -146,33 +146,44 @@ var errHashSize = errors.New("hash is not 32 bytes")
// databasePath resolves the database location: SFDUPES_DATABASE when // databasePath resolves the database location: SFDUPES_DATABASE when
// set and non-empty, the compiled-in default otherwise. It returns the // set and non-empty, the compiled-in default otherwise. It returns the
// path, with the working directory, its symlinks resolved, put in front // path, made absolute by absolutePath, and where the path came from,
// of it when it is relative, and where the path came from, "default" or // "default" or "from SFDUPES_DATABASE", for the line each subcommand
// "from SFDUPES_DATABASE", for the line each subcommand prints first on // prints first on stderr.
// stderr.
func databasePath() (string, string, error) { func databasePath() (string, string, error) {
path, source := defaultDatabasePath, "default" path, source := defaultDatabasePath, "default"
if p := os.Getenv(databaseEnv); p != "" { if p := os.Getenv(databaseEnv); p != "" {
path, source = p, "from "+databaseEnv path, source = p, "from "+databaseEnv
} }
if filepath.IsAbs(path) { abs, err := absolutePath(path)
return path, source, nil
}
// The working directory as the kernel reports it, with no symlink
// in it, as README "Database" says. os.Getwd may return PWD
// instead, which can name it through a symlink.
wd, err := unix.Getwd()
if err != nil { if err != nil {
return "", "", fmt.Errorf("resolve database path %s: %w", path, err) return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
} }
// Not filepath.Abs or filepath.Join: they clean the path, which return abs, source, nil
// drops each ".." together with the directory name before it. When }
// that directory is a symlink, the cleaned path names another file.
// absolutePath returns path unchanged when it is absolute, and
// otherwise the working directory, a "/" and path. The working
// directory is the one the kernel reports, with no symlink in it;
// os.Getwd may return PWD instead, which can name it through a symlink.
// The result is not cleaned, as filepath.Abs and filepath.Join would
// clean it. Cleaning drops each ".." together with the directory name
// before it. When that name is a symlink, the cleaned path names
// another file than the one the system reaches by following the symlink
// first.
func absolutePath(path string) (string, error) {
if filepath.IsAbs(path) {
return path, nil
}
wd, err := unix.Getwd()
if err != nil {
return "", err
}
// The working directory ends in "/" only when it is the root. // The working directory ends in "/" only when it is the root.
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil return strings.TrimSuffix(wd, "/") + "/" + path, nil
} }
// createDatabaseDirectory creates the directory the database at path // createDatabaseDirectory creates the directory the database at path
+250 -58
View File
@@ -14,6 +14,8 @@ import (
"strconv" "strconv"
"strings" "strings"
"testing" "testing"
"golang.org/x/sys/unix"
) )
// usageMarker is the first line of cobra's usage text, which a usage // usageMarker is the first line of cobra's usage text, which a usage
@@ -895,54 +897,262 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
assertNoSidecars(t, path) assertNoSidecars(t, path)
} }
func TestRunScanSkipsSymlinkOperand(t *testing.T) { // realTempDir returns a new temporary directory by its real path, so
path := testDBPath(t) // tests hold where the temporary directory is reached through a
t.Setenv(databaseEnv, path) // symlink.
func realTempDir(t *testing.T) string {
t.Helper()
dir := t.TempDir() dir, err := filepath.EvalSymlinks(t.TempDir())
writeFile(t, dir, "target/sub/f", pattern(1, 10))
link := filepath.Join(dir, "link")
err := os.Symlink(filepath.Join(dir, "target"), link)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
// Scanning a directory through the symlink stores a record beneath return dir
// the symlink's own path for a file beneath its target.
scanOK(t, filepath.Join(link, "sub"))
assertOperandSkipped(t, path, link, "symlink",
filepath.Join(link, "sub", "f"))
} }
// symlink creates link pointing at target.
func symlink(t *testing.T, target, link string) {
t.Helper()
err := os.Symlink(target, link)
if err != nil {
t.Fatal(err)
}
}
// assertSummary checks that stderr holds the scan summary line want.
func assertSummary(t *testing.T, stderr, want string) {
t.Helper()
if !strings.Contains(stderr, want) {
t.Errorf("stderr = %q, want %q", stderr, want)
}
}
// TestRunScanFollowsSymlinkOperand scans link, a symlink to target.
// The operand is scanned at its real path, and its file is recorded
// there.
func TestRunScanFollowsSymlinkOperand(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
link := filepath.Join(dir, "link")
symlink(t, filepath.Join(dir, "target"), link)
assertSummary(t, scanOK(t, link), "scan: 1 files seen (1 added, "+
"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f})
}
}
// TestRunScanWalksOperandUnderSymlinkOperand scans link and link/sub,
// with link a symlink to target. Both resolve under target, so
// target/sub is walked once, as part of target.
func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) { func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
dir := t.TempDir() dir := realTempDir(t)
writeFile(t, dir, "target/sub/f", pattern(1, 10)) f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
link := filepath.Join(dir, "link") link := filepath.Join(dir, "link")
symlink(t, filepath.Join(dir, "target"), link)
err := os.Symlink(filepath.Join(dir, "target"), link) assertSummary(t, scanOK(t, link, filepath.Join(link, "sub")),
"scan: 1 files seen (1 added, 0 updated, 0 removed, "+
"0 unchanged), 0 skipped\n")
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f})
}
}
// TestRunScanWalksSymlinkedOperandBelowOperand is the case of
// https://git.eeqj.de/sneak/sfdupes/issues/144: scan dir dir/link/sub,
// with dir/link a symlink to a directory outside dir. The walk of dir
// does not follow link, so dir/link/sub is walked on its own at its
// real path, and the record an earlier scan stored there survives.
func TestRunScanWalksSymlinkedOperandBelowOperand(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
a := writeFile(t, dir, "a", pattern(1, 10))
outside := realTempDir(t)
f := writeFile(t, outside, "sub/f", pattern(2, 10))
symlink(t, outside, filepath.Join(dir, "link"))
scanOK(t, filepath.Join(outside, "sub"))
stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub"))
assertSummary(t, stderr, "scan: 2 files seen (1 added, 0 updated, "+
"0 removed, 1 unchanged), 0 skipped\n")
want := []string{a, f}
slices.Sort(want)
if got := storedPaths(t, path); !slices.Equal(got, want) {
t.Errorf("records = %q, want %q", got, want)
}
}
// TestRunScanDropsSymlinkOperandInsideOperand scans dir and link, with
// link, outside dir, a symlink to dir/real. link resolves inside dir,
// so it is dropped as covered by dir, and its file is recorded once.
func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
f := writeFile(t, dir, "real/f", pattern(1, 10))
link := filepath.Join(realTempDir(t), "link")
symlink(t, filepath.Join(dir, "real"), link)
assertSummary(t, scanOK(t, dir, link), "scan: 1 files seen (1 added, "+
"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f})
}
}
// TestRunScanDanglingSymlinkOperandIsFatal scans a valid directory and
// a symlink whose target does not exist. The scan fails with exit 1 and
// an error naming the symlink as given, before it walks anything, so no
// database is created.
func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
writeFile(t, dir, "f", pattern(1, 10))
dangling := filepath.Join(realTempDir(t), "dangling")
symlink(t, filepath.Join(dir, "missing"), dangling)
stdout := captureStdout(t)
var stderr bytes.Buffer
code := run([]string{cmdScan, dir, dangling}, os.Stdout, &stderr)
if code != exitFatal {
t.Errorf("run(scan %s %s) = %d, want %d", dir, dangling, code,
exitFatal)
}
assertFatalOutput(t, stderr.String(), stdout())
want := "sfdupes: resolve " + dangling + ": "
if !strings.Contains(stderr.String(), want) {
t.Errorf("stderr = %q, want %q", stderr.String(), want)
}
_, err := os.Stat(path)
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("database: stat error %v, want it missing", err)
}
}
// TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath runs scan .
// after a cd through home/proj, a symlink to data/proj. The file is
// recorded under data/proj, its real path.
func TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
err := os.Mkdir(filepath.Join(dir, "home"), 0o750)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
// link is dropped as a symlink, but link/sub must still be scanned, symlink(t, filepath.Join(dir, "data", "proj"),
// not dropped as lying under link. filepath.Join(dir, "home", "proj"))
scanOK(t, link, filepath.Join(link, "sub"))
db, err := openDB(path, reportParams) // t.Chdir also sets PWD to the path through the symlink, as a
// shell does after cd.
t.Chdir(filepath.Join(dir, "home", "proj"))
scanOK(t, ".")
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f})
}
}
// TestRunScanFollowsSymlinkBeforeDotDot scans a path with a ".." after
// home/proj, a symlink to data/proj: home/proj/.. from dir, and .. after
// a cd through home/proj. As with realpath, the ".." counts from
// data/proj, so data is scanned and home is not.
func TestRunScanFollowsSymlinkBeforeDotDot(t *testing.T) {
for _, tc := range []struct{ wd, operand string }{
{"", "home/proj/.."},
{"home/proj", ".."},
} {
t.Run(tc.operand, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
g := writeFile(t, dir, "data/g", pattern(2, 10))
writeFile(t, dir, "home/h", pattern(3, 10))
symlink(t, filepath.Join(dir, "data", "proj"),
filepath.Join(dir, "home", "proj"))
// t.Chdir also sets PWD to the path given, as a shell does
// after cd.
t.Chdir(filepath.Join(dir, tc.wd))
scanOK(t, tc.operand)
want := []string{f, g}
slices.Sort(want)
if got := storedPaths(t, path); !slices.Equal(got, want) {
t.Errorf("records = %q, want %q", got, want)
}
})
}
}
// TestRunScanSkipsSymlinkToFIFOOperand scans link, a symlink to a FIFO.
// The scan warns about the FIFO by its real path, counts it as skipped,
// and exits 0.
func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
dir := realTempDir(t)
fifo := filepath.Join(dir, "fifo")
err := unix.Mkfifo(fifo, 0o600)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
t.Cleanup(func() { _ = db.Close() }) link := filepath.Join(dir, "link")
symlink(t, fifo, link)
recordByPath(t, dbRecords(t, db), filepath.Join(link, "sub", "f")) stderr := scanOK(t, link)
warning := "walk " + fifo + ": skipping FIFO operand\n"
if !strings.Contains(stderr, warning) {
t.Errorf("stderr = %q, want %q", stderr, warning)
}
assertSummary(t, stderr, "scan: 0 files seen (0 added, 0 updated, "+
"0 removed, 0 unchanged), 1 skipped\n")
} }
func TestRunScanSkipsZFSOperand(t *testing.T) { func TestRunScanSkipsZFSOperand(t *testing.T) {
@@ -1041,23 +1251,19 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where // TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about // snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
// by the path given, counted as skipped, and not walked. // by its real path, counted as skipped, and not walked.
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) { func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t)) t.Setenv(databaseEnv, testDBPath(t))
dir := t.TempDir() dir := realTempDir(t)
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10)) writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
snaps := filepath.Join(dir, "snaps") snaps := filepath.Join(dir, "snaps")
symlink(t, filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps) stderr := scanOK(t, filepath.Join(snaps, "daily"))
if err != nil {
t.Fatal(err)
}
daily := filepath.Join(snaps, "daily")
stderr := scanOK(t, daily)
daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily")
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n" warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " + summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n" "0 unchanged), 1 skipped\n"
@@ -1069,39 +1275,25 @@ func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
} }
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is // TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
// a symlink to a directory outside any .zfs directory. The operand is // a symlink to pool/data. The operand is scanned as usual, and its file
// scanned as usual, and its file is recorded by the path given. // is recorded by its real path.
func TestRunScanWalksOperandThroughSymlink(t *testing.T) { func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
dir := t.TempDir() dir := realTempDir(t)
writeFile(t, dir, "pool/data/daily/f", pattern(1, 10)) f := writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
data := filepath.Join(dir, "data") data := filepath.Join(dir, "data")
symlink(t, filepath.Join(dir, "pool", "data"), data)
err := os.Symlink(filepath.Join(dir, "pool", "data"), data) assertSummary(t, scanOK(t, filepath.Join(data, "daily")),
if err != nil { "scan: 1 files seen (1 added, 0 updated, 0 removed, "+
t.Fatal(err) "0 unchanged), 0 skipped\n")
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f})
} }
daily := filepath.Join(data, "daily")
stderr := scanOK(t, daily)
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
db, err := openDB(path, reportParams)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
} }
func TestRunReportSucceeds(t *testing.T) { func TestRunReportSucceeds(t *testing.T) {
+27 -30
View File
@@ -209,29 +209,30 @@ func interrupted(walked int) error {
return errInterrupted return errInterrupted
} }
// resolveRoots converts each PATH operand to an absolute, lexically // resolveRoots converts each PATH operand to its real path: absolute,
// cleaned path (symlinks are not resolved) and verifies that it // with every symlink in it followed. Database records are keyed by
// exists. Database records are keyed by absolute path, so scan results // that path, so scan results depend neither on the working directory
// must not depend on the working directory. // nor on the symlinks an operand was given through. An operand that
// cannot be resolved, because it or a symlink's target does not exist
// or cannot be read, is a fatal error before any scanning.
func resolveRoots(roots []string) ([]string, error) { func resolveRoots(roots []string) ([]string, error) {
abs := make([]string, 0, len(roots)) resolved := make([]string, 0, len(roots))
for _, root := range roots { for _, root := range roots {
a, err := filepath.Abs(root) a, err := absolutePath(root)
if err != nil { if err != nil {
return nil, fmt.Errorf("resolve %s: %w", root, err) return nil, fmt.Errorf("resolve %s: %w", root, err)
} }
// A nonexistent operand is a fatal error before any scanning. r, err := filepath.EvalSymlinks(a)
_, err = os.Lstat(a)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("resolve %s: %w", root, err)
} }
abs = append(abs, a) resolved = append(resolved, r)
} }
return abs, nil return resolved, nil
} }
// pruneRoots drops operands already covered by another operand: // pruneRoots drops operands already covered by another operand:
@@ -417,14 +418,16 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
// walkableRoots returns the operands the walk can start from: regular // walkableRoots returns the operands the walk can start from: regular
// files, and directories not named .zfs, none of them inside a // files, and directories not named .zfs, none of them inside a
// directory named .zfs. Every other operand is warned about, counted as // directory named .zfs. The operands are real paths from resolveRoots,
// skipped, and dropped. A dropped operand is no longer a root, so the // so these checks apply to what a symlink operand points at. Every
// records stored beneath it are treated as if it had not been given: // other operand is warned about, counted as skipped, and dropped. A
// those under no other root count as outside the roots and are not // dropped operand is no longer a root, so the records stored beneath it
// deleted as unverified. An operand on a filesystem in rules that is not // are treated as if it had not been given: those under no other root
// persistent is dropped the same way, with a line saying so that is not // count as outside the roots and are not deleted as unverified. An
// a warning, and is not counted as skipped. An operand that fails lstat // operand on a filesystem in rules that is not persistent is dropped
// here is kept, and the walk warns about it unless it no longer exists. // the same way, with a line saying so that is not a warning, and is not
// counted as skipped. An operand that fails lstat here is kept, and the
// walk warns about it unless it no longer exists.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
kept := make([]string, 0, len(roots)) kept := make([]string, 0, len(roots))
@@ -894,19 +897,13 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
// operandWarning returns the one-line warning for an operand the walk // operandWarning returns the one-line warning for an operand the walk
// does not start from, naming the path and what it is, or "" for one it // does not start from, naming the path and what it is, or "" for one it
// does: a regular file, or a directory not named .zfs, that is not // does: a regular file, or a directory not named .zfs, that is not
// inside a directory named .zfs, neither as given nor with its symlinks // inside a directory named .zfs. resolveRoots has already followed the
// resolved. Symlinks are never followed, including as operands. // operand's symlinks, so a symlink here is an operand replaced by one
// after it was resolved, and is not followed.
func operandWarning(root string, fi fs.FileInfo) string { func operandWarning(root string, fi fs.FileInfo) string {
// The walk never enters a .zfs directory, so it starts from nothing // The walk never enters a .zfs directory, so it starts from nothing
// inside one either, not even through a symlink. An operand whose // inside one either.
// symlinks cannot be resolved is checked as given. if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") {
resolved, err := filepath.EvalSymlinks(root)
if err != nil {
resolved = root
}
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory", return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
root) root)
} }
+35 -28
View File
@@ -2310,33 +2310,24 @@ func TestUnderRoot(t *testing.T) {
} }
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own // TestRunScanPassesOverDatabase scans a tree that holds the scan's own
// database, reached directly and through a symlinked directory. The // database, and in one case also a hard link to the database file. The
// walk passes over the database file, its -wal and -shm files and its // walk knows the database's files by device ID and inode number, so it
// .lock file with no line and no count, and the record an earlier scan // passes over the database file, its -wal and -shm files, its .lock
// left for the .lock file is deleted. A rescan of the unchanged tree // file and the hard link with no line and no count. The record an
// updates nothing, and report lists none of the database's files. // earlier scan left for the .lock file is deleted. A rescan of the
// unchanged tree updates nothing, and report lists none of the
// database's files.
func TestRunScanPassesOverDatabase(t *testing.T) { func TestRunScanPassesOverDatabase(t *testing.T) {
for _, viaSymlink := range []bool{false, true} { for _, viaHardLink := range []bool{false, true} {
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) { t.Run(fmt.Sprintf("via hard link %v", viaHardLink), func(t *testing.T) {
base := t.TempDir() tree := realTempDir(t)
tree := filepath.Join(base, "real", "tree") hardLink := filepath.Join(tree, "hardlink")
operand := tree
writeFile(t, tree, "a", pattern(1, 100)) writeFile(t, tree, "a", pattern(1, 100))
writeFile(t, tree, "b", pattern(1, 100)) writeFile(t, tree, "b", pattern(1, 100))
writeFile(t, tree, "empty1", nil) writeFile(t, tree, "empty1", nil)
writeFile(t, tree, "empty2", nil) writeFile(t, tree, "empty2", nil)
if viaSymlink {
err := os.Symlink(filepath.Join(base, "real"),
filepath.Join(base, "link"))
if err != nil {
t.Fatal(err)
}
operand = filepath.Join(base, "link", "tree")
}
dbPath := filepath.Join(tree, "db.sqlite") dbPath := filepath.Join(tree, "db.sqlite")
t.Setenv(databaseEnv, dbPath) t.Setenv(databaseEnv, dbPath)
@@ -2346,7 +2337,7 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
} }
err = applyChanges(t.Context(), db, []scanRec{ err = applyChanges(t.Context(), db, []scanRec{
{path: filepath.Join(operand, "db.sqlite.lock")}, {path: filepath.Join(tree, "db.sqlite.lock")},
}, nil, nil) }, nil, nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
@@ -2354,16 +2345,29 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
closeScanDatabase(t.Context(), db, dbPath) closeScanDatabase(t.Context(), db, dbPath)
assertScanSummary(t, if viaHardLink {
err = os.Link(dbPath, hardLink)
if err != nil {
t.Fatal(err)
}
}
stderr := assertScanSummary(t,
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)", "4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
operand) tree)
assertScanSummary(t, stderr += assertScanSummary(t,
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)", "4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
operand) tree)
if strings.Contains(stderr, hardLink) {
t.Errorf("scan stderr = %q, want no line naming %s",
stderr, hardLink)
}
report := runStdout(t, cmdReport) report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") || if strings.Contains(report, "db.sqlite") ||
!strings.Contains(report, filepath.Join(operand, "empty2")) { strings.Contains(report, hardLink) ||
!strings.Contains(report, filepath.Join(tree, "empty2")) {
t.Errorf("report = %q, want the empty files and none of "+ t.Errorf("report = %q, want the empty files and none of "+
"the database's files", report) "the database's files", report)
} }
@@ -2388,8 +2392,9 @@ func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
// assertScanSummary scans operands and fails the test unless the // assertScanSummary scans operands and fails the test unless the
// summary line gives counts with nothing skipped, and the line naming // summary line gives counts with nothing skipped, and the line naming
// the database is the only one that names one of its files. // the database is the only one that names one of its files. It returns
func assertScanSummary(t *testing.T, counts string, operands ...string) { // everything the scan printed to stderr.
func assertScanSummary(t *testing.T, counts string, operands ...string) string {
t.Helper() t.Helper()
stderr := scanOK(t, operands...) stderr := scanOK(t, operands...)
@@ -2403,4 +2408,6 @@ func assertScanSummary(t *testing.T, counts string, operands ...string) {
t.Errorf("scan %q: stderr = %q, want only the line naming the "+ t.Errorf("scan %q: stderr = %q, want only the line naming the "+
"database to name one of its files", operands, stderr) "database to name one of its files", operands, stderr)
} }
return stderr
} }