Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c6066afa6e | ||
|
|
f52bbe5d95 |
@@ -348,17 +348,24 @@ default. An operand may be a directory or a regular file.
|
|||||||
|
|
||||||
- `scan` with no operand is a usage error. The usage text goes to stderr, and
|
- `scan` with no operand is a usage error. The usage text goes to stderr, and
|
||||||
the exit code is 2.
|
the exit code is 2.
|
||||||
- An operand that does not exist is a fatal error. `scan` then stops before it
|
- An operand that does not exist is a fatal error, and so is a symlink whose
|
||||||
walks anything, and exits 1.
|
target does not exist. `scan` then stops before it walks anything, and exits
|
||||||
- Each operand is made absolute before the walk. A relative operand is joined to
|
with code 1.
|
||||||
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
|
- Each operand is resolved to its real path before the walk, as `realpath`
|
||||||
`..` in it is resolved as text, and symlinks are not followed. Each record is
|
prints it. A relative operand is joined to the working directory as `pwd -P`
|
||||||
stored under that absolute path.
|
prints it. Every symlink in the path is then followed. A `..` after a symlink
|
||||||
|
counts from the directory the symlink leads to. From then on the operand is
|
||||||
|
that real path. Its files are recorded under it, and warnings name it.
|
||||||
|
- An operand that is a symlink is scanned at its target's real path. After a
|
||||||
|
`cd` through a symlink, `scan .` records its files under the real path of the
|
||||||
|
working directory.
|
||||||
- All operands belong to one scan, and one walk covers them all at once (see
|
- All operands belong to one scan, and one walk covers them all at once (see
|
||||||
"Concurrency").
|
"Concurrency").
|
||||||
- An operand that repeats another is dropped before the walk.
|
- An operand that repeats another is dropped before the walk. Two operands
|
||||||
|
repeat each other when their real paths are the same.
|
||||||
- An operand under another operand is dropped the same way, unless one of the
|
- An operand under another operand is dropped the same way, unless one of the
|
||||||
cases below applies. No path is then walked twice.
|
cases below applies. This too is judged on real paths. No path is then walked
|
||||||
|
twice.
|
||||||
- An operand under another is kept, and walked on its own, when the inner
|
- An operand under another is kept, and walked on its own, when the inner
|
||||||
operand or a directory between the two is one of these:
|
operand or a directory between the two is one of these:
|
||||||
- a default exclude (see "Rules for the walk");
|
- a default exclude (see "Rules for the walk");
|
||||||
@@ -369,25 +376,21 @@ default. An operand may be a directory or a regular file.
|
|||||||
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
||||||
such a directory, so the inner operand is walked on its own.
|
such a directory, so the inner operand is walked on its own.
|
||||||
|
|
||||||
- No other case keeps an operand under another. An operand reached through a
|
- No other case keeps an operand under another.
|
||||||
symlinked directory below the outer operand is dropped too. The outer
|
- With `/srv/link` a symlink to a directory outside `/srv`,
|
||||||
operand's walk does not follow the symlink, so no file under the inner operand
|
`scan /srv /srv/link/sub` walks `sub` on its own, at its real path.
|
||||||
is scanned. The records that an earlier scan left under it are deleted. The
|
- An operand that is neither a regular file nor a directory is not scanned. Such
|
||||||
`-x` case above also covers an operand reached that way.
|
an operand is, for example, a symlink to a device node. `scan` prints a
|
||||||
- An operand that is neither a regular file nor a directory is not scanned.
|
warning naming the real path and what it is. It counts the operand as skipped
|
||||||
`scan` prints a warning naming the path and what it is. It counts the operand
|
and drops it. The scan keeps the records beneath a dropped operand, unless
|
||||||
as skipped and drops it. An operand beneath a symlink operand is scanned all
|
another operand covers them.
|
||||||
the same. The scan keeps the records beneath a dropped operand, unless another
|
|
||||||
operand covers them. After a `cd` through a symlink, `scan .` names the
|
|
||||||
symlink and is skipped as a symlink operand. `scan sub` in that directory
|
|
||||||
records its files under the symlink's path. `report` can then list each as a
|
|
||||||
duplicate of the same file recorded under its real path.
|
|
||||||
- An operand at or inside a directory named `.zfs` is never scanned, whether
|
- An operand at or inside a directory named `.zfs` is never scanned, whether
|
||||||
given alone or under another operand. The check looks at the path as given and
|
given alone or under another operand. The check looks at the real path, so a
|
||||||
at the path with its symlinks resolved. `scan` prints a warning naming the
|
symlink into a `.zfs` directory is not scanned either. `scan` prints a warning
|
||||||
path, counts the operand as skipped and drops it. The records beneath it are
|
naming the real path, counts the operand as skipped and drops it. The records
|
||||||
kept, unless it lies under another operand. Then they are deleted, because
|
beneath it are kept, unless it lies under another operand. Then they are
|
||||||
that operand's walk never enters `.zfs` (see "Rules for the walk").
|
deleted, because that operand's walk never enters `.zfs` (see "Rules for the
|
||||||
|
walk").
|
||||||
- An operand on a filesystem that is not persistent is dropped too. `scan`
|
- An operand on a filesystem that is not persistent is dropped too. `scan`
|
||||||
prints a line naming it and its filesystem type, and does not count it as
|
prints a line naming it and its filesystem type, and does not count it as
|
||||||
skipped.
|
skipped.
|
||||||
@@ -484,12 +487,12 @@ walk practical on large or busy storage pools.
|
|||||||
The walk knows them by device ID and inode number. The device ID is the
|
The walk knows them by device ID and inode number. The device ID is the
|
||||||
number by which the kernel tells filesystems apart. The inode number tells
|
number by which the kernel tells filesystems apart. The inode number tells
|
||||||
the files of one filesystem apart. So the walk passes over them whatever
|
the files of one filesystem apart. So the walk passes over them whatever
|
||||||
path leads to them. The walk follows no symlink it finds. It does open each
|
path leads to them. The walk follows no symlink it finds. It opens each
|
||||||
operand at the path given, even when that path runs through a symlinked
|
operand at its real path, not the path given. An operand given through a
|
||||||
directory. Such an operand can lead to the database files, and the walk
|
symlink can lead to the database files, and the walk passes over them there
|
||||||
passes over them there as well. The walk prints no line for the database
|
as well. The walk prints no line for the database files and does not count
|
||||||
files and does not count them as skipped. A record that an earlier scan left
|
them as skipped. A record that an earlier scan left for one of them is
|
||||||
for one of them is deleted.
|
deleted.
|
||||||
|
|
||||||
- By default the walk enters directories on other filesystems than its
|
- By default the walk enters directories on other filesystems than its
|
||||||
operand's. With `-x`, it never enters a directory on a different filesystem
|
operand's. With `-x`, it never enters a directory on a different filesystem
|
||||||
@@ -1138,7 +1141,7 @@ showing the count it stopped at.
|
|||||||
- `0`: success, even when files were skipped with warnings.
|
- `0`: success, even when files were skipped with warnings.
|
||||||
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal
|
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal
|
||||||
error is one of these:
|
error is one of these:
|
||||||
- an operand that does not exist, or whose details `scan` cannot read;
|
- an operand that does not exist, or whose real path `scan` cannot find;
|
||||||
- a list of mounted filesystems that `scan` cannot read;
|
- a list of mounted filesystems that `scan` cannot read;
|
||||||
- another `scan` holding the lock (see "The scan lock");
|
- another `scan` holding the lock (see "The scan lock");
|
||||||
- a database, its directory or its lock file that cannot be created or
|
- a database, its directory or its lock file that cannot be created or
|
||||||
|
|||||||
@@ -29,6 +29,12 @@
|
|||||||
Newest first. An entry ends with its date and its issue or branch, where those
|
Newest first. An entry ends with its date and its issue or branch, where those
|
||||||
are known.
|
are known.
|
||||||
|
|
||||||
|
- `scan` resolves each operand to its real path, symlinks followed, before the
|
||||||
|
walk. A symlink operand is scanned at its target instead of skipped. With
|
||||||
|
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
|
||||||
|
walks `sub` on its own instead of dropping it and deleting its records
|
||||||
|
(2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144)
|
||||||
|
|
||||||
- `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's
|
- `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's
|
||||||
blocks only seconds after the write. The count can land in under a second.
|
blocks only seconds after the write. The count can land in under a second.
|
||||||
Both now say up to seconds after the write, as the comment on `diskSizeOf` in
|
Both now say up to seconds after the write, as the comment on `diskSizeOf` in
|
||||||
|
|||||||
@@ -146,33 +146,44 @@ var errHashSize = errors.New("hash is not 32 bytes")
|
|||||||
|
|
||||||
// databasePath resolves the database location: SFDUPES_DATABASE when
|
// databasePath resolves the database location: SFDUPES_DATABASE when
|
||||||
// set and non-empty, the compiled-in default otherwise. It returns the
|
// set and non-empty, the compiled-in default otherwise. It returns the
|
||||||
// path, with the working directory, its symlinks resolved, put in front
|
// path, made absolute by absolutePath, and where the path came from,
|
||||||
// of it when it is relative, and where the path came from, "default" or
|
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
||||||
// "from SFDUPES_DATABASE", for the line each subcommand prints first on
|
// prints first on stderr.
|
||||||
// stderr.
|
|
||||||
func databasePath() (string, string, error) {
|
func databasePath() (string, string, error) {
|
||||||
path, source := defaultDatabasePath, "default"
|
path, source := defaultDatabasePath, "default"
|
||||||
if p := os.Getenv(databaseEnv); p != "" {
|
if p := os.Getenv(databaseEnv); p != "" {
|
||||||
path, source = p, "from "+databaseEnv
|
path, source = p, "from "+databaseEnv
|
||||||
}
|
}
|
||||||
|
|
||||||
if filepath.IsAbs(path) {
|
abs, err := absolutePath(path)
|
||||||
return path, source, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// The working directory as the kernel reports it, with no symlink
|
|
||||||
// in it, as README "Database" says. os.Getwd may return PWD
|
|
||||||
// instead, which can name it through a symlink.
|
|
||||||
wd, err := unix.Getwd()
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Not filepath.Abs or filepath.Join: they clean the path, which
|
return abs, source, nil
|
||||||
// drops each ".." together with the directory name before it. When
|
}
|
||||||
// that directory is a symlink, the cleaned path names another file.
|
|
||||||
|
// absolutePath returns path unchanged when it is absolute, and
|
||||||
|
// otherwise the working directory, a "/" and path. The working
|
||||||
|
// directory is the one the kernel reports, with no symlink in it;
|
||||||
|
// os.Getwd may return PWD instead, which can name it through a symlink.
|
||||||
|
// The result is not cleaned, as filepath.Abs and filepath.Join would
|
||||||
|
// clean it. Cleaning drops each ".." together with the directory name
|
||||||
|
// before it. When that name is a symlink, the cleaned path names
|
||||||
|
// another file than the one the system reaches by following the symlink
|
||||||
|
// first.
|
||||||
|
func absolutePath(path string) (string, error) {
|
||||||
|
if filepath.IsAbs(path) {
|
||||||
|
return path, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
wd, err := unix.Getwd()
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
// The working directory ends in "/" only when it is the root.
|
// The working directory ends in "/" only when it is the root.
|
||||||
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
|
return strings.TrimSuffix(wd, "/") + "/" + path, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// createDatabaseDirectory creates the directory the database at path
|
// createDatabaseDirectory creates the directory the database at path
|
||||||
|
|||||||
+250
-58
@@ -14,6 +14,8 @@ import (
|
|||||||
"strconv"
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"golang.org/x/sys/unix"
|
||||||
)
|
)
|
||||||
|
|
||||||
// usageMarker is the first line of cobra's usage text, which a usage
|
// usageMarker is the first line of cobra's usage text, which a usage
|
||||||
@@ -895,54 +897,262 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
|||||||
assertNoSidecars(t, path)
|
assertNoSidecars(t, path)
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunScanSkipsSymlinkOperand(t *testing.T) {
|
// realTempDir returns a new temporary directory by its real path, so
|
||||||
path := testDBPath(t)
|
// tests hold where the temporary directory is reached through a
|
||||||
t.Setenv(databaseEnv, path)
|
// symlink.
|
||||||
|
func realTempDir(t *testing.T) string {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir, err := filepath.EvalSymlinks(t.TempDir())
|
||||||
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
|
||||||
|
|
||||||
link := filepath.Join(dir, "link")
|
|
||||||
|
|
||||||
err := os.Symlink(filepath.Join(dir, "target"), link)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Scanning a directory through the symlink stores a record beneath
|
return dir
|
||||||
// the symlink's own path for a file beneath its target.
|
|
||||||
scanOK(t, filepath.Join(link, "sub"))
|
|
||||||
|
|
||||||
assertOperandSkipped(t, path, link, "symlink",
|
|
||||||
filepath.Join(link, "sub", "f"))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// symlink creates link pointing at target.
|
||||||
|
func symlink(t *testing.T, target, link string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
err := os.Symlink(target, link)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// assertSummary checks that stderr holds the scan summary line want.
|
||||||
|
func assertSummary(t *testing.T, stderr, want string) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
if !strings.Contains(stderr, want) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanFollowsSymlinkOperand scans link, a symlink to target.
|
||||||
|
// The operand is scanned at its real path, and its file is recorded
|
||||||
|
// there.
|
||||||
|
func TestRunScanFollowsSymlinkOperand(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||||
|
|
||||||
|
link := filepath.Join(dir, "link")
|
||||||
|
symlink(t, filepath.Join(dir, "target"), link)
|
||||||
|
|
||||||
|
assertSummary(t, scanOK(t, link), "scan: 1 files seen (1 added, "+
|
||||||
|
"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
|
||||||
|
|
||||||
|
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||||
|
t.Errorf("records = %q, want %q", got, []string{f})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanWalksOperandUnderSymlinkOperand scans link and link/sub,
|
||||||
|
// with link a symlink to target. Both resolve under target, so
|
||||||
|
// target/sub is walked once, as part of target.
|
||||||
func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
|
func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
|
||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := realTempDir(t)
|
||||||
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||||
|
|
||||||
link := filepath.Join(dir, "link")
|
link := filepath.Join(dir, "link")
|
||||||
|
symlink(t, filepath.Join(dir, "target"), link)
|
||||||
|
|
||||||
err := os.Symlink(filepath.Join(dir, "target"), link)
|
assertSummary(t, scanOK(t, link, filepath.Join(link, "sub")),
|
||||||
|
"scan: 1 files seen (1 added, 0 updated, 0 removed, "+
|
||||||
|
"0 unchanged), 0 skipped\n")
|
||||||
|
|
||||||
|
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||||
|
t.Errorf("records = %q, want %q", got, []string{f})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanWalksSymlinkedOperandBelowOperand is the case of
|
||||||
|
// https://git.eeqj.de/sneak/sfdupes/issues/144: scan dir dir/link/sub,
|
||||||
|
// with dir/link a symlink to a directory outside dir. The walk of dir
|
||||||
|
// does not follow link, so dir/link/sub is walked on its own at its
|
||||||
|
// real path, and the record an earlier scan stored there survives.
|
||||||
|
func TestRunScanWalksSymlinkedOperandBelowOperand(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
a := writeFile(t, dir, "a", pattern(1, 10))
|
||||||
|
|
||||||
|
outside := realTempDir(t)
|
||||||
|
f := writeFile(t, outside, "sub/f", pattern(2, 10))
|
||||||
|
|
||||||
|
symlink(t, outside, filepath.Join(dir, "link"))
|
||||||
|
|
||||||
|
scanOK(t, filepath.Join(outside, "sub"))
|
||||||
|
|
||||||
|
stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub"))
|
||||||
|
assertSummary(t, stderr, "scan: 2 files seen (1 added, 0 updated, "+
|
||||||
|
"0 removed, 1 unchanged), 0 skipped\n")
|
||||||
|
|
||||||
|
want := []string{a, f}
|
||||||
|
slices.Sort(want)
|
||||||
|
|
||||||
|
if got := storedPaths(t, path); !slices.Equal(got, want) {
|
||||||
|
t.Errorf("records = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanDropsSymlinkOperandInsideOperand scans dir and link, with
|
||||||
|
// link, outside dir, a symlink to dir/real. link resolves inside dir,
|
||||||
|
// so it is dropped as covered by dir, and its file is recorded once.
|
||||||
|
func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
f := writeFile(t, dir, "real/f", pattern(1, 10))
|
||||||
|
|
||||||
|
link := filepath.Join(realTempDir(t), "link")
|
||||||
|
symlink(t, filepath.Join(dir, "real"), link)
|
||||||
|
|
||||||
|
assertSummary(t, scanOK(t, dir, link), "scan: 1 files seen (1 added, "+
|
||||||
|
"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
|
||||||
|
|
||||||
|
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||||
|
t.Errorf("records = %q, want %q", got, []string{f})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanDanglingSymlinkOperandIsFatal scans a valid directory and
|
||||||
|
// a symlink whose target does not exist. The scan fails with exit 1 and
|
||||||
|
// an error naming the symlink as given, before it walks anything, so no
|
||||||
|
// database is created.
|
||||||
|
func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
writeFile(t, dir, "f", pattern(1, 10))
|
||||||
|
|
||||||
|
dangling := filepath.Join(realTempDir(t), "dangling")
|
||||||
|
symlink(t, filepath.Join(dir, "missing"), dangling)
|
||||||
|
|
||||||
|
stdout := captureStdout(t)
|
||||||
|
|
||||||
|
var stderr bytes.Buffer
|
||||||
|
|
||||||
|
code := run([]string{cmdScan, dir, dangling}, os.Stdout, &stderr)
|
||||||
|
if code != exitFatal {
|
||||||
|
t.Errorf("run(scan %s %s) = %d, want %d", dir, dangling, code,
|
||||||
|
exitFatal)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertFatalOutput(t, stderr.String(), stdout())
|
||||||
|
|
||||||
|
want := "sfdupes: resolve " + dangling + ": "
|
||||||
|
if !strings.Contains(stderr.String(), want) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr.String(), want)
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := os.Stat(path)
|
||||||
|
if !errors.Is(err, fs.ErrNotExist) {
|
||||||
|
t.Errorf("database: stat error %v, want it missing", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath runs scan .
|
||||||
|
// after a cd through home/proj, a symlink to data/proj. The file is
|
||||||
|
// recorded under data/proj, its real path.
|
||||||
|
func TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
|
||||||
|
|
||||||
|
err := os.Mkdir(filepath.Join(dir, "home"), 0o750)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// link is dropped as a symlink, but link/sub must still be scanned,
|
symlink(t, filepath.Join(dir, "data", "proj"),
|
||||||
// not dropped as lying under link.
|
filepath.Join(dir, "home", "proj"))
|
||||||
scanOK(t, link, filepath.Join(link, "sub"))
|
|
||||||
|
|
||||||
db, err := openDB(path, reportParams)
|
// t.Chdir also sets PWD to the path through the symlink, as a
|
||||||
|
// shell does after cd.
|
||||||
|
t.Chdir(filepath.Join(dir, "home", "proj"))
|
||||||
|
|
||||||
|
scanOK(t, ".")
|
||||||
|
|
||||||
|
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||||
|
t.Errorf("records = %q, want %q", got, []string{f})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanFollowsSymlinkBeforeDotDot scans a path with a ".." after
|
||||||
|
// home/proj, a symlink to data/proj: home/proj/.. from dir, and .. after
|
||||||
|
// a cd through home/proj. As with realpath, the ".." counts from
|
||||||
|
// data/proj, so data is scanned and home is not.
|
||||||
|
func TestRunScanFollowsSymlinkBeforeDotDot(t *testing.T) {
|
||||||
|
for _, tc := range []struct{ wd, operand string }{
|
||||||
|
{"", "home/proj/.."},
|
||||||
|
{"home/proj", ".."},
|
||||||
|
} {
|
||||||
|
t.Run(tc.operand, func(t *testing.T) {
|
||||||
|
path := testDBPath(t)
|
||||||
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
|
||||||
|
g := writeFile(t, dir, "data/g", pattern(2, 10))
|
||||||
|
writeFile(t, dir, "home/h", pattern(3, 10))
|
||||||
|
symlink(t, filepath.Join(dir, "data", "proj"),
|
||||||
|
filepath.Join(dir, "home", "proj"))
|
||||||
|
|
||||||
|
// t.Chdir also sets PWD to the path given, as a shell does
|
||||||
|
// after cd.
|
||||||
|
t.Chdir(filepath.Join(dir, tc.wd))
|
||||||
|
|
||||||
|
scanOK(t, tc.operand)
|
||||||
|
|
||||||
|
want := []string{f, g}
|
||||||
|
slices.Sort(want)
|
||||||
|
|
||||||
|
if got := storedPaths(t, path); !slices.Equal(got, want) {
|
||||||
|
t.Errorf("records = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestRunScanSkipsSymlinkToFIFOOperand scans link, a symlink to a FIFO.
|
||||||
|
// The scan warns about the FIFO by its real path, counts it as skipped,
|
||||||
|
// and exits 0.
|
||||||
|
func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) {
|
||||||
|
t.Setenv(databaseEnv, testDBPath(t))
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
fifo := filepath.Join(dir, "fifo")
|
||||||
|
|
||||||
|
err := unix.Mkfifo(fifo, 0o600)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
link := filepath.Join(dir, "link")
|
||||||
|
symlink(t, fifo, link)
|
||||||
|
|
||||||
recordByPath(t, dbRecords(t, db), filepath.Join(link, "sub", "f"))
|
stderr := scanOK(t, link)
|
||||||
|
|
||||||
|
warning := "walk " + fifo + ": skipping FIFO operand\n"
|
||||||
|
if !strings.Contains(stderr, warning) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertSummary(t, stderr, "scan: 0 files seen (0 added, 0 updated, "+
|
||||||
|
"0 removed, 0 unchanged), 1 skipped\n")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunScanSkipsZFSOperand(t *testing.T) {
|
func TestRunScanSkipsZFSOperand(t *testing.T) {
|
||||||
@@ -1041,23 +1251,19 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
|
|||||||
|
|
||||||
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
|
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
|
||||||
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
|
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
|
||||||
// by the path given, counted as skipped, and not walked.
|
// by its real path, counted as skipped, and not walked.
|
||||||
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
||||||
t.Setenv(databaseEnv, testDBPath(t))
|
t.Setenv(databaseEnv, testDBPath(t))
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := realTempDir(t)
|
||||||
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
|
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
|
||||||
|
|
||||||
snaps := filepath.Join(dir, "snaps")
|
snaps := filepath.Join(dir, "snaps")
|
||||||
|
symlink(t, filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
|
||||||
|
|
||||||
err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
|
stderr := scanOK(t, filepath.Join(snaps, "daily"))
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
daily := filepath.Join(snaps, "daily")
|
|
||||||
stderr := scanOK(t, daily)
|
|
||||||
|
|
||||||
|
daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily")
|
||||||
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
|
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
|
||||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
||||||
"0 unchanged), 1 skipped\n"
|
"0 unchanged), 1 skipped\n"
|
||||||
@@ -1069,39 +1275,25 @@ func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
|
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
|
||||||
// a symlink to a directory outside any .zfs directory. The operand is
|
// a symlink to pool/data. The operand is scanned as usual, and its file
|
||||||
// scanned as usual, and its file is recorded by the path given.
|
// is recorded by its real path.
|
||||||
func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
|
func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
|
||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := realTempDir(t)
|
||||||
writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
|
f := writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
|
||||||
|
|
||||||
data := filepath.Join(dir, "data")
|
data := filepath.Join(dir, "data")
|
||||||
|
symlink(t, filepath.Join(dir, "pool", "data"), data)
|
||||||
|
|
||||||
err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
|
assertSummary(t, scanOK(t, filepath.Join(data, "daily")),
|
||||||
if err != nil {
|
"scan: 1 files seen (1 added, 0 updated, 0 removed, "+
|
||||||
t.Fatal(err)
|
"0 unchanged), 0 skipped\n")
|
||||||
|
|
||||||
|
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||||
|
t.Errorf("records = %q, want %q", got, []string{f})
|
||||||
}
|
}
|
||||||
|
|
||||||
daily := filepath.Join(data, "daily")
|
|
||||||
stderr := scanOK(t, daily)
|
|
||||||
|
|
||||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
|
||||||
"0 unchanged), 0 skipped\n"
|
|
||||||
if !strings.Contains(stderr, summary) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
|
||||||
}
|
|
||||||
|
|
||||||
db, err := openDB(path, reportParams)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
t.Cleanup(func() { _ = db.Close() })
|
|
||||||
|
|
||||||
recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
|
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunReportSucceeds(t *testing.T) {
|
func TestRunReportSucceeds(t *testing.T) {
|
||||||
|
|||||||
@@ -209,29 +209,30 @@ func interrupted(walked int) error {
|
|||||||
return errInterrupted
|
return errInterrupted
|
||||||
}
|
}
|
||||||
|
|
||||||
// resolveRoots converts each PATH operand to an absolute, lexically
|
// resolveRoots converts each PATH operand to its real path: absolute,
|
||||||
// cleaned path (symlinks are not resolved) and verifies that it
|
// with every symlink in it followed. Database records are keyed by
|
||||||
// exists. Database records are keyed by absolute path, so scan results
|
// that path, so scan results depend neither on the working directory
|
||||||
// must not depend on the working directory.
|
// nor on the symlinks an operand was given through. An operand that
|
||||||
|
// cannot be resolved, because it or a symlink's target does not exist
|
||||||
|
// or cannot be read, is a fatal error before any scanning.
|
||||||
func resolveRoots(roots []string) ([]string, error) {
|
func resolveRoots(roots []string) ([]string, error) {
|
||||||
abs := make([]string, 0, len(roots))
|
resolved := make([]string, 0, len(roots))
|
||||||
|
|
||||||
for _, root := range roots {
|
for _, root := range roots {
|
||||||
a, err := filepath.Abs(root)
|
a, err := absolutePath(root)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// A nonexistent operand is a fatal error before any scanning.
|
r, err := filepath.EvalSymlinks(a)
|
||||||
_, err = os.Lstat(a)
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
abs = append(abs, a)
|
resolved = append(resolved, r)
|
||||||
}
|
}
|
||||||
|
|
||||||
return abs, nil
|
return resolved, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// pruneRoots drops operands already covered by another operand:
|
// pruneRoots drops operands already covered by another operand:
|
||||||
@@ -417,14 +418,16 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
|||||||
|
|
||||||
// walkableRoots returns the operands the walk can start from: regular
|
// walkableRoots returns the operands the walk can start from: regular
|
||||||
// files, and directories not named .zfs, none of them inside a
|
// files, and directories not named .zfs, none of them inside a
|
||||||
// directory named .zfs. Every other operand is warned about, counted as
|
// directory named .zfs. The operands are real paths from resolveRoots,
|
||||||
// skipped, and dropped. A dropped operand is no longer a root, so the
|
// so these checks apply to what a symlink operand points at. Every
|
||||||
// records stored beneath it are treated as if it had not been given:
|
// other operand is warned about, counted as skipped, and dropped. A
|
||||||
// those under no other root count as outside the roots and are not
|
// dropped operand is no longer a root, so the records stored beneath it
|
||||||
// deleted as unverified. An operand on a filesystem in rules that is not
|
// are treated as if it had not been given: those under no other root
|
||||||
// persistent is dropped the same way, with a line saying so that is not
|
// count as outside the roots and are not deleted as unverified. An
|
||||||
// a warning, and is not counted as skipped. An operand that fails lstat
|
// operand on a filesystem in rules that is not persistent is dropped
|
||||||
// here is kept, and the walk warns about it unless it no longer exists.
|
// the same way, with a line saying so that is not a warning, and is not
|
||||||
|
// counted as skipped. An operand that fails lstat here is kept, and the
|
||||||
|
// walk warns about it unless it no longer exists.
|
||||||
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
||||||
kept := make([]string, 0, len(roots))
|
kept := make([]string, 0, len(roots))
|
||||||
|
|
||||||
@@ -894,19 +897,13 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
|
|||||||
// operandWarning returns the one-line warning for an operand the walk
|
// operandWarning returns the one-line warning for an operand the walk
|
||||||
// does not start from, naming the path and what it is, or "" for one it
|
// does not start from, naming the path and what it is, or "" for one it
|
||||||
// does: a regular file, or a directory not named .zfs, that is not
|
// does: a regular file, or a directory not named .zfs, that is not
|
||||||
// inside a directory named .zfs, neither as given nor with its symlinks
|
// inside a directory named .zfs. resolveRoots has already followed the
|
||||||
// resolved. Symlinks are never followed, including as operands.
|
// operand's symlinks, so a symlink here is an operand replaced by one
|
||||||
|
// after it was resolved, and is not followed.
|
||||||
func operandWarning(root string, fi fs.FileInfo) string {
|
func operandWarning(root string, fi fs.FileInfo) string {
|
||||||
// The walk never enters a .zfs directory, so it starts from nothing
|
// The walk never enters a .zfs directory, so it starts from nothing
|
||||||
// inside one either, not even through a symlink. An operand whose
|
// inside one either.
|
||||||
// symlinks cannot be resolved is checked as given.
|
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") {
|
||||||
resolved, err := filepath.EvalSymlinks(root)
|
|
||||||
if err != nil {
|
|
||||||
resolved = root
|
|
||||||
}
|
|
||||||
|
|
||||||
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
|
|
||||||
slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
|
|
||||||
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
|
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
|
||||||
root)
|
root)
|
||||||
}
|
}
|
||||||
|
|||||||
+35
-28
@@ -2310,33 +2310,24 @@ func TestUnderRoot(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
|
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
|
||||||
// database, reached directly and through a symlinked directory. The
|
// database, and in one case also a hard link to the database file. The
|
||||||
// walk passes over the database file, its -wal and -shm files and its
|
// walk knows the database's files by device ID and inode number, so it
|
||||||
// .lock file with no line and no count, and the record an earlier scan
|
// passes over the database file, its -wal and -shm files, its .lock
|
||||||
// left for the .lock file is deleted. A rescan of the unchanged tree
|
// file and the hard link with no line and no count. The record an
|
||||||
// updates nothing, and report lists none of the database's files.
|
// earlier scan left for the .lock file is deleted. A rescan of the
|
||||||
|
// unchanged tree updates nothing, and report lists none of the
|
||||||
|
// database's files.
|
||||||
func TestRunScanPassesOverDatabase(t *testing.T) {
|
func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||||
for _, viaSymlink := range []bool{false, true} {
|
for _, viaHardLink := range []bool{false, true} {
|
||||||
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
|
t.Run(fmt.Sprintf("via hard link %v", viaHardLink), func(t *testing.T) {
|
||||||
base := t.TempDir()
|
tree := realTempDir(t)
|
||||||
tree := filepath.Join(base, "real", "tree")
|
hardLink := filepath.Join(tree, "hardlink")
|
||||||
operand := tree
|
|
||||||
|
|
||||||
writeFile(t, tree, "a", pattern(1, 100))
|
writeFile(t, tree, "a", pattern(1, 100))
|
||||||
writeFile(t, tree, "b", pattern(1, 100))
|
writeFile(t, tree, "b", pattern(1, 100))
|
||||||
writeFile(t, tree, "empty1", nil)
|
writeFile(t, tree, "empty1", nil)
|
||||||
writeFile(t, tree, "empty2", nil)
|
writeFile(t, tree, "empty2", nil)
|
||||||
|
|
||||||
if viaSymlink {
|
|
||||||
err := os.Symlink(filepath.Join(base, "real"),
|
|
||||||
filepath.Join(base, "link"))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
operand = filepath.Join(base, "link", "tree")
|
|
||||||
}
|
|
||||||
|
|
||||||
dbPath := filepath.Join(tree, "db.sqlite")
|
dbPath := filepath.Join(tree, "db.sqlite")
|
||||||
t.Setenv(databaseEnv, dbPath)
|
t.Setenv(databaseEnv, dbPath)
|
||||||
|
|
||||||
@@ -2346,7 +2337,7 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
err = applyChanges(t.Context(), db, []scanRec{
|
err = applyChanges(t.Context(), db, []scanRec{
|
||||||
{path: filepath.Join(operand, "db.sqlite.lock")},
|
{path: filepath.Join(tree, "db.sqlite.lock")},
|
||||||
}, nil, nil)
|
}, nil, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -2354,16 +2345,29 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
|||||||
|
|
||||||
closeScanDatabase(t.Context(), db, dbPath)
|
closeScanDatabase(t.Context(), db, dbPath)
|
||||||
|
|
||||||
assertScanSummary(t,
|
if viaHardLink {
|
||||||
|
err = os.Link(dbPath, hardLink)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stderr := assertScanSummary(t,
|
||||||
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
|
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
|
||||||
operand)
|
tree)
|
||||||
assertScanSummary(t,
|
stderr += assertScanSummary(t,
|
||||||
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
|
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
|
||||||
operand)
|
tree)
|
||||||
|
|
||||||
|
if strings.Contains(stderr, hardLink) {
|
||||||
|
t.Errorf("scan stderr = %q, want no line naming %s",
|
||||||
|
stderr, hardLink)
|
||||||
|
}
|
||||||
|
|
||||||
report := runStdout(t, cmdReport)
|
report := runStdout(t, cmdReport)
|
||||||
if strings.Contains(report, "db.sqlite") ||
|
if strings.Contains(report, "db.sqlite") ||
|
||||||
!strings.Contains(report, filepath.Join(operand, "empty2")) {
|
strings.Contains(report, hardLink) ||
|
||||||
|
!strings.Contains(report, filepath.Join(tree, "empty2")) {
|
||||||
t.Errorf("report = %q, want the empty files and none of "+
|
t.Errorf("report = %q, want the empty files and none of "+
|
||||||
"the database's files", report)
|
"the database's files", report)
|
||||||
}
|
}
|
||||||
@@ -2388,8 +2392,9 @@ func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
|
|||||||
|
|
||||||
// assertScanSummary scans operands and fails the test unless the
|
// assertScanSummary scans operands and fails the test unless the
|
||||||
// summary line gives counts with nothing skipped, and the line naming
|
// summary line gives counts with nothing skipped, and the line naming
|
||||||
// the database is the only one that names one of its files.
|
// the database is the only one that names one of its files. It returns
|
||||||
func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
// everything the scan printed to stderr.
|
||||||
|
func assertScanSummary(t *testing.T, counts string, operands ...string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
stderr := scanOK(t, operands...)
|
stderr := scanOK(t, operands...)
|
||||||
@@ -2403,4 +2408,6 @@ func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
|||||||
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
|
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
|
||||||
"database to name one of its files", operands, stderr)
|
"database to name one of its files", operands, stderr)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return stderr
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user