2 Commits
Author SHA1 Message Date
clawbot da75b713aa Add sfdupes dwim: scan / then print duplicate trees (closes #154)
check / check (push) Successful in 1m18s
sfdupes dwim, as sneak ordered: with no arguments it runs what scan / runs with no flags, default excludes on, then what trees runs, printing the duplicate trees largest first on stdout. Its last line, on stderr, says that sfdupes trees prints the report again. It needs no setup beyond what scan creates. SIGINT or SIGTERM during the scan stops it as it stops scan, exit 1, with no report; during the report a signal ends it at once, as it ends trees. A positional argument is a usage error, exit 2.

Model: opus-5-5
2026-11-06 20:27:24 +01:00
clawbot 24b0fc486d Resolve scan operands to their real paths (closes #144)
check / check (push) Canceled after 0s
scan now resolves each operand to its real path, every symlink followed, before the walk, as sneak ruled. Files are recorded and warnings printed under that path. With /srv/link a symlink to a directory outside /srv, scan /srv /srv/link/sub now walks sub on its own and keeps its records, where before it dropped sub and deleted them. A symlink operand is scanned at its target. A dangling symlink is a fatal error, exit 1. A .. after a symlink counts from the symlink's target, as realpath does. The .zfs rule applies to the real path. The walk still follows no symlink it finds.

Judgement call: records an earlier scan stored under a path through a symlink are not migrated; there is no installed base.

Model: opus-5-5
2026-11-06 19:56:33 +01:00
10 changed files with 378 additions and 103 deletions

No files matched your search

+47 -14
View File
@@ -17,12 +17,13 @@ gigabyte. A large file is never read in full. A match between two large files is
a strong candidate, not proof. a strong candidate, not proof.
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every `sfdupes` is meant for multi-terabyte servers, where reading every byte of every
file takes too long. It has four subcommands. `scan` walks the filesystem and file takes too long. It has five subcommands. `scan` walks the filesystem and
records the files it finds in an SQLite database. SQLite keeps a whole database records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` read the database alone and system's job scheduler. `report`, `trees` and `info` read the database alone and
can run at any time. can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
first look at a new machine with one command.
## Getting Started ## Getting Started
@@ -247,12 +248,15 @@ In order:
- `report` prints the file-level duplicate report from the database. - `report` prints the file-level duplicate report from the database.
- `trees` prints the duplicate-tree report from the database. - `trees` prints the duplicate-tree report from the database.
- `info` prints the database's details. - `info` prints the database's details.
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
report.
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv sfdupes trees > dupetrees.tsv
sfdupes info sfdupes info
sfdupes dwim
sfdupes --version sfdupes --version
sfdupes [command] --help sfdupes [command] --help
``` ```
@@ -352,9 +356,10 @@ default. An operand may be a directory or a regular file.
target does not exist. `scan` then stops before it walks anything, and exits target does not exist. `scan` then stops before it walks anything, and exits
with code 1. with code 1.
- Each operand is resolved to its real path before the walk, as `realpath` - Each operand is resolved to its real path before the walk, as `realpath`
prints it. The path is made absolute, and every symlink in it is followed. prints it. A relative operand is joined to the working directory as `pwd -P`
From then on the operand is that real path. Its files are recorded under it, prints it. Every symlink in the path is then followed. A `..` after a symlink
and warnings name it. counts from the directory the symlink leads to. From then on the operand is
that real path. Its files are recorded under it, and warnings name it.
- An operand that is a symlink is scanned at its target's real path. After a - An operand that is a symlink is scanned at its target's real path. After a
`cd` through a symlink, `scan .` records its files under the real path of the `cd` through a symlink, `scan .` records its files under the real path of the
working directory. working directory.
@@ -363,8 +368,10 @@ default. An operand may be a directory or a regular file.
- An operand that repeats another is dropped before the walk. Two operands - An operand that repeats another is dropped before the walk. Two operands
repeat each other when their real paths are the same. repeat each other when their real paths are the same.
- An operand under another operand is dropped the same way, unless one of the - An operand under another operand is dropped the same way, unless one of the
cases below applies. This too is judged on real paths. No path is then walked cases below applies. No path is then walked twice. Whether one operand is
twice. under another is judged on their real paths. With `/srv/link` a symlink to a
directory outside `/srv`, `scan /srv /srv/link/sub` walks `sub` on its own,
because the real path of `sub` is not under `/srv`.
- An operand under another is kept, and walked on its own, when the inner - An operand under another is kept, and walked on its own, when the inner
operand or a directory between the two is one of these: operand or a directory between the two is one of these:
- a default exclude (see "Rules for the walk"); - a default exclude (see "Rules for the walk");
@@ -376,9 +383,6 @@ default. An operand may be a directory or a regular file.
such a directory, so the inner operand is walked on its own. such a directory, so the inner operand is walked on its own.
- No other case keeps an operand under another. - No other case keeps an operand under another.
- An operand given through a symlinked directory below another operand, as in
`scan /srv /srv/link/sub`, is not under it. It resolves to the real path of
`sub`, which lies outside `/srv`, so it is walked on its own.
- An operand that is neither a regular file nor a directory is not scanned. Such - An operand that is neither a regular file nor a directory is not scanned. Such
an operand is, for example, a symlink to a device node. `scan` prints a an operand is, for example, a symlink to a device node. `scan` prints a
warning naming the real path and what it is. It counts the operand as skipped warning naming the real path and what it is. It counts the operand as skipped
@@ -800,6 +804,33 @@ The summary line on stderr gives:
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
``` ```
### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments and no flags of
its own. Like `scan`, it should run as root, so that it can read every file.
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
excludes and crosses into other filesystems. `--workers` keeps its default. The
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
what `sfdupes trees` runs. The largest trees come first.
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
`scan` prints, then what `trees` prints, then one last line:
```
dwim: run "sfdupes trees" to print this report again
```
The database keeps the scan, so `sfdupes trees` prints the same report later,
even after the terminal is gone.
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
`dwim` then prints no report and exits 1. A fatal scan error also ends it before
the report, with the same message and exit code as `scan`. Warnings for skipped
files do not stop it. `dwim` stops catching signals when the scan ends, so a
signal during the report ends it at once, as it ends `trees`.
### `info` mode ### `info` mode
`info` takes no positional arguments and prints the database's details on `info` takes no positional arguments and prints the database's details on
@@ -840,7 +871,7 @@ scan is writing. `info` writes its lines after the `count` display has ended.
### Database ### Database
All four subcommands work on one SQLite database file. All five subcommands work on one SQLite database file.
#### Location #### Location
@@ -856,6 +887,8 @@ scan: new database /var/lib/sfdupes/db.sqlite (default)
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE) trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
``` ```
`dwim` prints the `scan` line first, and the `trees` line after the scan.
A relative path is printed with the working directory in front of it. The A relative path is printed with the working directory in front of it. The
working directory is printed as `pwd -P` prints it, with its symlinks resolved. working directory is printed as `pwd -P` prints it, with its symlinks resolved.
The rest of the path is printed as set, so a `..` in it stays. The line names The rest of the path is printed as set, so a `..` in it stays. The line names
@@ -1139,8 +1172,8 @@ showing the count it stopped at.
### Error handling and exit codes ### Error handling and exit codes
- `0`: success, even when files were skipped with warnings. - `0`: success, even when files were skipped with warnings.
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal - `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
error is one of these: stopped by either during its scan. A fatal error is one of these:
- an operand that does not exist, or whose real path `scan` cannot find; - an operand that does not exist, or whose real path `scan` cannot find;
- a list of mounted filesystems that `scan` cannot read; - a list of mounted filesystems that `scan` cannot read;
- another `scan` holding the lock (see "The scan lock"); - another `scan` holding the lock (see "The scan lock");
@@ -1161,7 +1194,7 @@ showing the count it stopped at.
`--workers abc`; `--workers abc`;
- `scan` with no operand; - `scan` with no operand;
- `--workers` below 1; - `--workers` below 1;
- `report`, `trees` or `info` with a positional argument. - `report`, `trees`, `info` or `dwim` with a positional argument.
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
stdout, such as a full disk, is reported the same way. Two cases never reach stdout, such as a full disk, is reported the same way. Two cases never reach
+9 -4
View File
@@ -29,11 +29,16 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
prints the report again (2026-11-06,
https://git.eeqj.de/sneak/sfdupes/issues/154)
- `scan` resolves each operand to its real path, symlinks followed, before the - `scan` resolves each operand to its real path, symlinks followed, before the
walk. A symlink operand is scanned at its target instead of skipped. An walk. A symlink operand is scanned at its target instead of skipped. With
operand given through a symlinked directory below another operand is walked on `/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
its own instead of dropped with its records deleted (2026-11-06, walks `sub` on its own instead of dropping it and deleting its records
https://git.eeqj.de/sneak/sfdupes/issues/144) (2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144)
- `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's - `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's
blocks only seconds after the write. The count can land in under a second. blocks only seconds after the write. The count can land in under a second.
+27 -16
View File
@@ -146,33 +146,44 @@ var errHashSize = errors.New("hash is not 32 bytes")
// databasePath resolves the database location: SFDUPES_DATABASE when // databasePath resolves the database location: SFDUPES_DATABASE when
// set and non-empty, the compiled-in default otherwise. It returns the // set and non-empty, the compiled-in default otherwise. It returns the
// path, with the working directory, its symlinks resolved, put in front // path, made absolute by absolutePath, and where the path came from,
// of it when it is relative, and where the path came from, "default" or // "default" or "from SFDUPES_DATABASE", for the line each subcommand
// "from SFDUPES_DATABASE", for the line each subcommand prints first on // prints first on stderr.
// stderr.
func databasePath() (string, string, error) { func databasePath() (string, string, error) {
path, source := defaultDatabasePath, "default" path, source := defaultDatabasePath, "default"
if p := os.Getenv(databaseEnv); p != "" { if p := os.Getenv(databaseEnv); p != "" {
path, source = p, "from "+databaseEnv path, source = p, "from "+databaseEnv
} }
if filepath.IsAbs(path) { abs, err := absolutePath(path)
return path, source, nil
}
// The working directory as the kernel reports it, with no symlink
// in it, as README "Database" says. os.Getwd may return PWD
// instead, which can name it through a symlink.
wd, err := unix.Getwd()
if err != nil { if err != nil {
return "", "", fmt.Errorf("resolve database path %s: %w", path, err) return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
} }
// Not filepath.Abs or filepath.Join: they clean the path, which return abs, source, nil
// drops each ".." together with the directory name before it. When }
// that directory is a symlink, the cleaned path names another file.
// absolutePath returns path unchanged when it is absolute, and
// otherwise the working directory, a "/" and path. The working
// directory is the one the kernel reports, with no symlink in it;
// os.Getwd may return PWD instead, which can name it through a symlink.
// The result is not cleaned, as filepath.Abs and filepath.Join would
// clean it. Cleaning drops each ".." together with the directory name
// before it. When that name is a symlink, the cleaned path names
// another file than the one the system reaches by following the symlink
// first.
func absolutePath(path string) (string, error) {
if filepath.IsAbs(path) {
return path, nil
}
wd, err := unix.Getwd()
if err != nil {
return "", err
}
// The working directory ends in "/" only when it is the root. // The working directory ends in "/" only when it is the root.
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil return strings.TrimSuffix(wd, "/") + "/" + path, nil
} }
// createDatabaseDirectory creates the directory the database at path // createDatabaseDirectory creates the directory the database at path
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"context"
"fmt"
"io"
"os"
"runtime"
)
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
// root" runs with no flags, then what "sfdupes trees" runs, and ends
// with a line on stderr saying how to print the report again. The dwim
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
// does, but only during the scan. An error from the scan, an interrupt
// included, ends it before the report. During the report, a signal
// ends it at once, as it ends trees.
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
scanCtx, stop := interruptContext(ctx)
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
stop()
if err != nil {
return err
}
err = runTrees(ctx, stdout)
if err != nil {
return err
}
fmt.Fprintln(os.Stderr,
`dwim: run "sfdupes trees" to print this report again`)
return nil
}
+83
View File
@@ -0,0 +1,83 @@
package main
import (
"bytes"
"path/filepath"
"strings"
"testing"
)
func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
// dwim scans its root, then prints the trees report, the larger of
// two duplicate-tree pairs first, and ends by saying how to print
// the report again.
t.Setenv(databaseEnv, testDBPath(t))
root := realTempDir(t)
small := []string{
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
}
large := []string{
filepath.Dir(writeFile(t, root, "b1/f.bin", pattern(2, 5000))),
filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000))),
}
var stdout bytes.Buffer
stderr := captureStderr(t)
err := runDwim(t.Context(), root, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
if len(lines) != 3 {
t.Fatalf("stdout = %q, want a header and two groups",
stdout.String())
}
wantFirst := large[0] + "\t" + large[1] + "\t1\t5000\t"
if !strings.HasPrefix(lines[1], wantFirst) {
t.Errorf("first group = %q, want it to start %q", lines[1], wantFirst)
}
wantSecond := small[0] + "\t" + small[1] + "\t1\t300\t"
if !strings.HasPrefix(lines[2], wantSecond) {
t.Errorf("second group = %q, want it to start %q",
lines[2], wantSecond)
}
last := "\ndwim: run \"sfdupes trees\" to print this report again\n"
if got := stderr(); !strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to end %q", got, last)
}
}
func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
// A fatal scan error ends dwim before the report: a root that does
// not exist fails the scan as it fails scan itself.
t.Setenv(databaseEnv, testDBPath(t))
missing := filepath.Join(realTempDir(t), "missing")
var stdout bytes.Buffer
stderr := captureStderr(t)
err := runDwim(t.Context(), missing, &stdout)
if err == nil {
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
}
if got := stdout.String(); got != "" {
t.Errorf("stdout = %q, want no report", got)
}
if got := stderr(); strings.Contains(got, "trees:") ||
strings.Contains(got, "dwim:") {
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
}
}
+1 -1
View File
@@ -286,7 +286,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
dir := t.TempDir() dir := realTempDir(t)
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) { if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
t.Skip(dir + " is not inside a default exclude") t.Skip(dir + " is not inside a default exclude")
} }
+14 -3
View File
@@ -14,6 +14,7 @@
// sfdupes report > dupes.tsv // sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv // sfdupes trees > dupetrees.tsv
// sfdupes info // sfdupes info
// sfdupes dwim
// sfdupes --version // sfdupes --version
// //
// See README.md for the complete specification. // See README.md for the complete specification.
@@ -45,6 +46,7 @@ const (
cmdReport = "report" cmdReport = "report"
cmdTrees = "trees" cmdTrees = "trees"
cmdInfo = "info" cmdInfo = "info"
cmdDwim = "dwim"
) )
// errNoSubcommand is returned by the root command when it is invoked // errNoSubcommand is returned by the root command when it is invoked
@@ -96,8 +98,8 @@ func main() {
// exit code. It is the program's single exit point: the subcommands // exit code. It is the program's single exit point: the subcommands
// return their errors instead of exiting, so every deferred cleanup — // return their errors instead of exiting, so every deferred cleanup —
// above all closing the database, which checkpoints the SQLite WAL — // above all closing the database, which checkpoints the SQLite WAL —
// runs before the process ends. The report, trees and info subcommands // runs before the process ends. The report, trees, info and dwim
// write their data to stdout. // subcommands write their data to stdout.
func run(args []string, stdout, stderr io.Writer) int { func run(args []string, stdout, stderr io.Writer) int {
// A nil slice makes cobra fall back to os.Args, which would let a // A nil slice makes cobra fall back to os.Args, which would let a
// test binary's own flags reach the command tree. // test binary's own flags reach the command tree.
@@ -201,7 +203,16 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}), }),
} }
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd) dwimCmd := &cobra.Command{
Use: cmdDwim,
Short: "Scan / with the default excludes, then print the duplicate-tree report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", stdout)
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
return root return root
} }
+116 -27
View File
@@ -14,6 +14,8 @@ import (
"strconv" "strconv"
"strings" "strings"
"testing" "testing"
"golang.org/x/sys/unix"
) )
// usageMarker is the first line of cobra's usage text, which a usage // usageMarker is the first line of cobra's usage text, which a usage
@@ -620,6 +622,7 @@ func TestRunUsageErrors(t *testing.T) {
"report with args": {[]string{cmdReport, "x"}, usageMarker}, "report with args": {[]string{cmdReport, "x"}, usageMarker},
"trees with args": {[]string{cmdTrees, "x"}, usageMarker}, "trees with args": {[]string{cmdTrees, "x"}, usageMarker},
"info with args": {[]string{cmdInfo, "x"}, usageMarker}, "info with args": {[]string{cmdInfo, "x"}, usageMarker},
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker}, "unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
"unknown subcommand": {[]string{"nope"}, "unknown command"}, "unknown subcommand": {[]string{"nope"}, "unknown command"},
} }
@@ -797,7 +800,7 @@ func TestRunVersionWriteFailureIsFatal(t *testing.T) {
func scanFixture(t *testing.T) []string { func scanFixture(t *testing.T) []string {
t.Helper() t.Helper()
dir := t.TempDir() dir := realTempDir(t)
dupes := []string{ dupes := []string{
writeFile(t, dir, "one/a.bin", pattern(1, 300)), writeFile(t, dir, "one/a.bin", pattern(1, 300)),
@@ -867,7 +870,7 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
dir := t.TempDir() dir := realTempDir(t)
writeFile(t, dir, "a.bin", pattern(1, 300)) writeFile(t, dir, "a.bin", pattern(1, 300))
// Same size as a.bin, so the scan reads it, and the read fails. // Same size as a.bin, so the scan reads it, and the read fails.
@@ -919,15 +922,6 @@ func symlink(t *testing.T, target, link string) {
} }
} }
// assertSummary checks that stderr holds the scan summary line want.
func assertSummary(t *testing.T, stderr, want string) {
t.Helper()
if !strings.Contains(stderr, want) {
t.Errorf("stderr = %q, want %q", stderr, want)
}
}
// TestRunScanFollowsSymlinkOperand scans link, a symlink to target. // TestRunScanFollowsSymlinkOperand scans link, a symlink to target.
// The operand is scanned at its real path, and its file is recorded // The operand is scanned at its real path, and its file is recorded
// there. // there.
@@ -941,8 +935,13 @@ func TestRunScanFollowsSymlinkOperand(t *testing.T) {
link := filepath.Join(dir, "link") link := filepath.Join(dir, "link")
symlink(t, filepath.Join(dir, "target"), link) symlink(t, filepath.Join(dir, "target"), link)
assertSummary(t, scanOK(t, link), "scan: 1 files seen (1 added, "+ stderr := scanOK(t, link)
"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) { if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f}) t.Errorf("records = %q, want %q", got, []string{f})
@@ -962,9 +961,13 @@ func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
link := filepath.Join(dir, "link") link := filepath.Join(dir, "link")
symlink(t, filepath.Join(dir, "target"), link) symlink(t, filepath.Join(dir, "target"), link)
assertSummary(t, scanOK(t, link, filepath.Join(link, "sub")), stderr := scanOK(t, link, filepath.Join(link, "sub"))
"scan: 1 files seen (1 added, 0 updated, 0 removed, "+
"0 unchanged), 0 skipped\n") summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) { if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f}) t.Errorf("records = %q, want %q", got, []string{f})
@@ -991,8 +994,12 @@ func TestRunScanWalksSymlinkedOperandBelowOperand(t *testing.T) {
scanOK(t, filepath.Join(outside, "sub")) scanOK(t, filepath.Join(outside, "sub"))
stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub")) stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub"))
assertSummary(t, stderr, "scan: 2 files seen (1 added, 0 updated, "+
"0 removed, 1 unchanged), 0 skipped\n") summary := "scan: 2 files seen (1 added, 0 updated, 0 removed, " +
"1 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
want := []string{a, f} want := []string{a, f}
slices.Sort(want) slices.Sort(want)
@@ -1015,8 +1022,13 @@ func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) {
link := filepath.Join(realTempDir(t), "link") link := filepath.Join(realTempDir(t), "link")
symlink(t, filepath.Join(dir, "real"), link) symlink(t, filepath.Join(dir, "real"), link)
assertSummary(t, scanOK(t, dir, link), "scan: 1 files seen (1 added, "+ stderr := scanOK(t, dir, link)
"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) { if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f}) t.Errorf("records = %q, want %q", got, []string{f})
@@ -1024,8 +1036,9 @@ func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) {
} }
// TestRunScanDanglingSymlinkOperandIsFatal scans a valid directory and // TestRunScanDanglingSymlinkOperandIsFatal scans a valid directory and
// a symlink whose target does not exist. The scan fails with exit 1 // a symlink whose target does not exist. The scan fails with exit 1 and
// before it walks anything, so no database is created. // an error naming the symlink as given, before it walks anything, so no
// database is created.
func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) { func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
@@ -1048,6 +1061,11 @@ func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) {
assertFatalOutput(t, stderr.String(), stdout()) assertFatalOutput(t, stderr.String(), stdout())
want := "sfdupes: resolve " + dangling + ": "
if !strings.Contains(stderr.String(), want) {
t.Errorf("stderr = %q, want %q", stderr.String(), want)
}
_, err := os.Stat(path) _, err := os.Stat(path)
if !errors.Is(err, fs.ErrNotExist) { if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("database: stat error %v, want it missing", err) t.Errorf("database: stat error %v, want it missing", err)
@@ -1083,8 +1101,75 @@ func TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath(t *testing.T) {
} }
} }
// TestRunScanFollowsSymlinkBeforeDotDot scans a path with a ".." after
// home/proj, a symlink to data/proj: home/proj/.. from dir, and .. after
// a cd through home/proj. As with realpath, the ".." counts from
// data/proj, so data is scanned and home is not.
func TestRunScanFollowsSymlinkBeforeDotDot(t *testing.T) {
for _, tc := range []struct{ wd, operand string }{
{"", "home/proj/.."},
{"home/proj", ".."},
} {
t.Run(tc.operand, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := realTempDir(t)
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
g := writeFile(t, dir, "data/g", pattern(2, 10))
writeFile(t, dir, "home/h", pattern(3, 10))
symlink(t, filepath.Join(dir, "data", "proj"),
filepath.Join(dir, "home", "proj"))
// t.Chdir also sets PWD to the path given, as a shell does
// after cd.
t.Chdir(filepath.Join(dir, tc.wd))
scanOK(t, tc.operand)
want := []string{f, g}
slices.Sort(want)
if got := storedPaths(t, path); !slices.Equal(got, want) {
t.Errorf("records = %q, want %q", got, want)
}
})
}
}
// TestRunScanSkipsSymlinkToFIFOOperand scans link, a symlink to a FIFO.
// The scan warns about the FIFO by its real path, counts it as skipped,
// and exits 0.
func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
dir := realTempDir(t)
fifo := filepath.Join(dir, "fifo")
err := unix.Mkfifo(fifo, 0o600)
if err != nil {
t.Fatal(err)
}
link := filepath.Join(dir, "link")
symlink(t, fifo, link)
stderr := scanOK(t, link)
warning := "walk " + fifo + ": skipping FIFO operand\n"
if !strings.Contains(stderr, warning) {
t.Errorf("stderr = %q, want %q", stderr, warning)
}
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
}
func TestRunScanSkipsZFSOperand(t *testing.T) { func TestRunScanSkipsZFSOperand(t *testing.T) {
zfs := filepath.Join(t.TempDir(), ".zfs") zfs := filepath.Join(realTempDir(t), ".zfs")
f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10)) f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
// No scan records a file inside a .zfs directory, so the record // No scan records a file inside a .zfs directory, so the record
@@ -1135,7 +1220,7 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
dir := t.TempDir() dir := realTempDir(t)
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly") hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
writeFile(t, hourly, "f", pattern(1, 10)) writeFile(t, hourly, "f", pattern(1, 10))
f := writeFile(t, dir, "f", pattern(2, 10)) f := writeFile(t, dir, "f", pattern(2, 10))
@@ -1215,9 +1300,13 @@ func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
data := filepath.Join(dir, "data") data := filepath.Join(dir, "data")
symlink(t, filepath.Join(dir, "pool", "data"), data) symlink(t, filepath.Join(dir, "pool", "data"), data)
assertSummary(t, scanOK(t, filepath.Join(data, "daily")), stderr := scanOK(t, filepath.Join(data, "daily"))
"scan: 1 files seen (1 added, 0 updated, 0 removed, "+
"0 unchanged), 0 skipped\n") summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) { if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f}) t.Errorf("records = %q, want %q", got, []string{f})
+10 -10
View File
@@ -219,14 +219,14 @@ func resolveRoots(roots []string) ([]string, error) {
resolved := make([]string, 0, len(roots)) resolved := make([]string, 0, len(roots))
for _, root := range roots { for _, root := range roots {
a, err := filepath.Abs(root) a, err := absolutePath(root)
if err != nil { if err != nil {
return nil, fmt.Errorf("resolve %s: %w", root, err) return nil, fmt.Errorf("resolve %s: %w", root, err)
} }
r, err := filepath.EvalSymlinks(a) r, err := filepath.EvalSymlinks(a)
if err != nil { if err != nil {
return nil, err return nil, fmt.Errorf("resolve %s: %w", root, err)
} }
resolved = append(resolved, r) resolved = append(resolved, r)
@@ -420,14 +420,14 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
// files, and directories not named .zfs, none of them inside a // files, and directories not named .zfs, none of them inside a
// directory named .zfs. The operands are real paths from resolveRoots, // directory named .zfs. The operands are real paths from resolveRoots,
// so these checks apply to what a symlink operand points at. Every // so these checks apply to what a symlink operand points at. Every
// other operand is warned about, counted as // other operand is warned about, counted as skipped, and dropped. A
// skipped, and dropped. A dropped operand is no longer a root, so the // dropped operand is no longer a root, so the records stored beneath it
// records stored beneath it are treated as if it had not been given: // are treated as if it had not been given: those under no other root
// those under no other root count as outside the roots and are not // count as outside the roots and are not deleted as unverified. An
// deleted as unverified. An operand on a filesystem in rules that is not // operand on a filesystem in rules that is not persistent is dropped
// persistent is dropped the same way, with a line saying so that is not // the same way, with a line saying so that is not a warning, and is not
// a warning, and is not counted as skipped. An operand that fails lstat // counted as skipped. An operand that fails lstat here is kept, and the
// here is kept, and the walk warns about it unless it no longer exists. // walk warns about it unless it no longer exists.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string { func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
kept := make([]string, 0, len(roots)) kept := make([]string, 0, len(roots))
+34 -28
View File
@@ -1094,7 +1094,7 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
func buildSmokeTree(t *testing.T) string { func buildSmokeTree(t *testing.T) string {
t.Helper() t.Helper()
dir := t.TempDir() dir := realTempDir(t)
one := pattern(10, 2000) one := pattern(10, 2000)
f1 := pattern(30, 3000) f1 := pattern(30, 3000)
f2 := pattern(40, 100) f2 := pattern(40, 100)
@@ -2310,34 +2310,24 @@ func TestUnderRoot(t *testing.T) {
} }
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own // TestRunScanPassesOverDatabase scans a tree that holds the scan's own
// database, given directly and through a symlinked directory, which // database, and in one case also a hard link to the database file. The
// the scan resolves to the same real path. The walk passes over the // walk knows the database's files by device ID and inode number, so it
// database file, its -wal and -shm files and its .lock file with no // passes over the database file, its -wal and -shm files, its .lock
// line and no count, and the record an earlier scan left for the .lock // file and the hard link with no line and no count. The record an
// file is deleted. A rescan of the unchanged tree // earlier scan left for the .lock file is deleted. A rescan of the
// updates nothing, and report lists none of the database's files. // unchanged tree updates nothing, and report lists none of the
// database's files.
func TestRunScanPassesOverDatabase(t *testing.T) { func TestRunScanPassesOverDatabase(t *testing.T) {
for _, viaSymlink := range []bool{false, true} { for _, viaHardLink := range []bool{false, true} {
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) { t.Run(fmt.Sprintf("via hard link %v", viaHardLink), func(t *testing.T) {
base := realTempDir(t) tree := realTempDir(t)
tree := filepath.Join(base, "real", "tree") hardLink := filepath.Join(tree, "hardlink")
operand := tree
writeFile(t, tree, "a", pattern(1, 100)) writeFile(t, tree, "a", pattern(1, 100))
writeFile(t, tree, "b", pattern(1, 100)) writeFile(t, tree, "b", pattern(1, 100))
writeFile(t, tree, "empty1", nil) writeFile(t, tree, "empty1", nil)
writeFile(t, tree, "empty2", nil) writeFile(t, tree, "empty2", nil)
if viaSymlink {
err := os.Symlink(filepath.Join(base, "real"),
filepath.Join(base, "link"))
if err != nil {
t.Fatal(err)
}
operand = filepath.Join(base, "link", "tree")
}
dbPath := filepath.Join(tree, "db.sqlite") dbPath := filepath.Join(tree, "db.sqlite")
t.Setenv(databaseEnv, dbPath) t.Setenv(databaseEnv, dbPath)
@@ -2355,15 +2345,28 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
closeScanDatabase(t.Context(), db, dbPath) closeScanDatabase(t.Context(), db, dbPath)
assertScanSummary(t, if viaHardLink {
err = os.Link(dbPath, hardLink)
if err != nil {
t.Fatal(err)
}
}
stderr := assertScanSummary(t,
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)", "4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
operand) tree)
assertScanSummary(t, stderr += assertScanSummary(t,
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)", "4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
operand) tree)
if strings.Contains(stderr, hardLink) {
t.Errorf("scan stderr = %q, want no line naming %s",
stderr, hardLink)
}
report := runStdout(t, cmdReport) report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") || if strings.Contains(report, "db.sqlite") ||
strings.Contains(report, hardLink) ||
!strings.Contains(report, filepath.Join(tree, "empty2")) { !strings.Contains(report, filepath.Join(tree, "empty2")) {
t.Errorf("report = %q, want the empty files and none of "+ t.Errorf("report = %q, want the empty files and none of "+
"the database's files", report) "the database's files", report)
@@ -2389,8 +2392,9 @@ func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
// assertScanSummary scans operands and fails the test unless the // assertScanSummary scans operands and fails the test unless the
// summary line gives counts with nothing skipped, and the line naming // summary line gives counts with nothing skipped, and the line naming
// the database is the only one that names one of its files. // the database is the only one that names one of its files. It returns
func assertScanSummary(t *testing.T, counts string, operands ...string) { // everything the scan printed to stderr.
func assertScanSummary(t *testing.T, counts string, operands ...string) string {
t.Helper() t.Helper()
stderr := scanOK(t, operands...) stderr := scanOK(t, operands...)
@@ -2404,4 +2408,6 @@ func assertScanSummary(t *testing.T, counts string, operands ...string) {
t.Errorf("scan %q: stderr = %q, want only the line naming the "+ t.Errorf("scan %q: stderr = %q, want only the line naming the "+
"database to name one of its files", operands, stderr) "database to name one of its files", operands, stderr)
} }
return stderr
} }