Compare commits
2
Commits
8e39e1b293
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
da75b713aa | ||
|
|
24b0fc486d |
No files matched your search
@@ -17,12 +17,13 @@ gigabyte. A large file is never read in full. A match between two large files is
|
|||||||
a strong candidate, not proof.
|
a strong candidate, not proof.
|
||||||
|
|
||||||
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every
|
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every
|
||||||
file takes too long. It has four subcommands. `scan` walks the filesystem and
|
file takes too long. It has five subcommands. `scan` walks the filesystem and
|
||||||
records the files it finds in an SQLite database. SQLite keeps a whole database
|
records the files it finds in an SQLite database. SQLite keeps a whole database
|
||||||
in one file. The database is kept between runs. A rescan reads only new and
|
in one file. The database is kept between runs. A rescan reads only new and
|
||||||
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
||||||
system's job scheduler. `report`, `trees` and `info` read the database alone and
|
system's job scheduler. `report`, `trees` and `info` read the database alone and
|
||||||
can run at any time.
|
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
|
||||||
|
first look at a new machine with one command.
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
@@ -247,12 +248,15 @@ In order:
|
|||||||
- `report` prints the file-level duplicate report from the database.
|
- `report` prints the file-level duplicate report from the database.
|
||||||
- `trees` prints the duplicate-tree report from the database.
|
- `trees` prints the duplicate-tree report from the database.
|
||||||
- `info` prints the database's details.
|
- `info` prints the database's details.
|
||||||
|
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
|
||||||
|
report.
|
||||||
|
|
||||||
```
|
```
|
||||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||||
sfdupes report > dupes.tsv
|
sfdupes report > dupes.tsv
|
||||||
sfdupes trees > dupetrees.tsv
|
sfdupes trees > dupetrees.tsv
|
||||||
sfdupes info
|
sfdupes info
|
||||||
|
sfdupes dwim
|
||||||
sfdupes --version
|
sfdupes --version
|
||||||
sfdupes [command] --help
|
sfdupes [command] --help
|
||||||
```
|
```
|
||||||
@@ -364,8 +368,10 @@ default. An operand may be a directory or a regular file.
|
|||||||
- An operand that repeats another is dropped before the walk. Two operands
|
- An operand that repeats another is dropped before the walk. Two operands
|
||||||
repeat each other when their real paths are the same.
|
repeat each other when their real paths are the same.
|
||||||
- An operand under another operand is dropped the same way, unless one of the
|
- An operand under another operand is dropped the same way, unless one of the
|
||||||
cases below applies. This too is judged on real paths. No path is then walked
|
cases below applies. No path is then walked twice. Whether one operand is
|
||||||
twice.
|
under another is judged on their real paths. With `/srv/link` a symlink to a
|
||||||
|
directory outside `/srv`, `scan /srv /srv/link/sub` walks `sub` on its own,
|
||||||
|
because the real path of `sub` is not under `/srv`.
|
||||||
- An operand under another is kept, and walked on its own, when the inner
|
- An operand under another is kept, and walked on its own, when the inner
|
||||||
operand or a directory between the two is one of these:
|
operand or a directory between the two is one of these:
|
||||||
- a default exclude (see "Rules for the walk");
|
- a default exclude (see "Rules for the walk");
|
||||||
@@ -377,8 +383,6 @@ default. An operand may be a directory or a regular file.
|
|||||||
such a directory, so the inner operand is walked on its own.
|
such a directory, so the inner operand is walked on its own.
|
||||||
|
|
||||||
- No other case keeps an operand under another.
|
- No other case keeps an operand under another.
|
||||||
- With `/srv/link` a symlink to a directory outside `/srv`,
|
|
||||||
`scan /srv /srv/link/sub` walks `sub` on its own, at its real path.
|
|
||||||
- An operand that is neither a regular file nor a directory is not scanned. Such
|
- An operand that is neither a regular file nor a directory is not scanned. Such
|
||||||
an operand is, for example, a symlink to a device node. `scan` prints a
|
an operand is, for example, a symlink to a device node. `scan` prints a
|
||||||
warning naming the real path and what it is. It counts the operand as skipped
|
warning naming the real path and what it is. It counts the operand as skipped
|
||||||
@@ -800,6 +804,33 @@ The summary line on stderr gives:
|
|||||||
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
|
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### `dwim` mode
|
||||||
|
|
||||||
|
`dwim`, short for "do what I mean", scans a new machine and reports its
|
||||||
|
duplicate trees in one command. It takes no positional arguments and no flags of
|
||||||
|
its own. Like `scan`, it should run as root, so that it can read every file.
|
||||||
|
|
||||||
|
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
|
||||||
|
excludes and crosses into other filesystems. `--workers` keeps its default. The
|
||||||
|
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
|
||||||
|
what `sfdupes trees` runs. The largest trees come first.
|
||||||
|
|
||||||
|
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
|
||||||
|
`scan` prints, then what `trees` prints, then one last line:
|
||||||
|
|
||||||
|
```
|
||||||
|
dwim: run "sfdupes trees" to print this report again
|
||||||
|
```
|
||||||
|
|
||||||
|
The database keeps the scan, so `sfdupes trees` prints the same report later,
|
||||||
|
even after the terminal is gone.
|
||||||
|
|
||||||
|
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
|
||||||
|
`dwim` then prints no report and exits 1. A fatal scan error also ends it before
|
||||||
|
the report, with the same message and exit code as `scan`. Warnings for skipped
|
||||||
|
files do not stop it. `dwim` stops catching signals when the scan ends, so a
|
||||||
|
signal during the report ends it at once, as it ends `trees`.
|
||||||
|
|
||||||
### `info` mode
|
### `info` mode
|
||||||
|
|
||||||
`info` takes no positional arguments and prints the database's details on
|
`info` takes no positional arguments and prints the database's details on
|
||||||
@@ -840,7 +871,7 @@ scan is writing. `info` writes its lines after the `count` display has ended.
|
|||||||
|
|
||||||
### Database
|
### Database
|
||||||
|
|
||||||
All four subcommands work on one SQLite database file.
|
All five subcommands work on one SQLite database file.
|
||||||
|
|
||||||
#### Location
|
#### Location
|
||||||
|
|
||||||
@@ -856,6 +887,8 @@ scan: new database /var/lib/sfdupes/db.sqlite (default)
|
|||||||
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
|
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`dwim` prints the `scan` line first, and the `trees` line after the scan.
|
||||||
|
|
||||||
A relative path is printed with the working directory in front of it. The
|
A relative path is printed with the working directory in front of it. The
|
||||||
working directory is printed as `pwd -P` prints it, with its symlinks resolved.
|
working directory is printed as `pwd -P` prints it, with its symlinks resolved.
|
||||||
The rest of the path is printed as set, so a `..` in it stays. The line names
|
The rest of the path is printed as set, so a `..` in it stays. The line names
|
||||||
@@ -1139,8 +1172,8 @@ showing the count it stopped at.
|
|||||||
### Error handling and exit codes
|
### Error handling and exit codes
|
||||||
|
|
||||||
- `0`: success, even when files were skipped with warnings.
|
- `0`: success, even when files were skipped with warnings.
|
||||||
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal
|
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
|
||||||
error is one of these:
|
stopped by either during its scan. A fatal error is one of these:
|
||||||
- an operand that does not exist, or whose real path `scan` cannot find;
|
- an operand that does not exist, or whose real path `scan` cannot find;
|
||||||
- a list of mounted filesystems that `scan` cannot read;
|
- a list of mounted filesystems that `scan` cannot read;
|
||||||
- another `scan` holding the lock (see "The scan lock");
|
- another `scan` holding the lock (see "The scan lock");
|
||||||
@@ -1161,7 +1194,7 @@ showing the count it stopped at.
|
|||||||
`--workers abc`;
|
`--workers abc`;
|
||||||
- `scan` with no operand;
|
- `scan` with no operand;
|
||||||
- `--workers` below 1;
|
- `--workers` below 1;
|
||||||
- `report`, `trees` or `info` with a positional argument.
|
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
||||||
|
|
||||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||||
stdout, such as a full disk, is reported the same way. Two cases never reach
|
stdout, such as a full disk, is reported the same way. Two cases never reach
|
||||||
|
|||||||
@@ -29,6 +29,11 @@
|
|||||||
Newest first. An entry ends with its date and its issue or branch, where those
|
Newest first. An entry ends with its date and its issue or branch, where those
|
||||||
are known.
|
are known.
|
||||||
|
|
||||||
|
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
|
||||||
|
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
|
||||||
|
prints the report again (2026-11-06,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/154)
|
||||||
|
|
||||||
- `scan` resolves each operand to its real path, symlinks followed, before the
|
- `scan` resolves each operand to its real path, symlinks followed, before the
|
||||||
walk. A symlink operand is scanned at its target instead of skipped. With
|
walk. A symlink operand is scanned at its target instead of skipped. With
|
||||||
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
|
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
|
||||||
|
|||||||
@@ -146,33 +146,44 @@ var errHashSize = errors.New("hash is not 32 bytes")
|
|||||||
|
|
||||||
// databasePath resolves the database location: SFDUPES_DATABASE when
|
// databasePath resolves the database location: SFDUPES_DATABASE when
|
||||||
// set and non-empty, the compiled-in default otherwise. It returns the
|
// set and non-empty, the compiled-in default otherwise. It returns the
|
||||||
// path, with the working directory, its symlinks resolved, put in front
|
// path, made absolute by absolutePath, and where the path came from,
|
||||||
// of it when it is relative, and where the path came from, "default" or
|
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
||||||
// "from SFDUPES_DATABASE", for the line each subcommand prints first on
|
// prints first on stderr.
|
||||||
// stderr.
|
|
||||||
func databasePath() (string, string, error) {
|
func databasePath() (string, string, error) {
|
||||||
path, source := defaultDatabasePath, "default"
|
path, source := defaultDatabasePath, "default"
|
||||||
if p := os.Getenv(databaseEnv); p != "" {
|
if p := os.Getenv(databaseEnv); p != "" {
|
||||||
path, source = p, "from "+databaseEnv
|
path, source = p, "from "+databaseEnv
|
||||||
}
|
}
|
||||||
|
|
||||||
if filepath.IsAbs(path) {
|
abs, err := absolutePath(path)
|
||||||
return path, source, nil
|
|
||||||
}
|
|
||||||
|
|
||||||
// The working directory as the kernel reports it, with no symlink
|
|
||||||
// in it, as README "Database" says. os.Getwd may return PWD
|
|
||||||
// instead, which can name it through a symlink.
|
|
||||||
wd, err := unix.Getwd()
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Not filepath.Abs or filepath.Join: they clean the path, which
|
return abs, source, nil
|
||||||
// drops each ".." together with the directory name before it. When
|
}
|
||||||
// that directory is a symlink, the cleaned path names another file.
|
|
||||||
|
// absolutePath returns path unchanged when it is absolute, and
|
||||||
|
// otherwise the working directory, a "/" and path. The working
|
||||||
|
// directory is the one the kernel reports, with no symlink in it;
|
||||||
|
// os.Getwd may return PWD instead, which can name it through a symlink.
|
||||||
|
// The result is not cleaned, as filepath.Abs and filepath.Join would
|
||||||
|
// clean it. Cleaning drops each ".." together with the directory name
|
||||||
|
// before it. When that name is a symlink, the cleaned path names
|
||||||
|
// another file than the one the system reaches by following the symlink
|
||||||
|
// first.
|
||||||
|
func absolutePath(path string) (string, error) {
|
||||||
|
if filepath.IsAbs(path) {
|
||||||
|
return path, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
wd, err := unix.Getwd()
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
|
||||||
// The working directory ends in "/" only when it is the root.
|
// The working directory ends in "/" only when it is the root.
|
||||||
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
|
return strings.TrimSuffix(wd, "/") + "/" + path, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// createDatabaseDirectory creates the directory the database at path
|
// createDatabaseDirectory creates the directory the database at path
|
||||||
|
|||||||
@@ -0,0 +1,37 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"runtime"
|
||||||
|
)
|
||||||
|
|
||||||
|
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
|
||||||
|
// root" runs with no flags, then what "sfdupes trees" runs, and ends
|
||||||
|
// with a line on stderr saying how to print the report again. The dwim
|
||||||
|
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
|
||||||
|
// does, but only during the scan. An error from the scan, an interrupt
|
||||||
|
// included, ends it before the report. During the report, a signal
|
||||||
|
// ends it at once, as it ends trees.
|
||||||
|
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
||||||
|
scanCtx, stop := interruptContext(ctx)
|
||||||
|
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
|
||||||
|
|
||||||
|
stop()
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
err = runTrees(ctx, stdout)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Fprintln(os.Stderr,
|
||||||
|
`dwim: run "sfdupes trees" to print this report again`)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
|
||||||
|
// dwim scans its root, then prints the trees report, the larger of
|
||||||
|
// two duplicate-tree pairs first, and ends by saying how to print
|
||||||
|
// the report again.
|
||||||
|
t.Setenv(databaseEnv, testDBPath(t))
|
||||||
|
|
||||||
|
root := realTempDir(t)
|
||||||
|
|
||||||
|
small := []string{
|
||||||
|
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
|
||||||
|
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
|
||||||
|
}
|
||||||
|
large := []string{
|
||||||
|
filepath.Dir(writeFile(t, root, "b1/f.bin", pattern(2, 5000))),
|
||||||
|
filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000))),
|
||||||
|
}
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
|
err := runDwim(t.Context(), root, &stdout)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
||||||
|
}
|
||||||
|
|
||||||
|
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
|
||||||
|
if len(lines) != 3 {
|
||||||
|
t.Fatalf("stdout = %q, want a header and two groups",
|
||||||
|
stdout.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
wantFirst := large[0] + "\t" + large[1] + "\t1\t5000\t"
|
||||||
|
if !strings.HasPrefix(lines[1], wantFirst) {
|
||||||
|
t.Errorf("first group = %q, want it to start %q", lines[1], wantFirst)
|
||||||
|
}
|
||||||
|
|
||||||
|
wantSecond := small[0] + "\t" + small[1] + "\t1\t300\t"
|
||||||
|
if !strings.HasPrefix(lines[2], wantSecond) {
|
||||||
|
t.Errorf("second group = %q, want it to start %q",
|
||||||
|
lines[2], wantSecond)
|
||||||
|
}
|
||||||
|
|
||||||
|
last := "\ndwim: run \"sfdupes trees\" to print this report again\n"
|
||||||
|
if got := stderr(); !strings.HasSuffix(got, last) {
|
||||||
|
t.Errorf("stderr = %q, want it to end %q", got, last)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
||||||
|
// A fatal scan error ends dwim before the report: a root that does
|
||||||
|
// not exist fails the scan as it fails scan itself.
|
||||||
|
t.Setenv(databaseEnv, testDBPath(t))
|
||||||
|
|
||||||
|
missing := filepath.Join(realTempDir(t), "missing")
|
||||||
|
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
|
err := runDwim(t.Context(), missing, &stdout)
|
||||||
|
if err == nil {
|
||||||
|
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := stdout.String(); got != "" {
|
||||||
|
t.Errorf("stdout = %q, want no report", got)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := stderr(); strings.Contains(got, "trees:") ||
|
||||||
|
strings.Contains(got, "dwim:") {
|
||||||
|
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
+1
-1
@@ -286,7 +286,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
|
|||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := realTempDir(t)
|
||||||
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
|
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
|
||||||
t.Skip(dir + " is not inside a default exclude")
|
t.Skip(dir + " is not inside a default exclude")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,6 +14,7 @@
|
|||||||
// sfdupes report > dupes.tsv
|
// sfdupes report > dupes.tsv
|
||||||
// sfdupes trees > dupetrees.tsv
|
// sfdupes trees > dupetrees.tsv
|
||||||
// sfdupes info
|
// sfdupes info
|
||||||
|
// sfdupes dwim
|
||||||
// sfdupes --version
|
// sfdupes --version
|
||||||
//
|
//
|
||||||
// See README.md for the complete specification.
|
// See README.md for the complete specification.
|
||||||
@@ -45,6 +46,7 @@ const (
|
|||||||
cmdReport = "report"
|
cmdReport = "report"
|
||||||
cmdTrees = "trees"
|
cmdTrees = "trees"
|
||||||
cmdInfo = "info"
|
cmdInfo = "info"
|
||||||
|
cmdDwim = "dwim"
|
||||||
)
|
)
|
||||||
|
|
||||||
// errNoSubcommand is returned by the root command when it is invoked
|
// errNoSubcommand is returned by the root command when it is invoked
|
||||||
@@ -96,8 +98,8 @@ func main() {
|
|||||||
// exit code. It is the program's single exit point: the subcommands
|
// exit code. It is the program's single exit point: the subcommands
|
||||||
// return their errors instead of exiting, so every deferred cleanup —
|
// return their errors instead of exiting, so every deferred cleanup —
|
||||||
// above all closing the database, which checkpoints the SQLite WAL —
|
// above all closing the database, which checkpoints the SQLite WAL —
|
||||||
// runs before the process ends. The report, trees and info subcommands
|
// runs before the process ends. The report, trees, info and dwim
|
||||||
// write their data to stdout.
|
// subcommands write their data to stdout.
|
||||||
func run(args []string, stdout, stderr io.Writer) int {
|
func run(args []string, stdout, stderr io.Writer) int {
|
||||||
// A nil slice makes cobra fall back to os.Args, which would let a
|
// A nil slice makes cobra fall back to os.Args, which would let a
|
||||||
// test binary's own flags reach the command tree.
|
// test binary's own flags reach the command tree.
|
||||||
@@ -201,7 +203,16 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
|||||||
}),
|
}),
|
||||||
}
|
}
|
||||||
|
|
||||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
|
dwimCmd := &cobra.Command{
|
||||||
|
Use: cmdDwim,
|
||||||
|
Short: "Scan / with the default excludes, then print the duplicate-tree report",
|
||||||
|
Args: cobra.NoArgs,
|
||||||
|
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||||
|
return runDwim(ctx, "/", stdout)
|
||||||
|
}),
|
||||||
|
}
|
||||||
|
|
||||||
|
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
|
||||||
|
|
||||||
return root
|
return root
|
||||||
}
|
}
|
||||||
|
|||||||
+44
-27
@@ -622,6 +622,7 @@ func TestRunUsageErrors(t *testing.T) {
|
|||||||
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
||||||
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
||||||
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
||||||
|
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
|
||||||
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
||||||
"unknown subcommand": {[]string{"nope"}, "unknown command"},
|
"unknown subcommand": {[]string{"nope"}, "unknown command"},
|
||||||
}
|
}
|
||||||
@@ -799,7 +800,7 @@ func TestRunVersionWriteFailureIsFatal(t *testing.T) {
|
|||||||
func scanFixture(t *testing.T) []string {
|
func scanFixture(t *testing.T) []string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := realTempDir(t)
|
||||||
|
|
||||||
dupes := []string{
|
dupes := []string{
|
||||||
writeFile(t, dir, "one/a.bin", pattern(1, 300)),
|
writeFile(t, dir, "one/a.bin", pattern(1, 300)),
|
||||||
@@ -869,7 +870,7 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
|||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := realTempDir(t)
|
||||||
writeFile(t, dir, "a.bin", pattern(1, 300))
|
writeFile(t, dir, "a.bin", pattern(1, 300))
|
||||||
|
|
||||||
// Same size as a.bin, so the scan reads it, and the read fails.
|
// Same size as a.bin, so the scan reads it, and the read fails.
|
||||||
@@ -921,15 +922,6 @@ func symlink(t *testing.T, target, link string) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// assertSummary checks that stderr holds the scan summary line want.
|
|
||||||
func assertSummary(t *testing.T, stderr, want string) {
|
|
||||||
t.Helper()
|
|
||||||
|
|
||||||
if !strings.Contains(stderr, want) {
|
|
||||||
t.Errorf("stderr = %q, want %q", stderr, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestRunScanFollowsSymlinkOperand scans link, a symlink to target.
|
// TestRunScanFollowsSymlinkOperand scans link, a symlink to target.
|
||||||
// The operand is scanned at its real path, and its file is recorded
|
// The operand is scanned at its real path, and its file is recorded
|
||||||
// there.
|
// there.
|
||||||
@@ -943,8 +935,13 @@ func TestRunScanFollowsSymlinkOperand(t *testing.T) {
|
|||||||
link := filepath.Join(dir, "link")
|
link := filepath.Join(dir, "link")
|
||||||
symlink(t, filepath.Join(dir, "target"), link)
|
symlink(t, filepath.Join(dir, "target"), link)
|
||||||
|
|
||||||
assertSummary(t, scanOK(t, link), "scan: 1 files seen (1 added, "+
|
stderr := scanOK(t, link)
|
||||||
"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
|
|
||||||
|
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||||
|
"0 unchanged), 0 skipped\n"
|
||||||
|
if !strings.Contains(stderr, summary) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||||
|
}
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
t.Errorf("records = %q, want %q", got, []string{f})
|
||||||
@@ -964,9 +961,13 @@ func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
|
|||||||
link := filepath.Join(dir, "link")
|
link := filepath.Join(dir, "link")
|
||||||
symlink(t, filepath.Join(dir, "target"), link)
|
symlink(t, filepath.Join(dir, "target"), link)
|
||||||
|
|
||||||
assertSummary(t, scanOK(t, link, filepath.Join(link, "sub")),
|
stderr := scanOK(t, link, filepath.Join(link, "sub"))
|
||||||
"scan: 1 files seen (1 added, 0 updated, 0 removed, "+
|
|
||||||
"0 unchanged), 0 skipped\n")
|
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||||
|
"0 unchanged), 0 skipped\n"
|
||||||
|
if !strings.Contains(stderr, summary) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||||
|
}
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
t.Errorf("records = %q, want %q", got, []string{f})
|
||||||
@@ -993,8 +994,12 @@ func TestRunScanWalksSymlinkedOperandBelowOperand(t *testing.T) {
|
|||||||
scanOK(t, filepath.Join(outside, "sub"))
|
scanOK(t, filepath.Join(outside, "sub"))
|
||||||
|
|
||||||
stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub"))
|
stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub"))
|
||||||
assertSummary(t, stderr, "scan: 2 files seen (1 added, 0 updated, "+
|
|
||||||
"0 removed, 1 unchanged), 0 skipped\n")
|
summary := "scan: 2 files seen (1 added, 0 updated, 0 removed, " +
|
||||||
|
"1 unchanged), 0 skipped\n"
|
||||||
|
if !strings.Contains(stderr, summary) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||||
|
}
|
||||||
|
|
||||||
want := []string{a, f}
|
want := []string{a, f}
|
||||||
slices.Sort(want)
|
slices.Sort(want)
|
||||||
@@ -1017,8 +1022,13 @@ func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) {
|
|||||||
link := filepath.Join(realTempDir(t), "link")
|
link := filepath.Join(realTempDir(t), "link")
|
||||||
symlink(t, filepath.Join(dir, "real"), link)
|
symlink(t, filepath.Join(dir, "real"), link)
|
||||||
|
|
||||||
assertSummary(t, scanOK(t, dir, link), "scan: 1 files seen (1 added, "+
|
stderr := scanOK(t, dir, link)
|
||||||
"0 updated, 0 removed, 0 unchanged), 0 skipped\n")
|
|
||||||
|
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||||
|
"0 unchanged), 0 skipped\n"
|
||||||
|
if !strings.Contains(stderr, summary) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||||
|
}
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
t.Errorf("records = %q, want %q", got, []string{f})
|
||||||
@@ -1151,12 +1161,15 @@ func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) {
|
|||||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||||
}
|
}
|
||||||
|
|
||||||
assertSummary(t, stderr, "scan: 0 files seen (0 added, 0 updated, "+
|
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
||||||
"0 removed, 0 unchanged), 1 skipped\n")
|
"0 unchanged), 1 skipped\n"
|
||||||
|
if !strings.Contains(stderr, summary) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunScanSkipsZFSOperand(t *testing.T) {
|
func TestRunScanSkipsZFSOperand(t *testing.T) {
|
||||||
zfs := filepath.Join(t.TempDir(), ".zfs")
|
zfs := filepath.Join(realTempDir(t), ".zfs")
|
||||||
f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
|
f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
|
||||||
|
|
||||||
// No scan records a file inside a .zfs directory, so the record
|
// No scan records a file inside a .zfs directory, so the record
|
||||||
@@ -1207,7 +1220,7 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
|
|||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := realTempDir(t)
|
||||||
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
|
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
|
||||||
writeFile(t, hourly, "f", pattern(1, 10))
|
writeFile(t, hourly, "f", pattern(1, 10))
|
||||||
f := writeFile(t, dir, "f", pattern(2, 10))
|
f := writeFile(t, dir, "f", pattern(2, 10))
|
||||||
@@ -1287,9 +1300,13 @@ func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
|
|||||||
data := filepath.Join(dir, "data")
|
data := filepath.Join(dir, "data")
|
||||||
symlink(t, filepath.Join(dir, "pool", "data"), data)
|
symlink(t, filepath.Join(dir, "pool", "data"), data)
|
||||||
|
|
||||||
assertSummary(t, scanOK(t, filepath.Join(data, "daily")),
|
stderr := scanOK(t, filepath.Join(data, "daily"))
|
||||||
"scan: 1 files seen (1 added, 0 updated, 0 removed, "+
|
|
||||||
"0 unchanged), 0 skipped\n")
|
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||||
|
"0 unchanged), 0 skipped\n"
|
||||||
|
if !strings.Contains(stderr, summary) {
|
||||||
|
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||||
|
}
|
||||||
|
|
||||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||||
t.Errorf("records = %q, want %q", got, []string{f})
|
t.Errorf("records = %q, want %q", got, []string{f})
|
||||||
|
|||||||
@@ -18,8 +18,6 @@ import (
|
|||||||
"sync"
|
"sync"
|
||||||
"syscall"
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
"golang.org/x/sys/unix"
|
|
||||||
)
|
)
|
||||||
|
|
||||||
// The hash phase gives every file it reads three hashes (see
|
// The hash phase gives every file it reads three hashes (see
|
||||||
@@ -221,24 +219,11 @@ func resolveRoots(roots []string) ([]string, error) {
|
|||||||
resolved := make([]string, 0, len(roots))
|
resolved := make([]string, 0, len(roots))
|
||||||
|
|
||||||
for _, root := range roots {
|
for _, root := range roots {
|
||||||
a := root
|
a, err := absolutePath(root)
|
||||||
if !filepath.IsAbs(a) {
|
|
||||||
// The working directory as the kernel reports it, with no
|
|
||||||
// symlink in it. os.Getwd may return PWD instead, which can
|
|
||||||
// name it through a symlink.
|
|
||||||
wd, err := unix.Getwd()
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Not filepath.Abs or filepath.Join: they clean the path,
|
|
||||||
// which drops each ".." together with the directory name
|
|
||||||
// before it. EvalSymlinks follows a symlink before it
|
|
||||||
// applies the ".." after it, as realpath does. The working
|
|
||||||
// directory ends in "/" only when it is the root.
|
|
||||||
a = strings.TrimSuffix(wd, "/") + "/" + root
|
|
||||||
}
|
|
||||||
|
|
||||||
r, err := filepath.EvalSymlinks(a)
|
r, err := filepath.EvalSymlinks(a)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||||
|
|||||||
+34
-28
@@ -1094,7 +1094,7 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
|
|||||||
func buildSmokeTree(t *testing.T) string {
|
func buildSmokeTree(t *testing.T) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := realTempDir(t)
|
||||||
one := pattern(10, 2000)
|
one := pattern(10, 2000)
|
||||||
f1 := pattern(30, 3000)
|
f1 := pattern(30, 3000)
|
||||||
f2 := pattern(40, 100)
|
f2 := pattern(40, 100)
|
||||||
@@ -2310,34 +2310,24 @@ func TestUnderRoot(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
|
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
|
||||||
// database, given directly and through a symlinked directory, which
|
// database, and in one case also a hard link to the database file. The
|
||||||
// the scan resolves to the same real path. The walk passes over the
|
// walk knows the database's files by device ID and inode number, so it
|
||||||
// database file, its -wal and -shm files and its .lock file with no
|
// passes over the database file, its -wal and -shm files, its .lock
|
||||||
// line and no count, and the record an earlier scan left for the .lock
|
// file and the hard link with no line and no count. The record an
|
||||||
// file is deleted. A rescan of the unchanged tree updates nothing, and
|
// earlier scan left for the .lock file is deleted. A rescan of the
|
||||||
// report lists none of the database's files.
|
// unchanged tree updates nothing, and report lists none of the
|
||||||
|
// database's files.
|
||||||
func TestRunScanPassesOverDatabase(t *testing.T) {
|
func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||||
for _, viaSymlink := range []bool{false, true} {
|
for _, viaHardLink := range []bool{false, true} {
|
||||||
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
|
t.Run(fmt.Sprintf("via hard link %v", viaHardLink), func(t *testing.T) {
|
||||||
base := realTempDir(t)
|
tree := realTempDir(t)
|
||||||
tree := filepath.Join(base, "real", "tree")
|
hardLink := filepath.Join(tree, "hardlink")
|
||||||
operand := tree
|
|
||||||
|
|
||||||
writeFile(t, tree, "a", pattern(1, 100))
|
writeFile(t, tree, "a", pattern(1, 100))
|
||||||
writeFile(t, tree, "b", pattern(1, 100))
|
writeFile(t, tree, "b", pattern(1, 100))
|
||||||
writeFile(t, tree, "empty1", nil)
|
writeFile(t, tree, "empty1", nil)
|
||||||
writeFile(t, tree, "empty2", nil)
|
writeFile(t, tree, "empty2", nil)
|
||||||
|
|
||||||
if viaSymlink {
|
|
||||||
err := os.Symlink(filepath.Join(base, "real"),
|
|
||||||
filepath.Join(base, "link"))
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
operand = filepath.Join(base, "link", "tree")
|
|
||||||
}
|
|
||||||
|
|
||||||
dbPath := filepath.Join(tree, "db.sqlite")
|
dbPath := filepath.Join(tree, "db.sqlite")
|
||||||
t.Setenv(databaseEnv, dbPath)
|
t.Setenv(databaseEnv, dbPath)
|
||||||
|
|
||||||
@@ -2355,15 +2345,28 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
|||||||
|
|
||||||
closeScanDatabase(t.Context(), db, dbPath)
|
closeScanDatabase(t.Context(), db, dbPath)
|
||||||
|
|
||||||
assertScanSummary(t,
|
if viaHardLink {
|
||||||
|
err = os.Link(dbPath, hardLink)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
stderr := assertScanSummary(t,
|
||||||
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
|
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
|
||||||
operand)
|
tree)
|
||||||
assertScanSummary(t,
|
stderr += assertScanSummary(t,
|
||||||
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
|
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
|
||||||
operand)
|
tree)
|
||||||
|
|
||||||
|
if strings.Contains(stderr, hardLink) {
|
||||||
|
t.Errorf("scan stderr = %q, want no line naming %s",
|
||||||
|
stderr, hardLink)
|
||||||
|
}
|
||||||
|
|
||||||
report := runStdout(t, cmdReport)
|
report := runStdout(t, cmdReport)
|
||||||
if strings.Contains(report, "db.sqlite") ||
|
if strings.Contains(report, "db.sqlite") ||
|
||||||
|
strings.Contains(report, hardLink) ||
|
||||||
!strings.Contains(report, filepath.Join(tree, "empty2")) {
|
!strings.Contains(report, filepath.Join(tree, "empty2")) {
|
||||||
t.Errorf("report = %q, want the empty files and none of "+
|
t.Errorf("report = %q, want the empty files and none of "+
|
||||||
"the database's files", report)
|
"the database's files", report)
|
||||||
@@ -2389,8 +2392,9 @@ func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
|
|||||||
|
|
||||||
// assertScanSummary scans operands and fails the test unless the
|
// assertScanSummary scans operands and fails the test unless the
|
||||||
// summary line gives counts with nothing skipped, and the line naming
|
// summary line gives counts with nothing skipped, and the line naming
|
||||||
// the database is the only one that names one of its files.
|
// the database is the only one that names one of its files. It returns
|
||||||
func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
// everything the scan printed to stderr.
|
||||||
|
func assertScanSummary(t *testing.T, counts string, operands ...string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
stderr := scanOK(t, operands...)
|
stderr := scanOK(t, operands...)
|
||||||
@@ -2404,4 +2408,6 @@ func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
|||||||
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
|
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
|
||||||
"database to name one of its files", operands, stderr)
|
"database to name one of its files", operands, stderr)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return stderr
|
||||||
}
|
}
|
||||||
Reference in new issue
Block a user