Compare commits

..
Author SHA1 Message Date
clawbot 19a131ca15 TODO.md: keep only open work (closes #167)
check / check (push) Canceled after 0s
Remove the Completed Steps section and the Status sentence saying the
file records history. Git keeps the history. The Workflow step that
recorded finished work in Completed Steps now deletes Next Step instead.

Model: opus-5-5
2026-11-11 06:54:06 +00:00
10 changed files with 130 additions and 380 deletions

No files matched your search

+35 -59
View File
@@ -128,8 +128,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
- one line naming the database; - one line naming the database;
- progress lines, as described under "Progress"; - progress lines, as described under "Progress";
- a warning for each path it skips; - a warning for each path it skips;
- a line for each system, temporary or cache directory the walk leaves out - a line for each system or temporary directory the walk leaves out (see
(see "Rules for the walk"); "Rules for the walk");
- a line for each path left out because its filesystem is one the kernel, - a line for each path left out because its filesystem is one the kernel,
the core of the operating system, makes up or keeps in memory, such as the core of the operating system, makes up or keeps in memory, such as
`tmpfs`; `tmpfs`;
@@ -254,14 +254,15 @@ In order:
- `report` prints the file-level duplicate report from the database. - `report` prints the file-level duplicate report from the database.
- `trees` prints the duplicate-tree report from the database. - `trees` prints the duplicate-tree report from the database.
- `info` prints the database's details. - `info` prints the database's details.
- `dwim` scans `/`, then prints the duplicate-tree report. - `dwim` scans `/` with the default excludes, then prints the duplicate-tree
report.
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
sfdupes report > dupes.tsv sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv sfdupes trees > dupetrees.tsv
sfdupes info sfdupes info
sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... sfdupes dwim
sfdupes --version sfdupes --version
sfdupes [command] --help sfdupes [command] --help
``` ```
@@ -291,21 +292,20 @@ The reference below is also the program's specification.
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
it. The path names the file exactly, whatever characters it holds. A relative it. The path names the file exactly, whatever characters it holds. A relative
path is relative to the working directory. path is relative to the working directory.
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once - `--workers N` (`scan`) sets how many directories `scan` reads at once while it
while it lists the trees. It also sets how many files `scan` reads at once lists the trees. It also sets how many files `scan` reads at once while it
while it hashes them. The default is the number of CPUs. `N` must be at hashes them. The default is the number of CPUs. `N` must be at least 1. A
least 1. A smaller value is a usage error, reported in one line on stderr with smaller value is a usage error, reported in one line on stderr with exit 2
exit 2 before anything is scanned. before anything is scanned.
- `-x`, `--one-file-system` (`scan`, `dwim`) keeps the walk of each operand out - `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
of directories on other filesystems, as the same flag does for `du` and directories on other filesystems, as the same flag does for `du` and `rsync`.
`rsync`. By default the walk also enters directories on other filesystems. By default the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default - `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
excludes. Those are the system, temporary and cache directories the walk Those are the system and temporary directories the walk otherwise leaves out,
otherwise leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules such as `/var/tmp` on Linux (see "Rules for the walk").
for the walk"). - `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything it. `PATH` may be a directory or a file. The flag may be given more than once.
below it. `PATH` may be a directory or a file. The flag may be given more than The walk knows `PATH` by its device ID and inode number, as it knows the
once. The walk knows `PATH` by its device ID and inode number, as it knows the
database files (see "Rules for the walk"). So the walk leaves `PATH` out database files (see "Rules for the walk"). So the walk leaves `PATH` out
whatever path leads to it. That includes a hard link to a file given as whatever path leads to it. That includes a hard link to a file given as
`PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in `PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in
@@ -318,9 +318,9 @@ The reference below is also the program's specification.
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr - `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan` and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
names the database. The help of `report`, `trees`, `info` and `dwim` says what names the database. The help of `report`, `trees`, `info` and `dwim` says what
each prints. `sfdupes scan --help` and `sfdupes dwim --help` also list the each prints. `sfdupes scan --help` also lists the default excludes of the
default excludes of the system they run on, and the filesystem types the walk system it runs on, and the filesystem types the walk never records anything
never records anything on. on.
### Duplicate detection ### Duplicate detection
@@ -572,17 +572,12 @@ walk practical on large or busy storage pools.
`--no-default-excludes` does not change this rule. `--no-default-excludes` does not change this rule.
- Below an operand, the walk does not enter a default exclude. On Linux, the - Below an operand, the walk does not enter a default exclude. On Linux, the
default excludes hold the kernel's view of the running system, temporary default excludes hold temporary files and the kernel's view of the running
files, and caches. A cache here is data that programs and package managers system. They depend on the system, and `sfdupes scan --help` lists those of
download or build again on their own. The default excludes depend on the the system it runs on. On Linux they are:
system, and `sfdupes scan --help` lists those of the system it runs on. In the
lists, `*` stands for any one directory name, so `/home/*/.cache` matches
`/home/alice/.cache`. On Linux they are:
``` ```
/proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache /proc, /sys, /dev, /run, /tmp, /var/tmp
/root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
/home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
``` ```
On macOS they are: On macOS they are:
@@ -590,21 +585,13 @@ walk practical on large or busy storage pools.
``` ```
/dev, /private/tmp, /private/var/tmp, /private/var/vm /dev, /private/tmp, /private/var/tmp, /private/var/vm
/System/Volumes/VM, /System/Volumes/Data /System/Volumes/VM, /System/Volumes/Data
/private/var/folders, /Library/Caches
/private/var/root/Library/Caches, /private/var/root/.cache
/private/var/root/.npm, /private/var/root/.cargo/registry
/private/var/root/go/pkg/mod
/Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
/Users/*/.cargo/registry, /Users/*/go/pkg/mod
``` ```
`/private/var/folders` holds each user's temporary files and caches on A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already,
macOS. Root's home on macOS is `/var/root`, which the walk reaches as through firmlinks. Firmlinks are the links macOS uses to show the data
`/private/var/root`. A walk of `/` on macOS reaches everything in volume inside the system volume. Walking `/System/Volumes/Data` as well
`/System/Volumes/Data` already, through firmlinks. Firmlinks are the links would record each of those files twice. Any other system has no default
macOS uses to show the data volume inside the system volume. Walking excludes.
`/System/Volumes/Data` as well would record each of those files twice. Any
other system has no default excludes.
An operand at or inside a default exclude is walked, so An operand at or inside a default exclude is walked, so
`sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem. `sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem.
@@ -871,25 +858,14 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
### `dwim` mode ### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its `dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments. Like `scan`, duplicate trees in one command. It takes no positional arguments and no flags of
it should run as root, so that it can read every file. its own. Like `scan`, it should run as root, so that it can read every file.
`dwim` takes these flags of `scan`, with the same meaning (see "Settings"): `dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
excludes and crosses into other filesystems. `--workers` keeps its default. The
- `--workers N`;
- `-x`, `--one-file-system`;
- `--no-default-excludes`;
- `--exclude PATH`, which may be given more than once.
`dwim` runs what `sfdupes scan /` runs with the same flags. With no flags, the
walk skips the default excludes and crosses into other filesystems. The
database, its creation and the scan lock are those of `scan`. Then `dwim` runs database, its creation and the scan lock are those of `scan`. Then `dwim` runs
what `sfdupes trees` runs. The largest trees come first. what `sfdupes trees` runs. The largest trees come first.
```
sfdupes dwim -x --exclude /srv/cache
```
The report goes to stdout, exactly as `trees` prints it. On stderr comes what The report goes to stdout, exactly as `trees` prints it. On stderr comes what
`scan` prints, then what `trees` prints, then one last line: `scan` prints, then what `trees` prints, then one last line:
+7 -6
View File
@@ -358,9 +358,8 @@ func TestRunScanInterrupted(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
dir := buildSmokeTree(t) dir := buildSmokeTree(t)
opts := scanOptions{workers: walkCancelWorkers} err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false, nil)
err := runScan(t.Context(), []string{dir}, opts)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -383,7 +382,8 @@ func TestRunScanInterrupted(t *testing.T) {
go func() { go func() {
defer close(done) defer close(done)
err = runScan(cancelledContext(t), []string{dir}, opts) err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
false, false, nil)
}() }()
awaitReturn(t, done, "runScan") awaitReturn(t, done, "runScan")
@@ -410,7 +410,8 @@ func TestRunScanInterrupted(t *testing.T) {
got, before) got, before)
} }
err = runScan(t.Context(), []string{dir}, opts) err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false, nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -436,7 +437,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
dir := buildWalkCancelTree(t) dir := buildWalkCancelTree(t)
err := runScan(newWalkClock(hashCancelAtDone), []string{dir}, err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
scanOptions{workers: walkCancelWorkers}) walkCancelWorkers, false, false, nil)
if !errors.Is(err, errInterrupted) { if !errors.Is(err, errInterrupted) {
t.Fatalf("runScan interrupted mid-hash = %v, want %v", t.Fatalf("runScan interrupted mid-hash = %v, want %v",
err, errInterrupted) err, errInterrupted)
+53 -85
View File
@@ -88,10 +88,10 @@ value, separated by a tab:
size of recorded files their total size, in bytes, then rounded size of recorded files their total size, in bytes, then rounded
` `
dwimHelp = `dwim runs what "sfdupes scan /" runs with the same flags, then what dwimHelp = `dwim runs what "sfdupes scan /" runs, then what "sfdupes trees" runs.
"sfdupes trees" runs. Run it as root, so that the scan can read every Run it as root, so that the scan can read every file. The scan is stored
file. The scan is stored in the database, as scan stores it, so in the database, as scan stores it, so "sfdupes trees" prints the report
"sfdupes trees" prints the report again later. again later.
` `
) )
@@ -100,11 +100,12 @@ file. The scan is stored in the database, as scan stores it, so
// cobra prints for it is the whole message. // cobra prints for it is the whole message.
var errNoSubcommand = errors.New("no subcommand") var errNoSubcommand = errors.New("no subcommand")
// errWorkersBelowOne is the usage error for a --workers value below 1. // errWorkersBelowOne is the usage error for a scan --workers value
// below 1.
var errWorkersBelowOne = errors.New("--workers must be at least 1") var errWorkersBelowOne = errors.New("--workers must be at least 1")
// errEmptyExcludeValue is the usage error for an empty --exclude value, // errEmptyExcludeValue is the usage error for an empty scan --exclude
// as from --exclude "$DIR" with DIR unset. Taken as a path, it // value, as from --exclude "$DIR" with DIR unset. Taken as a path, it
// would name the working directory. // would name the working directory.
var errEmptyExcludeValue = errors.New("--exclude must not be empty") var errEmptyExcludeValue = errors.New("--exclude must not be empty")
@@ -255,67 +256,43 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}), }),
} }
dwimCmd := &cobra.Command{
Use: cmdDwim,
Short: "Scan / with the default excludes, then print the duplicate-tree report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", stdout)
}),
}
root.Long = root.Short + ".\n\n" + databaseHelp root.Long = root.Short + ".\n\n" + databaseHelp
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
dwimCmd.Long = dwimCmd.Short + ".\n\n" + dwimHelp
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
newDwimCommand(stdout))
return root return root
} }
// scanOptions holds the values of the flags that scan and dwim share: // newScanCommand builds the scan subcommand, with its --workers, -x,
// --workers, -x, --no-default-excludes and --exclude. // --no-default-excludes and --exclude flags. Its help names the
type scanOptions struct {
workers int
oneFS bool
noDefaultExcludes bool
excludePaths []string
}
// scanFlagsUsage names the flags that scan and dwim share, for their
// usage lines.
const scanFlagsUsage = "[--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]..."
// addScanFlags adds the flags that scan and dwim share to cmd, which
// stores their values in opts.
func addScanFlags(cmd *cobra.Command, opts *scanOptions) {
cmd.Flags().IntVar(&opts.workers, "workers", runtime.NumCPU(),
"concurrent workers for the walk and hash phases")
cmd.Flags().BoolVarP(&opts.oneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
cmd.Flags().BoolVar(&opts.noDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
// A string array, not a slice, so a comma in a path is not a
// separator.
cmd.Flags().StringArrayVar(&opts.excludePaths, "exclude", nil,
"never enter or record `PATH` or anything below it; repeatable")
}
// check rejects the flag values that are usage errors, before the
// command runs. It is the PreRunE of scan and dwim.
func (o *scanOptions) check(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, o.workers)
if err != nil {
return err
}
return rejectEmptyExcludeValue(cmd, o.excludePaths)
}
// newScanCommand builds the scan subcommand. Its help names the
// database, and lists the default excludes of the system it runs on and // database, and lists the default excludes of the system it runs on and
// the filesystem types it never records anything on. // the filesystem types it never records anything on.
func newScanCommand() *cobra.Command { func newScanCommand() *cobra.Command {
var opts scanOptions var (
scanWorkers int
scanOneFS bool
scanNoDefaultExcludes bool
scanExclude []string
)
short := "Walk trees and synchronize the scan database" short := "Walk trees and synchronize the scan database"
scanCmd := &cobra.Command{ scanCmd := &cobra.Command{
Use: cmdScan + " " + scanFlagsUsage + " PATH...", Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]... PATH...",
// The Use line already names every flag, so cobra must not // The Use line already names every flag, so cobra must not
// append its generic "[flags]" to it. // append its generic "[flags]" to it.
DisableFlagsInUseLine: true, DisableFlagsInUseLine: true,
@@ -323,48 +300,39 @@ func newScanCommand() *cobra.Command {
Long: short + ".\n\n" + databaseHelp + "\n" + Long: short + ".\n\n" + databaseHelp + "\n" +
defaultExcludesHelp(runtime.GOOS), defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1), Args: cobra.MinimumNArgs(1),
PreRunE: opts.check, PreRunE: func(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, scanWorkers)
if err != nil {
return err
}
return rejectEmptyExcludeValue(cmd, scanExclude)
},
RunE: runE(func(ctx context.Context, args []string) error { RunE: runE(func(ctx context.Context, args []string) error {
ctx, stop := interruptContext(ctx) ctx, stop := interruptContext(ctx)
defer stop() defer stop()
return runScan(ctx, args, opts) return runScan(ctx, args, scanWorkers, scanOneFS,
scanNoDefaultExcludes, scanExclude)
}), }),
} }
addScanFlags(scanCmd, &opts) scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
"concurrent workers for the walk and hash phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
// A string array, not a slice, so a comma in a path is not a
// separator.
scanCmd.Flags().StringArrayVar(&scanExclude, "exclude", nil,
"never enter or record `PATH` or anything below it; repeatable")
return scanCmd return scanCmd
} }
// newDwimCommand builds the dwim subcommand, which takes the scan flags // checkScanWorkers rejects a scan --workers value below 1. That is a
// with the meaning they have for scan. Its help says what dwim runs, // usage error reported in one line: cobra prints the returned message
// then lists the default excludes and the filesystem types the scan // without the usage text, and run exits 2.
// never records anything on.
func newDwimCommand(stdout io.Writer) *cobra.Command {
var opts scanOptions
short := "Scan /, then print the duplicate-tree report"
dwimCmd := &cobra.Command{
Use: cmdDwim + " " + scanFlagsUsage,
DisableFlagsInUseLine: true,
Short: short,
Long: short + ".\n\n" + dwimHelp + "\n" +
defaultExcludesHelp(runtime.GOOS),
Args: cobra.NoArgs,
PreRunE: opts.check,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", opts, stdout)
}),
}
addScanFlags(dwimCmd, &opts)
return dwimCmd
}
// checkScanWorkers rejects a --workers value below 1. That is a usage
// error reported in one line: cobra prints the returned message without
// the usage text, and run exits 2.
func checkScanWorkers(cmd *cobra.Command, workers int) error { func checkScanWorkers(cmd *cobra.Command, workers int) error {
if workers >= 1 { if workers >= 1 {
return nil return nil
@@ -375,7 +343,7 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers) return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
} }
// rejectEmptyExcludeValue rejects an empty --exclude value. That // rejectEmptyExcludeValue rejects an empty scan --exclude value. That
// is a usage error reported in one line, as for --workers below 1. // is a usage error reported in one line, as for --workers below 1.
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error { func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
if !slices.Contains(excludePaths, "") { if !slices.Contains(excludePaths, "") {
+3 -6
View File
@@ -25,9 +25,6 @@ const usageMarker = "Usage:"
// helpFlag asks for the help text. // helpFlag asks for the help text.
const helpFlag = "--help" const helpFlag = "--help"
// workersFlag is the scan and dwim flag that sets the worker count.
const workersFlag = "--workers"
// TestMain stops the tests at once, saying why, when their temporary // TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing // directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail // there (README "Rules for the walk"), so most scan tests would fail
@@ -576,8 +573,8 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}{ }{
cmdScan: { cmdScan: {
run: func() error { run: func() error {
return runScan(t.Context(), []string{missing}, return runScan(t.Context(), []string{missing}, 1, false, false,
scanOptions{workers: 1}) nil)
}, },
want: scanWant, want: scanWant,
}, },
@@ -668,7 +665,7 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
args := []string{cmdScan, workersFlag, workers, t.TempDir()} args := []string{cmdScan, "--workers", workers, t.TempDir()}
code := run(args, &stdout, &stderr) code := run(args, &stdout, &stderr)
if code != exitUsage { if code != exitUsage {
+10 -10
View File
@@ -5,20 +5,20 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"runtime"
) )
// runDwim implements the dwim subcommand: it runs what "sfdupes scan // runDwim implements the dwim subcommand: it runs what "sfdupes scan
// root" runs with the scan flags in opts, then what "sfdupes trees" // root" runs with no flags, then what "sfdupes trees" runs, and ends
// runs, and ends with a line on stderr saying how to print the report // with a line on stderr saying how to print the report again. The dwim
// again. The dwim command passes "/" as root. It catches SIGINT and // command passes "/" as root. It catches SIGINT and SIGTERM as scan
// SIGTERM as scan does, but only during the scan. An error from the // does, but only during the scan. An error from the scan, an interrupt
// scan, an interrupt included, ends it before the report. During the // included, ends it before the report. During the report, a signal
// report, a signal ends it at once, as it ends trees. // ends it at once, as it ends trees.
func runDwim( func runDwim(ctx context.Context, root string, stdout io.Writer) error {
ctx context.Context, root string, opts scanOptions, stdout io.Writer,
) error {
scanCtx, stop := interruptContext(ctx) scanCtx, stop := interruptContext(ctx)
err := runScan(scanCtx, []string{root}, opts) err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false,
nil)
stop() stop()
+2 -129
View File
@@ -2,10 +2,7 @@ package sfdupes //nolint:testpackage // calls the unexported runDwim
import ( import (
"bytes" "bytes"
"errors"
"os"
"path/filepath" "path/filepath"
"runtime"
"strings" "strings"
"testing" "testing"
) )
@@ -31,7 +28,7 @@ func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
err := runDwim(t.Context(), root, scanOptions{workers: 1}, &stdout) err := runDwim(t.Context(), root, &stdout)
if err != nil { if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr()) t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
} }
@@ -70,7 +67,7 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, &stdout) err := runDwim(t.Context(), missing, &stdout)
if err == nil { if err == nil {
t.Fatal("runDwim on a missing root: error = nil, want it fatal") t.Fatal("runDwim on a missing root: error = nil, want it fatal")
} }
@@ -84,127 +81,3 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
t.Errorf("stderr = %q, want nothing from trees or dwim", got) t.Errorf("stderr = %q, want nothing from trees or dwim", got)
} }
} }
func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
// dwim passes its --exclude paths to the scan: an excluded copy is
// not recorded, so its pair is not reported.
t.Setenv(databaseEnv, testDBPath(t))
root := realTempDir(t)
kept := []string{
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
}
writeFile(t, root, "b1/f.bin", pattern(2, 5000))
excluded := filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000)))
var stdout bytes.Buffer
stderr := captureStderr(t)
opts := scanOptions{workers: 1, excludePaths: []string{excluded}}
err := runDwim(t.Context(), root, opts, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
if len(lines) != 2 {
t.Fatalf("stdout = %q, want a header and one group",
stdout.String())
}
want := kept[0] + "\t" + kept[1] + "\t1\t300\t"
if !strings.HasPrefix(lines[1], want) {
t.Errorf("group = %q, want it to start %q", lines[1], want)
}
skipped := "scan: skipped " + excluded + " (--exclude)\n"
if got := stderr(); !strings.Contains(got, skipped) {
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
}
}
func TestRunDwimTakesScanFlags(t *testing.T) {
// The dwim command parses and checks the scan flags as scan does.
// Each case stops before the walk of / begins, and before the
// database is created. "dwim --exclude /" excludes its own operand.
cases := map[string]struct {
args []string
code int
want string
}{
"--exclude /": {
[]string{cmdDwim, excludeFlag, "/"},
exitFatal,
"sfdupes: scan /: operand excluded by --exclude /\n",
},
"--exclude empty": {
[]string{cmdDwim, excludeFlag, ""},
exitUsage,
"Error: --exclude must not be empty\n",
},
"--workers 0": {
[]string{cmdDwim, workersFlag, "0", "-x", "--no-default-excludes"},
exitUsage,
"Error: --workers must be at least 1, got 0\n",
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
stdout := captureStdout(t)
stderr := captureStderr(t)
code := run(tc.args, os.Stdout, os.Stderr)
if code != tc.code {
t.Errorf("run(%q) = %d, want %d", tc.args, code, tc.code)
}
if !strings.HasSuffix(stderr(), tc.want) || stdout() != "" {
t.Errorf("stdout, stderr = %q, %q, want \"\" and stderr "+
"ending %q", stdout(), stderr(), tc.want)
}
_, err := os.Lstat(path)
if !errors.Is(err, os.ErrNotExist) {
t.Errorf("lstat %s: %v, want no database", path, err)
}
})
}
}
func TestRunDwimHelpListsScanFlags(t *testing.T) {
t.Parallel()
// README "Subcommands" shows this usage line for dwim.
const want = "\n sfdupes dwim [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]...\n"
var stdout, stderr bytes.Buffer
run([]string{cmdDwim, "-h"}, &stdout, &stderr)
if !strings.Contains(stderr.String(), want) {
t.Errorf("dwim -h stderr = %q, want the usage line %q",
stderr.String(), want)
}
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
t.Errorf("dwim -h stderr = %q, want %s PATH", stderr.String(),
excludeFlag)
}
// README "Settings": the help lists the default excludes and the
// filesystem types the walk never records anything on.
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
if !strings.Contains(stderr.String(), help) {
t.Errorf("dwim -h stderr = %q, want it to hold %q", stderr.String(),
help)
}
}
+3 -20
View File
@@ -72,31 +72,14 @@ type walkRules struct {
func defaultExcludes(goos string) []string { func defaultExcludes(goos string) []string {
switch goos { switch goos {
case goosLinux: case goosLinux:
// The kernel's view of the running system, temporary files, and return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
// caches that programs and package managers fill again on their
// own: the system's, root's and those in each home in /home.
return []string{
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
"/root/go/pkg/mod",
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
"/home/*/go/pkg/mod",
}
case goosDarwin: case goosDarwin:
// A walk of / reaches everything in /System/Volumes/Data through // A walk of / reaches everything in /System/Volumes/Data through
// the firmlinks, so walking it as well would record each of // the firmlinks, so walking it as well would record each of
// those files twice. /private/var/folders holds each user's // those files twice.
// temporary files and caches. The walk reaches root's home,
// /var/root, as /private/var/root.
return []string{ return []string{
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm", "/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data", "/System/Volumes/VM", "/System/Volumes/Data",
"/private/var/folders", "/Library/Caches",
"/private/var/root/Library/Caches", "/private/var/root/.cache",
"/private/var/root/.npm", "/private/var/root/.cargo/registry",
"/private/var/root/go/pkg/mod",
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
} }
case goosWindows: case goosWindows:
// Relative to the root of the drive an operand lies on. The // Relative to the root of the drive an operand lies on. The
@@ -271,7 +254,7 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
return devices, lines.Err() return devices, lines.Err()
} }
// defaultExcludesHelp describes, for the scan and dwim help, the // defaultExcludesHelp describes, for the scan command's help, the
// default excludes of the system goos names and, where the scan lists // default excludes of the system goos names and, where the scan lists
// the mounted filesystems, the types it never records anything on. // the mounted filesystems, the types it never records anything on.
func defaultExcludesHelp(goos string) string { func defaultExcludesHelp(goos string) string {
+2 -51
View File
@@ -24,22 +24,10 @@ func TestDefaultExcludes(t *testing.T) {
// on any other system. The Windows list exists in the code only: the // on any other system. The Windows list exists in the code only: the
// binary does not build for Windows, so the README leaves it out. // binary does not build for Windows, so the README leaves it out.
cases := map[string][]string{ cases := map[string][]string{
goosLinux: { goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
"/root/go/pkg/mod",
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
"/home/*/go/pkg/mod",
},
goosDarwin: { goosDarwin: {
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm", "/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data", "/System/Volumes/VM", "/System/Volumes/Data",
"/private/var/folders", "/Library/Caches",
"/private/var/root/Library/Caches", "/private/var/root/.cache",
"/private/var/root/.npm", "/private/var/root/.cargo/registry",
"/private/var/root/go/pkg/mod",
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
}, },
goosWindows: { goosWindows: {
"pagefile.sys", "swapfile.sys", "hiberfil.sys", "pagefile.sys", "swapfile.sys", "hiberfil.sys",
@@ -75,29 +63,6 @@ func TestIsDefaultExclude(t *testing.T) {
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want) t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
} }
} }
// The Linux list matches the cache directories in /root and in each
// home in /home, but nothing below them and no .cache directory
// elsewhere.
linux := map[string]bool{
"/home/u/.cache": true,
"/home/u/go/pkg/mod": true,
"/root/.cache": true,
"/var/cache": true,
"/home/u/.cache/pip": false,
"/home/u/src/.cache": false,
"/srv/home/u/.cache": false,
"/home/u/.cargo/bin": false,
"/home/u/go/pkg/mod/cache": false,
}
for path, want := range linux {
got := isDefaultExclude(path, defaultExcludes(goosLinux))
if got != want {
t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
got, want)
}
}
} }
func TestReadMountInfo(t *testing.T) { func TestReadMountInfo(t *testing.T) {
@@ -325,7 +290,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
dir := realTempDir(t) dir := realTempDir(t)
if !insideDefaultExclude(dir, defaultExcludes(runtime.GOOS)) { if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
t.Skip(dir + " is not inside a default exclude") t.Skip(dir + " is not inside a default exclude")
} }
@@ -746,20 +711,6 @@ func shmTempDir(t *testing.T) (string, string, walkRules) {
return dir, fsType, walkRules{nonPersistent: nonPersistent} return dir, fsType, walkRules{nonPersistent: nonPersistent}
} }
// insideDefaultExclude reports whether path, which is absolute and
// clean, or a directory above it is one of excludes.
func insideDefaultExclude(path string, excludes []string) bool {
for dir := path; ; dir = filepath.Dir(dir) {
if isDefaultExclude(dir, excludes) {
return true
}
if dir == "/" {
return false
}
}
}
// assertRecorded checks that the database at dbPath holds a record for // assertRecorded checks that the database at dbPath holds a record for
// path when want is set, and none when it is not. // path when want is set, and none when it is not.
func assertRecorded(t *testing.T, dbPath, path string, want bool) { func assertRecorded(t *testing.T, dbPath, path string, want bool) {
+14 -13
View File
@@ -98,16 +98,17 @@ func scanPhase(name string) string {
// runs, and the lock is released after it. When ctx is cancelled, as by // runs, and the lock is released after it. When ctx is cancelled, as by
// the SIGINT or SIGTERM that interruptContext catches, the scan keeps // the SIGINT or SIGTERM that interruptContext catches, the scan keeps
// what it has hashed (see syncScan), prints how many files its walk // what it has hashed (see syncScan), prints how many files its walk
// reached, and returns errInterrupted. opts.workers must be at least 1; // reached, and returns errInterrupted. workers must be at least 1; the
// the scan and dwim commands reject anything less. Before the lock is // scan command rejects anything less. Before the lock is taken, the
// taken, the scan lists the filesystems that are not persistent, on // scan lists the filesystems that are not persistent, on which the walk
// which the walk records nothing; unless opts.noDefaultExcludes is set, // records nothing; unless noDefaultExcludes is set, the walk does not
// the walk does not enter the default excludes either. The walk neither // enter the default excludes either. The walk neither enters nor
// enters nor records the paths in opts.excludePaths, the --exclude // records the paths in excludePaths, the --exclude values; an operand
// values; an operand at or below one is a fatal error. Once the // at or below one is a fatal error. Once the database is open, the scan
// database is open, the scan notes its files, which the walk passes // notes its files, which the walk passes over.
// over. func runScan(ctx context.Context, roots []string, workers int,
func runScan(ctx context.Context, roots []string, opts scanOptions) error { oneFS, noDefaultExcludes bool, excludePaths []string,
) error {
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
return err return err
@@ -122,7 +123,7 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
var rules walkRules var rules walkRules
rules.excludePaths, err = excludePathInodes(opts.excludePaths) rules.excludePaths, err = excludePathInodes(excludePaths)
if err != nil { if err != nil {
return fmt.Errorf("--exclude: %w", err) return fmt.Errorf("--exclude: %w", err)
} }
@@ -137,7 +138,7 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
return fmt.Errorf("list the mounted filesystems: %w", err) return fmt.Errorf("list the mounted filesystems: %w", err)
} }
if !opts.noDefaultExcludes { if !noDefaultExcludes {
rules.defaultExcludes = defaultExcludes(runtime.GOOS) rules.defaultExcludes = defaultExcludes(runtime.GOOS)
} }
@@ -166,7 +167,7 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
return err return err
} }
st, err := syncScan(ctx, db, roots, opts.workers, opts.oneFS, rules) st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
if errors.Is(err, context.Canceled) { if errors.Is(err, context.Canceled) {
return interrupted(st.walked) return interrupted(st.walked)
} }
+1 -1
View File
@@ -2196,7 +2196,7 @@ func TestScanHashWriteFailureUnwindsPool(t *testing.T) {
var stderr bytes.Buffer var stderr bytes.Buffer
code := run([]string{ code := run([]string{
cmdScan, workersFlag, strconv.Itoa(hashLeakWorkers), dir, cmdScan, "--workers", strconv.Itoa(hashLeakWorkers), dir,
}, io.Discard, &stderr) }, io.Discard, &stderr)
if code != exitFatal { if code != exitFatal {
t.Fatalf("run(scan) = %d, want %d; stderr: %s", t.Fatalf("run(scan) = %d, want %d; stderr: %s",