Leave caches out of the walk by default (closes #181)

The default excludes now cover caches that programs and package
managers fill again on their own. On Linux: /var/cache and, in /root
and in each home in /home, .cache, .npm, .cargo/registry and
go/pkg/mod. On macOS: /private/var/folders, /Library/Caches and, in
/private/var/root and in each home in /Users, Library/Caches plus the
same four. README and scan --help list them; --no-default-excludes
still turns them all off.

Model: opus-5-5

README.md

@@ -43,9 +43,7 @@ as opposed to a directory, a symlink or a device node. A symlink is a file that
 holds the path of another file and stands in for it. A device node is a file
 that stands for a piece of hardware, such as a disk or a terminal. `report`
 prints the file-level duplicates. `trees` prints the duplicate directory trees.
-Both cover the whole database, or only the paths given after them, as in
+`info` prints the database's path, size and record count.
-`sfdupes report /srv/photos`. `info` prints the database's path, size and record
-count.
 
 The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
 another file. The first `scan` creates it.
@@ -130,8 +128,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
     - one line naming the database;
     - progress lines, as described under "Progress";
     - a warning for each path it skips;
-    - a line for each system or temporary directory the walk leaves out (see
+    - a line for each system, temporary or cache directory the walk leaves out
-      "Rules for the walk");
+      (see "Rules for the walk");
     - a line for each path left out because its filesystem is one the kernel,
       the core of the operating system, makes up or keeps in memory, such as
       `tmpfs`;
@@ -260,8 +258,8 @@ In order:
 
 ```
 sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
-sfdupes report [PATH...] > dupes.tsv
+sfdupes report > dupes.tsv
-sfdupes trees [PATH...] > dupetrees.tsv
+sfdupes trees > dupetrees.tsv
 sfdupes info
 sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]...
 sfdupes --version
@@ -302,8 +300,9 @@ The reference below is also the program's specification.
   of directories on other filesystems, as the same flag does for `du` and
   `rsync`. By default the walk also enters directories on other filesystems.
 - `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
-  excludes. Those are the system and temporary directories the walk otherwise
+  excludes. Those are the system, temporary and cache directories the walk
-  leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
+  otherwise leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules
+  for the walk").
 - `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
   below it. `PATH` may be a directory or a file. The flag may be given more than
   once. The walk knows `PATH` by its device ID and inode number, as it knows the
@@ -573,12 +572,17 @@ walk practical on large or busy storage pools.
     `--no-default-excludes` does not change this rule.
 
 - Below an operand, the walk does not enter a default exclude. On Linux, the
-  default excludes hold temporary files and the kernel's view of the running
+  default excludes hold the kernel's view of the running system, temporary
-  system. They depend on the system, and `sfdupes scan --help` lists those of
+  files, and caches. A cache here is data that programs and package managers
-  the system it runs on. On Linux they are:
+  download or build again on their own. The default excludes depend on the
+  system, and `sfdupes scan --help` lists those of the system it runs on. In the
+  lists, `*` stands for any one directory name, so `/home/*/.cache` matches
+  `/home/alice/.cache`. On Linux they are:
 
     ```
-    /proc, /sys, /dev, /run, /tmp, /var/tmp
+    /proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache
+    /root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
+    /home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
     ```
 
     On macOS they are:
@@ -586,13 +590,21 @@ walk practical on large or busy storage pools.
     ```
     /dev, /private/tmp, /private/var/tmp, /private/var/vm
     /System/Volumes/VM, /System/Volumes/Data
+    /private/var/folders, /Library/Caches
+    /private/var/root/Library/Caches, /private/var/root/.cache
+    /private/var/root/.npm, /private/var/root/.cargo/registry
+    /private/var/root/go/pkg/mod
+    /Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
+    /Users/*/.cargo/registry, /Users/*/go/pkg/mod
     ```
 
-    A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already,
+    `/private/var/folders` holds each user's temporary files and caches on
-    through firmlinks. Firmlinks are the links macOS uses to show the data
+    macOS. Root's home on macOS is `/var/root`, which the walk reaches as
-    volume inside the system volume. Walking `/System/Volumes/Data` as well
+    `/private/var/root`. A walk of `/` on macOS reaches everything in
-    would record each of those files twice. Any other system has no default
+    `/System/Volumes/Data` already, through firmlinks. Firmlinks are the links
-    excludes.
+    macOS uses to show the data volume inside the system volume. Walking
+    `/System/Volumes/Data` as well would record each of those files twice. Any
+    other system has no default excludes.
 
     An operand at or inside a default exclude is walked, so
     `sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem.
@@ -688,12 +700,11 @@ and `scan` leaves it ignored.
 
 ### `report` mode
 
-`report` reads every record from the database. It takes optional `PATH`
+`report` reads every record from the database. It takes no positional arguments.
-operands. With none, it uses every record. With one or more, it uses only the
+A positional argument is anything on the command line that is not a flag, such
-records at or below them (see "Operands of `report` and `trees`"). `report`
+as the paths `scan` takes. `report` never opens a scanned file. It checks that
-never opens a scanned file. It checks that each path in a group still exists,
+each path in a group still exists, and looks at no other path. The check is
-and looks at no other path. It never looks up an operand. The check is `lstat`,
+`lstat`, the system call that looks up a path without following a symlink:
-the system call that looks up a path without following a symlink:
 
 - A path is gone when `lstat` finds that it, or a directory above it, does not
   exist.
@@ -725,27 +736,6 @@ with one signature come together, and so do the records of one size. So `report`
 holds only the duplicate groups of the size it is reading. Once it reaches a
 smaller size, it writes them, ordered by `first` path.
 
-#### Operands of `report` and `trees`
-
-`report` and `trees` take the same `PATH` operands:
-
-- A record is used when its path is an operand or lies below one. `/srv/photos`
-  covers `/srv/photos/x` but not `/srv/photos2/x`.
-- The records used are grouped, ordered and totalled as they are without
-  operands. A copy outside the operands is not reported. With copies `/a/f`,
-  `/srv/photos/x` and `/srv/photos/y`, `sfdupes report /srv/photos` prints one
-  row, `/srv/photos/x` and `/srv/photos/y`.
-- A relative operand is joined to the working directory as `pwd -P` prints it.
-- Each operand is then cleaned as text, as `realpath -ms` prints it. Its `.`
-  elements and repeated or trailing slashes are dropped. Each `..` drops the
-  element before it.
-- No operand is looked up on the filesystem. A symlink in an operand is not
-  followed, and an operand need not exist. The records hold real paths (see
-  "Operands" under "`scan` mode"), so an operand that passes through a symlink
-  matches no record.
-- An operand that matches no record is not an error; it adds nothing to the
-  report.
-
 #### Report output format
 
 The report is TSV (tab-separated values) on stdout. It holds a header line, then
@@ -773,8 +763,7 @@ Grouping and ordering use the stored path. The warnings about skipped paths that
 
 The summary line on stderr gives:
 
-- the records read, which are all the records in the database, with or without
+- the records read;
-  operands;
 - the duplicate groups;
 - the dupe files, which is the number of rows;
 - the reclaimable bytes, the sum of `size` over all rows.
@@ -785,12 +774,11 @@ report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
 
 ### `trees` mode
 
-`trees` reads the same database as `report` and takes the same `PATH` operands
+`trees` reads the same database as `report` and takes no positional arguments.
-(see "Operands of `report` and `trees`"). It reports whole duplicate directory
+It reports whole duplicate directory trees. Two directories are duplicate trees
-trees. Two directories are duplicate trees when they hold the same relative
+when they hold the same relative paths, and each path holds a file with the same
-paths, and each path holds a file with the same signature in both. It rebuilds
+signature in both. It rebuilds the directory tree from the paths in the records,
-the directory tree from the paths in the records, split on `/`. Like `report`,
+split on `/`. Like `report`, it never opens a scanned file. It checks only that
-it never opens a scanned file and never looks up an operand. It checks only that
 the directory at the top of each copy still exists, as `report` checks a path.
 
 Definitions:
@@ -818,11 +806,6 @@ Processing:
   directories that all have the same digest. The parents' group, or an
   ancestor's, already implies it. A group is always reported when it holds
   sibling directories, or when its members' parents have different digests.
-- With operands, `trees` builds the tree from the records it uses. It reports
-  only directories at or below an operand. Any other directory lies above an
-  operand and can hold files that are not used, so its digest does not describe
-  it. Such a directory is never reported and never implies a group of its
-  children.
 - A directory that is gone is dropped from its group. A group left with fewer
   than two directories is not reported.
 - Within a group, paths are sorted in byte order. The first is `first`, and
@@ -876,8 +859,7 @@ directory's path is `/`.
 
 The summary line on stderr gives:
 
-- the records read, which are all the records in the database, with or without
+- the records read;
-  operands;
 - the duplicate-tree groups;
 - the dupe trees, which is the number of rows;
 - the reclaimable bytes, the sum of `size` over all rows.
@@ -889,10 +871,8 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
 ### `dwim` mode
 
 `dwim`, short for "do what I mean", scans a new machine and reports its
-duplicate trees in one command. It takes no positional arguments. A positional
+duplicate trees in one command. It takes no positional arguments. Like `scan`,
-argument is anything on the command line that is not a flag, such as the paths
+it should run as root, so that it can read every file.
-`scan` takes. Like `scan`, it should run as root, so that it can read every
-file.
 
 `dwim` takes these flags of `scan`, with the same meaning (see "Settings"):
 
@@ -1294,8 +1274,7 @@ showing the count it stopped at.
     - `scan` with no operand;
     - `--workers` below 1;
     - an empty `--exclude` value, as in `--exclude ""`;
-    - an empty `PATH` for `report` or `trees`, as in `sfdupes report ""`;
+    - `report`, `trees`, `info` or `dwim` with a positional argument.
-    - `info` or `dwim` with a positional argument.
 
 A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
 stdout, such as a full disk, is reported the same way. Two cases never reach

internal/sfdupes/cli.go

@@ -49,13 +49,6 @@ const escapesHelp = `Four characters in a path are escaped, so each row stays on
   \r  a carriage return
 `
 
-// operandsHelp is the paragraph of the report and trees help on their
-// PATH operands (README "Operands of `report` and `trees`").
-const operandsHelp = `With PATH operands, only the records at or below them are used. A
-relative PATH is joined to the working directory. No PATH is looked up
-on the filesystem, so a symlink in one is not followed.
-`
-
 // reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
 // summary in the help of their subcommands.
 const (
@@ -68,7 +61,7 @@ row per duplicate file. The columns are:
 
 Rows are ordered by size, largest first, then by path.
 
-` + escapesHelp + "\n" + operandsHelp
+` + escapesHelp
 
 	treesHelp = `The report is tab-separated text on stdout: a header line, then one
 row per duplicate directory tree. The columns are:
@@ -81,7 +74,7 @@ row per duplicate directory tree. The columns are:
 
 Rows are ordered by size, largest first, then by path.
 
-` + escapesHelp + "\n" + operandsHelp
+` + escapesHelp
 
 	infoHelp = `The details go to stdout, one per line. Each line holds a name and its
 value, separated by a tab:
@@ -115,11 +108,6 @@ var errWorkersBelowOne = errors.New("--workers must be at least 1")
 // would name the working directory.
 var errEmptyExcludeValue = errors.New("--exclude must not be empty")
 
-// errEmptyOperand is the usage error for an empty PATH operand of
-// report or trees, as from "$DIR" with DIR unset. Taken as a path, it
-// would name the working directory.
-var errEmptyOperand = errors.New("PATH must not be empty")
-
 // Version is the build version, injected at link time via -ldflags
 // (see the Makefile); empty when the linker set none, as for a binary
 // built by go install. buildVersion turns it into what --version prints.
@@ -241,20 +229,20 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
 		"print the version to stdout")
 
 	reportCmd := &cobra.Command{
-		Use:   cmdReport + " [PATH...]",
+		Use:   cmdReport,
 		Short: "Read the scan database and print the file-level duplicates report",
-		Args:  rejectEmptyOperand,
+		Args:  cobra.NoArgs,
-		RunE: runE(func(ctx context.Context, args []string) error {
+		RunE: runE(func(ctx context.Context, _ []string) error {
-			return runReport(ctx, args, stdout)
+			return runReport(ctx, stdout)
 		}),
 	}
 
 	treesCmd := &cobra.Command{
-		Use:   cmdTrees + " [PATH...]",
+		Use:   cmdTrees,
 		Short: "Read the scan database and print the duplicate-tree report",
-		Args:  rejectEmptyOperand,
+		Args:  cobra.NoArgs,
-		RunE: runE(func(ctx context.Context, args []string) error {
+		RunE: runE(func(ctx context.Context, _ []string) error {
-			return runTrees(ctx, args, stdout)
+			return runTrees(ctx, stdout)
 		}),
 	}
 
@@ -399,18 +387,6 @@ func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
 	return errEmptyExcludeValue
 }
 
-// rejectEmptyOperand rejects an empty PATH operand of report or trees.
-// That is a usage error reported in one line, as for --workers below 1.
-func rejectEmptyOperand(cmd *cobra.Command, args []string) error {
-	if !slices.Contains(args, "") {
-		return nil
-	}
-
-	cmd.SilenceUsage = true
-
-	return errEmptyOperand
-}
-
 // runE adapts a subcommand implementation, or the version print, to
 // cobra's RunE. Cobra prints the error and the command's usage text for
 // every error RunE returns, but a subcommand that ran and failed has no

internal/sfdupes/cli_test.go

@@ -583,14 +583,14 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
 		},
 		cmdReport: {
 			run: func() error {
-				return runReport(cancelledContext(t), nil, io.Discard)
+				return runReport(cancelledContext(t), io.Discard)
 			},
 			want: "report: database " + defaultDatabasePath + " (default)\n" +
 				reportOpenLine + "\n" + reportOpenLine + "\n",
 		},
 		cmdTrees: {
 			run: func() error {
-				return runTrees(cancelledContext(t), nil, io.Discard)
+				return runTrees(cancelledContext(t), io.Discard)
 			},
 			want: "trees: database " + defaultDatabasePath + " (default)\n" +
 				treesOpenLine + "\n" + treesOpenLine + "\n",
@@ -626,6 +626,8 @@ func TestRunUsageErrors(t *testing.T) {
 	}{
 		"no subcommand":      {[]string{}, usageMarker},
 		"scan without paths": {[]string{cmdScan}, usageMarker},
+		"report with args":   {[]string{cmdReport, "x"}, usageMarker},
+		"trees with args":    {[]string{cmdTrees, "x"}, usageMarker},
 		"info with args":     {[]string{cmdInfo, "x"}, usageMarker},
 		"dwim with args":     {[]string{cmdDwim, "x"}, usageMarker},
 		"unknown flag":       {[]string{cmdScan, "--nope", "/"}, usageMarker},
@@ -762,16 +764,11 @@ func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
 	t.Parallel()
 
 	// README §Settings: the help names the database, and says what
-	// report, trees, info and dwim print. The help of report and trees
+	// report, trees, info and dwim print.
-	// also explains their PATH operands.
 	database := []string{
 		"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
 	}
 	escapes := []string{`\\`, `\t`, `\n`, `\r`}
-	operands := []string{
-		"With PATH operands", "at or below them", "working directory",
-		"No PATH is looked up", "not followed",
-	}
 
 	// The subcommand whose help is printed; none for the root.
 	cases := []struct {
@@ -782,17 +779,17 @@ func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
 		{[]string{cmdScan}, database},
 		{
 			[]string{cmdReport},
-			slices.Concat([]string{
+			append([]string{
 				"tab-separated", "\n  first ", "\n  dupe ", "\n  size ",
 				"largest first",
-			}, escapes, operands),
+			}, escapes...),
 		},
 		{
 			[]string{cmdTrees},
-			slices.Concat([]string{
+			append([]string{
 				"tab-separated", "\n  first ", "\n  dupe ", "\n  files ",
 				"\n  size ", "\n  disk_size ", "largest first",
-			}, escapes, operands),
+			}, escapes...),
 		},
 		{
 			[]string{cmdInfo},
@@ -1842,12 +1839,8 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
 	scanFixture(t)
 
 	cases := map[string]func(context.Context, io.Writer) error{
-		cmdReport: func(ctx context.Context, w io.Writer) error {
+		cmdReport: runReport,
-			return runReport(ctx, nil, w)
+		cmdTrees:  runTrees,
-		},
-		cmdTrees: func(ctx context.Context, w io.Writer) error {
-			return runTrees(ctx, nil, w)
-		},
 		cmdInfo:   runInfo,
 	}
 

internal/sfdupes/dwim.go

@@ -26,7 +26,7 @@ func runDwim(
 		return err
 	}
 
-	err = runTrees(ctx, nil, stdout)
+	err = runTrees(ctx, stdout)
 	if err != nil {
 		return err
 	}

internal/sfdupes/exclude.go

@@ -72,14 +72,31 @@ type walkRules struct {
 func defaultExcludes(goos string) []string {
 	switch goos {
 	case goosLinux:
-		return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
+		// The kernel's view of the running system, temporary files, and
+		// caches that programs and package managers fill again on their
+		// own: the system's, root's and those in each home in /home.
+		return []string{
+			"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
+			"/root/.cache", "/root/.npm", "/root/.cargo/registry",
+			"/root/go/pkg/mod",
+			"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
+			"/home/*/go/pkg/mod",
+		}
 	case goosDarwin:
 		// A walk of / reaches everything in /System/Volumes/Data through
 		// the firmlinks, so walking it as well would record each of
-		// those files twice.
+		// those files twice. /private/var/folders holds each user's
+		// temporary files and caches. The walk reaches root's home,
+		// /var/root, as /private/var/root.
 		return []string{
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
+			"/private/var/folders", "/Library/Caches",
+			"/private/var/root/Library/Caches", "/private/var/root/.cache",
+			"/private/var/root/.npm", "/private/var/root/.cargo/registry",
+			"/private/var/root/go/pkg/mod",
+			"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
+			"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
 		}
 	case goosWindows:
 		// Relative to the root of the drive an operand lies on. The

internal/sfdupes/exclude_test.go

@@ -24,10 +24,22 @@ func TestDefaultExcludes(t *testing.T) {
 	// on any other system. The Windows list exists in the code only: the
 	// binary does not build for Windows, so the README leaves it out.
 	cases := map[string][]string{
-		goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
+		goosLinux: {
+			"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
+			"/root/.cache", "/root/.npm", "/root/.cargo/registry",
+			"/root/go/pkg/mod",
+			"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
+			"/home/*/go/pkg/mod",
+		},
 		goosDarwin: {
 			"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
 			"/System/Volumes/VM", "/System/Volumes/Data",
+			"/private/var/folders", "/Library/Caches",
+			"/private/var/root/Library/Caches", "/private/var/root/.cache",
+			"/private/var/root/.npm", "/private/var/root/.cargo/registry",
+			"/private/var/root/go/pkg/mod",
+			"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
+			"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
 		},
 		goosWindows: {
 			"pagefile.sys", "swapfile.sys", "hiberfil.sys",
@@ -63,6 +75,29 @@ func TestIsDefaultExclude(t *testing.T) {
 			t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
 		}
 	}
+
+	// The Linux list matches the cache directories in /root and in each
+	// home in /home, but nothing below them and no .cache directory
+	// elsewhere.
+	linux := map[string]bool{
+		"/home/u/.cache":           true,
+		"/home/u/go/pkg/mod":       true,
+		"/root/.cache":             true,
+		"/var/cache":               true,
+		"/home/u/.cache/pip":       false,
+		"/home/u/src/.cache":       false,
+		"/srv/home/u/.cache":       false,
+		"/home/u/.cargo/bin":       false,
+		"/home/u/go/pkg/mod/cache": false,
+	}
+
+	for path, want := range linux {
+		got := isDefaultExclude(path, defaultExcludes(goosLinux))
+		if got != want {
+			t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
+				got, want)
+		}
+	}
 }
 
 func TestReadMountInfo(t *testing.T) {
@@ -290,7 +325,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
 	t.Setenv(databaseEnv, path)
 
 	dir := realTempDir(t)
-	if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
+	if !insideDefaultExclude(dir, defaultExcludes(runtime.GOOS)) {
 		t.Skip(dir + " is not inside a default exclude")
 	}
 
@@ -711,6 +746,20 @@ func shmTempDir(t *testing.T) (string, string, walkRules) {
 	return dir, fsType, walkRules{nonPersistent: nonPersistent}
 }
 
+// insideDefaultExclude reports whether path, which is absolute and
+// clean, or a directory above it is one of excludes.
+func insideDefaultExclude(path string, excludes []string) bool {
+	for dir := path; ; dir = filepath.Dir(dir) {
+		if isDefaultExclude(dir, excludes) {
+			return true
+		}
+
+		if dir == "/" {
+			return false
+		}
+	}
+}
+
 // assertRecorded checks that the database at dbPath holds a record for
 // path when want is set, and none when it is not.
 func assertRecorded(t *testing.T, dbPath, path string, want bool) {

internal/sfdupes/report.go

@@ -9,7 +9,6 @@ import (
 	"io"
 	"io/fs"
 	"os"
-	"path/filepath"
 	"slices"
 	"strings"
 	"syscall"
@@ -45,12 +44,10 @@ func reportPhase(name string) string {
 }
 
 // runReport implements the report subcommand: it prints the file-level
-// duplicates report as TSV on stdout, from the records at or below the
+// duplicates report as TSV on stdout. Of the scanned filesystem it
-// operands in args, or from every record when args is empty. Of the
+// only checks, with pathExists, that each path in a duplicate group is
-// scanned filesystem it only checks, with pathExists, that each path in
+// still there. It names the database on stderr before it opens it. Any
-// a duplicate group is still there; it never looks up an operand. It
+// database problem, including a missing database, is fatal.
-// names the database on stderr before it opens it. Any database
-// problem, including a missing database, is fatal.
 //
 // The work runs in two phases, each under a progress display of its
 // own on stderr: open (open the database) and read (read the number of
@@ -58,12 +55,7 @@ func reportPhase(name string) string {
 // order, writing each size's duplicate groups to stdout once that size
 // is read; see readReport). Each display ends before the next phase
 // starts, and before an error is returned to be printed.
-func runReport(ctx context.Context, args []string, stdout io.Writer) error {
+func runReport(ctx context.Context, stdout io.Writer) error {
-	operands, err := cleanOperands(args)
-	if err != nil {
-		return err
-	}
-
 	dbPath, source, err := databasePath()
 	if err != nil {
 		return err
@@ -95,8 +87,7 @@ func runReport(ctx context.Context, args []string, stdout io.Writer) error {
 		label: reportPhase("read"), unit: "records", total: -1,
 		waitingFor: countingRecords, stalledOn: "db/disk/out",
 	})
-	report, err := readReport(ctx, db, dbPath, operands, stdout, pathExists,
+	report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog)
-		prog)
 
 	prog.finish()
 
@@ -117,14 +108,12 @@ func runReport(ctx context.Context, args []string, stdout io.Writer) error {
 // dbPath, as its total, then reads every record in signature order,
 // counting each on prog, and writes the report to stdout: the header
 // line, then the rows of each size's duplicate groups once that size is
-// read. Only the records that underOperand finds under operands join a
+// read. A path for which exists returns false is left out of its group.
-// group, and a path for which exists returns false is left out of its
+// stdout may be the terminal prog is drawn on, so the lines go through
-// group. stdout may be the terminal prog is drawn on, so the lines go
+// prog's writer. It returns what it wrote, with the totals for the
-// through prog's writer. It returns what it wrote, with the totals for
+// summary line.
-// the summary line.
 func readReport(ctx context.Context, db *sql.DB, dbPath string,
-	operands []string, stdout io.Writer, exists func(path string) bool,
+	stdout io.Writer, exists func(path string) bool, prog *progress,
-	prog *progress,
 ) (*reportWriter, error) {
 	out := prog.writer(stdout)
 	report := &reportWriter{out: out, exists: exists}
@@ -139,12 +128,6 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
 	err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
 		prog.increment()
 
-		report.records++
-
-		if !underOperand(r.path, operands) {
-			return nil
-		}
-
 		writeErr = report.add(r)
 
 		return writeErr
@@ -176,7 +159,7 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
 // records of one size, so it holds only the duplicate groups of the
 // size being read, and writes them, ordered by first path, once a
 // record of a smaller size comes. It counts what the summary line
-// reports; readReport counts the records read.
+// reports.
 type reportWriter struct {
 	out io.Writer
 
@@ -196,6 +179,8 @@ type reportWriter struct {
 // its paths; any other ends them, and one of a smaller size first
 // writes the groups of the size before.
 func (w *reportWriter) add(r scanRec) error {
+	w.records++
+
 	if r.size == w.sig.size && r.head == w.sig.head &&
 		r.tail == w.sig.tail && r.content == w.sig.content {
 		w.paths = append(w.paths, r.path)
@@ -288,47 +273,6 @@ func pathExists(path string) bool {
 	return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
 }
 
-// cleanOperands returns the operands of report or trees, each made
-// absolute by absolutePath and then cleaned. The records hold real
-// paths, with no "." or ".." element and no repeated or trailing
-// slash, so an operand is cleaned to match them. No operand is looked
-// up on the filesystem. None is empty: rejectEmptyOperand refuses an
-// empty operand on the command line.
-func cleanOperands(args []string) ([]string, error) {
-	operands := make([]string, 0, len(args))
-
-	for _, a := range args {
-		abs, err := absolutePath(a)
-		if err != nil {
-			return nil, fmt.Errorf("resolve path %s: %w", escapePath(a), err)
-		}
-
-		operands = append(operands, filepath.Clean(abs))
-	}
-
-	return operands, nil
-}
-
-// underOperand reports whether path is one of operands or lies below
-// one of them. Each operand is clean and absolute, as cleanOperands
-// returns it. With no operands, a report covers the whole database, so
-// underOperand is true for every path.
-func underOperand(path string, operands []string) bool {
-	if len(operands) == 0 {
-		return true
-	}
-
-	for _, o := range operands {
-		// Only the root directory's clean path ends in "/".
-		dir := strings.TrimSuffix(o, "/") + "/"
-		if path == o || strings.HasPrefix(path, dir) {
-			return true
-		}
-	}
-
-	return false
-}
-
 // escapePath returns a path as it is written in a report column (README
 // "Report output format"): a backslash, tab, newline or carriage return
 // becomes \\, \t, \n or \r, and every other byte is kept as it is.

internal/sfdupes/report_test.go

@@ -102,8 +102,7 @@ func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
 
 	var out bytes.Buffer
 
-	_, err := readReport(t.Context(), db, "", nil, &out, everyPathExists,
+	_, err := readReport(t.Context(), db, "", &out, everyPathExists, nil)
-		nil)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -149,136 +148,6 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
 	return dupeGroups(t, db)
 }
 
-// TestRunReportOperands runs report with operands on three copies of
-// one file, two of them under srv/photos, and a fourth copy in
-// srv/photos2, all in a new directory. Only the records at or below the
-// operands form the groups and the totals, however each operand is
-// written.
-//
-//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
-func TestRunReportOperands(t *testing.T) {
-	h := testHash("h")
-
-	recs := make([]scanRec, 0, 4)
-	for _, p := range []string{
-		"/a/copy", "/srv/photos/x", "/srv/photos/y", "/srv/photos2/z",
-	} {
-		recs = append(recs,
-			scanRec{size: 7, head: h, tail: h, content: h, path: p})
-	}
-
-	dir, recs := recsOnDisk(t, recs)
-	a, x, y, z := recs[0].path, recs[1].path, recs[2].path, recs[3].path
-
-	t.Setenv(databaseEnv, seedDatabase(t, recs))
-
-	header := "first\tdupe\tsize\n"
-	row := func(first, dupe string) string {
-		return first + "\t" + dupe + "\t7\n"
-	}
-	photos := header + row(x, y)
-
-	cases := map[string]struct {
-		operands []string
-		want     string
-		summary  string
-	}{
-		"directory": {[]string{dir + "/srv/photos"}, photos,
-			"1 duplicate groups, 1 dupe files, 7 B reclaimable"},
-		"trailing slash":  {[]string{dir + "/srv/photos/"}, photos, ""},
-		"dot and dot-dot": {[]string{dir + "/srv/./a/../photos"}, photos, ""},
-		"files":           {[]string{a, z}, header + row(a, z), ""},
-		"overlapping": {[]string{dir + "/srv", dir + "/srv/photos"},
-			header + row(x, y) + row(x, z), ""},
-		"root": {[]string{"/"},
-			header + row(a, x) + row(a, y) + row(a, z), ""},
-		"no record": {[]string{dir + "/srv/photo"}, header,
-			"0 duplicate groups, 0 dupe files, 0 B reclaimable"},
-	}
-
-	for name, tc := range cases {
-		t.Run(name, func(t *testing.T) {
-			var stdout bytes.Buffer
-
-			stderr := captureStderr(t)
-
-			code := run(append([]string{cmdReport}, tc.operands...),
-				&stdout, os.Stderr)
-			if code != exitOK {
-				t.Fatalf("run(report %q) = %d, want %d; stderr: %s",
-					tc.operands, code, exitOK, stderr())
-			}
-
-			if got := stdout.String(); got != tc.want {
-				t.Errorf("stdout = %q, want %q", got, tc.want)
-			}
-
-			// The records read are all four, whatever the operands.
-			summary := "report: 4 records read, " + tc.summary + "\n"
-			if got := stderr(); tc.summary != "" &&
-				!strings.HasSuffix(got, summary) {
-				t.Errorf("stderr = %q, want it to end %q", got, summary)
-			}
-		})
-	}
-}
-
-//nolint:paralleltest // t.Chdir changes the process-wide working directory
-func TestCleanOperandsMakesRelativePathsAbsolute(t *testing.T) {
-	dir := t.TempDir()
-	t.Chdir(dir)
-
-	wd, err := absolutePath(".")
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	got, err := cleanOperands([]string{"photos/", "a/../b", "/c//d"})
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	want := []string{
-		filepath.Join(wd, "photos"), filepath.Join(wd, "b"), "/c/d",
-	}
-	if !slices.Equal(got, want) {
-		t.Errorf("cleanOperands = %q, want %q", got, want)
-	}
-}
-
-// TestRunReportsRejectEmptyOperand checks that an empty PATH operand of
-// report or trees, alone or after another operand, is a usage error
-// reported in one line before the database is opened (README "Error
-// handling and exit codes").
-func TestRunReportsRejectEmptyOperand(t *testing.T) {
-	cases := map[string][]string{
-		"report":           {cmdReport, ""},
-		"trees":            {cmdTrees, ""},
-		"after an operand": {cmdTrees, t.TempDir(), ""},
-	}
-
-	for name, args := range cases {
-		t.Run(name, func(t *testing.T) {
-			// No database exists there, so opening one would be a fatal
-			// error, exit 1.
-			t.Setenv(databaseEnv, testDBPath(t))
-
-			var stdout, stderr bytes.Buffer
-
-			code := run(args, &stdout, &stderr)
-			if code != exitUsage {
-				t.Errorf("run(%q) = %d, want %d", args, code, exitUsage)
-			}
-
-			want := "Error: PATH must not be empty\n"
-			if stdout.String() != "" || stderr.String() != want {
-				t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
-					stdout.String(), stderr.String(), want)
-			}
-		})
-	}
-}
-
 func TestRunReportEscapesPaths(t *testing.T) {
 	dir, recs := recsOnDisk(t, awkwardPairRecs())
 	t.Setenv(databaseEnv, seedDatabase(t, recs))
@@ -326,7 +195,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
 	// Counts the records read, and draws nothing.
 	prog := &progress{now: time.Now}
 
-	_, err = readReport(t.Context(), db, "", nil, failingWriter{},
+	_, err = readReport(t.Context(), db, "", failingWriter{},
 		everyPathExists, prog)
 	if !errors.Is(err, errWriteFailed) ||
 		!strings.HasPrefix(err.Error(), "write stdout: ") {
@@ -423,8 +292,7 @@ func TestReadReportOnTerminal(t *testing.T) {
 		waitingFor: countingRecords,
 	}, terminalWidth, time.Hour, time.Now)
 
-	_, err = readReport(t.Context(), db, "", nil, os.Stderr,
+	_, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p)
-		everyPathExists, p)
 
 	p.finish()
 
@@ -646,7 +514,7 @@ func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
 		return true
 	}
 
-	_, err = readReport(t.Context(), db, "", nil, io.Discard, exists, nil)
+	_, err = readReport(t.Context(), db, "", io.Discard, exists, nil)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -680,8 +548,7 @@ func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
 
 	exists := func(path string) bool { return path != "/a/gone" }
 
-	report, err := readReport(t.Context(), db, "", nil, io.Discard, exists,
+	report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
-		nil)
 	if err != nil {
 		t.Fatal(err)
 	}
@@ -714,7 +581,7 @@ func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
 
 	var out bytes.Buffer
 
-	report, err := readReport(t.Context(), db, "", nil, &out, exists, nil)
+	report, err := readReport(t.Context(), db, "", &out, exists, nil)
 	if err != nil {
 		t.Fatal(err)
 	}

internal/sfdupes/trees.go

@@ -37,14 +37,12 @@ func treesPhase(name string) string {
 
 // runTrees implements the trees subcommand: it reads every record from
 // the database in path order, reconstructs the directory hierarchy from
-// the paths of the records at or below the operands in args (of every
+// the record paths, computes a Merkle-style digest per directory, and
-// record when args is empty), computes a Merkle-style digest per
+// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned
-// directory, and prints maximal duplicate-tree groups as TSV on stdout.
+// filesystem it only checks, with pathExists, that the root of each
-// Of the scanned filesystem it only checks, with pathExists, that the
+// duplicate tree is still there, never the files below it. It names the
-// root of each duplicate tree is still there, never the files below it;
+// database on stderr before it opens it. Any database problem,
-// it never looks up an operand. It names the database on stderr before
+// including a missing database, is fatal.
-// it opens it. Any database problem, including a missing database, is
-// fatal.
 //
 // The work runs in four phases, each under a progress display of its
 // own on stderr: open (open the database), read (read the number of
@@ -52,12 +50,7 @@ func treesPhase(name string) string {
 // hierarchy), group (group the directories by digest) and write (write
 // the groups to stdout). Each display ends before the next phase
 // starts, and before an error is returned to be printed.
-func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
+func runTrees(ctx context.Context, stdout io.Writer) error {
-	operands, err := cleanOperands(args)
-	if err != nil {
-		return err
-	}
-
 	dbPath, source, err := databasePath()
 	if err != nil {
 		return err
@@ -86,7 +79,7 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
 		label: treesPhase("read"), unit: "records", total: -1,
 		waitingFor: countingRecords, stalledOn: "db",
 	})
-	records, super, allDirs, err := readTree(ctx, db, operands, prog)
+	records, super, allDirs, err := readTree(ctx, db, prog)
 
 	prog.finish()
 
@@ -125,17 +118,10 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
 }
 
 // readTree gives prog the number of records in db as its total, then
-// reads every record in path order, counting each on prog. It adds the
+// reads every record in path order into the directory hierarchy,
-// records that underOperand finds under operands to the directory
+// counting each on prog. It returns the number of records read, the
-// hierarchy. It returns the number of records read, the super-root and
+// super-root and every directory.
-// every directory at or below one of operands.
+func readTree(ctx context.Context, db *sql.DB,
-//
-// Any other directory lies above an operand and can hold files that are
-// not used, so its digest does not describe it. It is left out of the
-// directories returned. A directory whose parent is left out gets the
-// super-root as its parent, so suppressed never judges its group by
-// that parent's digest.
-func readTree(ctx context.Context, db *sql.DB, operands []string,
 	prog *progress,
 ) (int, *treeNode, []*treeNode, error) {
 	records := 0
@@ -145,10 +131,7 @@ func readTree(ctx context.Context, db *sql.DB, operands []string,
 		records++
 
 		prog.increment()
-
-		if underOperand(r.path, operands) {
 		tree.add(r)
-		}
 	})
 	if err != nil {
 		return 0, nil, nil, err
@@ -156,21 +139,7 @@ func readTree(ctx context.Context, db *sql.DB, operands []string,
 
 	super, allDirs := tree.finish()
 
-	dirs := allDirs[:0]
+	return records, super, allDirs, nil
-
-	for _, d := range allDirs {
-		if !underOperand(d.path, operands) {
-			continue
-		}
-
-		if !underOperand(d.parent.path, operands) {
-			d.parent = super
-		}
-
-		dirs = append(dirs, d)
-	}
-
-	return records, super, dirs, nil
 }
 
 // writeTreeGroups writes dupes to stdout as TSV, a header line and then

internal/sfdupes/trees_test.go

@@ -214,57 +214,6 @@ func TestRunTreesEscapesPaths(t *testing.T) {
 	}
 }
 
-// TestRunTreesOperands runs trees with operands on records in a new
-// directory. x/a and y/a are copies, and so are y/a and srv/b. x and y
-// also hold files that differ. Given x/a and y/a, trees reads x and y
-// as holding one copy each of a, yet neither is reported: only the
-// directories at or below an operand are.
-func TestRunTreesOperands(t *testing.T) {
-	f, g, k := testHash("f"), testHash("g"), testHash("k")
-	dir, recs := recsOnDisk(t, []scanRec{
-		{size: 10, head: f, tail: f, content: f, path: "/srv/b/f"},
-		{size: 10, head: f, tail: f, content: f, path: "/x/a/f"},
-		{size: 20, head: g, tail: g, content: g, path: "/x/g"},
-		{size: 10, head: f, tail: f, content: f, path: "/y/a/f"},
-		{size: 30, head: k, tail: k, content: k, path: "/y/k"},
-	})
-	b, xa, ya := dir+"/srv/b", dir+"/x/a", dir+"/y/a"
-
-	header := "first\tdupe\tfiles\tsize\tdisk_size\n"
-	row := func(first, dupe string) string {
-		return first + "\t" + dupe + "\t1\t10\t0\n"
-	}
-
-	cases := map[string]struct {
-		operands []string
-		want     string
-	}{
-		"none":       {nil, header + row(b, xa) + row(b, ya)},
-		"two copies": {[]string{xa, ya}, header + row(xa, ya)},
-		"parents":    {[]string{dir + "/x", dir + "/y"}, header + row(xa, ya)},
-		"one copy":   {[]string{b}, header},
-	}
-
-	for name, tc := range cases {
-		t.Run(name, func(t *testing.T) {
-			t.Setenv(databaseEnv, seedDatabase(t, recs))
-
-			var stdout, stderr bytes.Buffer
-
-			code := run(append([]string{cmdTrees}, tc.operands...),
-				&stdout, &stderr)
-			if code != exitOK {
-				t.Fatalf("run(trees %q) = %d, want %d; stderr: %s",
-					tc.operands, code, exitOK, stderr.String())
-			}
-
-			if got := stdout.String(); got != tc.want {
-				t.Errorf("stdout = %q, want %q", got, tc.want)
-			}
-		})
-	}
-}
-
 // TestRunTreesDiskSize runs trees on two copies of a tree whose files
 // take different space on disk. The row's disk_size is the sum over the
 // dupe copy's files, those in its subdirectory included.