diff --git a/README.md b/README.md index dc51fc8..e3bff5a 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,9 @@ as opposed to a directory, a symlink or a device node. A symlink is a file that holds the path of another file and stands in for it. A device node is a file that stands for a piece of hardware, such as a disk or a terminal. `report` prints the file-level duplicates. `trees` prints the duplicate directory trees. -`info` prints the database's path, size and record count. +Both cover the whole database, or only the paths given after them, as in +`sfdupes report /srv/photos`. `info` prints the database's path, size and record +count. The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names another file. The first `scan` creates it. @@ -260,8 +262,8 @@ In order: ``` sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH... -sfdupes report > dupes.tsv -sfdupes trees > dupetrees.tsv +sfdupes report [PATH...] > dupes.tsv +sfdupes trees [PATH...] > dupetrees.tsv sfdupes info sfdupes dwim sfdupes --version @@ -686,11 +688,12 @@ and `scan` leaves it ignored. ### `report` mode -`report` reads every record from the database. It takes no positional arguments. -A positional argument is anything on the command line that is not a flag, such -as the paths `scan` takes. `report` never opens a scanned file. It checks that -each path in a group still exists, and looks at no other path. The check is -`lstat`, the system call that looks up a path without following a symlink: +`report` reads every record from the database. It takes optional `PATH` +operands. With none, it uses every record. With one or more, it uses only the +records at or below them (see "Operands of `report` and `trees`"). `report` +never opens a scanned file. It checks that each path in a group still exists, +and looks at no other path. It never looks up an operand. The check is `lstat`, +the system call that looks up a path without following a symlink: - A path is gone when `lstat` finds that it, or a directory above it, does not exist. @@ -722,6 +725,27 @@ with one signature come together, and so do the records of one size. So `report` holds only the duplicate groups of the size it is reading. Once it reaches a smaller size, it writes them, ordered by `first` path. +#### Operands of `report` and `trees` + +`report` and `trees` take the same `PATH` operands: + +- A record is used when its path is an operand or lies below one. `/srv/photos` + covers `/srv/photos/x` but not `/srv/photos2/x`. +- The records used are grouped, ordered and totalled as they are without + operands. A copy outside the operands is not reported. With copies `/a/f`, + `/srv/photos/x` and `/srv/photos/y`, `sfdupes report /srv/photos` prints one + row, `/srv/photos/x` and `/srv/photos/y`. +- A relative operand is joined to the working directory as `pwd -P` prints it. +- Each operand is then cleaned as text, as `realpath -ms` prints it. Its `.` + elements and repeated or trailing slashes are dropped. Each `..` drops the + element before it. +- No operand is looked up on the filesystem. A symlink in an operand is not + followed, and an operand need not exist. The records hold real paths (see + "Operands" under "`scan` mode"), so an operand that passes through a symlink + matches no record. +- An operand that matches no record is not an error. The report then has no + rows. + #### Report output format The report is TSV (tab-separated values) on stdout. It holds a header line, then @@ -749,7 +773,8 @@ Grouping and ordering use the stored path. The warnings about skipped paths that The summary line on stderr gives: -- the records read; +- the records read, which are all the records in the database, with or without + operands; - the duplicate groups; - the dupe files, which is the number of rows; - the reclaimable bytes, the sum of `size` over all rows. @@ -760,11 +785,12 @@ report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable ### `trees` mode -`trees` reads the same database as `report` and takes no positional arguments. -It reports whole duplicate directory trees. Two directories are duplicate trees -when they hold the same relative paths, and each path holds a file with the same -signature in both. It rebuilds the directory tree from the paths in the records, -split on `/`. Like `report`, it never opens a scanned file. It checks only that +`trees` reads the same database as `report` and takes the same `PATH` operands +(see "Operands of `report` and `trees`"). It reports whole duplicate directory +trees. Two directories are duplicate trees when they hold the same relative +paths, and each path holds a file with the same signature in both. It rebuilds +the directory tree from the paths in the records, split on `/`. Like `report`, +it never opens a scanned file and never looks up an operand. It checks only that the directory at the top of each copy still exists, as `report` checks a path. Definitions: @@ -792,6 +818,11 @@ Processing: directories that all have the same digest. The parents' group, or an ancestor's, already implies it. A group is always reported when it holds sibling directories, or when its members' parents have different digests. +- With operands, `trees` builds the tree from the records it uses. It reports + only directories at or below an operand. Any other directory lies above an + operand and can hold files that are not used, so its digest does not describe + it. Such a directory is never reported and never implies a group of its + children. - A directory that is gone is dropped from its group. A group left with fewer than two directories is not reported. - Within a group, paths are sorted in byte order. The first is `first`, and @@ -845,7 +876,8 @@ directory's path is `/`. The summary line on stderr gives: -- the records read; +- the records read, which are all the records in the database, with or without + operands; - the duplicate-tree groups; - the dupe trees, which is the number of rows; - the reclaimable bytes, the sum of `size` over all rows. @@ -858,7 +890,9 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab `dwim`, short for "do what I mean", scans a new machine and reports its duplicate trees in one command. It takes no positional arguments and no flags of -its own. Like `scan`, it should run as root, so that it can read every file. +its own. A positional argument is anything on the command line that is not a +flag, such as the paths `scan` takes. Like `scan`, it should run as root, so +that it can read every file. `dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default excludes and crosses into other filesystems. `--workers` keeps its default. The @@ -1249,7 +1283,7 @@ showing the count it stopped at. - `scan` with no operand; - `--workers` below 1; - an empty `--exclude` value, as in `--exclude ""`; - - `report`, `trees`, `info` or `dwim` with a positional argument. + - `info` or `dwim` with a positional argument. A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to stdout, such as a full disk, is reported the same way. Two cases never reach diff --git a/TODO.md b/TODO.md index 8fb200d..67a3fb3 100644 --- a/TODO.md +++ b/TODO.md @@ -29,6 +29,11 @@ Newest first. An entry ends with its date and its issue or branch, where those are known. +- `report` and `trees` take optional `PATH` operands. With operands, they use + only the records at or below them. `trees` then reports only directories at or + below an operand. Operands are made absolute and cleaned as text, never looked + up on the filesystem (2026-11-11, + https://git.eeqj.de/sneak/sfdupes/issues/174) - New `scan` flag `--exclude PATH`, which may be repeated. The walk neither enters nor records an excluded path, and keeps the records below it. An operand at or below one is a fatal error (2026-11-11, diff --git a/internal/sfdupes/cli.go b/internal/sfdupes/cli.go index d37754d..cc13c33 100644 --- a/internal/sfdupes/cli.go +++ b/internal/sfdupes/cli.go @@ -32,6 +32,13 @@ const ( cmdDwim = "dwim" ) +// operandsHelp explains the PATH operands of report and trees in their +// help. +const operandsHelp = "With PATH operands, only the records at or " + + "below them are used. A relative PATH is joined to the working " + + "directory. No PATH is looked up on the filesystem, so a symlink in " + + "one is not followed." + // errNoSubcommand is returned by the root command when it is invoked // without a subcommand. That is a usage error, and the usage text // cobra prints for it is the whole message. @@ -167,20 +174,24 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command { "print the version to stdout") reportCmd := &cobra.Command{ - Use: cmdReport, + Use: cmdReport + " [PATH...]", Short: "Read the scan database and print the file-level duplicates report", - Args: cobra.NoArgs, - RunE: runE(func(ctx context.Context, _ []string) error { - return runReport(ctx, stdout) + Long: "Read the scan database and print the file-level duplicates " + + "report.\n\n" + operandsHelp, + Args: cobra.ArbitraryArgs, + RunE: runE(func(ctx context.Context, args []string) error { + return runReport(ctx, args, stdout) }), } treesCmd := &cobra.Command{ - Use: cmdTrees, + Use: cmdTrees + " [PATH...]", Short: "Read the scan database and print the duplicate-tree report", - Args: cobra.NoArgs, - RunE: runE(func(ctx context.Context, _ []string) error { - return runTrees(ctx, stdout) + Long: "Read the scan database and print the duplicate-tree " + + "report.\n\n" + operandsHelp, + Args: cobra.ArbitraryArgs, + RunE: runE(func(ctx context.Context, args []string) error { + return runTrees(ctx, args, stdout) }), } diff --git a/internal/sfdupes/cli_test.go b/internal/sfdupes/cli_test.go index dc1bf26..30950ff 100644 --- a/internal/sfdupes/cli_test.go +++ b/internal/sfdupes/cli_test.go @@ -577,14 +577,14 @@ func TestRunNamesDefaultDatabase(t *testing.T) { }, cmdReport: { run: func() error { - return runReport(cancelledContext(t), io.Discard) + return runReport(cancelledContext(t), nil, io.Discard) }, want: "report: database " + defaultDatabasePath + " (default)\n" + reportOpenLine + "\n" + reportOpenLine + "\n", }, cmdTrees: { run: func() error { - return runTrees(cancelledContext(t), io.Discard) + return runTrees(cancelledContext(t), nil, io.Discard) }, want: "trees: database " + defaultDatabasePath + " (default)\n" + treesOpenLine + "\n" + treesOpenLine + "\n", @@ -620,8 +620,6 @@ func TestRunUsageErrors(t *testing.T) { }{ "no subcommand": {[]string{}, usageMarker}, "scan without paths": {[]string{cmdScan}, usageMarker}, - "report with args": {[]string{cmdReport, "x"}, usageMarker}, - "trees with args": {[]string{cmdTrees, "x"}, usageMarker}, "info with args": {[]string{cmdInfo, "x"}, usageMarker}, "dwim with args": {[]string{cmdDwim, "x"}, usageMarker}, "unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker}, @@ -1766,9 +1764,13 @@ func TestStdoutWriteErrorPropagates(t *testing.T) { scanFixture(t) cases := map[string]func(context.Context, io.Writer) error{ - cmdReport: runReport, - cmdTrees: runTrees, - cmdInfo: runInfo, + cmdReport: func(ctx context.Context, w io.Writer) error { + return runReport(ctx, nil, w) + }, + cmdTrees: func(ctx context.Context, w io.Writer) error { + return runTrees(ctx, nil, w) + }, + cmdInfo: runInfo, } for name, fn := range cases { diff --git a/internal/sfdupes/dwim.go b/internal/sfdupes/dwim.go index 7225a00..ccaf8c7 100644 --- a/internal/sfdupes/dwim.go +++ b/internal/sfdupes/dwim.go @@ -26,7 +26,7 @@ func runDwim(ctx context.Context, root string, stdout io.Writer) error { return err } - err = runTrees(ctx, stdout) + err = runTrees(ctx, nil, stdout) if err != nil { return err } diff --git a/internal/sfdupes/report.go b/internal/sfdupes/report.go index 84eb53f..f7f2d6c 100644 --- a/internal/sfdupes/report.go +++ b/internal/sfdupes/report.go @@ -9,6 +9,7 @@ import ( "io" "io/fs" "os" + "path/filepath" "slices" "strings" "syscall" @@ -44,10 +45,12 @@ func reportPhase(name string) string { } // runReport implements the report subcommand: it prints the file-level -// duplicates report as TSV on stdout. Of the scanned filesystem it -// only checks, with pathExists, that each path in a duplicate group is -// still there. It names the database on stderr before it opens it. Any -// database problem, including a missing database, is fatal. +// duplicates report as TSV on stdout, from the records at or below the +// operands in args, or from every record when args is empty. Of the +// scanned filesystem it only checks, with pathExists, that each path in +// a duplicate group is still there; it never looks up an operand. It +// names the database on stderr before it opens it. Any database +// problem, including a missing database, is fatal. // // The work runs in two phases, each under a progress display of its // own on stderr: open (open the database) and read (read the number of @@ -55,7 +58,12 @@ func reportPhase(name string) string { // order, writing each size's duplicate groups to stdout once that size // is read; see readReport). Each display ends before the next phase // starts, and before an error is returned to be printed. -func runReport(ctx context.Context, stdout io.Writer) error { +func runReport(ctx context.Context, args []string, stdout io.Writer) error { + operands, err := cleanOperands(args) + if err != nil { + return err + } + dbPath, source, err := databasePath() if err != nil { return err @@ -87,7 +95,8 @@ func runReport(ctx context.Context, stdout io.Writer) error { label: reportPhase("read"), unit: "records", total: -1, waitingFor: countingRecords, stalledOn: "db/disk/out", }) - report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog) + report, err := readReport(ctx, db, dbPath, operands, stdout, pathExists, + prog) prog.finish() @@ -108,12 +117,14 @@ func runReport(ctx context.Context, stdout io.Writer) error { // dbPath, as its total, then reads every record in signature order, // counting each on prog, and writes the report to stdout: the header // line, then the rows of each size's duplicate groups once that size is -// read. A path for which exists returns false is left out of its group. -// stdout may be the terminal prog is drawn on, so the lines go through -// prog's writer. It returns what it wrote, with the totals for the -// summary line. +// read. Only the records that underOperand finds under operands join a +// group, and a path for which exists returns false is left out of its +// group. stdout may be the terminal prog is drawn on, so the lines go +// through prog's writer. It returns what it wrote, with the totals for +// the summary line. func readReport(ctx context.Context, db *sql.DB, dbPath string, - stdout io.Writer, exists func(path string) bool, prog *progress, + operands []string, stdout io.Writer, exists func(path string) bool, + prog *progress, ) (*reportWriter, error) { out := prog.writer(stdout) report := &reportWriter{out: out, exists: exists} @@ -128,6 +139,12 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string, err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error { prog.increment() + report.records++ + + if !underOperand(r.path, operands) { + return nil + } + writeErr = report.add(r) return writeErr @@ -159,7 +176,7 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string, // records of one size, so it holds only the duplicate groups of the // size being read, and writes them, ordered by first path, once a // record of a smaller size comes. It counts what the summary line -// reports. +// reports; readReport counts the records read. type reportWriter struct { out io.Writer @@ -179,8 +196,6 @@ type reportWriter struct { // its paths; any other ends them, and one of a smaller size first // writes the groups of the size before. func (w *reportWriter) add(r scanRec) error { - w.records++ - if r.size == w.sig.size && r.head == w.sig.head && r.tail == w.sig.tail && r.content == w.sig.content { w.paths = append(w.paths, r.path) @@ -273,6 +288,46 @@ func pathExists(path string) bool { return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR) } +// cleanOperands returns the operands of report or trees, each made +// absolute by absolutePath and then cleaned. The records hold real +// paths, with no "." or ".." element and no repeated or trailing +// slash, so an operand is cleaned to match them. No operand is looked +// up on the filesystem. +func cleanOperands(args []string) ([]string, error) { + operands := make([]string, 0, len(args)) + + for _, a := range args { + abs, err := absolutePath(a) + if err != nil { + return nil, fmt.Errorf("resolve path %s: %w", escapePath(a), err) + } + + operands = append(operands, filepath.Clean(abs)) + } + + return operands, nil +} + +// underOperand reports whether path is one of operands or lies below +// one of them. Each operand is clean and absolute, as cleanOperands +// returns it. With no operands, a report covers the whole database, so +// underOperand is true for every path. +func underOperand(path string, operands []string) bool { + if len(operands) == 0 { + return true + } + + for _, o := range operands { + // Only the root directory's clean path ends in "/". + dir := strings.TrimSuffix(o, "/") + "/" + if path == o || strings.HasPrefix(path, dir) { + return true + } + } + + return false +} + // escapePath returns a path as it is written in a report column (README // "Report output format"): a backslash, tab, newline or carriage return // becomes \\, \t, \n or \r, and every other byte is kept as it is. diff --git a/internal/sfdupes/report_test.go b/internal/sfdupes/report_test.go index e976688..a8a578d 100644 --- a/internal/sfdupes/report_test.go +++ b/internal/sfdupes/report_test.go @@ -102,7 +102,8 @@ func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup { var out bytes.Buffer - _, err := readReport(t.Context(), db, "", &out, everyPathExists, nil) + _, err := readReport(t.Context(), db, "", nil, &out, everyPathExists, + nil) if err != nil { t.Fatal(err) } @@ -148,6 +149,103 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup { return dupeGroups(t, db) } +// TestRunReportOperands runs report with operands on three copies of +// one file, two of them under srv/photos, and a fourth copy in +// srv/photos2, all in a new directory. Only the records at or below the +// operands form the groups and the totals, however each operand is +// written. +// +//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr +func TestRunReportOperands(t *testing.T) { + h := testHash("h") + + recs := make([]scanRec, 0, 4) + for _, p := range []string{ + "/a/copy", "/srv/photos/x", "/srv/photos/y", "/srv/photos2/z", + } { + recs = append(recs, + scanRec{size: 7, head: h, tail: h, content: h, path: p}) + } + + dir, recs := recsOnDisk(t, recs) + a, x, y, z := recs[0].path, recs[1].path, recs[2].path, recs[3].path + + t.Setenv(databaseEnv, seedDatabase(t, recs)) + + header := "first\tdupe\tsize\n" + row := func(first, dupe string) string { + return first + "\t" + dupe + "\t7\n" + } + photos := header + row(x, y) + + cases := map[string]struct { + operands []string + want string + summary string + }{ + "directory": {[]string{dir + "/srv/photos"}, photos, + "1 duplicate groups, 1 dupe files, 7 B reclaimable"}, + "trailing slash": {[]string{dir + "/srv/photos/"}, photos, ""}, + "dot and dot-dot": {[]string{dir + "/srv/./a/../photos"}, photos, ""}, + "files": {[]string{a, z}, header + row(a, z), ""}, + "overlapping": {[]string{dir + "/srv", dir + "/srv/photos"}, + header + row(x, y) + row(x, z), ""}, + "root": {[]string{"/"}, + header + row(a, x) + row(a, y) + row(a, z), ""}, + "no record": {[]string{dir + "/srv/photo"}, header, + "0 duplicate groups, 0 dupe files, 0 B reclaimable"}, + } + + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + var stdout bytes.Buffer + + stderr := captureStderr(t) + + code := run(append([]string{cmdReport}, tc.operands...), + &stdout, os.Stderr) + if code != exitOK { + t.Fatalf("run(report %q) = %d, want %d; stderr: %s", + tc.operands, code, exitOK, stderr()) + } + + if got := stdout.String(); got != tc.want { + t.Errorf("stdout = %q, want %q", got, tc.want) + } + + // The records read are all four, whatever the operands. + summary := "report: 4 records read, " + tc.summary + "\n" + if got := stderr(); tc.summary != "" && + !strings.HasSuffix(got, summary) { + t.Errorf("stderr = %q, want it to end %q", got, summary) + } + }) + } +} + +//nolint:paralleltest // t.Chdir changes the process-wide working directory +func TestCleanOperandsMakesRelativePathsAbsolute(t *testing.T) { + dir := t.TempDir() + t.Chdir(dir) + + wd, err := absolutePath(".") + if err != nil { + t.Fatal(err) + } + + got, err := cleanOperands([]string{"photos/", "a/../b", "/c//d"}) + if err != nil { + t.Fatal(err) + } + + want := []string{ + filepath.Join(wd, "photos"), filepath.Join(wd, "b"), "/c/d", + } + if !slices.Equal(got, want) { + t.Errorf("cleanOperands = %q, want %q", got, want) + } +} + func TestRunReportEscapesPaths(t *testing.T) { dir, recs := recsOnDisk(t, awkwardPairRecs()) t.Setenv(databaseEnv, seedDatabase(t, recs)) @@ -195,7 +293,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) { // Counts the records read, and draws nothing. prog := &progress{now: time.Now} - _, err = readReport(t.Context(), db, "", failingWriter{}, + _, err = readReport(t.Context(), db, "", nil, failingWriter{}, everyPathExists, prog) if !errors.Is(err, errWriteFailed) || !strings.HasPrefix(err.Error(), "write stdout: ") { @@ -292,7 +390,8 @@ func TestReadReportOnTerminal(t *testing.T) { waitingFor: countingRecords, }, terminalWidth, time.Hour, time.Now) - _, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p) + _, err = readReport(t.Context(), db, "", nil, os.Stderr, + everyPathExists, p) p.finish() @@ -514,7 +613,7 @@ func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) { return true } - _, err = readReport(t.Context(), db, "", io.Discard, exists, nil) + _, err = readReport(t.Context(), db, "", nil, io.Discard, exists, nil) if err != nil { t.Fatal(err) } @@ -548,7 +647,8 @@ func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) { exists := func(path string) bool { return path != "/a/gone" } - report, err := readReport(t.Context(), db, "", io.Discard, exists, nil) + report, err := readReport(t.Context(), db, "", nil, io.Discard, exists, + nil) if err != nil { t.Fatal(err) } @@ -581,7 +681,7 @@ func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) { var out bytes.Buffer - report, err := readReport(t.Context(), db, "", &out, exists, nil) + report, err := readReport(t.Context(), db, "", nil, &out, exists, nil) if err != nil { t.Fatal(err) } diff --git a/internal/sfdupes/trees.go b/internal/sfdupes/trees.go index 36bfb12..b648a76 100644 --- a/internal/sfdupes/trees.go +++ b/internal/sfdupes/trees.go @@ -37,12 +37,14 @@ func treesPhase(name string) string { // runTrees implements the trees subcommand: it reads every record from // the database in path order, reconstructs the directory hierarchy from -// the record paths, computes a Merkle-style digest per directory, and -// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned -// filesystem it only checks, with pathExists, that the root of each -// duplicate tree is still there, never the files below it. It names the -// database on stderr before it opens it. Any database problem, -// including a missing database, is fatal. +// the paths of the records at or below the operands in args (of every +// record when args is empty), computes a Merkle-style digest per +// directory, and prints maximal duplicate-tree groups as TSV on stdout. +// Of the scanned filesystem it only checks, with pathExists, that the +// root of each duplicate tree is still there, never the files below it; +// it never looks up an operand. It names the database on stderr before +// it opens it. Any database problem, including a missing database, is +// fatal. // // The work runs in four phases, each under a progress display of its // own on stderr: open (open the database), read (read the number of @@ -50,7 +52,12 @@ func treesPhase(name string) string { // hierarchy), group (group the directories by digest) and write (write // the groups to stdout). Each display ends before the next phase // starts, and before an error is returned to be printed. -func runTrees(ctx context.Context, stdout io.Writer) error { +func runTrees(ctx context.Context, args []string, stdout io.Writer) error { + operands, err := cleanOperands(args) + if err != nil { + return err + } + dbPath, source, err := databasePath() if err != nil { return err @@ -79,7 +86,7 @@ func runTrees(ctx context.Context, stdout io.Writer) error { label: treesPhase("read"), unit: "records", total: -1, waitingFor: countingRecords, stalledOn: "db", }) - records, super, allDirs, err := readTree(ctx, db, prog) + records, super, allDirs, err := readTree(ctx, db, operands, prog) prog.finish() @@ -118,10 +125,17 @@ func runTrees(ctx context.Context, stdout io.Writer) error { } // readTree gives prog the number of records in db as its total, then -// reads every record in path order into the directory hierarchy, -// counting each on prog. It returns the number of records read, the -// super-root and every directory. -func readTree(ctx context.Context, db *sql.DB, +// reads every record in path order, counting each on prog. It adds the +// records that underOperand finds under operands to the directory +// hierarchy. It returns the number of records read, the super-root and +// every directory at or below one of operands. +// +// Any other directory lies above an operand and can hold files that are +// not used, so its digest does not describe it. It is left out of the +// directories returned. A directory whose parent is left out gets the +// super-root as its parent, so suppressed never judges its group by +// that parent's digest. +func readTree(ctx context.Context, db *sql.DB, operands []string, prog *progress, ) (int, *treeNode, []*treeNode, error) { records := 0 @@ -131,7 +145,10 @@ func readTree(ctx context.Context, db *sql.DB, records++ prog.increment() - tree.add(r) + + if underOperand(r.path, operands) { + tree.add(r) + } }) if err != nil { return 0, nil, nil, err @@ -139,7 +156,21 @@ func readTree(ctx context.Context, db *sql.DB, super, allDirs := tree.finish() - return records, super, allDirs, nil + dirs := allDirs[:0] + + for _, d := range allDirs { + if !underOperand(d.path, operands) { + continue + } + + if !underOperand(d.parent.path, operands) { + d.parent = super + } + + dirs = append(dirs, d) + } + + return records, super, dirs, nil } // writeTreeGroups writes dupes to stdout as TSV, a header line and then diff --git a/internal/sfdupes/trees_test.go b/internal/sfdupes/trees_test.go index 6a4d722..e0221f3 100644 --- a/internal/sfdupes/trees_test.go +++ b/internal/sfdupes/trees_test.go @@ -214,6 +214,57 @@ func TestRunTreesEscapesPaths(t *testing.T) { } } +// TestRunTreesOperands runs trees with operands on records in a new +// directory. x/a and y/a are copies, and so are y/a and srv/b. x and y +// also hold files that differ. Given x/a and y/a, trees reads x and y +// as holding one copy each of a, yet neither is reported: only the +// directories at or below an operand are. +func TestRunTreesOperands(t *testing.T) { + f, g, k := testHash("f"), testHash("g"), testHash("k") + dir, recs := recsOnDisk(t, []scanRec{ + {size: 10, head: f, tail: f, content: f, path: "/srv/b/f"}, + {size: 10, head: f, tail: f, content: f, path: "/x/a/f"}, + {size: 20, head: g, tail: g, content: g, path: "/x/g"}, + {size: 10, head: f, tail: f, content: f, path: "/y/a/f"}, + {size: 30, head: k, tail: k, content: k, path: "/y/k"}, + }) + b, xa, ya := dir+"/srv/b", dir+"/x/a", dir+"/y/a" + + header := "first\tdupe\tfiles\tsize\tdisk_size\n" + row := func(first, dupe string) string { + return first + "\t" + dupe + "\t1\t10\t0\n" + } + + cases := map[string]struct { + operands []string + want string + }{ + "none": {nil, header + row(b, xa) + row(b, ya)}, + "two copies": {[]string{xa, ya}, header + row(xa, ya)}, + "parents": {[]string{dir + "/x", dir + "/y"}, header + row(xa, ya)}, + "one copy": {[]string{b}, header}, + } + + for name, tc := range cases { + t.Run(name, func(t *testing.T) { + t.Setenv(databaseEnv, seedDatabase(t, recs)) + + var stdout, stderr bytes.Buffer + + code := run(append([]string{cmdTrees}, tc.operands...), + &stdout, &stderr) + if code != exitOK { + t.Fatalf("run(trees %q) = %d, want %d; stderr: %s", + tc.operands, code, exitOK, stderr.String()) + } + + if got := stdout.String(); got != tc.want { + t.Errorf("stdout = %q, want %q", got, tc.want) + } + }) + } +} + // TestRunTreesDiskSize runs trees on two copies of a tree whose files // take different space on disk. The row's disk_size is the sum over the // dupe copy's files, those in its subdirectory included.