Compare commits

...
2 Commits
Author SHA1 Message Date
sneak 97a02da837 Limit report and trees to given paths (closes #174)
check / check (push) Canceled after 0s
report and trees take optional PATH operands and use only the records
at or below them. Operands are made absolute and cleaned as text, never
looked up on the filesystem. With operands, trees reports only
directories at or below an operand: a directory above them holds
unused files, so its digest neither gets reported nor suppresses its
children's group. The summary still counts every record read.

Model: opus-5-5
2026-11-11 04:01:34 +00:00
clawbot 6c529fcc6d Leave empty files out of report and trees (closes #178) (#183)
check / check (push) Canceled after 0s
2026-11-11 03:52:45 +01:00
10 changed files with 409 additions and 72 deletions

No files matched your search

+57 -22
View File
@@ -43,7 +43,9 @@ as opposed to a directory, a symlink or a device node. A symlink is a file that
holds the path of another file and stands in for it. A device node is a file holds the path of another file and stands in for it. A device node is a file
that stands for a piece of hardware, such as a disk or a terminal. `report` that stands for a piece of hardware, such as a disk or a terminal. `report`
prints the file-level duplicates. `trees` prints the duplicate directory trees. prints the file-level duplicates. `trees` prints the duplicate directory trees.
`info` prints the database's path, size and record count. Both cover the whole database, or only the paths given after them, as in
`sfdupes report /srv/photos`. `info` prints the database's path, size and record
count.
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
another file. The first `scan` creates it. another file. The first `scan` creates it.
@@ -253,8 +255,8 @@ In order:
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv sfdupes report [PATH...] > dupes.tsv
sfdupes trees > dupetrees.tsv sfdupes trees [PATH...] > dupetrees.tsv
sfdupes info sfdupes info
sfdupes dwim sfdupes dwim
sfdupes --version sfdupes --version
@@ -653,18 +655,19 @@ and `scan` leaves it ignored.
### `report` mode ### `report` mode
`report` reads every record from the database. It takes no positional arguments. `report` reads every record from the database. It takes optional `PATH`
A positional argument is anything on the command line that is not a flag, such operands. With none, it uses every record. With one or more, it uses only the
as the paths `scan` takes. `report` never touches the scanned filesystem. It records at or below them (see "Report paths"). `report` never touches the
neither stats nor opens any path in the records. It reads the database and scanned filesystem. It neither stats nor opens any path in the records. It reads
writes to stdout and stderr, and nothing else. Its output is the same whether or the database and writes to stdout and stderr, and nothing else. Its output is
not the scanned filesystem is still mounted. the same whether or not the scanned filesystem is still mounted.
Processing: Processing:
- Records are grouped by signature, the four values - Records are grouped by signature, the four values
`(size, head, tail, content)`. `(size, head, tail, content)`.
- Every group with two or more paths is a duplicate group. - Every group with two or more paths is a duplicate group.
- Empty files form no group, because removing one frees nothing.
- Within a group, paths are sorted in byte order. The first is `first`, and - Within a group, paths are sorted in byte order. The first is `first`, and
every other path is a `dupe`. every other path is a `dupe`.
- Groups are ordered by size, largest first, then by `first` path. The output is - Groups are ordered by size, largest first, then by `first` path. The output is
@@ -680,6 +683,26 @@ with one signature come together, and so do the records of one size. So `report`
holds only the duplicate groups of the size it is reading. Once it reaches a holds only the duplicate groups of the size it is reading. Once it reaches a
smaller size, it writes them, ordered by `first` path. smaller size, it writes them, ordered by `first` path.
#### Report paths
`report` and `trees` take the same `PATH` operands:
- A record is used when its path is an operand or lies below one. `/srv/photos`
covers `/srv/photos/x` but not `/srv/photos2/x`.
- The records used are grouped, ordered and totalled as they are without
operands. A copy outside the operands is not reported. With copies `/a/f`,
`/srv/photos/x` and `/srv/photos/y`, `sfdupes report /srv/photos` prints one
row, `/srv/photos/x` and `/srv/photos/y`.
- A relative operand is joined to the working directory as `pwd -P` prints it.
- Each operand is then cleaned as text, as `realpath -ms` prints it. Its `.`
elements and repeated or trailing slashes are dropped. Each `..` drops the
element before it.
- No operand is looked up on the filesystem. A symlink in an operand is not
followed, and an operand need not exist. The records hold real paths (see
"Operands"), so an operand that passes through a symlink matches no record.
- An operand that matches no record is not an error. The report then has no
rows.
#### Report output format #### Report output format
The report is TSV (tab-separated values) on stdout. It holds a header line, then The report is TSV (tab-separated values) on stdout. It holds a header line, then
@@ -707,22 +730,24 @@ Grouping and ordering use the stored path. The warnings about skipped paths that
The summary line on stderr gives: The summary line on stderr gives:
- the records read; - the records read, which are all the records in the database, with or without
operands;
- the duplicate groups; - the duplicate groups;
- the dupe files, which is the number of rows; - the dupe files, which is the number of rows;
- the reclaimable bytes, the sum of `size` over all rows. - the reclaimable bytes, the sum of `size` over all rows.
``` ```
report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
``` ```
### `trees` mode ### `trees` mode
`trees` reads the same database as `report` and takes no positional arguments. `trees` reads the same database as `report` and takes the same `PATH` operands
It reports whole duplicate directory trees. Two directories are duplicate trees (see "Report paths"). It reports whole duplicate directory trees. Two
when they hold the same relative paths, and each path holds a file with the same directories are duplicate trees when they hold the same relative paths, and each
signature in both. Like `report`, it never touches the scanned filesystem. It path holds a file with the same signature in both. Like `report`, it never
rebuilds the directory tree from the paths in the records, split on `/`. touches the scanned filesystem. It rebuilds the directory tree from the paths in
the records, split on `/`.
Definitions: Definitions:
@@ -741,11 +766,18 @@ Processing:
- `trees` builds the directory tree and computes every directory's digest. It - `trees` builds the directory tree and computes every directory's digest. It
then groups the directories by digest. Every group with two or more then groups the directories by digest. Every group with two or more
directories is a duplicate-tree group. directories is a duplicate-tree group.
- `trees` leaves out a group whose trees hold 0 bytes, because removing one
frees nothing. An empty file still counts toward the digest of a tree that
holds other files.
- `trees` leaves out a group that a group of its members' parents already - `trees` leaves out a group that a group of its members' parents already
implies. A group is left out when its members' parents are distinct implies. A group is left out when its members' parents are distinct
directories that all have the same digest. The parents' group, or an directories that all have the same digest. The parents' group, or an
ancestor's, already implies it. A group is always reported when it holds ancestor's, already implies it. A group is always reported when it holds
sibling directories, or when its members' parents have different digests. sibling directories, or when its members' parents have different digests.
- With operands, `trees` builds the tree from the records it uses. It reports
only directories at or below an operand. A directory above every operand holds
files that are not used, so its digest does not describe it. Such a directory
is never reported and never implies a group of its children.
- Within a group, paths are sorted in byte order. The first is `first`, and - Within a group, paths are sorted in byte order. The first is `first`, and
every other path is a `dupe`. every other path is a `dupe`.
- Groups are ordered by total tree size, largest first, then by `first` path. - Groups are ordered by total tree size, largest first, then by `first` path.
@@ -795,7 +827,8 @@ directory's path is `/`.
The summary line on stderr gives: The summary line on stderr gives:
- the records read; - the records read, which are all the records in the database, with or without
operands;
- the duplicate-tree groups; - the duplicate-tree groups;
- the dupe trees, which is the number of rows; - the dupe trees, which is the number of rows;
- the reclaimable bytes, the sum of `size` over all rows. - the reclaimable bytes, the sum of `size` over all rows.
@@ -808,7 +841,9 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
`dwim`, short for "do what I mean", scans a new machine and reports its `dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments and no flags of duplicate trees in one command. It takes no positional arguments and no flags of
its own. Like `scan`, it should run as root, so that it can read every file. its own. A positional argument is anything on the command line that is not a
flag, such as the paths `scan` takes. Like `scan`, it should run as root, so
that it can read every file.
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default `dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
excludes and crosses into other filesystems. `--workers` keeps its default. The excludes and crosses into other filesystems. `--workers` keeps its default. The
@@ -1194,7 +1229,7 @@ showing the count it stopped at.
`--workers abc`; `--workers abc`;
- `scan` with no operand; - `scan` with no operand;
- `--workers` below 1; - `--workers` below 1;
- `report`, `trees`, `info` or `dwim` with a positional argument. - `info` or `dwim` with a positional argument.
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
stdout, such as a full disk, is reported the same way. Two cases never reach stdout, such as a full disk, is reported the same way. Two cases never reach
@@ -1382,7 +1417,7 @@ All of the following, run in this directory, must pass:
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
printf 'y' > "$d/tiny3" # 1-byte non-duplicate printf 'y' > "$d/tiny3" # 1-byte non-duplicate
: > "$d/empty1"; : > "$d/empty2" # empty duplicates : > "$d/empty1"; : > "$d/empty2" # empty files
# duplicate trees: t1 and t2 are identical; t3 differs by one filename # duplicate trees: t1 and t2 are identical; t3 differs by one filename
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub" mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
head -c 3000 /dev/urandom > "$d/t1/f1" head -c 3000 /dev/urandom > "$d/t1/f1"
@@ -1407,8 +1442,8 @@ All of the following, run in this directory, must pass:
is the path that sorts first. is the path that sorts first.
- `t1/f1`, `t2/f1` and `t3/f1` form one group. - `t1/f1`, `t2/f1` and `t3/f1` form one group.
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group. - `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
- `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`. - `tiny1` and `tiny2` form a pair.
- `unique.bin` and `tiny3` appear nowhere. - `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
- The groups come in order of size, largest first. - The groups come in order of size, largest first.
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`. `trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
+9
View File
@@ -29,6 +29,15 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- `report` and `trees` take optional `PATH` operands. With operands, they use
only the records at or below them. `trees` then reports only directories at or
below an operand. Operands are made absolute and cleaned as text, never looked
up on the filesystem (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/174)
- `report` and `trees` leave out empty files. `report` writes no group of size
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
counts toward a tree's digest (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/178)
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the - The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
command, and it calls `Main` in the package `internal/sfdupes`. `main.go` command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
became `internal/sfdupes/cli.go`. Every other file kept its name. The install became `internal/sfdupes/cli.go`. Every other file kept its name. The install
+19 -8
View File
@@ -31,6 +31,13 @@ const (
cmdDwim = "dwim" cmdDwim = "dwim"
) )
// reportPathsHelp explains the PATH operands of report and trees in
// their help.
const reportPathsHelp = "With PATH operands, only the records at or " +
"below them are used. A relative PATH is joined to the working " +
"directory. No PATH is looked up on the filesystem, so a symlink in " +
"one is not followed."
// errNoSubcommand is returned by the root command when it is invoked // errNoSubcommand is returned by the root command when it is invoked
// without a subcommand. That is a usage error, and the usage text // without a subcommand. That is a usage error, and the usage text
// cobra prints for it is the whole message. // cobra prints for it is the whole message.
@@ -161,20 +168,24 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
"print the version to stdout") "print the version to stdout")
reportCmd := &cobra.Command{ reportCmd := &cobra.Command{
Use: cmdReport, Use: cmdReport + " [PATH...]",
Short: "Read the scan database and print the file-level duplicates report", Short: "Read the scan database and print the file-level duplicates report",
Args: cobra.NoArgs, Long: "Read the scan database and print the file-level duplicates " +
RunE: runE(func(ctx context.Context, _ []string) error { "report.\n\n" + reportPathsHelp,
return runReport(ctx, stdout) Args: cobra.ArbitraryArgs,
RunE: runE(func(ctx context.Context, args []string) error {
return runReport(ctx, args, stdout)
}), }),
} }
treesCmd := &cobra.Command{ treesCmd := &cobra.Command{
Use: cmdTrees, Use: cmdTrees + " [PATH...]",
Short: "Read the scan database and print the duplicate-tree report", Short: "Read the scan database and print the duplicate-tree report",
Args: cobra.NoArgs, Long: "Read the scan database and print the duplicate-tree " +
RunE: runE(func(ctx context.Context, _ []string) error { "report.\n\n" + reportPathsHelp,
return runTrees(ctx, stdout) Args: cobra.ArbitraryArgs,
RunE: runE(func(ctx context.Context, args []string) error {
return runTrees(ctx, args, stdout)
}), }),
} }
+9 -7
View File
@@ -576,14 +576,14 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}, },
cmdReport: { cmdReport: {
run: func() error { run: func() error {
return runReport(cancelledContext(t), io.Discard) return runReport(cancelledContext(t), nil, io.Discard)
}, },
want: "report: database " + defaultDatabasePath + " (default)\n" + want: "report: database " + defaultDatabasePath + " (default)\n" +
reportOpenLine + "\n" + reportOpenLine + "\n", reportOpenLine + "\n" + reportOpenLine + "\n",
}, },
cmdTrees: { cmdTrees: {
run: func() error { run: func() error {
return runTrees(cancelledContext(t), io.Discard) return runTrees(cancelledContext(t), nil, io.Discard)
}, },
want: "trees: database " + defaultDatabasePath + " (default)\n" + want: "trees: database " + defaultDatabasePath + " (default)\n" +
treesOpenLine + "\n" + treesOpenLine + "\n", treesOpenLine + "\n" + treesOpenLine + "\n",
@@ -619,8 +619,6 @@ func TestRunUsageErrors(t *testing.T) {
}{ }{
"no subcommand": {[]string{}, usageMarker}, "no subcommand": {[]string{}, usageMarker},
"scan without paths": {[]string{cmdScan}, usageMarker}, "scan without paths": {[]string{cmdScan}, usageMarker},
"report with args": {[]string{cmdReport, "x"}, usageMarker},
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
"info with args": {[]string{cmdInfo, "x"}, usageMarker}, "info with args": {[]string{cmdInfo, "x"}, usageMarker},
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker}, "dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker}, "unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
@@ -1726,9 +1724,13 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
scanFixture(t) scanFixture(t)
cases := map[string]func(context.Context, io.Writer) error{ cases := map[string]func(context.Context, io.Writer) error{
cmdReport: runReport, cmdReport: func(ctx context.Context, w io.Writer) error {
cmdTrees: runTrees, return runReport(ctx, nil, w)
cmdInfo: runInfo, },
cmdTrees: func(ctx context.Context, w io.Writer) error {
return runTrees(ctx, nil, w)
},
cmdInfo: runInfo,
} }
for name, fn := range cases { for name, fn := range cases {
+1 -1
View File
@@ -25,7 +25,7 @@ func runDwim(ctx context.Context, root string, stdout io.Writer) error {
return err return err
} }
err = runTrees(ctx, stdout) err = runTrees(ctx, nil, stdout)
if err != nil { if err != nil {
return err return err
} }
+67 -14
View File
@@ -7,6 +7,7 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"path/filepath"
"slices" "slices"
"strings" "strings"
"time" "time"
@@ -41,10 +42,11 @@ func reportPhase(name string) string {
} }
// runReport implements the report subcommand: it prints the file-level // runReport implements the report subcommand: it prints the file-level
// duplicates report as TSV on stdout. It never touches the scanned // duplicates report as TSV on stdout, from the records at or below the
// filesystem; its only I/O is the database, stdout, and stderr. It // given paths, or from every record when none is given. It never
// names the database on stderr before it opens it. Any database // touches the scanned filesystem; its only I/O is the database,
// problem, including a missing database, is fatal. // stdout, and stderr. It names the database on stderr before it opens
// it. Any database problem, including a missing database, is fatal.
// //
// The work runs in two phases, each under a progress display of its // The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and read (read the number of // own on stderr: open (open the database) and read (read the number of
@@ -52,7 +54,12 @@ func reportPhase(name string) string {
// order, writing each size's duplicate groups to stdout once that size // order, writing each size's duplicate groups to stdout once that size
// is read; see readReport). Each display ends before the next phase // is read; see readReport). Each display ends before the next phase
// starts, and before an error is returned to be printed. // starts, and before an error is returned to be printed.
func runReport(ctx context.Context, stdout io.Writer) error { func runReport(ctx context.Context, args []string, stdout io.Writer) error {
paths, err := reportPaths(args)
if err != nil {
return err
}
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
return err return err
@@ -83,7 +90,7 @@ func runReport(ctx context.Context, stdout io.Writer) error {
label: reportPhase("read"), unit: "records", total: -1, label: reportPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords, stalledOn: "db/out", waitingFor: countingRecords, stalledOn: "db/out",
}) })
report, err := readReport(ctx, db, dbPath, stdout, prog) report, err := readReport(ctx, db, dbPath, paths, stdout, prog)
prog.finish() prog.finish()
@@ -104,11 +111,12 @@ func runReport(ctx context.Context, stdout io.Writer) error {
// dbPath, as its total, then reads every record in signature order, // dbPath, as its total, then reads every record in signature order,
// counting each on prog, and writes the report to stdout: the header // counting each on prog, and writes the report to stdout: the header
// line, then the rows of each size's duplicate groups once that size is // line, then the rows of each size's duplicate groups once that size is
// read. stdout may be the terminal prog is drawn on, so the lines go // read. Only the records that inReportPaths selects from paths join a
// group. stdout may be the terminal prog is drawn on, so the lines go
// through prog's writer. It returns what it wrote, with the totals for // through prog's writer. It returns what it wrote, with the totals for
// the summary line. // the summary line.
func readReport(ctx context.Context, db *sql.DB, dbPath string, func readReport(ctx context.Context, db *sql.DB, dbPath string,
stdout io.Writer, prog *progress, paths []string, stdout io.Writer, prog *progress,
) (*reportWriter, error) { ) (*reportWriter, error) {
out := prog.writer(stdout) out := prog.writer(stdout)
report := &reportWriter{out: out} report := &reportWriter{out: out}
@@ -123,6 +131,12 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error { err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
prog.increment() prog.increment()
report.records++
if !inReportPaths(r.path, paths) {
return nil
}
writeErr = report.add(r) writeErr = report.add(r)
return writeErr return writeErr
@@ -154,7 +168,7 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
// records of one size, so it holds only the duplicate groups of the // records of one size, so it holds only the duplicate groups of the
// size being read, and writes them, ordered by first path, once a // size being read, and writes them, ordered by first path, once a
// record of a smaller size comes. It counts what the summary line // record of a smaller size comes. It counts what the summary line
// reports. // reports; readReport counts the records read.
type reportWriter struct { type reportWriter struct {
out io.Writer out io.Writer
@@ -170,8 +184,6 @@ type reportWriter struct {
// its paths; any other ends them, and one of a smaller size first // its paths; any other ends them, and one of a smaller size first
// writes the groups of the size before. // writes the groups of the size before.
func (w *reportWriter) add(r scanRec) error { func (w *reportWriter) add(r scanRec) error {
w.records++
if r.size == w.sig.size && r.head == w.sig.head && if r.size == w.sig.size && r.head == w.sig.head &&
r.tail == w.sig.tail && r.content == w.sig.content { r.tail == w.sig.tail && r.content == w.sig.content {
w.paths = append(w.paths, r.path) w.paths = append(w.paths, r.path)
@@ -195,10 +207,11 @@ func (w *reportWriter) add(r scanRec) error {
} }
// endGroup ends the paths read with w.sig: two or more are a duplicate // endGroup ends the paths read with w.sig: two or more are a duplicate
// group, held with its paths sorted until writeGroups writes it. // group, held with its paths sorted until writeGroups writes it. Empty
// files form no group, since removing one frees nothing.
func (w *reportWriter) endGroup() { func (w *reportWriter) endGroup() {
if len(w.paths) < minGroupSize { if len(w.paths) < minGroupSize || w.sig.size == 0 {
// A path with no duplicate is dropped, and its slice reused. // The paths are dropped, and their slice reused.
w.paths = w.paths[:0] w.paths = w.paths[:0]
return return
@@ -245,6 +258,46 @@ func (w *reportWriter) finish() error {
return w.writeGroups() return w.writeGroups()
} }
// reportPaths returns the paths given to report or trees, each made
// absolute by absolutePath and then cleaned. The records hold real
// paths, with no "." or ".." element and no repeated or trailing
// slash, so a given path is cleaned to match them. No path is looked up
// on the filesystem.
func reportPaths(args []string) ([]string, error) {
paths := make([]string, 0, len(args))
for _, a := range args {
abs, err := absolutePath(a)
if err != nil {
return nil, fmt.Errorf("resolve path %s: %w", escapePath(a), err)
}
paths = append(paths, filepath.Clean(abs))
}
return paths, nil
}
// inReportPaths reports whether path is one of paths or lies below one
// of them. Each of paths is clean and absolute, as reportPaths returns
// it. With no paths, a report covers the whole database, so every path
// is in it.
func inReportPaths(path string, paths []string) bool {
if len(paths) == 0 {
return true
}
for _, p := range paths {
// Only the root directory's clean path ends in "/".
dir := strings.TrimSuffix(p, "/") + "/"
if path == p || strings.HasPrefix(path, dir) {
return true
}
}
return false
}
// escapePath returns a path as it is written in a report column (README // escapePath returns a path as it is written in a report column (README
// "Report output format"): a backslash, tab, newline or carriage return // "Report output format"): a backslash, tab, newline or carriage return
// becomes \\, \t, \n or \r, and every other byte is kept as it is. // becomes \\, \t, \n or \r, and every other byte is kept as it is.
+116 -3
View File
@@ -80,7 +80,7 @@ func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
var out bytes.Buffer var out bytes.Buffer
_, err := readReport(t.Context(), db, "", &out, nil) _, err := readReport(t.Context(), db, "", nil, &out, nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -126,6 +126,98 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
return dupeGroups(t, db) return dupeGroups(t, db)
} }
// TestRunReportPaths runs report with paths on three copies of one
// file, two of them under /srv/photos, and a fourth copy in
// /srv/photos2. Only the records at or below the given paths form the
// groups and the totals, however each path is written.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestRunReportPaths(t *testing.T) {
h := testHash("h")
a, x, y, z := "/a/copy", "/srv/photos/x", "/srv/photos/y", "/srv/photos2/z"
recs := make([]scanRec, 0, 4)
for _, p := range []string{a, x, y, z} {
recs = append(recs,
scanRec{size: 7, head: h, tail: h, content: h, path: p})
}
t.Setenv(databaseEnv, seedDatabase(t, recs))
header := "first\tdupe\tsize\n"
row := func(first, dupe string) string {
return first + "\t" + dupe + "\t7\n"
}
photos := header + row(x, y)
cases := map[string]struct {
paths []string
want string
summary string
}{
"directory": {[]string{"/srv/photos"}, photos,
"1 duplicate groups, 1 dupe files, 7 B reclaimable"},
"trailing slash": {[]string{"/srv/photos/"}, photos, ""},
"dot and dot-dot": {[]string{"/srv/./a/../photos"}, photos, ""},
"files": {[]string{a, z}, header + row(a, z), ""},
"overlapping": {[]string{"/srv", "/srv/photos"},
header + row(x, y) + row(x, z), ""},
"root": {[]string{"/"},
header + row(a, x) + row(a, y) + row(a, z), ""},
"no record": {[]string{"/srv/photo"}, header,
"0 duplicate groups, 0 dupe files, 0 B reclaimable"},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
var stdout bytes.Buffer
stderr := captureStderr(t)
code := run(append([]string{cmdReport}, tc.paths...),
&stdout, os.Stderr)
if code != exitOK {
t.Fatalf("run(report %q) = %d, want %d; stderr: %s",
tc.paths, code, exitOK, stderr())
}
if got := stdout.String(); got != tc.want {
t.Errorf("stdout = %q, want %q", got, tc.want)
}
// The records read are all four, whatever the paths.
summary := "report: 4 records read, " + tc.summary + "\n"
if got := stderr(); tc.summary != "" &&
!strings.HasSuffix(got, summary) {
t.Errorf("stderr = %q, want it to end %q", got, summary)
}
})
}
}
//nolint:paralleltest // t.Chdir changes the process-wide working directory
func TestReportPathsMakesRelativePathsAbsolute(t *testing.T) {
dir := t.TempDir()
t.Chdir(dir)
wd, err := absolutePath(".")
if err != nil {
t.Fatal(err)
}
got, err := reportPaths([]string{"photos/", "a/../b", "/c//d"})
if err != nil {
t.Fatal(err)
}
want := []string{
filepath.Join(wd, "photos"), filepath.Join(wd, "b"), "/c/d",
}
if !slices.Equal(got, want) {
t.Errorf("reportPaths = %q, want %q", got, want)
}
}
func TestRunReportEscapesPaths(t *testing.T) { func TestRunReportEscapesPaths(t *testing.T) {
t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs())) t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
@@ -172,7 +264,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
// Counts the records read, and draws nothing. // Counts the records read, and draws nothing.
prog := &progress{now: time.Now} prog := &progress{now: time.Now}
_, err = readReport(t.Context(), db, "", failingWriter{}, prog) _, err = readReport(t.Context(), db, "", nil, failingWriter{}, prog)
if !errors.Is(err, errWriteFailed) || if !errors.Is(err, errWriteFailed) ||
!strings.HasPrefix(err.Error(), "write stdout: ") { !strings.HasPrefix(err.Error(), "write stdout: ") {
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed) t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
@@ -266,7 +358,7 @@ func TestReadReportOnTerminal(t *testing.T) {
waitingFor: countingRecords, waitingFor: countingRecords,
}, terminalWidth, time.Hour, time.Now) }, terminalWidth, time.Hour, time.Now)
_, err = readReport(t.Context(), db, "", os.Stderr, p) _, err = readReport(t.Context(), db, "", nil, os.Stderr, p)
p.finish() p.finish()
@@ -434,6 +526,27 @@ func TestDupeGroups(t *testing.T) {
} }
} }
// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
// duplicate group, while 1-byte files still do.
func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
t.Parallel()
e, x := sha256.Sum256(nil), testHash("x")
recs := []scanRec{
{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
}
groups := dupeGroupsOf(t, recs)
if len(groups) != 1 || groups[0].size != 1 ||
!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
}
}
func TestDupeGroupsContentSeparates(t *testing.T) { func TestDupeGroupsContentSeparates(t *testing.T) {
t.Parallel() t.Parallel()
+6 -5
View File
@@ -1185,11 +1185,12 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
t.Helper() t.Helper()
groups := dupeGroups(t, db) groups := dupeGroups(t, db)
if len(groups) != 5 { if len(groups) != 4 {
t.Fatalf("len(groups) = %d, want 5", len(groups)) t.Fatalf("len(groups) = %d, want 4", len(groups))
} }
wantSizes := []int64{3000, 2000, 100, 1, 0} // The empty files empty1 and empty2 form no group.
wantSizes := []int64{3000, 2000, 100, 1}
for i, g := range groups { for i, g := range groups {
if g.size != wantSizes[i] { if g.size != wantSizes[i] {
t.Errorf("groups[%d].size = %d, want %d", t.Errorf("groups[%d].size = %d, want %d",
@@ -2367,8 +2368,8 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
report := runStdout(t, cmdReport) report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") || if strings.Contains(report, "db.sqlite") ||
strings.Contains(report, hardLink) || strings.Contains(report, hardLink) ||
!strings.Contains(report, filepath.Join(tree, "empty2")) { !strings.Contains(report, filepath.Join(tree, "b")) {
t.Errorf("report = %q, want the empty files and none of "+ t.Errorf("report = %q, want a and b and none of "+
"the database's files", report) "the database's files", report)
} }
}) })
+46 -12
View File
@@ -37,8 +37,9 @@ func treesPhase(name string) string {
// runTrees implements the trees subcommand: it reads every record from // runTrees implements the trees subcommand: it reads every record from
// the database in path order, reconstructs the directory hierarchy from // the database in path order, reconstructs the directory hierarchy from
// the record paths, computes a Merkle-style digest per directory, and // the paths of the records at or below the given paths (of every record
// prints maximal duplicate-tree groups as TSV on stdout. It never // when none is given), computes a Merkle-style digest per directory,
// and prints maximal duplicate-tree groups as TSV on stdout. It never
// touches the scanned filesystem; its only I/O is the database, stdout, // touches the scanned filesystem; its only I/O is the database, stdout,
// and stderr. It names the database on stderr before it opens it. Any // and stderr. It names the database on stderr before it opens it. Any
// database problem, including a missing database, is fatal. // database problem, including a missing database, is fatal.
@@ -49,7 +50,12 @@ func treesPhase(name string) string {
// hierarchy), group (group the directories by digest) and write (write // hierarchy), group (group the directories by digest) and write (write
// the groups to stdout). Each display ends before the next phase // the groups to stdout). Each display ends before the next phase
// starts, and before an error is returned to be printed. // starts, and before an error is returned to be printed.
func runTrees(ctx context.Context, stdout io.Writer) error { func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
paths, err := reportPaths(args)
if err != nil {
return err
}
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
return err return err
@@ -78,7 +84,7 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
label: treesPhase("read"), unit: "records", total: -1, label: treesPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords, stalledOn: "db", waitingFor: countingRecords, stalledOn: "db",
}) })
records, super, allDirs, err := readTree(ctx, db, prog) records, super, allDirs, err := readTree(ctx, db, paths, prog)
prog.finish() prog.finish()
@@ -117,10 +123,17 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
} }
// readTree gives prog the number of records in db as its total, then // readTree gives prog the number of records in db as its total, then
// reads every record in path order into the directory hierarchy, // reads every record in path order, counting each on prog. It adds the
// counting each on prog. It returns the number of records read, the // records that inReportPaths selects from paths to the directory
// super-root and every directory. // hierarchy. It returns the number of records read, the super-root and
func readTree(ctx context.Context, db *sql.DB, // every directory at or below one of paths.
//
// A directory above every one of paths holds only the selected records
// under it, so its digest does not describe it. It is left out of the
// directories returned. A directory whose parent is left out gets the
// super-root as its parent, so suppressed never judges its group by
// that parent's digest.
func readTree(ctx context.Context, db *sql.DB, paths []string,
prog *progress, prog *progress,
) (int, *treeNode, []*treeNode, error) { ) (int, *treeNode, []*treeNode, error) {
records := 0 records := 0
@@ -130,7 +143,10 @@ func readTree(ctx context.Context, db *sql.DB,
records++ records++
prog.increment() prog.increment()
tree.add(r)
if inReportPaths(r.path, paths) {
tree.add(r)
}
}) })
if err != nil { if err != nil {
return 0, nil, nil, err return 0, nil, nil, err
@@ -138,7 +154,21 @@ func readTree(ctx context.Context, db *sql.DB,
super, allDirs := tree.finish() super, allDirs := tree.finish()
return records, super, allDirs, nil dirs := allDirs[:0]
for _, d := range allDirs {
if !inReportPaths(d.path, paths) {
continue
}
if !inReportPaths(d.parent.path, paths) {
d.parent = super
}
dirs = append(dirs, d)
}
return records, super, dirs, nil
} }
// writeTreeGroups writes dupes to stdout as TSV, a header line and then // writeTreeGroups writes dupes to stdout as TSV, a header line and then
@@ -301,7 +331,9 @@ func fileEntry(name string, r scanRec) string {
// collectTreeGroups groups directories by digest, counting each one on // collectTreeGroups groups directories by digest, counting each one on
// prog as it is grouped, and returns every maximal group with two or // prog as it is grouped, and returns every maximal group with two or
// more members, each group's members sorted by path, groups ordered by // more members, each group's members sorted by path, groups ordered by
// tree size descending then by first path ascending. // tree size descending then by first path ascending. A group whose
// trees hold 0 bytes is left out, since removing one frees nothing. The
// groups it implies hold 0 bytes too, so leaving it out hides nothing.
func collectTreeGroups(allDirs []*treeNode, super *treeNode, func collectTreeGroups(allDirs []*treeNode, super *treeNode,
prog *progress, prog *progress,
) [][]*treeNode { ) [][]*treeNode {
@@ -315,7 +347,9 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
var dupes [][]*treeNode var dupes [][]*treeNode
for _, g := range groups { for _, g := range groups {
if len(g) < minGroupSize || suppressed(g, super) { // The members share a digest, so they share a total size.
if len(g) < minGroupSize || g[0].totalSize == 0 ||
suppressed(g, super) {
continue continue
} }
+79
View File
@@ -2,6 +2,7 @@ package sfdupes //nolint:testpackage // checks the unexported treeNode results
import ( import (
"bytes" "bytes"
"crypto/sha256"
"database/sql" "database/sql"
"os" "os"
"slices" "slices"
@@ -211,6 +212,55 @@ func TestRunTreesEscapesPaths(t *testing.T) {
} }
} }
// TestRunTreesPaths runs trees with paths. /x/a and /y/a are copies,
// and so are /y/a and /srv/b. /x and /y also hold files that differ.
// Given /x/a and /y/a, trees reads /x and /y as holding one copy each
// of a, yet neither is reported: only the directories at or below a
// given path are.
func TestRunTreesPaths(t *testing.T) {
f, g, k := testHash("f"), testHash("g"), testHash("k")
recs := []scanRec{
{size: 10, head: f, tail: f, content: f, path: "/srv/b/f"},
{size: 10, head: f, tail: f, content: f, path: "/x/a/f"},
{size: 20, head: g, tail: g, content: g, path: "/x/g"},
{size: 10, head: f, tail: f, content: f, path: "/y/a/f"},
{size: 30, head: k, tail: k, content: k, path: "/y/k"},
}
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
cases := map[string]struct {
paths []string
want string
}{
"none": {nil, header + "/srv/b\t/x/a\t1\t10\t0\n" +
"/srv/b\t/y/a\t1\t10\t0\n"},
"two copies": {[]string{"/x/a", "/y/a"},
header + "/x/a\t/y/a\t1\t10\t0\n"},
"parents": {[]string{"/x", "/y"}, header + "/x/a\t/y/a\t1\t10\t0\n"},
"one copy": {[]string{"/srv/b"}, header},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
t.Setenv(databaseEnv, seedDatabase(t, recs))
var stdout, stderr bytes.Buffer
code := run(append([]string{cmdTrees}, tc.paths...),
&stdout, &stderr)
if code != exitOK {
t.Fatalf("run(trees %q) = %d, want %d; stderr: %s",
tc.paths, code, exitOK, stderr.String())
}
if got := stdout.String(); got != tc.want {
t.Errorf("stdout = %q, want %q", got, tc.want)
}
})
}
}
// TestRunTreesDiskSize runs trees on two copies of a tree whose files // TestRunTreesDiskSize runs trees on two copies of a tree whose files
// take different space on disk. The row's disk_size is the sum over the // take different space on disk. The row's disk_size is the sum over the
// dupe copy's files, those in its subdirectory included. // dupe copy's files, those in its subdirectory included.
@@ -435,6 +485,35 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
} }
} }
// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
// holding only empty files form no group, and that an empty file still
// counts toward the digest of a tree that holds other files.
func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
t.Parallel()
h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
sha256.Sum256(nil)
recs := []scanRec{
{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
// Without the empty file, /p/x3 is no copy of /p/x1.
{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
}
super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/x1", "/p/x2"}}
if !slices.EqualFunc(got, want, slices.Equal) {
t.Fatalf("groups = %v, want %v", got, want)
}
}
func TestCollectTreeGroupsDifferingParents(t *testing.T) { func TestCollectTreeGroupsDifferingParents(t *testing.T) {
t.Parallel() t.Parallel()