Compare commits

..
Author SHA1 Message Date
sneak 729518c480 Limit report and trees to given paths (closes #174)
check / check (push) Waiting to run
report and trees take optional PATH operands and use only the records
at or below them. Operands are made absolute and cleaned as text, never
looked up on the filesystem; each reported copy is still checked on
disk. An empty operand is a usage error, as --exclude "" is, since it
would name the working directory. With operands, trees reports only
directories at or below an operand: any other directory lies above an
operand and can hold unused files, so its digest neither gets reported
nor suppresses its children's group. The summary still counts every
record read.

Model: opus-5-5
2026-11-11 08:23:01 +00:00
8 changed files with 416 additions and 84 deletions

No files matched your search

+57 -27
View File
@@ -43,7 +43,9 @@ as opposed to a directory, a symlink or a device node. A symlink is a file that
holds the path of another file and stands in for it. A device node is a file holds the path of another file and stands in for it. A device node is a file
that stands for a piece of hardware, such as a disk or a terminal. `report` that stands for a piece of hardware, such as a disk or a terminal. `report`
prints the file-level duplicates. `trees` prints the duplicate directory trees. prints the file-level duplicates. `trees` prints the duplicate directory trees.
`info` prints the database's path, size and record count. Both cover the whole database, or only the paths given after them, as in
`sfdupes report /srv/photos`. `info` prints the database's path, size and record
count.
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
another file. The first `scan` creates it. another file. The first `scan` creates it.
@@ -114,20 +116,15 @@ docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
Run `scan` as root, so that it can read every file. `scan` warns about a Run `scan` as root, so that it can read every file. `scan` warns about a
directory it cannot read, and about a new or changed file it cannot read. It directory it cannot read, and about a new or changed file it cannot read. It
skips the path and deletes the records at or below it (see "Rules for the skips the path and deletes the records at or below it (see "Rules for the
walk"). walk"). As the file `/etc/cron.d/sfdupes`:
An unmounted disk leaves its mount point empty, so `scan` deletes every record
under it, and the next scan reads every file on the disk again. Test the mount
point of each disk at or below `/srv` with its own `mountpoint -q`, joined with
`&&`, and test `/srv` only when it is one. As the file `/etc/cron.d/sfdupes`:
``` ```
30 3 * * * root if mountpoint -q /srv; then /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log; else echo "a disk is not mounted; scan of /srv skipped"; fi 30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
``` ```
- The first scan creates `/var/lib/sfdupes/db.sqlite` and its directory. To keep - The first scan creates `/var/lib/sfdupes/db.sqlite` and its directory. To keep
the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` directly the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` before the
before `/usr/local/bin/sfdupes`. command on the same line.
- `scan` writes nothing to stdout. Its stderr, appended here to - `scan` writes nothing to stdout. Its stderr, appended here to
`/var/log/sfdupes.log`, holds: `/var/log/sfdupes.log`, holds:
- one line naming the database; - one line naming the database;
@@ -263,8 +260,8 @@ In order:
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
sfdupes report > dupes.tsv sfdupes report [PATH...] > dupes.tsv
sfdupes trees > dupetrees.tsv sfdupes trees [PATH...] > dupetrees.tsv
sfdupes info sfdupes info
sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]...
sfdupes --version sfdupes --version
@@ -691,11 +688,12 @@ and `scan` leaves it ignored.
### `report` mode ### `report` mode
`report` reads every record from the database. It takes no positional arguments. `report` reads every record from the database. It takes optional `PATH`
A positional argument is anything on the command line that is not a flag, such operands. With none, it uses every record. With one or more, it uses only the
as the paths `scan` takes. `report` never opens a scanned file. It checks that records at or below them (see "Operands of `report` and `trees`"). `report`
each path in a group still exists, and looks at no other path. The check is never opens a scanned file. It checks that each path in a group still exists,
`lstat`, the system call that looks up a path without following a symlink: and looks at no other path. It never looks up an operand. The check is `lstat`,
the system call that looks up a path without following a symlink:
- A path is gone when `lstat` finds that it, or a directory above it, does not - A path is gone when `lstat` finds that it, or a directory above it, does not
exist. exist.
@@ -727,6 +725,27 @@ with one signature come together, and so do the records of one size. So `report`
holds only the duplicate groups of the size it is reading. Once it reaches a holds only the duplicate groups of the size it is reading. Once it reaches a
smaller size, it writes them, ordered by `first` path. smaller size, it writes them, ordered by `first` path.
#### Operands of `report` and `trees`
`report` and `trees` take the same `PATH` operands:
- A record is used when its path is an operand or lies below one. `/srv/photos`
covers `/srv/photos/x` but not `/srv/photos2/x`.
- The records used are grouped, ordered and totalled as they are without
operands. A copy outside the operands is not reported. With copies `/a/f`,
`/srv/photos/x` and `/srv/photos/y`, `sfdupes report /srv/photos` prints one
row, `/srv/photos/x` and `/srv/photos/y`.
- A relative operand is joined to the working directory as `pwd -P` prints it.
- Each operand is then cleaned as text, as `realpath -ms` prints it. Its `.`
elements and repeated or trailing slashes are dropped. Each `..` drops the
element before it.
- No operand is looked up on the filesystem. A symlink in an operand is not
followed, and an operand need not exist. The records hold real paths (see
"Operands" under "`scan` mode"), so an operand that passes through a symlink
matches no record.
- An operand that matches no record is not an error; it adds nothing to the
report.
#### Report output format #### Report output format
The report is TSV (tab-separated values) on stdout. It holds a header line, then The report is TSV (tab-separated values) on stdout. It holds a header line, then
@@ -754,7 +773,8 @@ Grouping and ordering use the stored path. The warnings about skipped paths that
The summary line on stderr gives: The summary line on stderr gives:
- the records read; - the records read, which are all the records in the database, with or without
operands;
- the duplicate groups; - the duplicate groups;
- the dupe files, which is the number of rows; - the dupe files, which is the number of rows;
- the reclaimable bytes, the sum of `size` over all rows. - the reclaimable bytes, the sum of `size` over all rows.
@@ -765,11 +785,12 @@ report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
### `trees` mode ### `trees` mode
`trees` reads the same database as `report` and takes no positional arguments. `trees` reads the same database as `report` and takes the same `PATH` operands
It reports whole duplicate directory trees. Two directories are duplicate trees (see "Operands of `report` and `trees`"). It reports whole duplicate directory
when they hold the same relative paths, and each path holds a file with the same trees. Two directories are duplicate trees when they hold the same relative
signature in both. It rebuilds the directory tree from the paths in the records, paths, and each path holds a file with the same signature in both. It rebuilds
split on `/`. Like `report`, it never opens a scanned file. It checks only that the directory tree from the paths in the records, split on `/`. Like `report`,
it never opens a scanned file and never looks up an operand. It checks only that
the directory at the top of each copy still exists, as `report` checks a path. the directory at the top of each copy still exists, as `report` checks a path.
Definitions: Definitions:
@@ -797,6 +818,11 @@ Processing:
directories that all have the same digest. The parents' group, or an directories that all have the same digest. The parents' group, or an
ancestor's, already implies it. A group is always reported when it holds ancestor's, already implies it. A group is always reported when it holds
sibling directories, or when its members' parents have different digests. sibling directories, or when its members' parents have different digests.
- With operands, `trees` builds the tree from the records it uses. It reports
only directories at or below an operand. Any other directory lies above an
operand and can hold files that are not used, so its digest does not describe
it. Such a directory is never reported and never implies a group of its
children.
- A directory that is gone is dropped from its group. A group left with fewer - A directory that is gone is dropped from its group. A group left with fewer
than two directories is not reported. than two directories is not reported.
- Within a group, paths are sorted in byte order. The first is `first`, and - Within a group, paths are sorted in byte order. The first is `first`, and
@@ -850,7 +876,8 @@ directory's path is `/`.
The summary line on stderr gives: The summary line on stderr gives:
- the records read; - the records read, which are all the records in the database, with or without
operands;
- the duplicate-tree groups; - the duplicate-tree groups;
- the dupe trees, which is the number of rows; - the dupe trees, which is the number of rows;
- the reclaimable bytes, the sum of `size` over all rows. - the reclaimable bytes, the sum of `size` over all rows.
@@ -862,8 +889,10 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
### `dwim` mode ### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its `dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments. Like `scan`, duplicate trees in one command. It takes no positional arguments. A positional
it should run as root, so that it can read every file. argument is anything on the command line that is not a flag, such as the paths
`scan` takes. Like `scan`, it should run as root, so that it can read every
file.
`dwim` takes these flags of `scan`, with the same meaning (see "Settings"): `dwim` takes these flags of `scan`, with the same meaning (see "Settings"):
@@ -1265,7 +1294,8 @@ showing the count it stopped at.
- `scan` with no operand; - `scan` with no operand;
- `--workers` below 1; - `--workers` below 1;
- an empty `--exclude` value, as in `--exclude ""`; - an empty `--exclude` value, as in `--exclude ""`;
- `report`, `trees`, `info` or `dwim` with a positional argument. - an empty `PATH` for `report` or `trees`, as in `sfdupes report ""`;
- `info` or `dwim` with a positional argument.
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
stdout, such as a full disk, is reported the same way. Two cases never reach stdout, such as a full disk, is reported the same way. Two cases never reach
+34 -10
View File
@@ -49,6 +49,13 @@ const escapesHelp = `Four characters in a path are escaped, so each row stays on
\r a carriage return \r a carriage return
` `
// operandsHelp is the paragraph of the report and trees help on their
// PATH operands (README "Operands of `report` and `trees`").
const operandsHelp = `With PATH operands, only the records at or below them are used. A
relative PATH is joined to the working directory. No PATH is looked up
on the filesystem, so a symlink in one is not followed.
`
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line // reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
// summary in the help of their subcommands. // summary in the help of their subcommands.
const ( const (
@@ -61,7 +68,7 @@ row per duplicate file. The columns are:
Rows are ordered by size, largest first, then by path. Rows are ordered by size, largest first, then by path.
` + escapesHelp ` + escapesHelp + "\n" + operandsHelp
treesHelp = `The report is tab-separated text on stdout: a header line, then one treesHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate directory tree. The columns are: row per duplicate directory tree. The columns are:
@@ -74,7 +81,7 @@ row per duplicate directory tree. The columns are:
Rows are ordered by size, largest first, then by path. Rows are ordered by size, largest first, then by path.
` + escapesHelp ` + escapesHelp + "\n" + operandsHelp
infoHelp = `The details go to stdout, one per line. Each line holds a name and its infoHelp = `The details go to stdout, one per line. Each line holds a name and its
value, separated by a tab: value, separated by a tab:
@@ -108,6 +115,11 @@ var errWorkersBelowOne = errors.New("--workers must be at least 1")
// would name the working directory. // would name the working directory.
var errEmptyExcludeValue = errors.New("--exclude must not be empty") var errEmptyExcludeValue = errors.New("--exclude must not be empty")
// errEmptyOperand is the usage error for an empty PATH operand of
// report or trees, as from "$DIR" with DIR unset. Taken as a path, it
// would name the working directory.
var errEmptyOperand = errors.New("PATH must not be empty")
// Version is the build version, injected at link time via -ldflags // Version is the build version, injected at link time via -ldflags
// (see the Makefile); empty when the linker set none, as for a binary // (see the Makefile); empty when the linker set none, as for a binary
// built by go install. buildVersion turns it into what --version prints. // built by go install. buildVersion turns it into what --version prints.
@@ -229,20 +241,20 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
"print the version to stdout") "print the version to stdout")
reportCmd := &cobra.Command{ reportCmd := &cobra.Command{
Use: cmdReport, Use: cmdReport + " [PATH...]",
Short: "Read the scan database and print the file-level duplicates report", Short: "Read the scan database and print the file-level duplicates report",
Args: cobra.NoArgs, Args: rejectEmptyOperand,
RunE: runE(func(ctx context.Context, _ []string) error { RunE: runE(func(ctx context.Context, args []string) error {
return runReport(ctx, stdout) return runReport(ctx, args, stdout)
}), }),
} }
treesCmd := &cobra.Command{ treesCmd := &cobra.Command{
Use: cmdTrees, Use: cmdTrees + " [PATH...]",
Short: "Read the scan database and print the duplicate-tree report", Short: "Read the scan database and print the duplicate-tree report",
Args: cobra.NoArgs, Args: rejectEmptyOperand,
RunE: runE(func(ctx context.Context, _ []string) error { RunE: runE(func(ctx context.Context, args []string) error {
return runTrees(ctx, stdout) return runTrees(ctx, args, stdout)
}), }),
} }
@@ -387,6 +399,18 @@ func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
return errEmptyExcludeValue return errEmptyExcludeValue
} }
// rejectEmptyOperand rejects an empty PATH operand of report or trees.
// That is a usage error reported in one line, as for --workers below 1.
func rejectEmptyOperand(cmd *cobra.Command, args []string) error {
if !slices.Contains(args, "") {
return nil
}
cmd.SilenceUsage = true
return errEmptyOperand
}
// runE adapts a subcommand implementation, or the version print, to // runE adapts a subcommand implementation, or the version print, to
// cobra's RunE. Cobra prints the error and the command's usage text for // cobra's RunE. Cobra prints the error and the command's usage text for
// every error RunE returns, but a subcommand that ran and failed has no // every error RunE returns, but a subcommand that ran and failed has no
+19 -12
View File
@@ -583,14 +583,14 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}, },
cmdReport: { cmdReport: {
run: func() error { run: func() error {
return runReport(cancelledContext(t), io.Discard) return runReport(cancelledContext(t), nil, io.Discard)
}, },
want: "report: database " + defaultDatabasePath + " (default)\n" + want: "report: database " + defaultDatabasePath + " (default)\n" +
reportOpenLine + "\n" + reportOpenLine + "\n", reportOpenLine + "\n" + reportOpenLine + "\n",
}, },
cmdTrees: { cmdTrees: {
run: func() error { run: func() error {
return runTrees(cancelledContext(t), io.Discard) return runTrees(cancelledContext(t), nil, io.Discard)
}, },
want: "trees: database " + defaultDatabasePath + " (default)\n" + want: "trees: database " + defaultDatabasePath + " (default)\n" +
treesOpenLine + "\n" + treesOpenLine + "\n", treesOpenLine + "\n" + treesOpenLine + "\n",
@@ -626,8 +626,6 @@ func TestRunUsageErrors(t *testing.T) {
}{ }{
"no subcommand": {[]string{}, usageMarker}, "no subcommand": {[]string{}, usageMarker},
"scan without paths": {[]string{cmdScan}, usageMarker}, "scan without paths": {[]string{cmdScan}, usageMarker},
"report with args": {[]string{cmdReport, "x"}, usageMarker},
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
"info with args": {[]string{cmdInfo, "x"}, usageMarker}, "info with args": {[]string{cmdInfo, "x"}, usageMarker},
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker}, "dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker}, "unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
@@ -764,11 +762,16 @@ func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
t.Parallel() t.Parallel()
// README §Settings: the help names the database, and says what // README §Settings: the help names the database, and says what
// report, trees, info and dwim print. // report, trees, info and dwim print. The help of report and trees
// also explains their PATH operands.
database := []string{ database := []string{
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root", "scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
} }
escapes := []string{`\\`, `\t`, `\n`, `\r`} escapes := []string{`\\`, `\t`, `\n`, `\r`}
operands := []string{
"With PATH operands", "at or below them", "working directory",
"No PATH is looked up", "not followed",
}
// The subcommand whose help is printed; none for the root. // The subcommand whose help is printed; none for the root.
cases := []struct { cases := []struct {
@@ -779,17 +782,17 @@ func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
{[]string{cmdScan}, database}, {[]string{cmdScan}, database},
{ {
[]string{cmdReport}, []string{cmdReport},
append([]string{ slices.Concat([]string{
"tab-separated", "\n first ", "\n dupe ", "\n size ", "tab-separated", "\n first ", "\n dupe ", "\n size ",
"largest first", "largest first",
}, escapes...), }, escapes, operands),
}, },
{ {
[]string{cmdTrees}, []string{cmdTrees},
append([]string{ slices.Concat([]string{
"tab-separated", "\n first ", "\n dupe ", "\n files ", "tab-separated", "\n first ", "\n dupe ", "\n files ",
"\n size ", "\n disk_size ", "largest first", "\n size ", "\n disk_size ", "largest first",
}, escapes...), }, escapes, operands),
}, },
{ {
[]string{cmdInfo}, []string{cmdInfo},
@@ -1839,9 +1842,13 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
scanFixture(t) scanFixture(t)
cases := map[string]func(context.Context, io.Writer) error{ cases := map[string]func(context.Context, io.Writer) error{
cmdReport: runReport, cmdReport: func(ctx context.Context, w io.Writer) error {
cmdTrees: runTrees, return runReport(ctx, nil, w)
cmdInfo: runInfo, },
cmdTrees: func(ctx context.Context, w io.Writer) error {
return runTrees(ctx, nil, w)
},
cmdInfo: runInfo,
} }
for name, fn := range cases { for name, fn := range cases {
+1 -1
View File
@@ -26,7 +26,7 @@ func runDwim(
return err return err
} }
err = runTrees(ctx, stdout) err = runTrees(ctx, nil, stdout)
if err != nil { if err != nil {
return err return err
} }
+70 -14
View File
@@ -9,6 +9,7 @@ import (
"io" "io"
"io/fs" "io/fs"
"os" "os"
"path/filepath"
"slices" "slices"
"strings" "strings"
"syscall" "syscall"
@@ -44,10 +45,12 @@ func reportPhase(name string) string {
} }
// runReport implements the report subcommand: it prints the file-level // runReport implements the report subcommand: it prints the file-level
// duplicates report as TSV on stdout. Of the scanned filesystem it // duplicates report as TSV on stdout, from the records at or below the
// only checks, with pathExists, that each path in a duplicate group is // operands in args, or from every record when args is empty. Of the
// still there. It names the database on stderr before it opens it. Any // scanned filesystem it only checks, with pathExists, that each path in
// database problem, including a missing database, is fatal. // a duplicate group is still there; it never looks up an operand. It
// names the database on stderr before it opens it. Any database
// problem, including a missing database, is fatal.
// //
// The work runs in two phases, each under a progress display of its // The work runs in two phases, each under a progress display of its
// own on stderr: open (open the database) and read (read the number of // own on stderr: open (open the database) and read (read the number of
@@ -55,7 +58,12 @@ func reportPhase(name string) string {
// order, writing each size's duplicate groups to stdout once that size // order, writing each size's duplicate groups to stdout once that size
// is read; see readReport). Each display ends before the next phase // is read; see readReport). Each display ends before the next phase
// starts, and before an error is returned to be printed. // starts, and before an error is returned to be printed.
func runReport(ctx context.Context, stdout io.Writer) error { func runReport(ctx context.Context, args []string, stdout io.Writer) error {
operands, err := cleanOperands(args)
if err != nil {
return err
}
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
return err return err
@@ -87,7 +95,8 @@ func runReport(ctx context.Context, stdout io.Writer) error {
label: reportPhase("read"), unit: "records", total: -1, label: reportPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords, stalledOn: "db/disk/out", waitingFor: countingRecords, stalledOn: "db/disk/out",
}) })
report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog) report, err := readReport(ctx, db, dbPath, operands, stdout, pathExists,
prog)
prog.finish() prog.finish()
@@ -108,12 +117,14 @@ func runReport(ctx context.Context, stdout io.Writer) error {
// dbPath, as its total, then reads every record in signature order, // dbPath, as its total, then reads every record in signature order,
// counting each on prog, and writes the report to stdout: the header // counting each on prog, and writes the report to stdout: the header
// line, then the rows of each size's duplicate groups once that size is // line, then the rows of each size's duplicate groups once that size is
// read. A path for which exists returns false is left out of its group. // read. Only the records that underOperand finds under operands join a
// stdout may be the terminal prog is drawn on, so the lines go through // group, and a path for which exists returns false is left out of its
// prog's writer. It returns what it wrote, with the totals for the // group. stdout may be the terminal prog is drawn on, so the lines go
// summary line. // through prog's writer. It returns what it wrote, with the totals for
// the summary line.
func readReport(ctx context.Context, db *sql.DB, dbPath string, func readReport(ctx context.Context, db *sql.DB, dbPath string,
stdout io.Writer, exists func(path string) bool, prog *progress, operands []string, stdout io.Writer, exists func(path string) bool,
prog *progress,
) (*reportWriter, error) { ) (*reportWriter, error) {
out := prog.writer(stdout) out := prog.writer(stdout)
report := &reportWriter{out: out, exists: exists} report := &reportWriter{out: out, exists: exists}
@@ -128,6 +139,12 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error { err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
prog.increment() prog.increment()
report.records++
if !underOperand(r.path, operands) {
return nil
}
writeErr = report.add(r) writeErr = report.add(r)
return writeErr return writeErr
@@ -159,7 +176,7 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
// records of one size, so it holds only the duplicate groups of the // records of one size, so it holds only the duplicate groups of the
// size being read, and writes them, ordered by first path, once a // size being read, and writes them, ordered by first path, once a
// record of a smaller size comes. It counts what the summary line // record of a smaller size comes. It counts what the summary line
// reports. // reports; readReport counts the records read.
type reportWriter struct { type reportWriter struct {
out io.Writer out io.Writer
@@ -179,8 +196,6 @@ type reportWriter struct {
// its paths; any other ends them, and one of a smaller size first // its paths; any other ends them, and one of a smaller size first
// writes the groups of the size before. // writes the groups of the size before.
func (w *reportWriter) add(r scanRec) error { func (w *reportWriter) add(r scanRec) error {
w.records++
if r.size == w.sig.size && r.head == w.sig.head && if r.size == w.sig.size && r.head == w.sig.head &&
r.tail == w.sig.tail && r.content == w.sig.content { r.tail == w.sig.tail && r.content == w.sig.content {
w.paths = append(w.paths, r.path) w.paths = append(w.paths, r.path)
@@ -273,6 +288,47 @@ func pathExists(path string) bool {
return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR) return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
} }
// cleanOperands returns the operands of report or trees, each made
// absolute by absolutePath and then cleaned. The records hold real
// paths, with no "." or ".." element and no repeated or trailing
// slash, so an operand is cleaned to match them. No operand is looked
// up on the filesystem. None is empty: rejectEmptyOperand refuses an
// empty operand on the command line.
func cleanOperands(args []string) ([]string, error) {
operands := make([]string, 0, len(args))
for _, a := range args {
abs, err := absolutePath(a)
if err != nil {
return nil, fmt.Errorf("resolve path %s: %w", escapePath(a), err)
}
operands = append(operands, filepath.Clean(abs))
}
return operands, nil
}
// underOperand reports whether path is one of operands or lies below
// one of them. Each operand is clean and absolute, as cleanOperands
// returns it. With no operands, a report covers the whole database, so
// underOperand is true for every path.
func underOperand(path string, operands []string) bool {
if len(operands) == 0 {
return true
}
for _, o := range operands {
// Only the root directory's clean path ends in "/".
dir := strings.TrimSuffix(o, "/") + "/"
if path == o || strings.HasPrefix(path, dir) {
return true
}
}
return false
}
// escapePath returns a path as it is written in a report column (README // escapePath returns a path as it is written in a report column (README
// "Report output format"): a backslash, tab, newline or carriage return // "Report output format"): a backslash, tab, newline or carriage return
// becomes \\, \t, \n or \r, and every other byte is kept as it is. // becomes \\, \t, \n or \r, and every other byte is kept as it is.
+139 -6
View File
@@ -102,7 +102,8 @@ func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
var out bytes.Buffer var out bytes.Buffer
_, err := readReport(t.Context(), db, "", &out, everyPathExists, nil) _, err := readReport(t.Context(), db, "", nil, &out, everyPathExists,
nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -148,6 +149,136 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
return dupeGroups(t, db) return dupeGroups(t, db)
} }
// TestRunReportOperands runs report with operands on three copies of
// one file, two of them under srv/photos, and a fourth copy in
// srv/photos2, all in a new directory. Only the records at or below the
// operands form the groups and the totals, however each operand is
// written.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestRunReportOperands(t *testing.T) {
h := testHash("h")
recs := make([]scanRec, 0, 4)
for _, p := range []string{
"/a/copy", "/srv/photos/x", "/srv/photos/y", "/srv/photos2/z",
} {
recs = append(recs,
scanRec{size: 7, head: h, tail: h, content: h, path: p})
}
dir, recs := recsOnDisk(t, recs)
a, x, y, z := recs[0].path, recs[1].path, recs[2].path, recs[3].path
t.Setenv(databaseEnv, seedDatabase(t, recs))
header := "first\tdupe\tsize\n"
row := func(first, dupe string) string {
return first + "\t" + dupe + "\t7\n"
}
photos := header + row(x, y)
cases := map[string]struct {
operands []string
want string
summary string
}{
"directory": {[]string{dir + "/srv/photos"}, photos,
"1 duplicate groups, 1 dupe files, 7 B reclaimable"},
"trailing slash": {[]string{dir + "/srv/photos/"}, photos, ""},
"dot and dot-dot": {[]string{dir + "/srv/./a/../photos"}, photos, ""},
"files": {[]string{a, z}, header + row(a, z), ""},
"overlapping": {[]string{dir + "/srv", dir + "/srv/photos"},
header + row(x, y) + row(x, z), ""},
"root": {[]string{"/"},
header + row(a, x) + row(a, y) + row(a, z), ""},
"no record": {[]string{dir + "/srv/photo"}, header,
"0 duplicate groups, 0 dupe files, 0 B reclaimable"},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
var stdout bytes.Buffer
stderr := captureStderr(t)
code := run(append([]string{cmdReport}, tc.operands...),
&stdout, os.Stderr)
if code != exitOK {
t.Fatalf("run(report %q) = %d, want %d; stderr: %s",
tc.operands, code, exitOK, stderr())
}
if got := stdout.String(); got != tc.want {
t.Errorf("stdout = %q, want %q", got, tc.want)
}
// The records read are all four, whatever the operands.
summary := "report: 4 records read, " + tc.summary + "\n"
if got := stderr(); tc.summary != "" &&
!strings.HasSuffix(got, summary) {
t.Errorf("stderr = %q, want it to end %q", got, summary)
}
})
}
}
//nolint:paralleltest // t.Chdir changes the process-wide working directory
func TestCleanOperandsMakesRelativePathsAbsolute(t *testing.T) {
dir := t.TempDir()
t.Chdir(dir)
wd, err := absolutePath(".")
if err != nil {
t.Fatal(err)
}
got, err := cleanOperands([]string{"photos/", "a/../b", "/c//d"})
if err != nil {
t.Fatal(err)
}
want := []string{
filepath.Join(wd, "photos"), filepath.Join(wd, "b"), "/c/d",
}
if !slices.Equal(got, want) {
t.Errorf("cleanOperands = %q, want %q", got, want)
}
}
// TestRunReportsRejectEmptyOperand checks that an empty PATH operand of
// report or trees, alone or after another operand, is a usage error
// reported in one line before the database is opened (README "Error
// handling and exit codes").
func TestRunReportsRejectEmptyOperand(t *testing.T) {
cases := map[string][]string{
"report": {cmdReport, ""},
"trees": {cmdTrees, ""},
"after an operand": {cmdTrees, t.TempDir(), ""},
}
for name, args := range cases {
t.Run(name, func(t *testing.T) {
// No database exists there, so opening one would be a fatal
// error, exit 1.
t.Setenv(databaseEnv, testDBPath(t))
var stdout, stderr bytes.Buffer
code := run(args, &stdout, &stderr)
if code != exitUsage {
t.Errorf("run(%q) = %d, want %d", args, code, exitUsage)
}
want := "Error: PATH must not be empty\n"
if stdout.String() != "" || stderr.String() != want {
t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
stdout.String(), stderr.String(), want)
}
})
}
}
func TestRunReportEscapesPaths(t *testing.T) { func TestRunReportEscapesPaths(t *testing.T) {
dir, recs := recsOnDisk(t, awkwardPairRecs()) dir, recs := recsOnDisk(t, awkwardPairRecs())
t.Setenv(databaseEnv, seedDatabase(t, recs)) t.Setenv(databaseEnv, seedDatabase(t, recs))
@@ -195,7 +326,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
// Counts the records read, and draws nothing. // Counts the records read, and draws nothing.
prog := &progress{now: time.Now} prog := &progress{now: time.Now}
_, err = readReport(t.Context(), db, "", failingWriter{}, _, err = readReport(t.Context(), db, "", nil, failingWriter{},
everyPathExists, prog) everyPathExists, prog)
if !errors.Is(err, errWriteFailed) || if !errors.Is(err, errWriteFailed) ||
!strings.HasPrefix(err.Error(), "write stdout: ") { !strings.HasPrefix(err.Error(), "write stdout: ") {
@@ -292,7 +423,8 @@ func TestReadReportOnTerminal(t *testing.T) {
waitingFor: countingRecords, waitingFor: countingRecords,
}, terminalWidth, time.Hour, time.Now) }, terminalWidth, time.Hour, time.Now)
_, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p) _, err = readReport(t.Context(), db, "", nil, os.Stderr,
everyPathExists, p)
p.finish() p.finish()
@@ -514,7 +646,7 @@ func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
return true return true
} }
_, err = readReport(t.Context(), db, "", io.Discard, exists, nil) _, err = readReport(t.Context(), db, "", nil, io.Discard, exists, nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -548,7 +680,8 @@ func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
exists := func(path string) bool { return path != "/a/gone" } exists := func(path string) bool { return path != "/a/gone" }
report, err := readReport(t.Context(), db, "", io.Discard, exists, nil) report, err := readReport(t.Context(), db, "", nil, io.Discard, exists,
nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -581,7 +714,7 @@ func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
var out bytes.Buffer var out bytes.Buffer
report, err := readReport(t.Context(), db, "", &out, exists, nil) report, err := readReport(t.Context(), db, "", nil, &out, exists, nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
+45 -14
View File
@@ -37,12 +37,14 @@ func treesPhase(name string) string {
// runTrees implements the trees subcommand: it reads every record from // runTrees implements the trees subcommand: it reads every record from
// the database in path order, reconstructs the directory hierarchy from // the database in path order, reconstructs the directory hierarchy from
// the record paths, computes a Merkle-style digest per directory, and // the paths of the records at or below the operands in args (of every
// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned // record when args is empty), computes a Merkle-style digest per
// filesystem it only checks, with pathExists, that the root of each // directory, and prints maximal duplicate-tree groups as TSV on stdout.
// duplicate tree is still there, never the files below it. It names the // Of the scanned filesystem it only checks, with pathExists, that the
// database on stderr before it opens it. Any database problem, // root of each duplicate tree is still there, never the files below it;
// including a missing database, is fatal. // it never looks up an operand. It names the database on stderr before
// it opens it. Any database problem, including a missing database, is
// fatal.
// //
// The work runs in four phases, each under a progress display of its // The work runs in four phases, each under a progress display of its
// own on stderr: open (open the database), read (read the number of // own on stderr: open (open the database), read (read the number of
@@ -50,7 +52,12 @@ func treesPhase(name string) string {
// hierarchy), group (group the directories by digest) and write (write // hierarchy), group (group the directories by digest) and write (write
// the groups to stdout). Each display ends before the next phase // the groups to stdout). Each display ends before the next phase
// starts, and before an error is returned to be printed. // starts, and before an error is returned to be printed.
func runTrees(ctx context.Context, stdout io.Writer) error { func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
operands, err := cleanOperands(args)
if err != nil {
return err
}
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
return err return err
@@ -79,7 +86,7 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
label: treesPhase("read"), unit: "records", total: -1, label: treesPhase("read"), unit: "records", total: -1,
waitingFor: countingRecords, stalledOn: "db", waitingFor: countingRecords, stalledOn: "db",
}) })
records, super, allDirs, err := readTree(ctx, db, prog) records, super, allDirs, err := readTree(ctx, db, operands, prog)
prog.finish() prog.finish()
@@ -118,10 +125,17 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
} }
// readTree gives prog the number of records in db as its total, then // readTree gives prog the number of records in db as its total, then
// reads every record in path order into the directory hierarchy, // reads every record in path order, counting each on prog. It adds the
// counting each on prog. It returns the number of records read, the // records that underOperand finds under operands to the directory
// super-root and every directory. // hierarchy. It returns the number of records read, the super-root and
func readTree(ctx context.Context, db *sql.DB, // every directory at or below one of operands.
//
// Any other directory lies above an operand and can hold files that are
// not used, so its digest does not describe it. It is left out of the
// directories returned. A directory whose parent is left out gets the
// super-root as its parent, so suppressed never judges its group by
// that parent's digest.
func readTree(ctx context.Context, db *sql.DB, operands []string,
prog *progress, prog *progress,
) (int, *treeNode, []*treeNode, error) { ) (int, *treeNode, []*treeNode, error) {
records := 0 records := 0
@@ -131,7 +145,10 @@ func readTree(ctx context.Context, db *sql.DB,
records++ records++
prog.increment() prog.increment()
tree.add(r)
if underOperand(r.path, operands) {
tree.add(r)
}
}) })
if err != nil { if err != nil {
return 0, nil, nil, err return 0, nil, nil, err
@@ -139,7 +156,21 @@ func readTree(ctx context.Context, db *sql.DB,
super, allDirs := tree.finish() super, allDirs := tree.finish()
return records, super, allDirs, nil dirs := allDirs[:0]
for _, d := range allDirs {
if !underOperand(d.path, operands) {
continue
}
if !underOperand(d.parent.path, operands) {
d.parent = super
}
dirs = append(dirs, d)
}
return records, super, dirs, nil
} }
// writeTreeGroups writes dupes to stdout as TSV, a header line and then // writeTreeGroups writes dupes to stdout as TSV, a header line and then
+51
View File
@@ -214,6 +214,57 @@ func TestRunTreesEscapesPaths(t *testing.T) {
} }
} }
// TestRunTreesOperands runs trees with operands on records in a new
// directory. x/a and y/a are copies, and so are y/a and srv/b. x and y
// also hold files that differ. Given x/a and y/a, trees reads x and y
// as holding one copy each of a, yet neither is reported: only the
// directories at or below an operand are.
func TestRunTreesOperands(t *testing.T) {
f, g, k := testHash("f"), testHash("g"), testHash("k")
dir, recs := recsOnDisk(t, []scanRec{
{size: 10, head: f, tail: f, content: f, path: "/srv/b/f"},
{size: 10, head: f, tail: f, content: f, path: "/x/a/f"},
{size: 20, head: g, tail: g, content: g, path: "/x/g"},
{size: 10, head: f, tail: f, content: f, path: "/y/a/f"},
{size: 30, head: k, tail: k, content: k, path: "/y/k"},
})
b, xa, ya := dir+"/srv/b", dir+"/x/a", dir+"/y/a"
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
row := func(first, dupe string) string {
return first + "\t" + dupe + "\t1\t10\t0\n"
}
cases := map[string]struct {
operands []string
want string
}{
"none": {nil, header + row(b, xa) + row(b, ya)},
"two copies": {[]string{xa, ya}, header + row(xa, ya)},
"parents": {[]string{dir + "/x", dir + "/y"}, header + row(xa, ya)},
"one copy": {[]string{b}, header},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
t.Setenv(databaseEnv, seedDatabase(t, recs))
var stdout, stderr bytes.Buffer
code := run(append([]string{cmdTrees}, tc.operands...),
&stdout, &stderr)
if code != exitOK {
t.Fatalf("run(trees %q) = %d, want %d; stderr: %s",
tc.operands, code, exitOK, stderr.String())
}
if got := stdout.String(); got != tc.want {
t.Errorf("stdout = %q, want %q", got, tc.want)
}
})
}
}
// TestRunTreesDiskSize runs trees on two copies of a tree whose files // TestRunTreesDiskSize runs trees on two copies of a tree whose files
// take different space on disk. The row's disk_size is the sum over the // take different space on disk. The row's disk_size is the sum over the
// dupe copy's files, those in its subdirectory included. // dupe copy's files, those in its subdirectory included.