Compare commits
1
Commits
next
...
784a7b7a74
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
784a7b7a74 |
No files matched your search
@@ -44,7 +44,9 @@ as opposed to a directory, a symlink or a device node. A symlink is a file that
|
|||||||
holds the path of another file and stands in for it. A device node is a file
|
holds the path of another file and stands in for it. A device node is a file
|
||||||
that stands for a piece of hardware, such as a disk or a terminal. `report`
|
that stands for a piece of hardware, such as a disk or a terminal. `report`
|
||||||
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
||||||
`info` prints the database's path, size and record count.
|
Both cover the whole database, or only the paths given after them, as in
|
||||||
|
`sfdupes report /srv/photos`. `info` prints the database's path, size and record
|
||||||
|
count.
|
||||||
|
|
||||||
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
||||||
another file. The first `scan` creates it.
|
another file. The first `scan` creates it.
|
||||||
@@ -260,8 +262,8 @@ In order:
|
|||||||
|
|
||||||
```
|
```
|
||||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
|
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
|
||||||
sfdupes report > dupes.tsv
|
sfdupes report [PATH...] > dupes.tsv
|
||||||
sfdupes trees > dupetrees.tsv
|
sfdupes trees [PATH...] > dupetrees.tsv
|
||||||
sfdupes info
|
sfdupes info
|
||||||
sfdupes dwim
|
sfdupes dwim
|
||||||
sfdupes --version
|
sfdupes --version
|
||||||
@@ -686,11 +688,12 @@ and `scan` leaves it ignored.
|
|||||||
|
|
||||||
### `report` mode
|
### `report` mode
|
||||||
|
|
||||||
`report` reads every record from the database. It takes no positional arguments.
|
`report` reads every record from the database. It takes optional `PATH`
|
||||||
A positional argument is anything on the command line that is not a flag, such
|
operands. With none, it uses every record. With one or more, it uses only the
|
||||||
as the paths `scan` takes. `report` never opens a scanned file. It checks that
|
records at or below them (see "Operands of `report` and `trees`"). `report`
|
||||||
each path in a group still exists, and looks at no other path. The check is
|
never opens a scanned file. It checks that each path in a group still exists,
|
||||||
`lstat`, the system call that looks up a path without following a symlink:
|
and looks at no other path. It never looks up an operand. The check is `lstat`,
|
||||||
|
the system call that looks up a path without following a symlink:
|
||||||
|
|
||||||
- A path is gone when `lstat` finds that it, or a directory above it, does not
|
- A path is gone when `lstat` finds that it, or a directory above it, does not
|
||||||
exist.
|
exist.
|
||||||
@@ -722,6 +725,27 @@ with one signature come together, and so do the records of one size. So `report`
|
|||||||
holds only the duplicate groups of the size it is reading. Once it reaches a
|
holds only the duplicate groups of the size it is reading. Once it reaches a
|
||||||
smaller size, it writes them, ordered by `first` path.
|
smaller size, it writes them, ordered by `first` path.
|
||||||
|
|
||||||
|
#### Operands of `report` and `trees`
|
||||||
|
|
||||||
|
`report` and `trees` take the same `PATH` operands:
|
||||||
|
|
||||||
|
- A record is used when its path is an operand or lies below one. `/srv/photos`
|
||||||
|
covers `/srv/photos/x` but not `/srv/photos2/x`.
|
||||||
|
- The records used are grouped, ordered and totalled as they are without
|
||||||
|
operands. A copy outside the operands is not reported. With copies `/a/f`,
|
||||||
|
`/srv/photos/x` and `/srv/photos/y`, `sfdupes report /srv/photos` prints one
|
||||||
|
row, `/srv/photos/x` and `/srv/photos/y`.
|
||||||
|
- A relative operand is joined to the working directory as `pwd -P` prints it.
|
||||||
|
- Each operand is then cleaned as text, as `realpath -ms` prints it. Its `.`
|
||||||
|
elements and repeated or trailing slashes are dropped. Each `..` drops the
|
||||||
|
element before it.
|
||||||
|
- No operand is looked up on the filesystem. A symlink in an operand is not
|
||||||
|
followed, and an operand need not exist. The records hold real paths (see
|
||||||
|
"Operands" under "`scan` mode"), so an operand that passes through a symlink
|
||||||
|
matches no record.
|
||||||
|
- An operand that matches no record is not an error. The report then has no
|
||||||
|
rows.
|
||||||
|
|
||||||
#### Report output format
|
#### Report output format
|
||||||
|
|
||||||
The report is TSV (tab-separated values) on stdout. It holds a header line, then
|
The report is TSV (tab-separated values) on stdout. It holds a header line, then
|
||||||
@@ -749,7 +773,8 @@ Grouping and ordering use the stored path. The warnings about skipped paths that
|
|||||||
|
|
||||||
The summary line on stderr gives:
|
The summary line on stderr gives:
|
||||||
|
|
||||||
- the records read;
|
- the records read, which are all the records in the database, with or without
|
||||||
|
operands;
|
||||||
- the duplicate groups;
|
- the duplicate groups;
|
||||||
- the dupe files, which is the number of rows;
|
- the dupe files, which is the number of rows;
|
||||||
- the reclaimable bytes, the sum of `size` over all rows.
|
- the reclaimable bytes, the sum of `size` over all rows.
|
||||||
@@ -760,11 +785,12 @@ report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
|||||||
|
|
||||||
### `trees` mode
|
### `trees` mode
|
||||||
|
|
||||||
`trees` reads the same database as `report` and takes no positional arguments.
|
`trees` reads the same database as `report` and takes the same `PATH` operands
|
||||||
It reports whole duplicate directory trees. Two directories are duplicate trees
|
(see "Operands of `report` and `trees`"). It reports whole duplicate directory
|
||||||
when they hold the same relative paths, and each path holds a file with the same
|
trees. Two directories are duplicate trees when they hold the same relative
|
||||||
signature in both. It rebuilds the directory tree from the paths in the records,
|
paths, and each path holds a file with the same signature in both. It rebuilds
|
||||||
split on `/`. Like `report`, it never opens a scanned file. It checks only that
|
the directory tree from the paths in the records, split on `/`. Like `report`,
|
||||||
|
it never opens a scanned file and never looks up an operand. It checks only that
|
||||||
the directory at the top of each copy still exists, as `report` checks a path.
|
the directory at the top of each copy still exists, as `report` checks a path.
|
||||||
|
|
||||||
Definitions:
|
Definitions:
|
||||||
@@ -792,6 +818,11 @@ Processing:
|
|||||||
directories that all have the same digest. The parents' group, or an
|
directories that all have the same digest. The parents' group, or an
|
||||||
ancestor's, already implies it. A group is always reported when it holds
|
ancestor's, already implies it. A group is always reported when it holds
|
||||||
sibling directories, or when its members' parents have different digests.
|
sibling directories, or when its members' parents have different digests.
|
||||||
|
- With operands, `trees` builds the tree from the records it uses. It reports
|
||||||
|
only directories at or below an operand. Any other directory lies above an
|
||||||
|
operand and can hold files that are not used, so its digest does not describe
|
||||||
|
it. Such a directory is never reported and never implies a group of its
|
||||||
|
children.
|
||||||
- A directory that is gone is dropped from its group. A group left with fewer
|
- A directory that is gone is dropped from its group. A group left with fewer
|
||||||
than two directories is not reported.
|
than two directories is not reported.
|
||||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||||
@@ -845,7 +876,8 @@ directory's path is `/`.
|
|||||||
|
|
||||||
The summary line on stderr gives:
|
The summary line on stderr gives:
|
||||||
|
|
||||||
- the records read;
|
- the records read, which are all the records in the database, with or without
|
||||||
|
operands;
|
||||||
- the duplicate-tree groups;
|
- the duplicate-tree groups;
|
||||||
- the dupe trees, which is the number of rows;
|
- the dupe trees, which is the number of rows;
|
||||||
- the reclaimable bytes, the sum of `size` over all rows.
|
- the reclaimable bytes, the sum of `size` over all rows.
|
||||||
@@ -858,7 +890,9 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
|
|||||||
|
|
||||||
`dwim`, short for "do what I mean", scans a new machine and reports its
|
`dwim`, short for "do what I mean", scans a new machine and reports its
|
||||||
duplicate trees in one command. It takes no positional arguments and no flags of
|
duplicate trees in one command. It takes no positional arguments and no flags of
|
||||||
its own. Like `scan`, it should run as root, so that it can read every file.
|
its own. A positional argument is anything on the command line that is not a
|
||||||
|
flag, such as the paths `scan` takes. Like `scan`, it should run as root, so
|
||||||
|
that it can read every file.
|
||||||
|
|
||||||
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
|
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
|
||||||
excludes and crosses into other filesystems. `--workers` keeps its default. The
|
excludes and crosses into other filesystems. `--workers` keeps its default. The
|
||||||
@@ -1249,7 +1283,7 @@ showing the count it stopped at.
|
|||||||
- `scan` with no operand;
|
- `scan` with no operand;
|
||||||
- `--workers` below 1;
|
- `--workers` below 1;
|
||||||
- an empty `--exclude` value, as in `--exclude ""`;
|
- an empty `--exclude` value, as in `--exclude ""`;
|
||||||
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
- `info` or `dwim` with a positional argument.
|
||||||
|
|
||||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||||
stdout, such as a full disk, is reported the same way. Two cases never reach
|
stdout, such as a full disk, is reported the same way. Two cases never reach
|
||||||
|
|||||||
@@ -29,6 +29,11 @@
|
|||||||
Newest first. An entry ends with its date and its issue or branch, where those
|
Newest first. An entry ends with its date and its issue or branch, where those
|
||||||
are known.
|
are known.
|
||||||
|
|
||||||
|
- `report` and `trees` take optional `PATH` operands. With operands, they use
|
||||||
|
only the records at or below them. `trees` then reports only directories at or
|
||||||
|
below an operand. Operands are made absolute and cleaned as text, never looked
|
||||||
|
up on the filesystem (2026-11-11,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/174)
|
||||||
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
|
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
|
||||||
enters nor records an excluded path, and keeps the records below it. An
|
enters nor records an excluded path, and keeps the records below it. An
|
||||||
operand at or below one is a fatal error (2026-11-11,
|
operand at or below one is a fatal error (2026-11-11,
|
||||||
|
|||||||
+19
-8
@@ -32,6 +32,13 @@ const (
|
|||||||
cmdDwim = "dwim"
|
cmdDwim = "dwim"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
// operandsHelp explains the PATH operands of report and trees in their
|
||||||
|
// help.
|
||||||
|
const operandsHelp = "With PATH operands, only the records at or " +
|
||||||
|
"below them are used. A relative PATH is joined to the working " +
|
||||||
|
"directory. No PATH is looked up on the filesystem, so a symlink in " +
|
||||||
|
"one is not followed."
|
||||||
|
|
||||||
// errNoSubcommand is returned by the root command when it is invoked
|
// errNoSubcommand is returned by the root command when it is invoked
|
||||||
// without a subcommand. That is a usage error, and the usage text
|
// without a subcommand. That is a usage error, and the usage text
|
||||||
// cobra prints for it is the whole message.
|
// cobra prints for it is the whole message.
|
||||||
@@ -167,20 +174,24 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
|||||||
"print the version to stdout")
|
"print the version to stdout")
|
||||||
|
|
||||||
reportCmd := &cobra.Command{
|
reportCmd := &cobra.Command{
|
||||||
Use: cmdReport,
|
Use: cmdReport + " [PATH...]",
|
||||||
Short: "Read the scan database and print the file-level duplicates report",
|
Short: "Read the scan database and print the file-level duplicates report",
|
||||||
Args: cobra.NoArgs,
|
Long: "Read the scan database and print the file-level duplicates " +
|
||||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
"report.\n\n" + operandsHelp,
|
||||||
return runReport(ctx, stdout)
|
Args: cobra.ArbitraryArgs,
|
||||||
|
RunE: runE(func(ctx context.Context, args []string) error {
|
||||||
|
return runReport(ctx, args, stdout)
|
||||||
}),
|
}),
|
||||||
}
|
}
|
||||||
|
|
||||||
treesCmd := &cobra.Command{
|
treesCmd := &cobra.Command{
|
||||||
Use: cmdTrees,
|
Use: cmdTrees + " [PATH...]",
|
||||||
Short: "Read the scan database and print the duplicate-tree report",
|
Short: "Read the scan database and print the duplicate-tree report",
|
||||||
Args: cobra.NoArgs,
|
Long: "Read the scan database and print the duplicate-tree " +
|
||||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
"report.\n\n" + operandsHelp,
|
||||||
return runTrees(ctx, stdout)
|
Args: cobra.ArbitraryArgs,
|
||||||
|
RunE: runE(func(ctx context.Context, args []string) error {
|
||||||
|
return runTrees(ctx, args, stdout)
|
||||||
}),
|
}),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -577,14 +577,14 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
|||||||
},
|
},
|
||||||
cmdReport: {
|
cmdReport: {
|
||||||
run: func() error {
|
run: func() error {
|
||||||
return runReport(cancelledContext(t), io.Discard)
|
return runReport(cancelledContext(t), nil, io.Discard)
|
||||||
},
|
},
|
||||||
want: "report: database " + defaultDatabasePath + " (default)\n" +
|
want: "report: database " + defaultDatabasePath + " (default)\n" +
|
||||||
reportOpenLine + "\n" + reportOpenLine + "\n",
|
reportOpenLine + "\n" + reportOpenLine + "\n",
|
||||||
},
|
},
|
||||||
cmdTrees: {
|
cmdTrees: {
|
||||||
run: func() error {
|
run: func() error {
|
||||||
return runTrees(cancelledContext(t), io.Discard)
|
return runTrees(cancelledContext(t), nil, io.Discard)
|
||||||
},
|
},
|
||||||
want: "trees: database " + defaultDatabasePath + " (default)\n" +
|
want: "trees: database " + defaultDatabasePath + " (default)\n" +
|
||||||
treesOpenLine + "\n" + treesOpenLine + "\n",
|
treesOpenLine + "\n" + treesOpenLine + "\n",
|
||||||
@@ -620,8 +620,6 @@ func TestRunUsageErrors(t *testing.T) {
|
|||||||
}{
|
}{
|
||||||
"no subcommand": {[]string{}, usageMarker},
|
"no subcommand": {[]string{}, usageMarker},
|
||||||
"scan without paths": {[]string{cmdScan}, usageMarker},
|
"scan without paths": {[]string{cmdScan}, usageMarker},
|
||||||
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
|
||||||
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
|
||||||
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
||||||
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
|
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
|
||||||
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
||||||
@@ -1766,8 +1764,12 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
|
|||||||
scanFixture(t)
|
scanFixture(t)
|
||||||
|
|
||||||
cases := map[string]func(context.Context, io.Writer) error{
|
cases := map[string]func(context.Context, io.Writer) error{
|
||||||
cmdReport: runReport,
|
cmdReport: func(ctx context.Context, w io.Writer) error {
|
||||||
cmdTrees: runTrees,
|
return runReport(ctx, nil, w)
|
||||||
|
},
|
||||||
|
cmdTrees: func(ctx context.Context, w io.Writer) error {
|
||||||
|
return runTrees(ctx, nil, w)
|
||||||
|
},
|
||||||
cmdInfo: runInfo,
|
cmdInfo: runInfo,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|
||||||
err = runTrees(ctx, stdout)
|
err = runTrees(ctx, nil, stdout)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
|||||||
+69
-14
@@ -9,6 +9,7 @@ import (
|
|||||||
"io"
|
"io"
|
||||||
"io/fs"
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
|
"path/filepath"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"syscall"
|
"syscall"
|
||||||
@@ -44,10 +45,12 @@ func reportPhase(name string) string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// runReport implements the report subcommand: it prints the file-level
|
// runReport implements the report subcommand: it prints the file-level
|
||||||
// duplicates report as TSV on stdout. Of the scanned filesystem it
|
// duplicates report as TSV on stdout, from the records at or below the
|
||||||
// only checks, with pathExists, that each path in a duplicate group is
|
// operands in args, or from every record when args is empty. Of the
|
||||||
// still there. It names the database on stderr before it opens it. Any
|
// scanned filesystem it only checks, with pathExists, that each path in
|
||||||
// database problem, including a missing database, is fatal.
|
// a duplicate group is still there; it never looks up an operand. It
|
||||||
|
// names the database on stderr before it opens it. Any database
|
||||||
|
// problem, including a missing database, is fatal.
|
||||||
//
|
//
|
||||||
// The work runs in two phases, each under a progress display of its
|
// The work runs in two phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database) and read (read the number of
|
// own on stderr: open (open the database) and read (read the number of
|
||||||
@@ -55,7 +58,12 @@ func reportPhase(name string) string {
|
|||||||
// order, writing each size's duplicate groups to stdout once that size
|
// order, writing each size's duplicate groups to stdout once that size
|
||||||
// is read; see readReport). Each display ends before the next phase
|
// is read; see readReport). Each display ends before the next phase
|
||||||
// starts, and before an error is returned to be printed.
|
// starts, and before an error is returned to be printed.
|
||||||
func runReport(ctx context.Context, stdout io.Writer) error {
|
func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||||
|
operands, err := cleanOperands(args)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -87,7 +95,8 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
|||||||
label: reportPhase("read"), unit: "records", total: -1,
|
label: reportPhase("read"), unit: "records", total: -1,
|
||||||
waitingFor: countingRecords, stalledOn: "db/disk/out",
|
waitingFor: countingRecords, stalledOn: "db/disk/out",
|
||||||
})
|
})
|
||||||
report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog)
|
report, err := readReport(ctx, db, dbPath, operands, stdout, pathExists,
|
||||||
|
prog)
|
||||||
|
|
||||||
prog.finish()
|
prog.finish()
|
||||||
|
|
||||||
@@ -108,12 +117,14 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
|||||||
// dbPath, as its total, then reads every record in signature order,
|
// dbPath, as its total, then reads every record in signature order,
|
||||||
// counting each on prog, and writes the report to stdout: the header
|
// counting each on prog, and writes the report to stdout: the header
|
||||||
// line, then the rows of each size's duplicate groups once that size is
|
// line, then the rows of each size's duplicate groups once that size is
|
||||||
// read. A path for which exists returns false is left out of its group.
|
// read. Only the records that underOperand finds under operands join a
|
||||||
// stdout may be the terminal prog is drawn on, so the lines go through
|
// group, and a path for which exists returns false is left out of its
|
||||||
// prog's writer. It returns what it wrote, with the totals for the
|
// group. stdout may be the terminal prog is drawn on, so the lines go
|
||||||
// summary line.
|
// through prog's writer. It returns what it wrote, with the totals for
|
||||||
|
// the summary line.
|
||||||
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||||
stdout io.Writer, exists func(path string) bool, prog *progress,
|
operands []string, stdout io.Writer, exists func(path string) bool,
|
||||||
|
prog *progress,
|
||||||
) (*reportWriter, error) {
|
) (*reportWriter, error) {
|
||||||
out := prog.writer(stdout)
|
out := prog.writer(stdout)
|
||||||
report := &reportWriter{out: out, exists: exists}
|
report := &reportWriter{out: out, exists: exists}
|
||||||
@@ -128,6 +139,12 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
|||||||
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
|
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
|
||||||
prog.increment()
|
prog.increment()
|
||||||
|
|
||||||
|
report.records++
|
||||||
|
|
||||||
|
if !underOperand(r.path, operands) {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
writeErr = report.add(r)
|
writeErr = report.add(r)
|
||||||
|
|
||||||
return writeErr
|
return writeErr
|
||||||
@@ -159,7 +176,7 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
|||||||
// records of one size, so it holds only the duplicate groups of the
|
// records of one size, so it holds only the duplicate groups of the
|
||||||
// size being read, and writes them, ordered by first path, once a
|
// size being read, and writes them, ordered by first path, once a
|
||||||
// record of a smaller size comes. It counts what the summary line
|
// record of a smaller size comes. It counts what the summary line
|
||||||
// reports.
|
// reports; readReport counts the records read.
|
||||||
type reportWriter struct {
|
type reportWriter struct {
|
||||||
out io.Writer
|
out io.Writer
|
||||||
|
|
||||||
@@ -179,8 +196,6 @@ type reportWriter struct {
|
|||||||
// its paths; any other ends them, and one of a smaller size first
|
// its paths; any other ends them, and one of a smaller size first
|
||||||
// writes the groups of the size before.
|
// writes the groups of the size before.
|
||||||
func (w *reportWriter) add(r scanRec) error {
|
func (w *reportWriter) add(r scanRec) error {
|
||||||
w.records++
|
|
||||||
|
|
||||||
if r.size == w.sig.size && r.head == w.sig.head &&
|
if r.size == w.sig.size && r.head == w.sig.head &&
|
||||||
r.tail == w.sig.tail && r.content == w.sig.content {
|
r.tail == w.sig.tail && r.content == w.sig.content {
|
||||||
w.paths = append(w.paths, r.path)
|
w.paths = append(w.paths, r.path)
|
||||||
@@ -273,6 +288,46 @@ func pathExists(path string) bool {
|
|||||||
return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
|
return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// cleanOperands returns the operands of report or trees, each made
|
||||||
|
// absolute by absolutePath and then cleaned. The records hold real
|
||||||
|
// paths, with no "." or ".." element and no repeated or trailing
|
||||||
|
// slash, so an operand is cleaned to match them. No operand is looked
|
||||||
|
// up on the filesystem.
|
||||||
|
func cleanOperands(args []string) ([]string, error) {
|
||||||
|
operands := make([]string, 0, len(args))
|
||||||
|
|
||||||
|
for _, a := range args {
|
||||||
|
abs, err := absolutePath(a)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("resolve path %s: %w", escapePath(a), err)
|
||||||
|
}
|
||||||
|
|
||||||
|
operands = append(operands, filepath.Clean(abs))
|
||||||
|
}
|
||||||
|
|
||||||
|
return operands, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// underOperand reports whether path is one of operands or lies below
|
||||||
|
// one of them. Each operand is clean and absolute, as cleanOperands
|
||||||
|
// returns it. With no operands, a report covers the whole database, so
|
||||||
|
// underOperand is true for every path.
|
||||||
|
func underOperand(path string, operands []string) bool {
|
||||||
|
if len(operands) == 0 {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, o := range operands {
|
||||||
|
// Only the root directory's clean path ends in "/".
|
||||||
|
dir := strings.TrimSuffix(o, "/") + "/"
|
||||||
|
if path == o || strings.HasPrefix(path, dir) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
// escapePath returns a path as it is written in a report column (README
|
// escapePath returns a path as it is written in a report column (README
|
||||||
// "Report output format"): a backslash, tab, newline or carriage return
|
// "Report output format"): a backslash, tab, newline or carriage return
|
||||||
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
||||||
|
|||||||
@@ -102,7 +102,8 @@ func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
|
|||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
|
|
||||||
_, err := readReport(t.Context(), db, "", &out, everyPathExists, nil)
|
_, err := readReport(t.Context(), db, "", nil, &out, everyPathExists,
|
||||||
|
nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -148,6 +149,103 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
|
|||||||
return dupeGroups(t, db)
|
return dupeGroups(t, db)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRunReportOperands runs report with operands on three copies of
|
||||||
|
// one file, two of them under srv/photos, and a fourth copy in
|
||||||
|
// srv/photos2, all in a new directory. Only the records at or below the
|
||||||
|
// operands form the groups and the totals, however each operand is
|
||||||
|
// written.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
|
func TestRunReportOperands(t *testing.T) {
|
||||||
|
h := testHash("h")
|
||||||
|
|
||||||
|
recs := make([]scanRec, 0, 4)
|
||||||
|
for _, p := range []string{
|
||||||
|
"/a/copy", "/srv/photos/x", "/srv/photos/y", "/srv/photos2/z",
|
||||||
|
} {
|
||||||
|
recs = append(recs,
|
||||||
|
scanRec{size: 7, head: h, tail: h, content: h, path: p})
|
||||||
|
}
|
||||||
|
|
||||||
|
dir, recs := recsOnDisk(t, recs)
|
||||||
|
a, x, y, z := recs[0].path, recs[1].path, recs[2].path, recs[3].path
|
||||||
|
|
||||||
|
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||||
|
|
||||||
|
header := "first\tdupe\tsize\n"
|
||||||
|
row := func(first, dupe string) string {
|
||||||
|
return first + "\t" + dupe + "\t7\n"
|
||||||
|
}
|
||||||
|
photos := header + row(x, y)
|
||||||
|
|
||||||
|
cases := map[string]struct {
|
||||||
|
operands []string
|
||||||
|
want string
|
||||||
|
summary string
|
||||||
|
}{
|
||||||
|
"directory": {[]string{dir + "/srv/photos"}, photos,
|
||||||
|
"1 duplicate groups, 1 dupe files, 7 B reclaimable"},
|
||||||
|
"trailing slash": {[]string{dir + "/srv/photos/"}, photos, ""},
|
||||||
|
"dot and dot-dot": {[]string{dir + "/srv/./a/../photos"}, photos, ""},
|
||||||
|
"files": {[]string{a, z}, header + row(a, z), ""},
|
||||||
|
"overlapping": {[]string{dir + "/srv", dir + "/srv/photos"},
|
||||||
|
header + row(x, y) + row(x, z), ""},
|
||||||
|
"root": {[]string{"/"},
|
||||||
|
header + row(a, x) + row(a, y) + row(a, z), ""},
|
||||||
|
"no record": {[]string{dir + "/srv/photo"}, header,
|
||||||
|
"0 duplicate groups, 0 dupe files, 0 B reclaimable"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, tc := range cases {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
var stdout bytes.Buffer
|
||||||
|
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
|
code := run(append([]string{cmdReport}, tc.operands...),
|
||||||
|
&stdout, os.Stderr)
|
||||||
|
if code != exitOK {
|
||||||
|
t.Fatalf("run(report %q) = %d, want %d; stderr: %s",
|
||||||
|
tc.operands, code, exitOK, stderr())
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := stdout.String(); got != tc.want {
|
||||||
|
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||||
|
}
|
||||||
|
|
||||||
|
// The records read are all four, whatever the operands.
|
||||||
|
summary := "report: 4 records read, " + tc.summary + "\n"
|
||||||
|
if got := stderr(); tc.summary != "" &&
|
||||||
|
!strings.HasSuffix(got, summary) {
|
||||||
|
t.Errorf("stderr = %q, want it to end %q", got, summary)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
//nolint:paralleltest // t.Chdir changes the process-wide working directory
|
||||||
|
func TestCleanOperandsMakesRelativePathsAbsolute(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
t.Chdir(dir)
|
||||||
|
|
||||||
|
wd, err := absolutePath(".")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
got, err := cleanOperands([]string{"photos/", "a/../b", "/c//d"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := []string{
|
||||||
|
filepath.Join(wd, "photos"), filepath.Join(wd, "b"), "/c/d",
|
||||||
|
}
|
||||||
|
if !slices.Equal(got, want) {
|
||||||
|
t.Errorf("cleanOperands = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunReportEscapesPaths(t *testing.T) {
|
func TestRunReportEscapesPaths(t *testing.T) {
|
||||||
dir, recs := recsOnDisk(t, awkwardPairRecs())
|
dir, recs := recsOnDisk(t, awkwardPairRecs())
|
||||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||||
@@ -195,7 +293,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
|
|||||||
// Counts the records read, and draws nothing.
|
// Counts the records read, and draws nothing.
|
||||||
prog := &progress{now: time.Now}
|
prog := &progress{now: time.Now}
|
||||||
|
|
||||||
_, err = readReport(t.Context(), db, "", failingWriter{},
|
_, err = readReport(t.Context(), db, "", nil, failingWriter{},
|
||||||
everyPathExists, prog)
|
everyPathExists, prog)
|
||||||
if !errors.Is(err, errWriteFailed) ||
|
if !errors.Is(err, errWriteFailed) ||
|
||||||
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
||||||
@@ -292,7 +390,8 @@ func TestReadReportOnTerminal(t *testing.T) {
|
|||||||
waitingFor: countingRecords,
|
waitingFor: countingRecords,
|
||||||
}, terminalWidth, time.Hour, time.Now)
|
}, terminalWidth, time.Hour, time.Now)
|
||||||
|
|
||||||
_, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p)
|
_, err = readReport(t.Context(), db, "", nil, os.Stderr,
|
||||||
|
everyPathExists, p)
|
||||||
|
|
||||||
p.finish()
|
p.finish()
|
||||||
|
|
||||||
@@ -514,7 +613,7 @@ func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
|
|||||||
return true
|
return true
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err = readReport(t.Context(), db, "", io.Discard, exists, nil)
|
_, err = readReport(t.Context(), db, "", nil, io.Discard, exists, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -548,7 +647,8 @@ func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
|
|||||||
|
|
||||||
exists := func(path string) bool { return path != "/a/gone" }
|
exists := func(path string) bool { return path != "/a/gone" }
|
||||||
|
|
||||||
report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
|
report, err := readReport(t.Context(), db, "", nil, io.Discard, exists,
|
||||||
|
nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -581,7 +681,7 @@ func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
|
|||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
|
|
||||||
report, err := readReport(t.Context(), db, "", &out, exists, nil)
|
report, err := readReport(t.Context(), db, "", nil, &out, exists, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|||||||
+44
-13
@@ -37,12 +37,14 @@ func treesPhase(name string) string {
|
|||||||
|
|
||||||
// runTrees implements the trees subcommand: it reads every record from
|
// runTrees implements the trees subcommand: it reads every record from
|
||||||
// the database in path order, reconstructs the directory hierarchy from
|
// the database in path order, reconstructs the directory hierarchy from
|
||||||
// the record paths, computes a Merkle-style digest per directory, and
|
// the paths of the records at or below the operands in args (of every
|
||||||
// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned
|
// record when args is empty), computes a Merkle-style digest per
|
||||||
// filesystem it only checks, with pathExists, that the root of each
|
// directory, and prints maximal duplicate-tree groups as TSV on stdout.
|
||||||
// duplicate tree is still there, never the files below it. It names the
|
// Of the scanned filesystem it only checks, with pathExists, that the
|
||||||
// database on stderr before it opens it. Any database problem,
|
// root of each duplicate tree is still there, never the files below it;
|
||||||
// including a missing database, is fatal.
|
// it never looks up an operand. It names the database on stderr before
|
||||||
|
// it opens it. Any database problem, including a missing database, is
|
||||||
|
// fatal.
|
||||||
//
|
//
|
||||||
// The work runs in four phases, each under a progress display of its
|
// The work runs in four phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database), read (read the number of
|
// own on stderr: open (open the database), read (read the number of
|
||||||
@@ -50,7 +52,12 @@ func treesPhase(name string) string {
|
|||||||
// hierarchy), group (group the directories by digest) and write (write
|
// hierarchy), group (group the directories by digest) and write (write
|
||||||
// the groups to stdout). Each display ends before the next phase
|
// the groups to stdout). Each display ends before the next phase
|
||||||
// starts, and before an error is returned to be printed.
|
// starts, and before an error is returned to be printed.
|
||||||
func runTrees(ctx context.Context, stdout io.Writer) error {
|
func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||||
|
operands, err := cleanOperands(args)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
dbPath, source, err := databasePath()
|
dbPath, source, err := databasePath()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
@@ -79,7 +86,7 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
|||||||
label: treesPhase("read"), unit: "records", total: -1,
|
label: treesPhase("read"), unit: "records", total: -1,
|
||||||
waitingFor: countingRecords, stalledOn: "db",
|
waitingFor: countingRecords, stalledOn: "db",
|
||||||
})
|
})
|
||||||
records, super, allDirs, err := readTree(ctx, db, prog)
|
records, super, allDirs, err := readTree(ctx, db, operands, prog)
|
||||||
|
|
||||||
prog.finish()
|
prog.finish()
|
||||||
|
|
||||||
@@ -118,10 +125,17 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// readTree gives prog the number of records in db as its total, then
|
// readTree gives prog the number of records in db as its total, then
|
||||||
// reads every record in path order into the directory hierarchy,
|
// reads every record in path order, counting each on prog. It adds the
|
||||||
// counting each on prog. It returns the number of records read, the
|
// records that underOperand finds under operands to the directory
|
||||||
// super-root and every directory.
|
// hierarchy. It returns the number of records read, the super-root and
|
||||||
func readTree(ctx context.Context, db *sql.DB,
|
// every directory at or below one of operands.
|
||||||
|
//
|
||||||
|
// Any other directory lies above an operand and can hold files that are
|
||||||
|
// not used, so its digest does not describe it. It is left out of the
|
||||||
|
// directories returned. A directory whose parent is left out gets the
|
||||||
|
// super-root as its parent, so suppressed never judges its group by
|
||||||
|
// that parent's digest.
|
||||||
|
func readTree(ctx context.Context, db *sql.DB, operands []string,
|
||||||
prog *progress,
|
prog *progress,
|
||||||
) (int, *treeNode, []*treeNode, error) {
|
) (int, *treeNode, []*treeNode, error) {
|
||||||
records := 0
|
records := 0
|
||||||
@@ -131,7 +145,10 @@ func readTree(ctx context.Context, db *sql.DB,
|
|||||||
records++
|
records++
|
||||||
|
|
||||||
prog.increment()
|
prog.increment()
|
||||||
|
|
||||||
|
if underOperand(r.path, operands) {
|
||||||
tree.add(r)
|
tree.add(r)
|
||||||
|
}
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, nil, nil, err
|
return 0, nil, nil, err
|
||||||
@@ -139,7 +156,21 @@ func readTree(ctx context.Context, db *sql.DB,
|
|||||||
|
|
||||||
super, allDirs := tree.finish()
|
super, allDirs := tree.finish()
|
||||||
|
|
||||||
return records, super, allDirs, nil
|
dirs := allDirs[:0]
|
||||||
|
|
||||||
|
for _, d := range allDirs {
|
||||||
|
if !underOperand(d.path, operands) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if !underOperand(d.parent.path, operands) {
|
||||||
|
d.parent = super
|
||||||
|
}
|
||||||
|
|
||||||
|
dirs = append(dirs, d)
|
||||||
|
}
|
||||||
|
|
||||||
|
return records, super, dirs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||||
|
|||||||
@@ -214,6 +214,57 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestRunTreesOperands runs trees with operands on records in a new
|
||||||
|
// directory. x/a and y/a are copies, and so are y/a and srv/b. x and y
|
||||||
|
// also hold files that differ. Given x/a and y/a, trees reads x and y
|
||||||
|
// as holding one copy each of a, yet neither is reported: only the
|
||||||
|
// directories at or below an operand are.
|
||||||
|
func TestRunTreesOperands(t *testing.T) {
|
||||||
|
f, g, k := testHash("f"), testHash("g"), testHash("k")
|
||||||
|
dir, recs := recsOnDisk(t, []scanRec{
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/srv/b/f"},
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/x/a/f"},
|
||||||
|
{size: 20, head: g, tail: g, content: g, path: "/x/g"},
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/y/a/f"},
|
||||||
|
{size: 30, head: k, tail: k, content: k, path: "/y/k"},
|
||||||
|
})
|
||||||
|
b, xa, ya := dir+"/srv/b", dir+"/x/a", dir+"/y/a"
|
||||||
|
|
||||||
|
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
|
||||||
|
row := func(first, dupe string) string {
|
||||||
|
return first + "\t" + dupe + "\t1\t10\t0\n"
|
||||||
|
}
|
||||||
|
|
||||||
|
cases := map[string]struct {
|
||||||
|
operands []string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
"none": {nil, header + row(b, xa) + row(b, ya)},
|
||||||
|
"two copies": {[]string{xa, ya}, header + row(xa, ya)},
|
||||||
|
"parents": {[]string{dir + "/x", dir + "/y"}, header + row(xa, ya)},
|
||||||
|
"one copy": {[]string{b}, header},
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, tc := range cases {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
|
code := run(append([]string{cmdTrees}, tc.operands...),
|
||||||
|
&stdout, &stderr)
|
||||||
|
if code != exitOK {
|
||||||
|
t.Fatalf("run(trees %q) = %d, want %d; stderr: %s",
|
||||||
|
tc.operands, code, exitOK, stderr.String())
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := stdout.String(); got != tc.want {
|
||||||
|
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
||||||
// take different space on disk. The row's disk_size is the sum over the
|
// take different space on disk. The row's disk_size is the sum over the
|
||||||
// dupe copy's files, those in its subdirectory included.
|
// dupe copy's files, those in its subdirectory included.
|
||||||
|
|||||||
Reference in new issue
Block a user