Compare commits
1
Commits
97a02da837
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8ab5fd621e |
No files matched your search
@@ -43,9 +43,7 @@ as opposed to a directory, a symlink or a device node. A symlink is a file that
|
||||
holds the path of another file and stands in for it. A device node is a file
|
||||
that stands for a piece of hardware, such as a disk or a terminal. `report`
|
||||
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
||||
Both cover the whole database, or only the paths given after them, as in
|
||||
`sfdupes report /srv/photos`. `info` prints the database's path, size and record
|
||||
count.
|
||||
`info` prints the database's path, size and record count.
|
||||
|
||||
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
||||
another file. The first `scan` creates it.
|
||||
@@ -255,8 +253,8 @@ In order:
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
sfdupes report [PATH...] > dupes.tsv
|
||||
sfdupes trees [PATH...] > dupetrees.tsv
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
sfdupes dwim
|
||||
sfdupes --version
|
||||
@@ -655,19 +653,18 @@ and `scan` leaves it ignored.
|
||||
|
||||
### `report` mode
|
||||
|
||||
`report` reads every record from the database. It takes optional `PATH`
|
||||
operands. With none, it uses every record. With one or more, it uses only the
|
||||
records at or below them (see "Report paths"). `report` never touches the
|
||||
scanned filesystem. It neither stats nor opens any path in the records. It reads
|
||||
the database and writes to stdout and stderr, and nothing else. Its output is
|
||||
the same whether or not the scanned filesystem is still mounted.
|
||||
`report` reads every record from the database. It takes no positional arguments.
|
||||
A positional argument is anything on the command line that is not a flag, such
|
||||
as the paths `scan` takes. `report` never touches the scanned filesystem. It
|
||||
neither stats nor opens any path in the records. It reads the database and
|
||||
writes to stdout and stderr, and nothing else. Its output is the same whether or
|
||||
not the scanned filesystem is still mounted.
|
||||
|
||||
Processing:
|
||||
|
||||
- Records are grouped by signature, the four values
|
||||
`(size, head, tail, content)`.
|
||||
- Every group with two or more paths is a duplicate group.
|
||||
- Empty files form no group, because removing one frees nothing.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
every other path is a `dupe`.
|
||||
- Groups are ordered by size, largest first, then by `first` path. The output is
|
||||
@@ -683,26 +680,6 @@ with one signature come together, and so do the records of one size. So `report`
|
||||
holds only the duplicate groups of the size it is reading. Once it reaches a
|
||||
smaller size, it writes them, ordered by `first` path.
|
||||
|
||||
#### Report paths
|
||||
|
||||
`report` and `trees` take the same `PATH` operands:
|
||||
|
||||
- A record is used when its path is an operand or lies below one. `/srv/photos`
|
||||
covers `/srv/photos/x` but not `/srv/photos2/x`.
|
||||
- The records used are grouped, ordered and totalled as they are without
|
||||
operands. A copy outside the operands is not reported. With copies `/a/f`,
|
||||
`/srv/photos/x` and `/srv/photos/y`, `sfdupes report /srv/photos` prints one
|
||||
row, `/srv/photos/x` and `/srv/photos/y`.
|
||||
- A relative operand is joined to the working directory as `pwd -P` prints it.
|
||||
- Each operand is then cleaned as text, as `realpath -ms` prints it. Its `.`
|
||||
elements and repeated or trailing slashes are dropped. Each `..` drops the
|
||||
element before it.
|
||||
- No operand is looked up on the filesystem. A symlink in an operand is not
|
||||
followed, and an operand need not exist. The records hold real paths (see
|
||||
"Operands"), so an operand that passes through a symlink matches no record.
|
||||
- An operand that matches no record is not an error. The report then has no
|
||||
rows.
|
||||
|
||||
#### Report output format
|
||||
|
||||
The report is TSV (tab-separated values) on stdout. It holds a header line, then
|
||||
@@ -730,24 +707,22 @@ Grouping and ordering use the stored path. The warnings about skipped paths that
|
||||
|
||||
The summary line on stderr gives:
|
||||
|
||||
- the records read, which are all the records in the database, with or without
|
||||
operands;
|
||||
- the records read;
|
||||
- the duplicate groups;
|
||||
- the dupe files, which is the number of rows;
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
|
||||
```
|
||||
report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
||||
report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable
|
||||
```
|
||||
|
||||
### `trees` mode
|
||||
|
||||
`trees` reads the same database as `report` and takes the same `PATH` operands
|
||||
(see "Report paths"). It reports whole duplicate directory trees. Two
|
||||
directories are duplicate trees when they hold the same relative paths, and each
|
||||
path holds a file with the same signature in both. Like `report`, it never
|
||||
touches the scanned filesystem. It rebuilds the directory tree from the paths in
|
||||
the records, split on `/`.
|
||||
`trees` reads the same database as `report` and takes no positional arguments.
|
||||
It reports whole duplicate directory trees. Two directories are duplicate trees
|
||||
when they hold the same relative paths, and each path holds a file with the same
|
||||
signature in both. Like `report`, it never touches the scanned filesystem. It
|
||||
rebuilds the directory tree from the paths in the records, split on `/`.
|
||||
|
||||
Definitions:
|
||||
|
||||
@@ -766,18 +741,11 @@ Processing:
|
||||
- `trees` builds the directory tree and computes every directory's digest. It
|
||||
then groups the directories by digest. Every group with two or more
|
||||
directories is a duplicate-tree group.
|
||||
- `trees` leaves out a group whose trees hold 0 bytes, because removing one
|
||||
frees nothing. An empty file still counts toward the digest of a tree that
|
||||
holds other files.
|
||||
- `trees` leaves out a group that a group of its members' parents already
|
||||
implies. A group is left out when its members' parents are distinct
|
||||
directories that all have the same digest. The parents' group, or an
|
||||
ancestor's, already implies it. A group is always reported when it holds
|
||||
sibling directories, or when its members' parents have different digests.
|
||||
- With operands, `trees` builds the tree from the records it uses. It reports
|
||||
only directories at or below an operand. A directory above every operand holds
|
||||
files that are not used, so its digest does not describe it. Such a directory
|
||||
is never reported and never implies a group of its children.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
every other path is a `dupe`.
|
||||
- Groups are ordered by total tree size, largest first, then by `first` path.
|
||||
@@ -827,8 +795,7 @@ directory's path is `/`.
|
||||
|
||||
The summary line on stderr gives:
|
||||
|
||||
- the records read, which are all the records in the database, with or without
|
||||
operands;
|
||||
- the records read;
|
||||
- the duplicate-tree groups;
|
||||
- the dupe trees, which is the number of rows;
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
@@ -841,9 +808,7 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
|
||||
|
||||
`dwim`, short for "do what I mean", scans a new machine and reports its
|
||||
duplicate trees in one command. It takes no positional arguments and no flags of
|
||||
its own. A positional argument is anything on the command line that is not a
|
||||
flag, such as the paths `scan` takes. Like `scan`, it should run as root, so
|
||||
that it can read every file.
|
||||
its own. Like `scan`, it should run as root, so that it can read every file.
|
||||
|
||||
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
|
||||
excludes and crosses into other filesystems. `--workers` keeps its default. The
|
||||
@@ -1229,7 +1194,7 @@ showing the count it stopped at.
|
||||
`--workers abc`;
|
||||
- `scan` with no operand;
|
||||
- `--workers` below 1;
|
||||
- `info` or `dwim` with a positional argument.
|
||||
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
||||
|
||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||
stdout, such as a full disk, is reported the same way. Two cases never reach
|
||||
@@ -1417,7 +1382,7 @@ All of the following, run in this directory, must pass:
|
||||
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
||||
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
||||
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
||||
: > "$d/empty1"; : > "$d/empty2" # empty files
|
||||
: > "$d/empty1"; : > "$d/empty2" # empty duplicates
|
||||
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
||||
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
||||
head -c 3000 /dev/urandom > "$d/t1/f1"
|
||||
@@ -1442,8 +1407,8 @@ All of the following, run in this directory, must pass:
|
||||
is the path that sorts first.
|
||||
- `t1/f1`, `t2/f1` and `t3/f1` form one group.
|
||||
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
|
||||
- `tiny1` and `tiny2` form a pair.
|
||||
- `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
|
||||
- `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`.
|
||||
- `unique.bin` and `tiny3` appear nowhere.
|
||||
- The groups come in order of size, largest first.
|
||||
|
||||
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
|
||||
|
||||
@@ -29,15 +29,6 @@
|
||||
Newest first. An entry ends with its date and its issue or branch, where those
|
||||
are known.
|
||||
|
||||
- `report` and `trees` take optional `PATH` operands. With operands, they use
|
||||
only the records at or below them. `trees` then reports only directories at or
|
||||
below an operand. Operands are made absolute and cleaned as text, never looked
|
||||
up on the filesystem (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/174)
|
||||
- `report` and `trees` leave out empty files. `report` writes no group of size
|
||||
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
|
||||
counts toward a tree's digest (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/178)
|
||||
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
|
||||
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
|
||||
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
|
||||
|
||||
+8
-19
@@ -31,13 +31,6 @@ const (
|
||||
cmdDwim = "dwim"
|
||||
)
|
||||
|
||||
// reportPathsHelp explains the PATH operands of report and trees in
|
||||
// their help.
|
||||
const reportPathsHelp = "With PATH operands, only the records at or " +
|
||||
"below them are used. A relative PATH is joined to the working " +
|
||||
"directory. No PATH is looked up on the filesystem, so a symlink in " +
|
||||
"one is not followed."
|
||||
|
||||
// errNoSubcommand is returned by the root command when it is invoked
|
||||
// without a subcommand. That is a usage error, and the usage text
|
||||
// cobra prints for it is the whole message.
|
||||
@@ -168,24 +161,20 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
"print the version to stdout")
|
||||
|
||||
reportCmd := &cobra.Command{
|
||||
Use: cmdReport + " [PATH...]",
|
||||
Use: cmdReport,
|
||||
Short: "Read the scan database and print the file-level duplicates report",
|
||||
Long: "Read the scan database and print the file-level duplicates " +
|
||||
"report.\n\n" + reportPathsHelp,
|
||||
Args: cobra.ArbitraryArgs,
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
return runReport(ctx, args, stdout)
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runReport(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
treesCmd := &cobra.Command{
|
||||
Use: cmdTrees + " [PATH...]",
|
||||
Use: cmdTrees,
|
||||
Short: "Read the scan database and print the duplicate-tree report",
|
||||
Long: "Read the scan database and print the duplicate-tree " +
|
||||
"report.\n\n" + reportPathsHelp,
|
||||
Args: cobra.ArbitraryArgs,
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
return runTrees(ctx, args, stdout)
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runTrees(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
|
||||
@@ -576,14 +576,14 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
},
|
||||
cmdReport: {
|
||||
run: func() error {
|
||||
return runReport(cancelledContext(t), nil, io.Discard)
|
||||
return runReport(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "report: database " + defaultDatabasePath + " (default)\n" +
|
||||
reportOpenLine + "\n" + reportOpenLine + "\n",
|
||||
},
|
||||
cmdTrees: {
|
||||
run: func() error {
|
||||
return runTrees(cancelledContext(t), nil, io.Discard)
|
||||
return runTrees(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "trees: database " + defaultDatabasePath + " (default)\n" +
|
||||
treesOpenLine + "\n" + treesOpenLine + "\n",
|
||||
@@ -619,6 +619,8 @@ func TestRunUsageErrors(t *testing.T) {
|
||||
}{
|
||||
"no subcommand": {[]string{}, usageMarker},
|
||||
"scan without paths": {[]string{cmdScan}, usageMarker},
|
||||
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
||||
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
||||
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
||||
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
|
||||
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
||||
@@ -1724,13 +1726,9 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
|
||||
scanFixture(t)
|
||||
|
||||
cases := map[string]func(context.Context, io.Writer) error{
|
||||
cmdReport: func(ctx context.Context, w io.Writer) error {
|
||||
return runReport(ctx, nil, w)
|
||||
},
|
||||
cmdTrees: func(ctx context.Context, w io.Writer) error {
|
||||
return runTrees(ctx, nil, w)
|
||||
},
|
||||
cmdInfo: runInfo,
|
||||
cmdReport: runReport,
|
||||
cmdTrees: runTrees,
|
||||
cmdInfo: runInfo,
|
||||
}
|
||||
|
||||
for name, fn := range cases {
|
||||
|
||||
@@ -25,7 +25,7 @@ func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
||||
return err
|
||||
}
|
||||
|
||||
err = runTrees(ctx, nil, stdout)
|
||||
err = runTrees(ctx, stdout)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
+14
-67
@@ -7,7 +7,6 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -42,11 +41,10 @@ func reportPhase(name string) string {
|
||||
}
|
||||
|
||||
// runReport implements the report subcommand: it prints the file-level
|
||||
// duplicates report as TSV on stdout, from the records at or below the
|
||||
// given paths, or from every record when none is given. It never
|
||||
// touches the scanned filesystem; its only I/O is the database,
|
||||
// stdout, and stderr. It names the database on stderr before it opens
|
||||
// it. Any database problem, including a missing database, is fatal.
|
||||
// duplicates report as TSV on stdout. It never touches the scanned
|
||||
// filesystem; its only I/O is the database, stdout, and stderr. It
|
||||
// names the database on stderr before it opens it. Any database
|
||||
// problem, including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in two phases, each under a progress display of its
|
||||
// own on stderr: open (open the database) and read (read the number of
|
||||
@@ -54,12 +52,7 @@ func reportPhase(name string) string {
|
||||
// order, writing each size's duplicate groups to stdout once that size
|
||||
// is read; see readReport). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
paths, err := reportPaths(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -90,7 +83,7 @@ func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
label: reportPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db/out",
|
||||
})
|
||||
report, err := readReport(ctx, db, dbPath, paths, stdout, prog)
|
||||
report, err := readReport(ctx, db, dbPath, stdout, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -111,12 +104,11 @@ func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
// dbPath, as its total, then reads every record in signature order,
|
||||
// counting each on prog, and writes the report to stdout: the header
|
||||
// line, then the rows of each size's duplicate groups once that size is
|
||||
// read. Only the records that inReportPaths selects from paths join a
|
||||
// group. stdout may be the terminal prog is drawn on, so the lines go
|
||||
// read. stdout may be the terminal prog is drawn on, so the lines go
|
||||
// through prog's writer. It returns what it wrote, with the totals for
|
||||
// the summary line.
|
||||
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
paths []string, stdout io.Writer, prog *progress,
|
||||
stdout io.Writer, prog *progress,
|
||||
) (*reportWriter, error) {
|
||||
out := prog.writer(stdout)
|
||||
report := &reportWriter{out: out}
|
||||
@@ -131,12 +123,6 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
|
||||
prog.increment()
|
||||
|
||||
report.records++
|
||||
|
||||
if !inReportPaths(r.path, paths) {
|
||||
return nil
|
||||
}
|
||||
|
||||
writeErr = report.add(r)
|
||||
|
||||
return writeErr
|
||||
@@ -168,7 +154,7 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
// records of one size, so it holds only the duplicate groups of the
|
||||
// size being read, and writes them, ordered by first path, once a
|
||||
// record of a smaller size comes. It counts what the summary line
|
||||
// reports; readReport counts the records read.
|
||||
// reports.
|
||||
type reportWriter struct {
|
||||
out io.Writer
|
||||
|
||||
@@ -184,6 +170,8 @@ type reportWriter struct {
|
||||
// its paths; any other ends them, and one of a smaller size first
|
||||
// writes the groups of the size before.
|
||||
func (w *reportWriter) add(r scanRec) error {
|
||||
w.records++
|
||||
|
||||
if r.size == w.sig.size && r.head == w.sig.head &&
|
||||
r.tail == w.sig.tail && r.content == w.sig.content {
|
||||
w.paths = append(w.paths, r.path)
|
||||
@@ -207,11 +195,10 @@ func (w *reportWriter) add(r scanRec) error {
|
||||
}
|
||||
|
||||
// endGroup ends the paths read with w.sig: two or more are a duplicate
|
||||
// group, held with its paths sorted until writeGroups writes it. Empty
|
||||
// files form no group, since removing one frees nothing.
|
||||
// group, held with its paths sorted until writeGroups writes it.
|
||||
func (w *reportWriter) endGroup() {
|
||||
if len(w.paths) < minGroupSize || w.sig.size == 0 {
|
||||
// The paths are dropped, and their slice reused.
|
||||
if len(w.paths) < minGroupSize {
|
||||
// A path with no duplicate is dropped, and its slice reused.
|
||||
w.paths = w.paths[:0]
|
||||
|
||||
return
|
||||
@@ -258,46 +245,6 @@ func (w *reportWriter) finish() error {
|
||||
return w.writeGroups()
|
||||
}
|
||||
|
||||
// reportPaths returns the paths given to report or trees, each made
|
||||
// absolute by absolutePath and then cleaned. The records hold real
|
||||
// paths, with no "." or ".." element and no repeated or trailing
|
||||
// slash, so a given path is cleaned to match them. No path is looked up
|
||||
// on the filesystem.
|
||||
func reportPaths(args []string) ([]string, error) {
|
||||
paths := make([]string, 0, len(args))
|
||||
|
||||
for _, a := range args {
|
||||
abs, err := absolutePath(a)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve path %s: %w", escapePath(a), err)
|
||||
}
|
||||
|
||||
paths = append(paths, filepath.Clean(abs))
|
||||
}
|
||||
|
||||
return paths, nil
|
||||
}
|
||||
|
||||
// inReportPaths reports whether path is one of paths or lies below one
|
||||
// of them. Each of paths is clean and absolute, as reportPaths returns
|
||||
// it. With no paths, a report covers the whole database, so every path
|
||||
// is in it.
|
||||
func inReportPaths(path string, paths []string) bool {
|
||||
if len(paths) == 0 {
|
||||
return true
|
||||
}
|
||||
|
||||
for _, p := range paths {
|
||||
// Only the root directory's clean path ends in "/".
|
||||
dir := strings.TrimSuffix(p, "/") + "/"
|
||||
if path == p || strings.HasPrefix(path, dir) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// escapePath returns a path as it is written in a report column (README
|
||||
// "Report output format"): a backslash, tab, newline or carriage return
|
||||
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
||||
|
||||
@@ -80,7 +80,7 @@ func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
_, err := readReport(t.Context(), db, "", nil, &out, nil)
|
||||
_, err := readReport(t.Context(), db, "", &out, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -126,98 +126,6 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
|
||||
return dupeGroups(t, db)
|
||||
}
|
||||
|
||||
// TestRunReportPaths runs report with paths on three copies of one
|
||||
// file, two of them under /srv/photos, and a fourth copy in
|
||||
// /srv/photos2. Only the records at or below the given paths form the
|
||||
// groups and the totals, however each path is written.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestRunReportPaths(t *testing.T) {
|
||||
h := testHash("h")
|
||||
a, x, y, z := "/a/copy", "/srv/photos/x", "/srv/photos/y", "/srv/photos2/z"
|
||||
|
||||
recs := make([]scanRec, 0, 4)
|
||||
for _, p := range []string{a, x, y, z} {
|
||||
recs = append(recs,
|
||||
scanRec{size: 7, head: h, tail: h, content: h, path: p})
|
||||
}
|
||||
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
header := "first\tdupe\tsize\n"
|
||||
row := func(first, dupe string) string {
|
||||
return first + "\t" + dupe + "\t7\n"
|
||||
}
|
||||
photos := header + row(x, y)
|
||||
|
||||
cases := map[string]struct {
|
||||
paths []string
|
||||
want string
|
||||
summary string
|
||||
}{
|
||||
"directory": {[]string{"/srv/photos"}, photos,
|
||||
"1 duplicate groups, 1 dupe files, 7 B reclaimable"},
|
||||
"trailing slash": {[]string{"/srv/photos/"}, photos, ""},
|
||||
"dot and dot-dot": {[]string{"/srv/./a/../photos"}, photos, ""},
|
||||
"files": {[]string{a, z}, header + row(a, z), ""},
|
||||
"overlapping": {[]string{"/srv", "/srv/photos"},
|
||||
header + row(x, y) + row(x, z), ""},
|
||||
"root": {[]string{"/"},
|
||||
header + row(a, x) + row(a, y) + row(a, z), ""},
|
||||
"no record": {[]string{"/srv/photo"}, header,
|
||||
"0 duplicate groups, 0 dupe files, 0 B reclaimable"},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run(append([]string{cmdReport}, tc.paths...),
|
||||
&stdout, os.Stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(report %q) = %d, want %d; stderr: %s",
|
||||
tc.paths, code, exitOK, stderr())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != tc.want {
|
||||
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||
}
|
||||
|
||||
// The records read are all four, whatever the paths.
|
||||
summary := "report: 4 records read, " + tc.summary + "\n"
|
||||
if got := stderr(); tc.summary != "" &&
|
||||
!strings.HasSuffix(got, summary) {
|
||||
t.Errorf("stderr = %q, want it to end %q", got, summary)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
//nolint:paralleltest // t.Chdir changes the process-wide working directory
|
||||
func TestReportPathsMakesRelativePathsAbsolute(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Chdir(dir)
|
||||
|
||||
wd, err := absolutePath(".")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got, err := reportPaths([]string{"photos/", "a/../b", "/c//d"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
want := []string{
|
||||
filepath.Join(wd, "photos"), filepath.Join(wd, "b"), "/c/d",
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("reportPaths = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReportEscapesPaths(t *testing.T) {
|
||||
t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
|
||||
|
||||
@@ -264,7 +172,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
|
||||
// Counts the records read, and draws nothing.
|
||||
prog := &progress{now: time.Now}
|
||||
|
||||
_, err = readReport(t.Context(), db, "", nil, failingWriter{}, prog)
|
||||
_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
|
||||
if !errors.Is(err, errWriteFailed) ||
|
||||
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
||||
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
|
||||
@@ -358,7 +266,7 @@ func TestReadReportOnTerminal(t *testing.T) {
|
||||
waitingFor: countingRecords,
|
||||
}, terminalWidth, time.Hour, time.Now)
|
||||
|
||||
_, err = readReport(t.Context(), db, "", nil, os.Stderr, p)
|
||||
_, err = readReport(t.Context(), db, "", os.Stderr, p)
|
||||
|
||||
p.finish()
|
||||
|
||||
@@ -526,27 +434,6 @@ func TestDupeGroups(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
|
||||
// duplicate group, while 1-byte files still do.
|
||||
func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e, x := sha256.Sum256(nil), testHash("x")
|
||||
recs := []scanRec{
|
||||
{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
|
||||
{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
|
||||
{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
|
||||
}
|
||||
|
||||
groups := dupeGroupsOf(t, recs)
|
||||
if len(groups) != 1 || groups[0].size != 1 ||
|
||||
!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
|
||||
t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDupeGroupsContentSeparates(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
@@ -1185,12 +1185,11 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
groups := dupeGroups(t, db)
|
||||
if len(groups) != 4 {
|
||||
t.Fatalf("len(groups) = %d, want 4", len(groups))
|
||||
if len(groups) != 5 {
|
||||
t.Fatalf("len(groups) = %d, want 5", len(groups))
|
||||
}
|
||||
|
||||
// The empty files empty1 and empty2 form no group.
|
||||
wantSizes := []int64{3000, 2000, 100, 1}
|
||||
wantSizes := []int64{3000, 2000, 100, 1, 0}
|
||||
for i, g := range groups {
|
||||
if g.size != wantSizes[i] {
|
||||
t.Errorf("groups[%d].size = %d, want %d",
|
||||
@@ -2368,8 +2367,8 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||
report := runStdout(t, cmdReport)
|
||||
if strings.Contains(report, "db.sqlite") ||
|
||||
strings.Contains(report, hardLink) ||
|
||||
!strings.Contains(report, filepath.Join(tree, "b")) {
|
||||
t.Errorf("report = %q, want a and b and none of "+
|
||||
!strings.Contains(report, filepath.Join(tree, "empty2")) {
|
||||
t.Errorf("report = %q, want the empty files and none of "+
|
||||
"the database's files", report)
|
||||
}
|
||||
})
|
||||
|
||||
+12
-46
@@ -37,9 +37,8 @@ func treesPhase(name string) string {
|
||||
|
||||
// runTrees implements the trees subcommand: it reads every record from
|
||||
// the database in path order, reconstructs the directory hierarchy from
|
||||
// the paths of the records at or below the given paths (of every record
|
||||
// when none is given), computes a Merkle-style digest per directory,
|
||||
// and prints maximal duplicate-tree groups as TSV on stdout. It never
|
||||
// the record paths, computes a Merkle-style digest per directory, and
|
||||
// prints maximal duplicate-tree groups as TSV on stdout. It never
|
||||
// touches the scanned filesystem; its only I/O is the database, stdout,
|
||||
// and stderr. It names the database on stderr before it opens it. Any
|
||||
// database problem, including a missing database, is fatal.
|
||||
@@ -50,12 +49,7 @@ func treesPhase(name string) string {
|
||||
// hierarchy), group (group the directories by digest) and write (write
|
||||
// the groups to stdout). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
paths, err := reportPaths(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -84,7 +78,7 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
label: treesPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db",
|
||||
})
|
||||
records, super, allDirs, err := readTree(ctx, db, paths, prog)
|
||||
records, super, allDirs, err := readTree(ctx, db, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -123,17 +117,10 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
}
|
||||
|
||||
// readTree gives prog the number of records in db as its total, then
|
||||
// reads every record in path order, counting each on prog. It adds the
|
||||
// records that inReportPaths selects from paths to the directory
|
||||
// hierarchy. It returns the number of records read, the super-root and
|
||||
// every directory at or below one of paths.
|
||||
//
|
||||
// A directory above every one of paths holds only the selected records
|
||||
// under it, so its digest does not describe it. It is left out of the
|
||||
// directories returned. A directory whose parent is left out gets the
|
||||
// super-root as its parent, so suppressed never judges its group by
|
||||
// that parent's digest.
|
||||
func readTree(ctx context.Context, db *sql.DB, paths []string,
|
||||
// reads every record in path order into the directory hierarchy,
|
||||
// counting each on prog. It returns the number of records read, the
|
||||
// super-root and every directory.
|
||||
func readTree(ctx context.Context, db *sql.DB,
|
||||
prog *progress,
|
||||
) (int, *treeNode, []*treeNode, error) {
|
||||
records := 0
|
||||
@@ -143,10 +130,7 @@ func readTree(ctx context.Context, db *sql.DB, paths []string,
|
||||
records++
|
||||
|
||||
prog.increment()
|
||||
|
||||
if inReportPaths(r.path, paths) {
|
||||
tree.add(r)
|
||||
}
|
||||
tree.add(r)
|
||||
})
|
||||
if err != nil {
|
||||
return 0, nil, nil, err
|
||||
@@ -154,21 +138,7 @@ func readTree(ctx context.Context, db *sql.DB, paths []string,
|
||||
|
||||
super, allDirs := tree.finish()
|
||||
|
||||
dirs := allDirs[:0]
|
||||
|
||||
for _, d := range allDirs {
|
||||
if !inReportPaths(d.path, paths) {
|
||||
continue
|
||||
}
|
||||
|
||||
if !inReportPaths(d.parent.path, paths) {
|
||||
d.parent = super
|
||||
}
|
||||
|
||||
dirs = append(dirs, d)
|
||||
}
|
||||
|
||||
return records, super, dirs, nil
|
||||
return records, super, allDirs, nil
|
||||
}
|
||||
|
||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||
@@ -331,9 +301,7 @@ func fileEntry(name string, r scanRec) string {
|
||||
// collectTreeGroups groups directories by digest, counting each one on
|
||||
// prog as it is grouped, and returns every maximal group with two or
|
||||
// more members, each group's members sorted by path, groups ordered by
|
||||
// tree size descending then by first path ascending. A group whose
|
||||
// trees hold 0 bytes is left out, since removing one frees nothing. The
|
||||
// groups it implies hold 0 bytes too, so leaving it out hides nothing.
|
||||
// tree size descending then by first path ascending.
|
||||
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||
prog *progress,
|
||||
) [][]*treeNode {
|
||||
@@ -347,9 +315,7 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||
var dupes [][]*treeNode
|
||||
|
||||
for _, g := range groups {
|
||||
// The members share a digest, so they share a total size.
|
||||
if len(g) < minGroupSize || g[0].totalSize == 0 ||
|
||||
suppressed(g, super) {
|
||||
if len(g) < minGroupSize || suppressed(g, super) {
|
||||
continue
|
||||
}
|
||||
|
||||
|
||||
@@ -2,7 +2,6 @@ package sfdupes //nolint:testpackage // checks the unexported treeNode results
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"os"
|
||||
"slices"
|
||||
@@ -212,55 +211,6 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesPaths runs trees with paths. /x/a and /y/a are copies,
|
||||
// and so are /y/a and /srv/b. /x and /y also hold files that differ.
|
||||
// Given /x/a and /y/a, trees reads /x and /y as holding one copy each
|
||||
// of a, yet neither is reported: only the directories at or below a
|
||||
// given path are.
|
||||
func TestRunTreesPaths(t *testing.T) {
|
||||
f, g, k := testHash("f"), testHash("g"), testHash("k")
|
||||
recs := []scanRec{
|
||||
{size: 10, head: f, tail: f, content: f, path: "/srv/b/f"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/x/a/f"},
|
||||
{size: 20, head: g, tail: g, content: g, path: "/x/g"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/y/a/f"},
|
||||
{size: 30, head: k, tail: k, content: k, path: "/y/k"},
|
||||
}
|
||||
|
||||
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
|
||||
|
||||
cases := map[string]struct {
|
||||
paths []string
|
||||
want string
|
||||
}{
|
||||
"none": {nil, header + "/srv/b\t/x/a\t1\t10\t0\n" +
|
||||
"/srv/b\t/y/a\t1\t10\t0\n"},
|
||||
"two copies": {[]string{"/x/a", "/y/a"},
|
||||
header + "/x/a\t/y/a\t1\t10\t0\n"},
|
||||
"parents": {[]string{"/x", "/y"}, header + "/x/a\t/y/a\t1\t10\t0\n"},
|
||||
"one copy": {[]string{"/srv/b"}, header},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := run(append([]string{cmdTrees}, tc.paths...),
|
||||
&stdout, &stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(trees %q) = %d, want %d; stderr: %s",
|
||||
tc.paths, code, exitOK, stderr.String())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != tc.want {
|
||||
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
||||
// take different space on disk. The row's disk_size is the sum over the
|
||||
// dupe copy's files, those in its subdirectory included.
|
||||
@@ -485,35 +435,6 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
|
||||
// holding only empty files form no group, and that an empty file still
|
||||
// counts toward the digest of a tree that holds other files.
|
||||
func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
|
||||
sha256.Sum256(nil)
|
||||
recs := []scanRec{
|
||||
{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
|
||||
// Without the empty file, /p/x3 is no copy of /p/x1.
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
|
||||
}
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||
|
||||
want := [][]string{{"/p/x1", "/p/x2"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
t.Fatalf("groups = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
|
||||
Reference in new issue
Block a user