Add sfdupes dwim: scan / then print duplicate trees (closes #154) #156

Merged
clawbot merged 1 commits from issue-154-dwim into next 2026-11-06 20:27:25 +01:00
6 changed files with 179 additions and 9 deletions
Showing only changes of commit e726c18ac7 - Show all commits
+39 -6
View File
@@ -17,12 +17,13 @@ gigabyte. A large file is never read in full. A match between two large files is
a strong candidate, not proof.
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every
file takes too long. It has four subcommands. `scan` walks the filesystem and
file takes too long. It has five subcommands. `scan` walks the filesystem and
records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` read the database alone and
can run at any time.
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
first look at a new machine with one command.
## Getting Started
@@ -247,12 +248,15 @@ In order:
- `report` prints the file-level duplicate report from the database.
- `trees` prints the duplicate-tree report from the database.
- `info` prints the database's details.
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
report.
```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
sfdupes info
sfdupes dwim
sfdupes --version
sfdupes [command] --help
```
@@ -800,6 +804,33 @@ The summary line on stderr gives:
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
```
### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments and no flags of
its own. Like `scan`, it should run as root, so that it can read every file.
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
excludes and crosses into other filesystems. `--workers` keeps its default. The
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
what `sfdupes trees` runs. The largest trees come first.
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
`scan` prints, then what `trees` prints, then one last line:
```
dwim: run "sfdupes trees" to print this report again
```
The database keeps the scan, so `sfdupes trees` prints the same report later,
even after the terminal is gone.
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
`dwim` then prints no report and exits 1. A fatal scan error also ends it before
the report, with the same message and exit code as `scan`. Warnings for skipped
files do not stop it. `dwim` stops catching signals when the scan ends, so a
signal during the report ends it at once, as it ends `trees`.
### `info` mode
`info` takes no positional arguments and prints the database's details on
@@ -840,7 +871,7 @@ scan is writing. `info` writes its lines after the `count` display has ended.
### Database
All four subcommands work on one SQLite database file.
All five subcommands work on one SQLite database file.
#### Location
@@ -856,6 +887,8 @@ scan: new database /var/lib/sfdupes/db.sqlite (default)
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
```
`dwim` prints the `scan` line first, and the `trees` line after the scan.
A relative path is printed with the working directory in front of it. The
working directory is printed as `pwd -P` prints it, with its symlinks resolved.
The rest of the path is printed as set, so a `..` in it stays. The line names
@@ -1139,8 +1172,8 @@ showing the count it stopped at.
### Error handling and exit codes
- `0`: success, even when files were skipped with warnings.
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal
error is one of these:
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
stopped by either during its scan. A fatal error is one of these:
- an operand that does not exist, or whose real path `scan` cannot find;
- a list of mounted filesystems that `scan` cannot read;
- another `scan` holding the lock (see "The scan lock");
@@ -1161,7 +1194,7 @@ showing the count it stopped at.
`--workers abc`;
- `scan` with no operand;
- `--workers` below 1;
- `report`, `trees` or `info` with a positional argument.
- `report`, `trees`, `info` or `dwim` with a positional argument.
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
stdout, such as a full disk, is reported the same way. Two cases never reach
+5
View File
@@ -29,6 +29,11 @@
Newest first. An entry ends with its date and its issue or branch, where those
are known.
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
prints the report again (2026-11-06,
https://git.eeqj.de/sneak/sfdupes/issues/154)
- `scan` resolves each operand to its real path, symlinks followed, before the
walk. A symlink operand is scanned at its target instead of skipped. With
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"context"
"fmt"
"io"
"os"
"runtime"
)
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
// root" runs with no flags, then what "sfdupes trees" runs, and ends
// with a line on stderr saying how to print the report again. The dwim
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
// does, but only during the scan. An error from the scan, an interrupt
// included, ends it before the report. During the report, a signal
// ends it at once, as it ends trees.
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
scanCtx, stop := interruptContext(ctx)
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
stop()
if err != nil {
return err
}
err = runTrees(ctx, stdout)
if err != nil {
return err
}
fmt.Fprintln(os.Stderr,
`dwim: run "sfdupes trees" to print this report again`)
return nil
}
+83
View File
@@ -0,0 +1,83 @@
package main
import (
"bytes"
"path/filepath"
"strings"
"testing"
)
func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
// dwim scans its root, then prints the trees report, the larger of
// two duplicate-tree pairs first, and ends by saying how to print
// the report again.
t.Setenv(databaseEnv, testDBPath(t))
root := realTempDir(t)
small := []string{
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
}
large := []string{
filepath.Dir(writeFile(t, root, "b1/f.bin", pattern(2, 5000))),
filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000))),
}
var stdout bytes.Buffer
stderr := captureStderr(t)
err := runDwim(t.Context(), root, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
if len(lines) != 3 {
t.Fatalf("stdout = %q, want a header and two groups",
stdout.String())
}
wantFirst := large[0] + "\t" + large[1] + "\t1\t5000\t"
if !strings.HasPrefix(lines[1], wantFirst) {
t.Errorf("first group = %q, want it to start %q", lines[1], wantFirst)
}
wantSecond := small[0] + "\t" + small[1] + "\t1\t300\t"
if !strings.HasPrefix(lines[2], wantSecond) {
t.Errorf("second group = %q, want it to start %q",
lines[2], wantSecond)
}
last := "\ndwim: run \"sfdupes trees\" to print this report again\n"
if got := stderr(); !strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to end %q", got, last)
}
}
func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
// A fatal scan error ends dwim before the report: a root that does
// not exist fails the scan as it fails scan itself.
t.Setenv(databaseEnv, testDBPath(t))
missing := filepath.Join(realTempDir(t), "missing")
var stdout bytes.Buffer
stderr := captureStderr(t)
err := runDwim(t.Context(), missing, &stdout)
if err == nil {
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
}
if got := stdout.String(); got != "" {
t.Errorf("stdout = %q, want no report", got)
}
if got := stderr(); strings.Contains(got, "trees:") ||
strings.Contains(got, "dwim:") {
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
}
}
+14 -3
View File
@@ -14,6 +14,7 @@
// sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv
// sfdupes info
// sfdupes dwim
// sfdupes --version
//
// See README.md for the complete specification.
@@ -45,6 +46,7 @@ const (
cmdReport = "report"
cmdTrees = "trees"
cmdInfo = "info"
cmdDwim = "dwim"
)
// errNoSubcommand is returned by the root command when it is invoked
@@ -96,8 +98,8 @@ func main() {
// exit code. It is the program's single exit point: the subcommands
// return their errors instead of exiting, so every deferred cleanup —
// above all closing the database, which checkpoints the SQLite WAL —
// runs before the process ends. The report, trees and info subcommands
// write their data to stdout.
// runs before the process ends. The report, trees, info and dwim
// subcommands write their data to stdout.
func run(args []string, stdout, stderr io.Writer) int {
// A nil slice makes cobra fall back to os.Args, which would let a
// test binary's own flags reach the command tree.
@@ -201,7 +203,16 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
dwimCmd := &cobra.Command{
Use: cmdDwim,
Short: "Scan / with the default excludes, then print the duplicate-tree report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", stdout)
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
return root
}
+1
View File
@@ -622,6 +622,7 @@ func TestRunUsageErrors(t *testing.T) {
"report with args": {[]string{cmdReport, "x"}, usageMarker},
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
"unknown subcommand": {[]string{"nope"}, "unknown command"},
}