Add sfdupes dwim: scan / then print duplicate trees (closes #154)
check / check (push) Waiting to run

sfdupes dwim, as sneak ordered: with no arguments it runs what scan / runs with no flags, default excludes on, then what trees runs, printing the duplicate trees largest first on stdout. Its last line, on stderr, says that sfdupes trees prints the report again. It needs no setup beyond what scan creates. SIGINT or SIGTERM during the scan stops it as it stops scan, exit 1, with no report; during the report a signal ends it at once, as it ends trees. A positional argument is a usage error, exit 2.

Model: opus-5-5
This commit was merged in pull request #156.
This commit is contained in:
2026-11-06 20:27:24 +01:00
parent 24b0fc486d
commit da75b713aa
6 changed files with 179 additions and 9 deletions
+39 -6
View File
@@ -17,12 +17,13 @@ gigabyte. A large file is never read in full. A match between two large files is
a strong candidate, not proof.
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every
file takes too long. It has four subcommands. `scan` walks the filesystem and
file takes too long. It has five subcommands. `scan` walks the filesystem and
records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` read the database alone and
can run at any time.
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
first look at a new machine with one command.
## Getting Started
@@ -247,12 +248,15 @@ In order:
- `report` prints the file-level duplicate report from the database.
- `trees` prints the duplicate-tree report from the database.
- `info` prints the database's details.
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
report.
```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
sfdupes info
sfdupes dwim
sfdupes --version
sfdupes [command] --help
```
@@ -800,6 +804,33 @@ The summary line on stderr gives:
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
```
### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments and no flags of
its own. Like `scan`, it should run as root, so that it can read every file.
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
excludes and crosses into other filesystems. `--workers` keeps its default. The
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
what `sfdupes trees` runs. The largest trees come first.
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
`scan` prints, then what `trees` prints, then one last line:
```
dwim: run "sfdupes trees" to print this report again
```
The database keeps the scan, so `sfdupes trees` prints the same report later,
even after the terminal is gone.
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
`dwim` then prints no report and exits 1. A fatal scan error also ends it before
the report, with the same message and exit code as `scan`. Warnings for skipped
files do not stop it. `dwim` stops catching signals when the scan ends, so a
signal during the report ends it at once, as it ends `trees`.
### `info` mode
`info` takes no positional arguments and prints the database's details on
@@ -840,7 +871,7 @@ scan is writing. `info` writes its lines after the `count` display has ended.
### Database
All four subcommands work on one SQLite database file.
All five subcommands work on one SQLite database file.
#### Location
@@ -856,6 +887,8 @@ scan: new database /var/lib/sfdupes/db.sqlite (default)
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
```
`dwim` prints the `scan` line first, and the `trees` line after the scan.
A relative path is printed with the working directory in front of it. The
working directory is printed as `pwd -P` prints it, with its symlinks resolved.
The rest of the path is printed as set, so a `..` in it stays. The line names
@@ -1139,8 +1172,8 @@ showing the count it stopped at.
### Error handling and exit codes
- `0`: success, even when files were skipped with warnings.
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal
error is one of these:
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
stopped by either during its scan. A fatal error is one of these:
- an operand that does not exist, or whose real path `scan` cannot find;
- a list of mounted filesystems that `scan` cannot read;
- another `scan` holding the lock (see "The scan lock");
@@ -1161,7 +1194,7 @@ showing the count it stopped at.
`--workers abc`;
- `scan` with no operand;
- `--workers` below 1;
- `report`, `trees` or `info` with a positional argument.
- `report`, `trees`, `info` or `dwim` with a positional argument.
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
stdout, such as a full disk, is reported the same way. Two cases never reach
+5
View File
@@ -29,6 +29,11 @@
Newest first. An entry ends with its date and its issue or branch, where those
are known.
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
prints the report again (2026-11-06,
https://git.eeqj.de/sneak/sfdupes/issues/154)
- `scan` resolves each operand to its real path, symlinks followed, before the
walk. A symlink operand is scanned at its target instead of skipped. With
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"context"
"fmt"
"io"
"os"
"runtime"
)
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
// root" runs with no flags, then what "sfdupes trees" runs, and ends
// with a line on stderr saying how to print the report again. The dwim
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
// does, but only during the scan. An error from the scan, an interrupt
// included, ends it before the report. During the report, a signal
// ends it at once, as it ends trees.
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
scanCtx, stop := interruptContext(ctx)
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
stop()
if err != nil {
return err
}
err = runTrees(ctx, stdout)
if err != nil {
return err
}
fmt.Fprintln(os.Stderr,
`dwim: run "sfdupes trees" to print this report again`)
return nil
}
+83
View File
@@ -0,0 +1,83 @@
package main
import (
"bytes"
"path/filepath"
"strings"
"testing"
)
func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
// dwim scans its root, then prints the trees report, the larger of
// two duplicate-tree pairs first, and ends by saying how to print
// the report again.
t.Setenv(databaseEnv, testDBPath(t))
root := realTempDir(t)
small := []string{
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
}
large := []string{
filepath.Dir(writeFile(t, root, "b1/f.bin", pattern(2, 5000))),
filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000))),
}
var stdout bytes.Buffer
stderr := captureStderr(t)
err := runDwim(t.Context(), root, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
if len(lines) != 3 {
t.Fatalf("stdout = %q, want a header and two groups",
stdout.String())
}
wantFirst := large[0] + "\t" + large[1] + "\t1\t5000\t"
if !strings.HasPrefix(lines[1], wantFirst) {
t.Errorf("first group = %q, want it to start %q", lines[1], wantFirst)
}
wantSecond := small[0] + "\t" + small[1] + "\t1\t300\t"
if !strings.HasPrefix(lines[2], wantSecond) {
t.Errorf("second group = %q, want it to start %q",
lines[2], wantSecond)
}
last := "\ndwim: run \"sfdupes trees\" to print this report again\n"
if got := stderr(); !strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to end %q", got, last)
}
}
func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
// A fatal scan error ends dwim before the report: a root that does
// not exist fails the scan as it fails scan itself.
t.Setenv(databaseEnv, testDBPath(t))
missing := filepath.Join(realTempDir(t), "missing")
var stdout bytes.Buffer
stderr := captureStderr(t)
err := runDwim(t.Context(), missing, &stdout)
if err == nil {
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
}
if got := stdout.String(); got != "" {
t.Errorf("stdout = %q, want no report", got)
}
if got := stderr(); strings.Contains(got, "trees:") ||
strings.Contains(got, "dwim:") {
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
}
}
+14 -3
View File
@@ -14,6 +14,7 @@
// sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv
// sfdupes info
// sfdupes dwim
// sfdupes --version
//
// See README.md for the complete specification.
@@ -45,6 +46,7 @@ const (
cmdReport = "report"
cmdTrees = "trees"
cmdInfo = "info"
cmdDwim = "dwim"
)
// errNoSubcommand is returned by the root command when it is invoked
@@ -96,8 +98,8 @@ func main() {
// exit code. It is the program's single exit point: the subcommands
// return their errors instead of exiting, so every deferred cleanup —
// above all closing the database, which checkpoints the SQLite WAL —
// runs before the process ends. The report, trees and info subcommands
// write their data to stdout.
// runs before the process ends. The report, trees, info and dwim
// subcommands write their data to stdout.
func run(args []string, stdout, stderr io.Writer) int {
// A nil slice makes cobra fall back to os.Args, which would let a
// test binary's own flags reach the command tree.
@@ -201,7 +203,16 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
dwimCmd := &cobra.Command{
Use: cmdDwim,
Short: "Scan / with the default excludes, then print the duplicate-tree report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", stdout)
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
return root
}
+1
View File
@@ -622,6 +622,7 @@ func TestRunUsageErrors(t *testing.T) {
"report with args": {[]string{cmdReport, "x"}, usageMarker},
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
"unknown subcommand": {[]string{"nope"}, "unknown command"},
}