Add sfdupes dwim: scan / then print duplicate trees (closes #154)
check / check (push) Successful in 1m21s

dwim runs what "sfdupes scan /" runs with no flags, then what
"sfdupes trees" runs, and ends with a stderr line saying that
"sfdupes trees" prints the report again. An interrupt or fatal scan
error ends it before the report, as scan ends. It catches signals only
during the scan, so a signal during the report ends it at once, as it
ends trees.

Model: opus-5-5
This commit is contained in:
2026-11-06 19:21:15 +00:00
parent 24b0fc486d
commit e726c18ac7
6 changed files with 179 additions and 9 deletions
+39 -6
View File
@@ -17,12 +17,13 @@ gigabyte. A large file is never read in full. A match between two large files is
a strong candidate, not proof. a strong candidate, not proof.
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every `sfdupes` is meant for multi-terabyte servers, where reading every byte of every
file takes too long. It has four subcommands. `scan` walks the filesystem and file takes too long. It has five subcommands. `scan` walks the filesystem and
records the files it finds in an SQLite database. SQLite keeps a whole database records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` read the database alone and system's job scheduler. `report`, `trees` and `info` read the database alone and
can run at any time. can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
first look at a new machine with one command.
## Getting Started ## Getting Started
@@ -247,12 +248,15 @@ In order:
- `report` prints the file-level duplicate report from the database. - `report` prints the file-level duplicate report from the database.
- `trees` prints the duplicate-tree report from the database. - `trees` prints the duplicate-tree report from the database.
- `info` prints the database's details. - `info` prints the database's details.
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
report.
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
sfdupes report > dupes.tsv sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv sfdupes trees > dupetrees.tsv
sfdupes info sfdupes info
sfdupes dwim
sfdupes --version sfdupes --version
sfdupes [command] --help sfdupes [command] --help
``` ```
@@ -800,6 +804,33 @@ The summary line on stderr gives:
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
``` ```
### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments and no flags of
its own. Like `scan`, it should run as root, so that it can read every file.
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
excludes and crosses into other filesystems. `--workers` keeps its default. The
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
what `sfdupes trees` runs. The largest trees come first.
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
`scan` prints, then what `trees` prints, then one last line:
```
dwim: run "sfdupes trees" to print this report again
```
The database keeps the scan, so `sfdupes trees` prints the same report later,
even after the terminal is gone.
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
`dwim` then prints no report and exits 1. A fatal scan error also ends it before
the report, with the same message and exit code as `scan`. Warnings for skipped
files do not stop it. `dwim` stops catching signals when the scan ends, so a
signal during the report ends it at once, as it ends `trees`.
### `info` mode ### `info` mode
`info` takes no positional arguments and prints the database's details on `info` takes no positional arguments and prints the database's details on
@@ -840,7 +871,7 @@ scan is writing. `info` writes its lines after the `count` display has ended.
### Database ### Database
All four subcommands work on one SQLite database file. All five subcommands work on one SQLite database file.
#### Location #### Location
@@ -856,6 +887,8 @@ scan: new database /var/lib/sfdupes/db.sqlite (default)
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE) trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
``` ```
`dwim` prints the `scan` line first, and the `trees` line after the scan.
A relative path is printed with the working directory in front of it. The A relative path is printed with the working directory in front of it. The
working directory is printed as `pwd -P` prints it, with its symlinks resolved. working directory is printed as `pwd -P` prints it, with its symlinks resolved.
The rest of the path is printed as set, so a `..` in it stays. The line names The rest of the path is printed as set, so a `..` in it stays. The line names
@@ -1139,8 +1172,8 @@ showing the count it stopped at.
### Error handling and exit codes ### Error handling and exit codes
- `0`: success, even when files were skipped with warnings. - `0`: success, even when files were skipped with warnings.
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal - `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
error is one of these: stopped by either during its scan. A fatal error is one of these:
- an operand that does not exist, or whose real path `scan` cannot find; - an operand that does not exist, or whose real path `scan` cannot find;
- a list of mounted filesystems that `scan` cannot read; - a list of mounted filesystems that `scan` cannot read;
- another `scan` holding the lock (see "The scan lock"); - another `scan` holding the lock (see "The scan lock");
@@ -1161,7 +1194,7 @@ showing the count it stopped at.
`--workers abc`; `--workers abc`;
- `scan` with no operand; - `scan` with no operand;
- `--workers` below 1; - `--workers` below 1;
- `report`, `trees` or `info` with a positional argument. - `report`, `trees`, `info` or `dwim` with a positional argument.
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
stdout, such as a full disk, is reported the same way. Two cases never reach stdout, such as a full disk, is reported the same way. Two cases never reach
+5
View File
@@ -29,6 +29,11 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
prints the report again (2026-11-06,
https://git.eeqj.de/sneak/sfdupes/issues/154)
- `scan` resolves each operand to its real path, symlinks followed, before the - `scan` resolves each operand to its real path, symlinks followed, before the
walk. A symlink operand is scanned at its target instead of skipped. With walk. A symlink operand is scanned at its target instead of skipped. With
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub` `/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
+37
View File
@@ -0,0 +1,37 @@
package main
import (
"context"
"fmt"
"io"
"os"
"runtime"
)
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
// root" runs with no flags, then what "sfdupes trees" runs, and ends
// with a line on stderr saying how to print the report again. The dwim
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
// does, but only during the scan. An error from the scan, an interrupt
// included, ends it before the report. During the report, a signal
// ends it at once, as it ends trees.
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
scanCtx, stop := interruptContext(ctx)
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
stop()
if err != nil {
return err
}
err = runTrees(ctx, stdout)
if err != nil {
return err
}
fmt.Fprintln(os.Stderr,
`dwim: run "sfdupes trees" to print this report again`)
return nil
}
+83
View File
@@ -0,0 +1,83 @@
package main
import (
"bytes"
"path/filepath"
"strings"
"testing"
)
func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
// dwim scans its root, then prints the trees report, the larger of
// two duplicate-tree pairs first, and ends by saying how to print
// the report again.
t.Setenv(databaseEnv, testDBPath(t))
root := realTempDir(t)
small := []string{
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
}
large := []string{
filepath.Dir(writeFile(t, root, "b1/f.bin", pattern(2, 5000))),
filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000))),
}
var stdout bytes.Buffer
stderr := captureStderr(t)
err := runDwim(t.Context(), root, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
if len(lines) != 3 {
t.Fatalf("stdout = %q, want a header and two groups",
stdout.String())
}
wantFirst := large[0] + "\t" + large[1] + "\t1\t5000\t"
if !strings.HasPrefix(lines[1], wantFirst) {
t.Errorf("first group = %q, want it to start %q", lines[1], wantFirst)
}
wantSecond := small[0] + "\t" + small[1] + "\t1\t300\t"
if !strings.HasPrefix(lines[2], wantSecond) {
t.Errorf("second group = %q, want it to start %q",
lines[2], wantSecond)
}
last := "\ndwim: run \"sfdupes trees\" to print this report again\n"
if got := stderr(); !strings.HasSuffix(got, last) {
t.Errorf("stderr = %q, want it to end %q", got, last)
}
}
func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
// A fatal scan error ends dwim before the report: a root that does
// not exist fails the scan as it fails scan itself.
t.Setenv(databaseEnv, testDBPath(t))
missing := filepath.Join(realTempDir(t), "missing")
var stdout bytes.Buffer
stderr := captureStderr(t)
err := runDwim(t.Context(), missing, &stdout)
if err == nil {
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
}
if got := stdout.String(); got != "" {
t.Errorf("stdout = %q, want no report", got)
}
if got := stderr(); strings.Contains(got, "trees:") ||
strings.Contains(got, "dwim:") {
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
}
}
+14 -3
View File
@@ -14,6 +14,7 @@
// sfdupes report > dupes.tsv // sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv // sfdupes trees > dupetrees.tsv
// sfdupes info // sfdupes info
// sfdupes dwim
// sfdupes --version // sfdupes --version
// //
// See README.md for the complete specification. // See README.md for the complete specification.
@@ -45,6 +46,7 @@ const (
cmdReport = "report" cmdReport = "report"
cmdTrees = "trees" cmdTrees = "trees"
cmdInfo = "info" cmdInfo = "info"
cmdDwim = "dwim"
) )
// errNoSubcommand is returned by the root command when it is invoked // errNoSubcommand is returned by the root command when it is invoked
@@ -96,8 +98,8 @@ func main() {
// exit code. It is the program's single exit point: the subcommands // exit code. It is the program's single exit point: the subcommands
// return their errors instead of exiting, so every deferred cleanup — // return their errors instead of exiting, so every deferred cleanup —
// above all closing the database, which checkpoints the SQLite WAL — // above all closing the database, which checkpoints the SQLite WAL —
// runs before the process ends. The report, trees and info subcommands // runs before the process ends. The report, trees, info and dwim
// write their data to stdout. // subcommands write their data to stdout.
func run(args []string, stdout, stderr io.Writer) int { func run(args []string, stdout, stderr io.Writer) int {
// A nil slice makes cobra fall back to os.Args, which would let a // A nil slice makes cobra fall back to os.Args, which would let a
// test binary's own flags reach the command tree. // test binary's own flags reach the command tree.
@@ -201,7 +203,16 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}), }),
} }
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd) dwimCmd := &cobra.Command{
Use: cmdDwim,
Short: "Scan / with the default excludes, then print the duplicate-tree report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", stdout)
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
return root return root
} }
+1
View File
@@ -622,6 +622,7 @@ func TestRunUsageErrors(t *testing.T) {
"report with args": {[]string{cmdReport, "x"}, usageMarker}, "report with args": {[]string{cmdReport, "x"}, usageMarker},
"trees with args": {[]string{cmdTrees, "x"}, usageMarker}, "trees with args": {[]string{cmdTrees, "x"}, usageMarker},
"info with args": {[]string{cmdInfo, "x"}, usageMarker}, "info with args": {[]string{cmdInfo, "x"}, usageMarker},
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker}, "unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
"unknown subcommand": {[]string{"nope"}, "unknown command"}, "unknown subcommand": {[]string{"nope"}, "unknown command"},
} }