Compare commits

..
3 Commits
Author SHA1 Message Date
sneak 5cb336039c Keep a non-root user's database in the XDG state directory (closes #171)
check / check (push) Canceled after 0s
Without SFDUPES_DATABASE, a user other than root now uses
sfdupes/db.sqlite in the XDG state directory (~/.local/state by
default) instead of failing on /var/lib/sfdupes. scan creates the
missing directories there with mode 0700, as the XDG Base Directory
specification asks. An XDG_STATE_HOME that is not an absolute path is
a fatal error for every subcommand. Root keeps
/var/lib/sfdupes/db.sqlite. The help of sfdupes and of scan, and
README, say where the database is. Adds github.com/adrg/xdg, the
decided XDG library.

Model: opus-5-5
2026-11-11 08:01:23 +00:00
clawbot a9ab6124ae dwim: take the scan flags (closes #162) (#193)
check / check (push) Canceled after 0s
2026-11-11 08:37:18 +01:00
clawbot db56fd0e4c Say in --help where the database is and what each report prints (closes #177) (#189)
check / check (push) Canceled after 0s
2026-11-11 07:22:28 +01:00
10 changed files with 472 additions and 139 deletions

No files matched your search

+37 -26
View File
@@ -22,9 +22,8 @@ records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` work from the database and
can run at any time. `report` and `trees` leave out a copy that was deleted
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first
look at a new machine with one command.
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
first look at a new machine with one command.
## Getting Started
@@ -257,15 +256,14 @@ In order:
- `report` prints the file-level duplicate report from the database.
- `trees` prints the duplicate-tree report from the database.
- `info` prints the database's details.
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
report.
- `dwim` scans `/`, then prints the duplicate-tree report.
```
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv
sfdupes info
sfdupes dwim
sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]...
sfdupes --version
sfdupes [command] --help
```
@@ -295,20 +293,20 @@ The reference below is also the program's specification.
database is at its default path (see "Location"). There is no command-line
flag for it. The path names the file exactly, whatever characters it holds. A
relative path is relative to the working directory.
- `--workers N` (`scan`) sets how many directories `scan` reads at once while it
lists the trees. It also sets how many files `scan` reads at once while it
hashes them. The default is the number of CPUs. `N` must be at least 1. A
smaller value is a usage error, reported in one line on stderr with exit 2
before anything is scanned.
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
directories on other filesystems, as the same flag does for `du` and `rsync`.
By default the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
Those are the system and temporary directories the walk otherwise leaves out,
such as `/var/tmp` on Linux (see "Rules for the walk").
- `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
it. `PATH` may be a directory or a file. The flag may be given more than once.
The walk knows `PATH` by its device ID and inode number, as it knows the
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once
while it lists the trees. It also sets how many files `scan` reads at once
while it hashes them. The default is the number of CPUs. `N` must be at
least 1. A smaller value is a usage error, reported in one line on stderr with
exit 2 before anything is scanned.
- `-x`, `--one-file-system` (`scan`, `dwim`) keeps the walk of each operand out
of directories on other filesystems, as the same flag does for `du` and
`rsync`. By default the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
excludes. Those are the system and temporary directories the walk otherwise
leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
below it. `PATH` may be a directory or a file. The flag may be given more than
once. The walk knows `PATH` by its device ID and inode number, as it knows the
database files (see "Rules for the walk"). So the walk leaves `PATH` out
whatever path leads to it. That includes a hard link to a file given as
`PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in
@@ -319,8 +317,10 @@ The reference below is also the program's specification.
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
exits 0. It writes nothing to stderr.
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the
default excludes of the system it runs on, and the filesystem types the walk
and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
names the database. The help of `report`, `trees`, `info` and `dwim` says what
each prints. `sfdupes scan --help` and `sfdupes dwim --help` also list the
default excludes of the system they run on, and the filesystem types the walk
never records anything on.
### Duplicate detection
@@ -859,14 +859,25 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments and no flags of
its own. Like `scan`, it should run as root, so that it can read every file.
duplicate trees in one command. It takes no positional arguments. Like `scan`,
it should run as root, so that it can read every file.
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
excludes and crosses into other filesystems. `--workers` keeps its default. The
`dwim` takes these flags of `scan`, with the same meaning (see "Settings"):
- `--workers N`;
- `-x`, `--one-file-system`;
- `--no-default-excludes`;
- `--exclude PATH`, which may be given more than once.
`dwim` runs what `sfdupes scan /` runs with the same flags. With no flags, the
walk skips the default excludes and crosses into other filesystems. The
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
what `sfdupes trees` runs. The largest trees come first.
```
sfdupes dwim -x --exclude /srv/cache
```
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
`scan` prints, then what `trees` prints, then one last line:
+13 -2
View File
@@ -31,8 +31,19 @@ are known.
- Without `SFDUPES_DATABASE`, a user other than root keeps the database in
`sfdupes/db.sqlite` under the XDG state directory, `~/.local/state` by
default. Root keeps `/var/lib/sfdupes/db.sqlite`. `sfdupes --help` says where
the database is (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/171)
default. Root keeps `/var/lib/sfdupes/db.sqlite`. `sfdupes --help` and
`sfdupes scan --help` say so (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/171)
- `dwim` takes the `scan` flags `--workers`, `-x`, `--no-default-excludes` and
`--exclude`, with the same meaning. Both commands register and check them with
the same code (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/162)
- The help of `sfdupes` and of `scan` names the database, its default path and
`SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
order of the rows and the escapes. The help of `info` lists its lines. The
help of `dwim` says that it scans `/`, should run as root and keeps the scan
in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
enters nor records an excluded path, and keeps the records below it. An
+6 -7
View File
@@ -358,8 +358,9 @@ func TestRunScanInterrupted(t *testing.T) {
stderr := captureStderr(t)
dir := buildSmokeTree(t)
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false, nil)
opts := scanOptions{workers: walkCancelWorkers}
err := runScan(t.Context(), []string{dir}, opts)
if err != nil {
t.Fatal(err)
}
@@ -382,8 +383,7 @@ func TestRunScanInterrupted(t *testing.T) {
go func() {
defer close(done)
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
false, false, nil)
err = runScan(cancelledContext(t), []string{dir}, opts)
}()
awaitReturn(t, done, "runScan")
@@ -410,8 +410,7 @@ func TestRunScanInterrupted(t *testing.T) {
got, before)
}
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false, nil)
err = runScan(t.Context(), []string{dir}, opts)
if err != nil {
t.Fatal(err)
}
@@ -437,7 +436,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
dir := buildWalkCancelTree(t)
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
walkCancelWorkers, false, false, nil)
scanOptions{workers: walkCancelWorkers})
if !errors.Is(err, errInterrupted) {
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
err, errInterrupted)
+158 -63
View File
@@ -32,17 +32,83 @@ const (
cmdDwim = "dwim"
)
// databaseHelp is the paragraph of the root and scan help that names
// the database (README "Location").
const databaseHelp = `scan and dwim store what they find in a database file. report,
trees and info read it. The file is $SFDUPES_DATABASE when that is set
and not empty. Otherwise root uses /var/lib/sfdupes/db.sqlite. Any other
user uses sfdupes/db.sqlite in the XDG state directory: $XDG_STATE_HOME,
or ~/.local/state when that is unset or empty
(~/Library/Application Support on macOS). An XDG_STATE_HOME that is
not an absolute path is an error.
`
// escapesHelp is the paragraph of the report and trees help on how
// paths are escaped (README "Report output format").
const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
\\ a backslash
\t a tab
\n a newline
\r a carriage return
`
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
// summary in the help of their subcommands.
const (
reportHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate file. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that file
size the file's size in bytes
Rows are ordered by size, largest first, then by path.
` + escapesHelp
treesHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate directory tree. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that tree
files the number of regular files in one copy
size their total size in bytes
disk_size the bytes the dupe copy takes on disk
Rows are ordered by size, largest first, then by path.
` + escapesHelp
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
value, separated by a tab:
database the path, and where it came from
database file size in bytes, then rounded, as 2147483648 (2.0 GiB)
-wal file size in bytes, then rounded; only if a -wal file exists
modified when the database file last changed, in UTC
schema version the version of the database's layout
records the number of records
size of recorded files their total size, in bytes, then rounded
`
dwimHelp = `dwim runs what "sfdupes scan /" runs with the same flags, then what
"sfdupes trees" runs. Run it as root, so that the scan can read every
file. The scan is stored in the database, as scan stores it, so
"sfdupes trees" prints the report again later.
`
)
// errNoSubcommand is returned by the root command when it is invoked
// without a subcommand. That is a usage error, and the usage text
// cobra prints for it is the whole message.
var errNoSubcommand = errors.New("no subcommand")
// errWorkersBelowOne is the usage error for a scan --workers value
// below 1.
// errWorkersBelowOne is the usage error for a --workers value below 1.
var errWorkersBelowOne = errors.New("--workers must be at least 1")
// errEmptyExcludeValue is the usage error for an empty scan --exclude
// value, as from --exclude "$DIR" with DIR unset. Taken as a path, it
// errEmptyExcludeValue is the usage error for an empty --exclude value,
// as from --exclude "$DIR" with DIR unset. Taken as a path, it
// would name the working directory.
var errEmptyExcludeValue = errors.New("--exclude must not be empty")
@@ -122,14 +188,6 @@ func run(args []string, stdout, stderr io.Writer) int {
}
}
// databaseHelp says in sfdupes --help where the database is.
const databaseHelp = `The database is the file SFDUPES_DATABASE names.
When that is unset or empty, root uses /var/lib/sfdupes/db.sqlite. Any
other user uses sfdupes/db.sqlite in the XDG state directory:
$XDG_STATE_HOME, or ~/.local/state when that is unset or empty
(~/Library/Application Support on macOS). An XDG_STATE_HOME that is
not an absolute path is an error.`
// newRootCommand builds the command tree. Everything on stdout is
// machine-readable data, the version line included; all human-facing
// output (help, usage, errors) goes to stderr.
@@ -148,12 +206,9 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
return nil
})
short := "Find candidate duplicate files by size and head/tail/content SHA-256"
root := &cobra.Command{
Use: "sfdupes",
Short: short,
Long: short + ".\n\n" + databaseHelp,
Short: "Find candidate duplicate files by size and head/tail/content SHA-256",
Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error {
if showVersion {
@@ -204,76 +259,116 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}),
}
dwimCmd := &cobra.Command{
Use: cmdDwim,
Short: "Scan / with the default excludes, then print the duplicate-tree report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", stdout)
}),
}
root.Long = root.Short + ".\n\n" + databaseHelp
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd,
newDwimCommand(stdout))
return root
}
// newScanCommand builds the scan subcommand, with its --workers, -x,
// --no-default-excludes and --exclude flags. Its help lists the default
// excludes of the system it runs on, and the filesystem types it never
// records anything on.
// scanOptions holds the values of the flags that scan and dwim share:
// --workers, -x, --no-default-excludes and --exclude.
type scanOptions struct {
workers int
oneFS bool
noDefaultExcludes bool
excludePaths []string
}
// scanFlagsUsage names the flags that scan and dwim share, for their
// usage lines.
const scanFlagsUsage = "[--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]..."
// addScanFlags adds the flags that scan and dwim share to cmd, which
// stores their values in opts.
func addScanFlags(cmd *cobra.Command, opts *scanOptions) {
cmd.Flags().IntVar(&opts.workers, "workers", runtime.NumCPU(),
"concurrent workers for the walk and hash phases")
cmd.Flags().BoolVarP(&opts.oneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
cmd.Flags().BoolVar(&opts.noDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
// A string array, not a slice, so a comma in a path is not a
// separator.
cmd.Flags().StringArrayVar(&opts.excludePaths, "exclude", nil,
"never enter or record `PATH` or anything below it; repeatable")
}
// check rejects the flag values that are usage errors, before the
// command runs. It is the PreRunE of scan and dwim.
func (o *scanOptions) check(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, o.workers)
if err != nil {
return err
}
return rejectEmptyExcludeValue(cmd, o.excludePaths)
}
// newScanCommand builds the scan subcommand. Its help names the
// database, and lists the default excludes of the system it runs on and
// the filesystem types it never records anything on.
func newScanCommand() *cobra.Command {
var (
scanWorkers int
scanOneFS bool
scanNoDefaultExcludes bool
scanExclude []string
)
var opts scanOptions
short := "Walk trees and synchronize the scan database"
scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]... PATH...",
Use: cmdScan + " " + scanFlagsUsage + " PATH...",
// The Use line already names every flag, so cobra must not
// append its generic "[flags]" to it.
DisableFlagsInUseLine: true,
Short: short,
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1),
PreRunE: func(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, scanWorkers)
if err != nil {
return err
}
return rejectEmptyExcludeValue(cmd, scanExclude)
},
Long: short + ".\n\n" + databaseHelp + "\n" +
defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1),
PreRunE: opts.check,
RunE: runE(func(ctx context.Context, args []string) error {
ctx, stop := interruptContext(ctx)
defer stop()
return runScan(ctx, args, scanWorkers, scanOneFS,
scanNoDefaultExcludes, scanExclude)
return runScan(ctx, args, opts)
}),
}
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
"concurrent workers for the walk and hash phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
// A string array, not a slice, so a comma in a path is not a
// separator.
scanCmd.Flags().StringArrayVar(&scanExclude, "exclude", nil,
"never enter or record `PATH` or anything below it; repeatable")
addScanFlags(scanCmd, &opts)
return scanCmd
}
// checkScanWorkers rejects a scan --workers value below 1. That is a
// usage error reported in one line: cobra prints the returned message
// without the usage text, and run exits 2.
// newDwimCommand builds the dwim subcommand, which takes the scan flags
// with the meaning they have for scan. Its help says what dwim runs,
// then lists the default excludes and the filesystem types the scan
// never records anything on.
func newDwimCommand(stdout io.Writer) *cobra.Command {
var opts scanOptions
short := "Scan /, then print the duplicate-tree report"
dwimCmd := &cobra.Command{
Use: cmdDwim + " " + scanFlagsUsage,
DisableFlagsInUseLine: true,
Short: short,
Long: short + ".\n\n" + dwimHelp + "\n" +
defaultExcludesHelp(runtime.GOOS),
Args: cobra.NoArgs,
PreRunE: opts.check,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", opts, stdout)
}),
}
addScanFlags(dwimCmd, &opts)
return dwimCmd
}
// checkScanWorkers rejects a --workers value below 1. That is a usage
// error reported in one line: cobra prints the returned message without
// the usage text, and run exits 2.
func checkScanWorkers(cmd *cobra.Command, workers int) error {
if workers >= 1 {
return nil
@@ -284,7 +379,7 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
}
// rejectEmptyExcludeValue rejects an empty scan --exclude value. That
// rejectEmptyExcludeValue rejects an empty --exclude value. That
// is a usage error reported in one line, as for --workers below 1.
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
if !slices.Contains(excludePaths, "") {
+104 -13
View File
@@ -23,6 +23,12 @@ import (
// error prints and a runtime failure must not.
const usageMarker = "Usage:"
// helpFlag asks for the help text.
const helpFlag = "--help"
// workersFlag is the scan and dwim flag that sets the worker count.
const workersFlag = "--workers"
// TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail
@@ -575,8 +581,8 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}{
cmdScan: {
run: func() error {
return runScan(t.Context(), []string{missing}, 1, false, false,
nil)
return runScan(t.Context(), []string{missing},
scanOptions{workers: 1})
},
want: scanWant,
},
@@ -667,7 +673,7 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
var stdout, stderr bytes.Buffer
args := []string{cmdScan, "--workers", workers, t.TempDir()}
args := []string{cmdScan, workersFlag, workers, t.TempDir()}
code := run(args, &stdout, &stderr)
if code != exitUsage {
@@ -698,7 +704,7 @@ func TestRunHelp(t *testing.T) {
// README §Settings: help goes to stderr, exits 0, and leaves stdout
// empty.
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
for _, args := range cases {
var stdout, stderr bytes.Buffer
@@ -718,15 +724,12 @@ func TestRunHelp(t *testing.T) {
args, got)
}
if args[0] == cmdScan {
continue
}
// README §Location: sfdupes --help says where the database is,
// and that XDG_STATE_HOME must be an absolute path.
// README §Location: the help of sfdupes and of scan says where
// the database is, and that XDG_STATE_HOME must be an absolute
// path.
for _, want := range []string{
databaseEnv, rootDatabasePath, stateHomeEnv, "~/.local/state",
"not an absolute path is an error",
"scan and dwim store", databaseEnv, rootDatabasePath,
stateHomeEnv, "~/.local/state", "not an absolute path is an error",
} {
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) stderr = %q, want it to name %q",
@@ -775,6 +778,69 @@ func TestRunScanUsageLine(t *testing.T) {
}
}
func TestRunHelpDescribesOutput(t *testing.T) {
t.Parallel()
// README §Settings: the help of report, trees, info and dwim says
// what each prints. TestRunHelp checks the database paragraph of
// the root and scan help.
escapes := []string{`\\`, `\t`, `\n`, `\r`}
// The subcommand whose help is printed.
cases := []struct {
subcommand []string
want []string
}{
{
[]string{cmdReport},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n size ",
"largest first",
}, escapes...),
},
{
[]string{cmdTrees},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n files ",
"\n size ", "\n disk_size ", "largest first",
}, escapes...),
},
{
[]string{cmdInfo},
[]string{
"\n database ", "\n database file size ",
"\n -wal file size ", "\n modified ",
"\n schema version ", "\n records ",
"\n size of recorded files ",
},
},
{
[]string{cmdDwim},
[]string{
"sfdupes scan /", "as root", "sfdupes trees", "database",
},
},
}
for _, tc := range cases {
var stdout, stderr bytes.Buffer
args := slices.Concat(tc.subcommand, []string{helpFlag})
code := run(args, &stdout, &stderr)
if code != exitOK {
t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
}
for _, want := range tc.want {
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) help lacks %q:\n%s",
args, want, stderr.String())
}
}
}
}
func TestRunVersion(t *testing.T) {
t.Parallel()
@@ -1587,6 +1653,31 @@ func TestRunNonRootUsesStateDirectory(t *testing.T) {
}
}
func TestRunNonRootDatabaseEnvDirectoryMode(t *testing.T) {
// README §Location: for SFDUPES_DATABASE, scan creates each missing
// directory with mode 755, for a user other than root too.
if os.Geteuid() == 0 {
t.Skip("needs a user other than root")
}
top := filepath.Join(t.TempDir(), "missing")
sub := filepath.Join(top, "sub")
t.Setenv(databaseEnv, filepath.Join(sub, "db.sqlite"))
scanOK(t, t.TempDir())
for _, dir := range []string{top, sub} {
fi, err := os.Stat(dir)
if err != nil {
t.Fatal(err)
}
if got := fi.Mode().Perm(); got != dbDirPerm {
t.Errorf("%s mode = %#o, want %#o", dir, got, dbDirPerm)
}
}
}
func TestRunRelativeStateHomeFails(t *testing.T) {
// README §Location: without SFDUPES_DATABASE, a user other than
// root whose XDG_STATE_HOME is not an absolute path gets an error
@@ -1619,7 +1710,7 @@ func TestRunRelativeStateHomeFails(t *testing.T) {
}
// dwim scans /, so runDwim gets the missing operand instead.
err := runDwim(t.Context(), missing, io.Discard)
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, io.Discard)
if !errors.Is(err, errStateHomeNotAbsolute) {
t.Errorf("runDwim: error = %v, want %v", err,
errStateHomeNotAbsolute)
+10 -10
View File
@@ -5,20 +5,20 @@ import (
"fmt"
"io"
"os"
"runtime"
)
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
// root" runs with no flags, then what "sfdupes trees" runs, and ends
// with a line on stderr saying how to print the report again. The dwim
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
// does, but only during the scan. An error from the scan, an interrupt
// included, ends it before the report. During the report, a signal
// ends it at once, as it ends trees.
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
// root" runs with the scan flags in opts, then what "sfdupes trees"
// runs, and ends with a line on stderr saying how to print the report
// again. The dwim command passes "/" as root. It catches SIGINT and
// SIGTERM as scan does, but only during the scan. An error from the
// scan, an interrupt included, ends it before the report. During the
// report, a signal ends it at once, as it ends trees.
func runDwim(
ctx context.Context, root string, opts scanOptions, stdout io.Writer,
) error {
scanCtx, stop := interruptContext(ctx)
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false,
nil)
err := runScan(scanCtx, []string{root}, opts)
stop()
+129 -2
View File
@@ -2,7 +2,10 @@ package sfdupes //nolint:testpackage // calls the unexported runDwim
import (
"bytes"
"errors"
"os"
"path/filepath"
"runtime"
"strings"
"testing"
)
@@ -28,7 +31,7 @@ func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
stderr := captureStderr(t)
err := runDwim(t.Context(), root, &stdout)
err := runDwim(t.Context(), root, scanOptions{workers: 1}, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
@@ -67,7 +70,7 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
stderr := captureStderr(t)
err := runDwim(t.Context(), missing, &stdout)
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, &stdout)
if err == nil {
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
}
@@ -81,3 +84,127 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
}
}
func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
// dwim passes its --exclude paths to the scan: an excluded copy is
// not recorded, so its pair is not reported.
t.Setenv(databaseEnv, testDBPath(t))
root := realTempDir(t)
kept := []string{
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
}
writeFile(t, root, "b1/f.bin", pattern(2, 5000))
excluded := filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000)))
var stdout bytes.Buffer
stderr := captureStderr(t)
opts := scanOptions{workers: 1, excludePaths: []string{excluded}}
err := runDwim(t.Context(), root, opts, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
if len(lines) != 2 {
t.Fatalf("stdout = %q, want a header and one group",
stdout.String())
}
want := kept[0] + "\t" + kept[1] + "\t1\t300\t"
if !strings.HasPrefix(lines[1], want) {
t.Errorf("group = %q, want it to start %q", lines[1], want)
}
skipped := "scan: skipped " + excluded + " (--exclude)\n"
if got := stderr(); !strings.Contains(got, skipped) {
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
}
}
func TestRunDwimTakesScanFlags(t *testing.T) {
// The dwim command parses and checks the scan flags as scan does.
// Each case stops before the walk of / begins, and before the
// database is created. "dwim --exclude /" excludes its own operand.
cases := map[string]struct {
args []string
code int
want string
}{
"--exclude /": {
[]string{cmdDwim, excludeFlag, "/"},
exitFatal,
"sfdupes: scan /: operand excluded by --exclude /\n",
},
"--exclude empty": {
[]string{cmdDwim, excludeFlag, ""},
exitUsage,
"Error: --exclude must not be empty\n",
},
"--workers 0": {
[]string{cmdDwim, workersFlag, "0", "-x", "--no-default-excludes"},
exitUsage,
"Error: --workers must be at least 1, got 0\n",
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
stdout := captureStdout(t)
stderr := captureStderr(t)
code := run(tc.args, os.Stdout, os.Stderr)
if code != tc.code {
t.Errorf("run(%q) = %d, want %d", tc.args, code, tc.code)
}
if !strings.HasSuffix(stderr(), tc.want) || stdout() != "" {
t.Errorf("stdout, stderr = %q, %q, want \"\" and stderr "+
"ending %q", stdout(), stderr(), tc.want)
}
_, err := os.Lstat(path)
if !errors.Is(err, os.ErrNotExist) {
t.Errorf("lstat %s: %v, want no database", path, err)
}
})
}
}
func TestRunDwimHelpListsScanFlags(t *testing.T) {
t.Parallel()
// README "Subcommands" shows this usage line for dwim.
const want = "\n sfdupes dwim [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]...\n"
var stdout, stderr bytes.Buffer
run([]string{cmdDwim, "-h"}, &stdout, &stderr)
if !strings.Contains(stderr.String(), want) {
t.Errorf("dwim -h stderr = %q, want the usage line %q",
stderr.String(), want)
}
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
t.Errorf("dwim -h stderr = %q, want %s PATH", stderr.String(),
excludeFlag)
}
// README "Settings": the help lists the default excludes and the
// filesystem types the walk never records anything on.
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
if !strings.Contains(stderr.String(), help) {
t.Errorf("dwim -h stderr = %q, want it to hold %q", stderr.String(),
help)
}
}
+1 -1
View File
@@ -254,7 +254,7 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
return devices, lines.Err()
}
// defaultExcludesHelp describes, for the scan command's help, the
// defaultExcludesHelp describes, for the scan and dwim help, the
// default excludes of the system goos names and, where the scan lists
// the mounted filesystems, the types it never records anything on.
func defaultExcludesHelp(goos string) string {
+13 -14
View File
@@ -98,17 +98,16 @@ func scanPhase(name string) string {
// runs, and the lock is released after it. When ctx is cancelled, as by
// the SIGINT or SIGTERM that interruptContext catches, the scan keeps
// what it has hashed (see syncScan), prints how many files its walk
// reached, and returns errInterrupted. workers must be at least 1; the
// scan command rejects anything less. Before the lock is taken, the
// scan lists the filesystems that are not persistent, on which the walk
// records nothing; unless noDefaultExcludes is set, the walk does not
// enter the default excludes either. The walk neither enters nor
// records the paths in excludePaths, the --exclude values; an operand
// at or below one is a fatal error. Once the database is open, the scan
// notes its files, which the walk passes over.
func runScan(ctx context.Context, roots []string, workers int,
oneFS, noDefaultExcludes bool, excludePaths []string,
) error {
// reached, and returns errInterrupted. opts.workers must be at least 1;
// the scan and dwim commands reject anything less. Before the lock is
// taken, the scan lists the filesystems that are not persistent, on
// which the walk records nothing; unless opts.noDefaultExcludes is set,
// the walk does not enter the default excludes either. The walk neither
// enters nor records the paths in opts.excludePaths, the --exclude
// values; an operand at or below one is a fatal error. Once the
// database is open, the scan notes its files, which the walk passes
// over.
func runScan(ctx context.Context, roots []string, opts scanOptions) error {
dbPath, source, err := databasePath()
if err != nil {
return err
@@ -123,7 +122,7 @@ func runScan(ctx context.Context, roots []string, workers int,
var rules walkRules
rules.excludePaths, err = excludePathInodes(excludePaths)
rules.excludePaths, err = excludePathInodes(opts.excludePaths)
if err != nil {
return fmt.Errorf("--exclude: %w", err)
}
@@ -138,7 +137,7 @@ func runScan(ctx context.Context, roots []string, workers int,
return fmt.Errorf("list the mounted filesystems: %w", err)
}
if !noDefaultExcludes {
if !opts.noDefaultExcludes {
rules.defaultExcludes = defaultExcludes(runtime.GOOS)
}
@@ -167,7 +166,7 @@ func runScan(ctx context.Context, roots []string, workers int,
return err
}
st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
st, err := syncScan(ctx, db, roots, opts.workers, opts.oneFS, rules)
if errors.Is(err, context.Canceled) {
return interrupted(st.walked)
}
+1 -1
View File
@@ -2196,7 +2196,7 @@ func TestScanHashWriteFailureUnwindsPool(t *testing.T) {
var stderr bytes.Buffer
code := run([]string{
cmdScan, "--workers", strconv.Itoa(hashLeakWorkers), dir,
cmdScan, workersFlag, strconv.Itoa(hashLeakWorkers), dir,
}, io.Discard, &stderr)
if code != exitFatal {
t.Fatalf("run(scan) = %d, want %d; stderr: %s",