Compare commits

..
Author SHA1 Message Date
sneak 9e6e3b9287 Say in --help where the database is and what each report prints (closes #177)
check / check (push) Successful in 3m4s
The help of sfdupes and of scan names the database, its default path
and SFDUPES_DATABASE. The help of report and trees lists the columns,
the order of the rows and the four escapes. The help of info lists its
lines. The help of dwim says that it scans /, should run as root and
keeps the scan in the database.

The README's opening drops its sentence on copies deleted after the
scan, an edge case that its report and trees sections already cover.

Model: opus-5-5
2026-11-11 05:57:25 +00:00
15 changed files with 166 additions and 584 deletions

No files matched your search

+36 -63
View File
@@ -29,6 +29,7 @@ first look at a new machine with one command.
```sh ```sh
make build make build
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes scan /srv ./sfdupes scan /srv
./sfdupes report > dupes.tsv ./sfdupes report > dupes.tsv
./sfdupes trees > dupetrees.tsv ./sfdupes trees > dupetrees.tsv
@@ -44,10 +45,8 @@ that stands for a piece of hardware, such as a disk or a terminal. `report`
prints the file-level duplicates. `trees` prints the duplicate directory trees. prints the file-level duplicates. `trees` prints the duplicate directory trees.
`info` prints the database's path, size and record count. `info` prints the database's path, size and record count.
Root keeps the database in `/var/lib/sfdupes/db.sqlite`. Any other user keeps it The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
in `sfdupes/db.sqlite` in the XDG state directory, `~/.local/state` by default another file. The first `scan` creates it.
on Linux. `SFDUPES_DATABASE` names another file (see "Location"). The first
`scan` creates the database.
`report`, `trees` and `info` write their data to stdout. Progress, warnings and `report`, `trees` and `info` write their data to stdout. Progress, warnings and
errors go to stderr. stdout and stderr are a program's two output streams. The errors go to stderr. stdout and stderr are a program's two output streams. The
@@ -149,9 +148,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
- `report`, `trees` and `info` need only read access to the database. The umask - `report`, `trees` and `info` need only read access to the database. The umask
is the setting that removes permissions from the files a process creates. is the setting that removes permissions from the files a process creates.
Under the usual umask of `022`, the first scan creates the database readable Under the usual umask of `022`, the first scan creates the database readable
by every user. Another user can then run those three commands against root's by every user. An unprivileged user can then run those three commands against
database, with `SFDUPES_DATABASE=/var/lib/sfdupes/db.sqlite` before the root's database.
command.
### Reading the reports ### Reading the reports
@@ -256,14 +254,15 @@ In order:
- `report` prints the file-level duplicate report from the database. - `report` prints the file-level duplicate report from the database.
- `trees` prints the duplicate-tree report from the database. - `trees` prints the duplicate-tree report from the database.
- `info` prints the database's details. - `info` prints the database's details.
- `dwim` scans `/`, then prints the duplicate-tree report. - `dwim` scans `/` with the default excludes, then prints the duplicate-tree
report.
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
sfdupes report > dupes.tsv sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv sfdupes trees > dupetrees.tsv
sfdupes info sfdupes info
sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... sfdupes dwim
sfdupes --version sfdupes --version
sfdupes [command] --help sfdupes [command] --help
``` ```
@@ -290,23 +289,23 @@ The reference below is also the program's specification.
### Settings ### Settings
- `SFDUPES_DATABASE` names the database file. When it is unset or empty, the - `SFDUPES_DATABASE` names the database file. When it is unset or empty, the
database is at its default path (see "Location"). There is no command-line database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
flag for it. The path names the file exactly, whatever characters it holds. A it. The path names the file exactly, whatever characters it holds. A relative
relative path is relative to the working directory. path is relative to the working directory.
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once - `--workers N` (`scan`) sets how many directories `scan` reads at once while it
while it lists the trees. It also sets how many files `scan` reads at once lists the trees. It also sets how many files `scan` reads at once while it
while it hashes them. The default is the number of CPUs. `N` must be at hashes them. The default is the number of CPUs. `N` must be at least 1. A
least 1. A smaller value is a usage error, reported in one line on stderr with smaller value is a usage error, reported in one line on stderr with exit 2
exit 2 before anything is scanned. before anything is scanned.
- `-x`, `--one-file-system` (`scan`, `dwim`) keeps the walk of each operand out - `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
of directories on other filesystems, as the same flag does for `du` and directories on other filesystems, as the same flag does for `du` and `rsync`.
`rsync`. By default the walk also enters directories on other filesystems. By default the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default - `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
excludes. Those are the system and temporary directories the walk otherwise Those are the system and temporary directories the walk otherwise leaves out,
leaves out, such as `/var/tmp` on Linux (see "Rules for the walk"). such as `/var/tmp` on Linux (see "Rules for the walk").
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything - `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
below it. `PATH` may be a directory or a file. The flag may be given more than it. `PATH` may be a directory or a file. The flag may be given more than once.
once. The walk knows `PATH` by its device ID and inode number, as it knows the The walk knows `PATH` by its device ID and inode number, as it knows the
database files (see "Rules for the walk"). So the walk leaves `PATH` out database files (see "Rules for the walk"). So the walk leaves `PATH` out
whatever path leads to it. That includes a hard link to a file given as whatever path leads to it. That includes a hard link to a file given as
`PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in `PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in
@@ -319,9 +318,9 @@ The reference below is also the program's specification.
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr - `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan` and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
names the database. The help of `report`, `trees`, `info` and `dwim` says what names the database. The help of `report`, `trees`, `info` and `dwim` says what
each prints. `sfdupes scan --help` and `sfdupes dwim --help` also list the each prints. `sfdupes scan --help` also lists the default excludes of the
default excludes of the system they run on, and the filesystem types the walk system it runs on, and the filesystem types the walk never records anything
never records anything on. on.
### Duplicate detection ### Duplicate detection
@@ -859,25 +858,14 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
### `dwim` mode ### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its `dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments. Like `scan`, duplicate trees in one command. It takes no positional arguments and no flags of
it should run as root, so that it can read every file. its own. Like `scan`, it should run as root, so that it can read every file.
`dwim` takes these flags of `scan`, with the same meaning (see "Settings"): `dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
excludes and crosses into other filesystems. `--workers` keeps its default. The
- `--workers N`;
- `-x`, `--one-file-system`;
- `--no-default-excludes`;
- `--exclude PATH`, which may be given more than once.
`dwim` runs what `sfdupes scan /` runs with the same flags. With no flags, the
walk skips the default excludes and crosses into other filesystems. The
database, its creation and the scan lock are those of `scan`. Then `dwim` runs database, its creation and the scan lock are those of `scan`. Then `dwim` runs
what `sfdupes trees` runs. The largest trees come first. what `sfdupes trees` runs. The largest trees come first.
```
sfdupes dwim -x --exclude /srv/cache
```
The report goes to stdout, exactly as `trees` prints it. On stderr comes what The report goes to stdout, exactly as `trees` prints it. On stderr comes what
`scan` prints, then what `trees` prints, then one last line: `scan` prints, then what `trees` prints, then one last line:
@@ -939,22 +927,7 @@ All five subcommands work on one SQLite database file.
#### Location #### Location
The database is the file named by `SFDUPES_DATABASE`, when that is set and not The database is the file named by `SFDUPES_DATABASE`, when that is set and not
empty. Otherwise it is at its default path, which depends on the user: empty, and otherwise `/var/lib/sfdupes/db.sqlite`.
- For root, it is `/var/lib/sfdupes/db.sqlite`.
- For any other user, it is `sfdupes/db.sqlite` in the XDG state directory. That
directory is `$XDG_STATE_HOME` when that is set and not empty. Otherwise it is
`~/.local/state` on Linux and `~/Library/Application Support` on macOS. An
`XDG_STATE_HOME` that is not an absolute path is a fatal error, exit 1, for
every subcommand. The error names `XDG_STATE_HOME`.
`scan` creates each missing directory on the way to the database with mode 755.
At the default path of a user other than root, it uses mode 700 instead, as the
XDG Base Directory specification asks. Other users cannot enter a directory with
mode 700, so they cannot read a database in it.
All five subcommands use the same default, so a user's `report` reads the
database that user's `scan` wrote.
Each subcommand prints one line on stderr naming the database it uses, before Each subcommand prints one line on stderr naming the database it uses, before
anything else and whether or not stderr is a terminal. The line gives the path anything else and whether or not stderr is a terminal. The line gives the path
@@ -994,9 +967,9 @@ Only one `scan` at a time can hold the lock.
- For its whole run, `scan` holds a lock on a lock file beside the database. The - For its whole run, `scan` holds a lock on a lock file beside the database. The
lock is taken with the `flock(2)` system call, a request a program makes to lock is taken with the `flock(2)` system call, a request a program makes to
the kernel, and only one process can hold it. The lock file's name is the the kernel, and only one process can hold it. The lock file's name is the
database's path, with symlinks resolved, plus `.lock`. For root, by default, database's path, with symlinks resolved, plus `.lock`. By default that is
that is `/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the `/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same
same lock file. lock file.
- `scan` takes the lock before it walks the filesystem or opens the database. - `scan` takes the lock before it walks the filesystem or opens the database.
Just before that, it creates the database file, empty, if it is missing. So Just before that, it creates the database file, empty, if it is missing. So
the lock file is named the same way before the first scan as after. A symlink the lock file is named the same way before the first scan as after. A symlink
-11
View File
@@ -29,22 +29,11 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- Without `SFDUPES_DATABASE`, a user other than root keeps the database in
`sfdupes/db.sqlite` under the XDG state directory, `~/.local/state` by
default. Root keeps `/var/lib/sfdupes/db.sqlite`. `sfdupes --help` and
`sfdupes scan --help` say so (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/171)
- `dwim` takes the `scan` flags `--workers`, `-x`, `--no-default-excludes` and
`--exclude`, with the same meaning. Both commands register and check them with
the same code (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/162)
- The help of `sfdupes` and of `scan` names the database, its default path and - The help of `sfdupes` and of `scan` names the database, its default path and
`SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the `SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
order of the rows and the escapes. The help of `info` lists its lines. The order of the rows and the escapes. The help of `info` lists its lines. The
help of `dwim` says that it scans `/`, should run as root and keeps the scan help of `dwim` says that it scans `/`, should run as root and keeps the scan
in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177) in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither - New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
enters nor records an excluded path, and keeps the records below it. An enters nor records an excluded path, and keeps the records below it. An
operand at or below one is a fatal error (2026-11-11, operand at or below one is a fatal error (2026-11-11,
+2 -4
View File
@@ -5,10 +5,8 @@
// KiB, and of the content. The content hash covers the whole file under // KiB, and of the content. The content hash covers the whole file under
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large // 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
// file is never read in full. scan maintains a persistent SQLite // file is never read in full. scan maintains a persistent SQLite
// database of file signatures that the other subcommands read. The // database of file signatures (SFDUPES_DATABASE, default
// database is SFDUPES_DATABASE, or by default /var/lib/sfdupes/db.sqlite // /var/lib/sfdupes/db.sqlite) that the other subcommands read.
// for root and sfdupes/db.sqlite in the XDG state directory for any
// other user.
// //
// Usage: // Usage:
// //
-1
View File
@@ -3,7 +3,6 @@ module sneak.berlin/go/sfdupes
go 1.25.7 go 1.25.7
require ( require (
github.com/adrg/xdg v0.5.3
github.com/schollz/progressbar/v3 v3.19.1 github.com/schollz/progressbar/v3 v3.19.1
github.com/spf13/cobra v1.10.2 github.com/spf13/cobra v1.10.2
golang.org/x/sys v0.46.0 golang.org/x/sys v0.46.0
-2
View File
@@ -1,5 +1,3 @@
github.com/adrg/xdg v0.5.3 h1:xRnxJXne7+oWDatRhR1JLnvuccuIeCoBu2rtuLqQB78=
github.com/adrg/xdg v0.5.3/go.mod h1:nlTsY+NNiCBGCK2tpm09vRqfVzrc2fLmXGpBLF0zlTQ=
github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM= github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM=
github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY= github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g= github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
+7 -6
View File
@@ -358,9 +358,8 @@ func TestRunScanInterrupted(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
dir := buildSmokeTree(t) dir := buildSmokeTree(t)
opts := scanOptions{workers: walkCancelWorkers} err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false, nil)
err := runScan(t.Context(), []string{dir}, opts)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -383,7 +382,8 @@ func TestRunScanInterrupted(t *testing.T) {
go func() { go func() {
defer close(done) defer close(done)
err = runScan(cancelledContext(t), []string{dir}, opts) err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
false, false, nil)
}() }()
awaitReturn(t, done, "runScan") awaitReturn(t, done, "runScan")
@@ -410,7 +410,8 @@ func TestRunScanInterrupted(t *testing.T) {
got, before) got, before)
} }
err = runScan(t.Context(), []string{dir}, opts) err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
false, nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -436,7 +437,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
dir := buildWalkCancelTree(t) dir := buildWalkCancelTree(t)
err := runScan(newWalkClock(hashCancelAtDone), []string{dir}, err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
scanOptions{workers: walkCancelWorkers}) walkCancelWorkers, false, false, nil)
if !errors.Is(err, errInterrupted) { if !errors.Is(err, errInterrupted) {
t.Fatalf("runScan interrupted mid-hash = %v, want %v", t.Fatalf("runScan interrupted mid-hash = %v, want %v",
err, errInterrupted) err, errInterrupted)
+58 -94
View File
@@ -34,14 +34,10 @@ const (
// databaseHelp is the paragraph of the root and scan help that names // databaseHelp is the paragraph of the root and scan help that names
// the database (README "Location"). // the database (README "Location").
const databaseHelp = `scan and dwim store what they find in a database file. report, const databaseHelp = "scan and dwim store what they find in a database " +
trees and info read it. The file is $SFDUPES_DATABASE when that is set "file. report, trees\nand info read it. The file is $" + databaseEnv +
and not empty. Otherwise root uses /var/lib/sfdupes/db.sqlite. Any other " when that is set and\nnot empty, and otherwise " +
user uses sfdupes/db.sqlite in the XDG state directory: $XDG_STATE_HOME, defaultDatabasePath + ", where creating it\nneeds root.\n"
or ~/.local/state when that is unset or empty
(~/Library/Application Support on macOS). An XDG_STATE_HOME that is
not an absolute path is an error.
`
// escapesHelp is the paragraph of the report and trees help on how // escapesHelp is the paragraph of the report and trees help on how
// paths are escaped (README "Report output format"). // paths are escaped (README "Report output format").
@@ -92,10 +88,10 @@ value, separated by a tab:
size of recorded files their total size, in bytes, then rounded size of recorded files their total size, in bytes, then rounded
` `
dwimHelp = `dwim runs what "sfdupes scan /" runs with the same flags, then what dwimHelp = `dwim runs what "sfdupes scan /" runs, then what "sfdupes trees" runs.
"sfdupes trees" runs. Run it as root, so that the scan can read every Run it as root, so that the scan can read every file. The scan is stored
file. The scan is stored in the database, as scan stores it, so in the database, as scan stores it, so "sfdupes trees" prints the report
"sfdupes trees" prints the report again later. again later.
` `
) )
@@ -104,11 +100,12 @@ file. The scan is stored in the database, as scan stores it, so
// cobra prints for it is the whole message. // cobra prints for it is the whole message.
var errNoSubcommand = errors.New("no subcommand") var errNoSubcommand = errors.New("no subcommand")
// errWorkersBelowOne is the usage error for a --workers value below 1. // errWorkersBelowOne is the usage error for a scan --workers value
// below 1.
var errWorkersBelowOne = errors.New("--workers must be at least 1") var errWorkersBelowOne = errors.New("--workers must be at least 1")
// errEmptyExcludeValue is the usage error for an empty --exclude value, // errEmptyExcludeValue is the usage error for an empty scan --exclude
// as from --exclude "$DIR" with DIR unset. Taken as a path, it // value, as from --exclude "$DIR" with DIR unset. Taken as a path, it
// would name the working directory. // would name the working directory.
var errEmptyExcludeValue = errors.New("--exclude must not be empty") var errEmptyExcludeValue = errors.New("--exclude must not be empty")
@@ -259,116 +256,83 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}), }),
} }
dwimCmd := &cobra.Command{
Use: cmdDwim,
Short: "Scan / with the default excludes, then print the duplicate-tree report",
Args: cobra.NoArgs,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", stdout)
}),
}
root.Long = root.Short + ".\n\n" + databaseHelp root.Long = root.Short + ".\n\n" + databaseHelp
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
dwimCmd.Long = dwimCmd.Short + ".\n\n" + dwimHelp
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
newDwimCommand(stdout))
return root return root
} }
// scanOptions holds the values of the flags that scan and dwim share: // newScanCommand builds the scan subcommand, with its --workers, -x,
// --workers, -x, --no-default-excludes and --exclude. // --no-default-excludes and --exclude flags. Its help names the
type scanOptions struct {
workers int
oneFS bool
noDefaultExcludes bool
excludePaths []string
}
// scanFlagsUsage names the flags that scan and dwim share, for their
// usage lines.
const scanFlagsUsage = "[--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]..."
// addScanFlags adds the flags that scan and dwim share to cmd, which
// stores their values in opts.
func addScanFlags(cmd *cobra.Command, opts *scanOptions) {
cmd.Flags().IntVar(&opts.workers, "workers", runtime.NumCPU(),
"concurrent workers for the walk and hash phases")
cmd.Flags().BoolVarP(&opts.oneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
cmd.Flags().BoolVar(&opts.noDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
// A string array, not a slice, so a comma in a path is not a
// separator.
cmd.Flags().StringArrayVar(&opts.excludePaths, "exclude", nil,
"never enter or record `PATH` or anything below it; repeatable")
}
// check rejects the flag values that are usage errors, before the
// command runs. It is the PreRunE of scan and dwim.
func (o *scanOptions) check(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, o.workers)
if err != nil {
return err
}
return rejectEmptyExcludeValue(cmd, o.excludePaths)
}
// newScanCommand builds the scan subcommand. Its help names the
// database, and lists the default excludes of the system it runs on and // database, and lists the default excludes of the system it runs on and
// the filesystem types it never records anything on. // the filesystem types it never records anything on.
func newScanCommand() *cobra.Command { func newScanCommand() *cobra.Command {
var opts scanOptions var (
scanWorkers int
scanOneFS bool
scanNoDefaultExcludes bool
scanExclude []string
)
short := "Walk trees and synchronize the scan database" short := "Walk trees and synchronize the scan database"
scanCmd := &cobra.Command{ scanCmd := &cobra.Command{
Use: cmdScan + " " + scanFlagsUsage + " PATH...", Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]... PATH...",
// The Use line already names every flag, so cobra must not // The Use line already names every flag, so cobra must not
// append its generic "[flags]" to it. // append its generic "[flags]" to it.
DisableFlagsInUseLine: true, DisableFlagsInUseLine: true,
Short: short, Short: short,
Long: short + ".\n\n" + databaseHelp + "\n" + Long: short + ".\n\n" + databaseHelp + "\n" +
defaultExcludesHelp(runtime.GOOS), defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1), Args: cobra.MinimumNArgs(1),
PreRunE: opts.check, PreRunE: func(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, scanWorkers)
if err != nil {
return err
}
return rejectEmptyExcludeValue(cmd, scanExclude)
},
RunE: runE(func(ctx context.Context, args []string) error { RunE: runE(func(ctx context.Context, args []string) error {
ctx, stop := interruptContext(ctx) ctx, stop := interruptContext(ctx)
defer stop() defer stop()
return runScan(ctx, args, opts) return runScan(ctx, args, scanWorkers, scanOneFS,
scanNoDefaultExcludes, scanExclude)
}), }),
} }
addScanFlags(scanCmd, &opts) scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
"concurrent workers for the walk and hash phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
// A string array, not a slice, so a comma in a path is not a
// separator.
scanCmd.Flags().StringArrayVar(&scanExclude, "exclude", nil,
"never enter or record `PATH` or anything below it; repeatable")
return scanCmd return scanCmd
} }
// newDwimCommand builds the dwim subcommand, which takes the scan flags // checkScanWorkers rejects a scan --workers value below 1. That is a
// with the meaning they have for scan. Its help says what dwim runs, // usage error reported in one line: cobra prints the returned message
// then lists the default excludes and the filesystem types the scan // without the usage text, and run exits 2.
// never records anything on.
func newDwimCommand(stdout io.Writer) *cobra.Command {
var opts scanOptions
short := "Scan /, then print the duplicate-tree report"
dwimCmd := &cobra.Command{
Use: cmdDwim + " " + scanFlagsUsage,
DisableFlagsInUseLine: true,
Short: short,
Long: short + ".\n\n" + dwimHelp + "\n" +
defaultExcludesHelp(runtime.GOOS),
Args: cobra.NoArgs,
PreRunE: opts.check,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", opts, stdout)
}),
}
addScanFlags(dwimCmd, &opts)
return dwimCmd
}
// checkScanWorkers rejects a --workers value below 1. That is a usage
// error reported in one line: cobra prints the returned message without
// the usage text, and run exits 2.
func checkScanWorkers(cmd *cobra.Command, workers int) error { func checkScanWorkers(cmd *cobra.Command, workers int) error {
if workers >= 1 { if workers >= 1 {
return nil return nil
@@ -379,7 +343,7 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers) return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
} }
// rejectEmptyExcludeValue rejects an empty --exclude value. That // rejectEmptyExcludeValue rejects an empty scan --exclude value. That
// is a usage error reported in one line, as for --workers below 1. // is a usage error reported in one line, as for --workers below 1.
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error { func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
if !slices.Contains(excludePaths, "") { if !slices.Contains(excludePaths, "") {
+19 -130
View File
@@ -15,7 +15,6 @@ import (
"strings" "strings"
"testing" "testing"
"github.com/adrg/xdg"
"golang.org/x/sys/unix" "golang.org/x/sys/unix"
) )
@@ -26,9 +25,6 @@ const usageMarker = "Usage:"
// helpFlag asks for the help text. // helpFlag asks for the help text.
const helpFlag = "--help" const helpFlag = "--help"
// workersFlag is the scan and dwim flag that sets the worker count.
const workersFlag = "--workers"
// TestMain stops the tests at once, saying why, when their temporary // TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing // directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail // there (README "Rules for the walk"), so most scan tests would fail
@@ -560,17 +556,13 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
// line. // line.
t.Setenv(databaseEnv, "") t.Setenv(databaseEnv, "")
dbPath, err := defaultDatabasePath(os.Geteuid())
if err != nil {
t.Fatal(err)
}
// Whether scan calls the database new depends on that machine too. // Whether scan calls the database new depends on that machine too.
scanWant := "scan: database " + dbPath + " (default)\n" scanWant := "scan: database " + defaultDatabasePath + " (default)\n"
_, err = os.Stat(dbPath) _, err := os.Stat(defaultDatabasePath)
if errors.Is(err, fs.ErrNotExist) { if errors.Is(err, fs.ErrNotExist) {
scanWant = "scan: new database " + dbPath + " (default)\n" scanWant = "scan: new database " + defaultDatabasePath +
" (default)\n"
} }
missing := filepath.Join(t.TempDir(), "nope") missing := filepath.Join(t.TempDir(), "nope")
@@ -581,8 +573,8 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}{ }{
cmdScan: { cmdScan: {
run: func() error { run: func() error {
return runScan(t.Context(), []string{missing}, return runScan(t.Context(), []string{missing}, 1, false, false,
scanOptions{workers: 1}) nil)
}, },
want: scanWant, want: scanWant,
}, },
@@ -590,21 +582,21 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
run: func() error { run: func() error {
return runReport(cancelledContext(t), io.Discard) return runReport(cancelledContext(t), io.Discard)
}, },
want: "report: database " + dbPath + " (default)\n" + want: "report: database " + defaultDatabasePath + " (default)\n" +
reportOpenLine + "\n" + reportOpenLine + "\n", reportOpenLine + "\n" + reportOpenLine + "\n",
}, },
cmdTrees: { cmdTrees: {
run: func() error { run: func() error {
return runTrees(cancelledContext(t), io.Discard) return runTrees(cancelledContext(t), io.Discard)
}, },
want: "trees: database " + dbPath + " (default)\n" + want: "trees: database " + defaultDatabasePath + " (default)\n" +
treesOpenLine + "\n" + treesOpenLine + "\n", treesOpenLine + "\n" + treesOpenLine + "\n",
}, },
cmdInfo: { cmdInfo: {
run: func() error { run: func() error {
return runInfo(cancelledContext(t), io.Discard) return runInfo(cancelledContext(t), io.Discard)
}, },
want: "info: database " + dbPath + " (default)\n" + want: "info: database " + defaultDatabasePath + " (default)\n" +
infoOpenLine + "\n" + infoOpenLine + "\n", infoOpenLine + "\n" + infoOpenLine + "\n",
}, },
} }
@@ -673,7 +665,7 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
args := []string{cmdScan, workersFlag, workers, t.TempDir()} args := []string{cmdScan, "--workers", workers, t.TempDir()}
code := run(args, &stdout, &stderr) code := run(args, &stdout, &stderr)
if code != exitUsage { if code != exitUsage {
@@ -723,19 +715,6 @@ func TestRunHelp(t *testing.T) {
t.Errorf("run(%v) stdout = %q, want nothing (data only)", t.Errorf("run(%v) stdout = %q, want nothing (data only)",
args, got) args, got)
} }
// README §Location: the help of sfdupes and of scan says where
// the database is, and that XDG_STATE_HOME must be an absolute
// path.
for _, want := range []string{
"scan and dwim store", databaseEnv, rootDatabasePath,
stateHomeEnv, "~/.local/state", "not an absolute path is an error",
} {
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) stderr = %q, want it to name %q",
args, stderr.String(), want)
}
}
} }
// The scan help lists --exclude with its argument. // The scan help lists --exclude with its argument.
@@ -778,19 +757,23 @@ func TestRunScanUsageLine(t *testing.T) {
} }
} }
func TestRunHelpDescribesOutput(t *testing.T) { func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
t.Parallel() t.Parallel()
// README §Settings: the help of report, trees, info and dwim says // README §Settings: the help names the database, and says what
// what each prints. TestRunHelp checks the database paragraph of // report, trees, info and dwim print.
// the root and scan help. database := []string{
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
}
escapes := []string{`\\`, `\t`, `\n`, `\r`} escapes := []string{`\\`, `\t`, `\n`, `\r`}
// The subcommand whose help is printed. // The subcommand whose help is printed; none for the root.
cases := []struct { cases := []struct {
subcommand []string subcommand []string
want []string want []string
}{ }{
{nil, database},
{[]string{cmdScan}, database},
{ {
[]string{cmdReport}, []string{cmdReport},
append([]string{ append([]string{
@@ -1623,100 +1606,6 @@ func TestRunDatabasePathUsedAsGiven(t *testing.T) {
}) })
} }
func TestRunNonRootUsesStateDirectory(t *testing.T) {
// README §Location: without SFDUPES_DATABASE, a user other than
// root keeps the database in the XDG state directory. scan creates
// the missing directories with mode 0700, and report and trees read
// the same file.
if os.Geteuid() == 0 {
t.Skip("root uses " + rootDatabasePath)
}
t.Cleanup(xdg.Reload)
state := filepath.Join(t.TempDir(), "state")
t.Setenv(stateHomeEnv, state)
t.Setenv(databaseEnv, "")
xdg.Reload()
assertRunsUseDatabase(t, filepath.Join(state, "sfdupes"), "db.sqlite")
for _, dir := range []string{state, filepath.Join(state, "sfdupes")} {
fi, err := os.Stat(dir)
if err != nil {
t.Fatal(err)
}
if got := fi.Mode().Perm(); got != stateDirPerm {
t.Errorf("%s mode = %#o, want %#o", dir, got, stateDirPerm)
}
}
}
func TestRunNonRootDatabaseEnvDirectoryMode(t *testing.T) {
// README §Location: for SFDUPES_DATABASE, scan creates each missing
// directory with mode 755, for a user other than root too.
if os.Geteuid() == 0 {
t.Skip("needs a user other than root")
}
top := filepath.Join(t.TempDir(), "missing")
sub := filepath.Join(top, "sub")
t.Setenv(databaseEnv, filepath.Join(sub, "db.sqlite"))
scanOK(t, t.TempDir())
for _, dir := range []string{top, sub} {
fi, err := os.Stat(dir)
if err != nil {
t.Fatal(err)
}
if got := fi.Mode().Perm(); got != dbDirPerm {
t.Errorf("%s mode = %#o, want %#o", dir, got, dbDirPerm)
}
}
}
func TestRunRelativeStateHomeFails(t *testing.T) {
// README §Location: without SFDUPES_DATABASE, a user other than
// root whose XDG_STATE_HOME is not an absolute path gets an error
// naming it from every subcommand, which exits 1 before it does
// anything else.
if os.Geteuid() == 0 {
t.Skip("root does not use " + stateHomeEnv)
}
t.Setenv(stateHomeEnv, "relative/state")
t.Setenv(databaseEnv, "")
want := "sfdupes: " + stateHomeEnv +
" must be an absolute path, got \"relative/state\"\n"
// A scan that got past the check would stop at this missing
// operand, before it creates anything.
missing := filepath.Join(t.TempDir(), "missing")
for _, args := range [][]string{
{cmdScan, missing}, {cmdReport}, {cmdTrees}, {cmdInfo},
} {
var stdout, stderr bytes.Buffer
code := run(args, &stdout, &stderr)
if code != exitFatal || stderr.String() != want {
t.Errorf("run(%v) = %d with stderr %q, want %d and %q",
args, code, stderr.String(), exitFatal, want)
}
}
// dwim scans /, so runDwim gets the missing operand instead.
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, io.Discard)
if !errors.Is(err, errStateHomeNotAbsolute) {
t.Errorf("runDwim: error = %v, want %v", err,
errStateHomeNotAbsolute)
}
}
// holdScanLock takes the lock on the database at path, as a running // holdScanLock takes the lock on the database at path, as a running
// scan does, and holds it until the test ends. It fails the test when // scan does, and holds it until the test ends. It fails the test when
// the lock is already held. // the lock is already held.
+15 -68
View File
@@ -15,40 +15,28 @@ import (
"strings" "strings"
"time" "time"
"github.com/adrg/xdg"
"golang.org/x/sys/unix" "golang.org/x/sys/unix"
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo // The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
// disabled. // disabled.
_ "modernc.org/sqlite" _ "modernc.org/sqlite"
) )
// rootDatabasePath is the database root uses when SFDUPES_DATABASE is // defaultDatabasePath is where the persistent scan database lives when
// not set. // SFDUPES_DATABASE is not set.
const rootDatabasePath = "/var/lib/sfdupes/db.sqlite" const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
// databaseEnv is the environment variable that overrides the database // databaseEnv is the environment variable that overrides the database
// path. // path.
const databaseEnv = "SFDUPES_DATABASE" const databaseEnv = "SFDUPES_DATABASE"
// stateHomeEnv is the environment variable that names the XDG state
// directory.
const stateHomeEnv = "XDG_STATE_HOME"
// schemaVersion is the database schema version this build reads and // schemaVersion is the database schema version this build reads and
// writes, stored in PRAGMA user_version. // writes, stored in PRAGMA user_version.
const schemaVersion = 1 const schemaVersion = 1
// dbDirPerm is the mode for a directory scan creates for root's // dbDirPerm is the mode for a database parent directory created by
// default database or for SFDUPES_DATABASE. // scan.
const dbDirPerm = 0o755 const dbDirPerm = 0o755
// stateDirPerm is the mode for a directory scan creates for the
// database at a user's default path in the XDG state directory. It is
// the mode the XDG Base Directory specification asks for, and it keeps
// other users from reading the database, which lists every scanned
// path.
const stateDirPerm = 0o700
// dbFilePerm is the mode for a database file created by scan, the mode // dbFilePerm is the mode for a database file created by scan, the mode
// SQLite gives a database file it creates. // SQLite gives a database file it creates.
const dbFilePerm = 0o644 const dbFilePerm = 0o644
@@ -156,48 +144,15 @@ var errScanRunning = errors.New("another scan is running")
// 32 bytes of a SHA-256. // 32 bytes of a SHA-256.
var errHashSize = errors.New("hash is not 32 bytes") var errHashSize = errors.New("hash is not 32 bytes")
// errStateHomeNotAbsolute reports an XDG_STATE_HOME that is set to a
// path that is not absolute.
var errStateHomeNotAbsolute = errors.New(
stateHomeEnv + " must be an absolute path")
// defaultDatabasePath is the database used when SFDUPES_DATABASE is not
// set: rootDatabasePath for root (euid 0), and db.sqlite in the
// directory sfdupes under the XDG state directory for any other user.
// xdg reads XDG_STATE_HOME and HOME once, when the program starts. For
// any other user, an XDG_STATE_HOME that is set and not an absolute
// path is an error, where xdg would use its default instead. The path
// is joined as text, not cleaned, for the reason absolutePath gives.
func defaultDatabasePath(euid int) (string, error) {
if euid == 0 {
return rootDatabasePath, nil
}
stateHome := os.Getenv(stateHomeEnv)
if stateHome != "" && !filepath.IsAbs(stateHome) {
return "", fmt.Errorf("%w, got %q", errStateHomeNotAbsolute,
stateHome)
}
stateDir := strings.TrimSuffix(xdg.StateHome, "/")
return stateDir + "/sfdupes/db.sqlite", nil
}
// databasePath resolves the database location: SFDUPES_DATABASE when // databasePath resolves the database location: SFDUPES_DATABASE when
// set and non-empty, defaultDatabasePath otherwise. It returns the // set and non-empty, the compiled-in default otherwise. It returns the
// path, made absolute by absolutePath, and where the path came from, // path, made absolute by absolutePath, and where the path came from,
// "default" or "from SFDUPES_DATABASE", for the line each subcommand // "default" or "from SFDUPES_DATABASE", for the line each subcommand
// prints first on stderr. // prints first on stderr.
func databasePath() (string, string, error) { func databasePath() (string, string, error) {
path, source := os.Getenv(databaseEnv), "from "+databaseEnv path, source := defaultDatabasePath, "default"
if path == "" { if p := os.Getenv(databaseEnv); p != "" {
def, err := defaultDatabasePath(os.Geteuid()) path, source = p, "from "+databaseEnv
if err != nil {
return "", "", err
}
path, source = def, "default"
} }
abs, err := absolutePath(path) abs, err := absolutePath(path)
@@ -233,23 +188,15 @@ func absolutePath(path string) (string, error) {
// createDatabaseDirectory creates the directory the database at path // createDatabaseDirectory creates the directory the database at path
// goes in, and any directory above it, if missing. path is absolute, as // goes in, and any directory above it, if missing. path is absolute, as
// databasePath returns it. Each directory gets mode dbDirPerm, or // databasePath returns it. The directory is path with its last element
// stateDirPerm when databasePath uses a user's default path in the XDG // cut off as text, by filepath.Split. filepath.Dir would clean it,
// state directory: the user is not root and SFDUPES_DATABASE is unset // dropping each ".." together with the directory name before it; when
// or empty. The directory is path with its last element cut off as // that name is a symlink, the cleaned path names another directory than
// text, by filepath.Split. filepath.Dir would clean it, dropping each // the one the system reaches when it opens path.
// ".." together with the directory name before it; when that name is a
// symlink, the cleaned path names another directory than the one the
// system reaches when it opens path.
func createDatabaseDirectory(path string) error { func createDatabaseDirectory(path string) error {
dir, _ := filepath.Split(path) dir, _ := filepath.Split(path)
perm := os.FileMode(dbDirPerm) err := os.MkdirAll(dir, dbDirPerm)
if os.Geteuid() != 0 && os.Getenv(databaseEnv) == "" {
perm = stateDirPerm
}
err := os.MkdirAll(dir, perm)
if err != nil { if err != nil {
return fmt.Errorf("create database directory: %w", err) return fmt.Errorf("create database directory: %w", err)
} }
+1 -51
View File
@@ -9,13 +9,10 @@ import (
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"runtime"
"slices" "slices"
"strings" "strings"
"testing" "testing"
"time" "time"
"github.com/adrg/xdg"
) )
// testDBPath returns a database path inside a fresh temp dir, with // testDBPath returns a database path inside a fresh temp dir, with
@@ -59,16 +56,11 @@ func TestDatabasePath(t *testing.T) {
t.Chdir(dir) t.Chdir(dir)
defaultPath, err := defaultDatabasePath(os.Geteuid())
if err != nil {
t.Fatal(err)
}
fromEnv := "from " + databaseEnv fromEnv := "from " + databaseEnv
cases := []struct { cases := []struct {
env, path, source string env, path, source string
}{ }{
{"", defaultPath, "default"}, {"", defaultDatabasePath, "default"},
{"/custom/place.sqlite", "/custom/place.sqlite", fromEnv}, {"/custom/place.sqlite", "/custom/place.sqlite", fromEnv},
{"/custom/link/../place.sqlite", "/custom/link/../place.sqlite", {"/custom/link/../place.sqlite", "/custom/link/../place.sqlite",
fromEnv}, fromEnv},
@@ -97,48 +89,6 @@ func TestDatabasePath(t *testing.T) {
} }
} }
func TestDefaultDatabasePath(t *testing.T) {
// README §Location: root's database is /var/lib/sfdupes/db.sqlite.
// Any other user's is db.sqlite in sfdupes under XDG_STATE_HOME, or
// under ~/.local/state when XDG_STATE_HOME is unset or empty. For
// that user, an XDG_STATE_HOME that is not an absolute path is an
// error.
if runtime.GOOS != goosLinux {
t.Skip("macOS has another default state directory")
}
t.Cleanup(xdg.Reload)
t.Setenv("HOME", "/home/someone")
cases := []struct {
euid int
stateHome string
want string
wantErr error
}{
{0, "", rootDatabasePath, nil},
{0, "/elsewhere/state", rootDatabasePath, nil},
{0, "relative/state", rootDatabasePath, nil},
{1000, "", "/home/someone/.local/state/sfdupes/db.sqlite", nil},
{1000, "relative/state", "", errStateHomeNotAbsolute},
{1000, "~/state", "", errStateHomeNotAbsolute},
{1000, "/elsewhere/state", "/elsewhere/state/sfdupes/db.sqlite", nil},
{1000, "/elsewhere/state/", "/elsewhere/state/sfdupes/db.sqlite", nil},
}
for _, tc := range cases {
t.Setenv(stateHomeEnv, tc.stateHome)
xdg.Reload()
got, err := defaultDatabasePath(tc.euid)
if got != tc.want || !errors.Is(err, tc.wantErr) {
t.Errorf("defaultDatabasePath(%d) with %s=%q = %q, %v; want %q, %v",
tc.euid, stateHomeEnv, tc.stateHome, got, err, tc.want,
tc.wantErr)
}
}
}
func TestOpenScanDatabaseCreates(t *testing.T) { func TestOpenScanDatabaseCreates(t *testing.T) {
t.Parallel() t.Parallel()
+10 -10
View File
@@ -5,20 +5,20 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"runtime"
) )
// runDwim implements the dwim subcommand: it runs what "sfdupes scan // runDwim implements the dwim subcommand: it runs what "sfdupes scan
// root" runs with the scan flags in opts, then what "sfdupes trees" // root" runs with no flags, then what "sfdupes trees" runs, and ends
// runs, and ends with a line on stderr saying how to print the report // with a line on stderr saying how to print the report again. The dwim
// again. The dwim command passes "/" as root. It catches SIGINT and // command passes "/" as root. It catches SIGINT and SIGTERM as scan
// SIGTERM as scan does, but only during the scan. An error from the // does, but only during the scan. An error from the scan, an interrupt
// scan, an interrupt included, ends it before the report. During the // included, ends it before the report. During the report, a signal
// report, a signal ends it at once, as it ends trees. // ends it at once, as it ends trees.
func runDwim( func runDwim(ctx context.Context, root string, stdout io.Writer) error {
ctx context.Context, root string, opts scanOptions, stdout io.Writer,
) error {
scanCtx, stop := interruptContext(ctx) scanCtx, stop := interruptContext(ctx)
err := runScan(scanCtx, []string{root}, opts) err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false,
nil)
stop() stop()
+2 -129
View File
@@ -2,10 +2,7 @@ package sfdupes //nolint:testpackage // calls the unexported runDwim
import ( import (
"bytes" "bytes"
"errors"
"os"
"path/filepath" "path/filepath"
"runtime"
"strings" "strings"
"testing" "testing"
) )
@@ -31,7 +28,7 @@ func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
err := runDwim(t.Context(), root, scanOptions{workers: 1}, &stdout) err := runDwim(t.Context(), root, &stdout)
if err != nil { if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr()) t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
} }
@@ -70,7 +67,7 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, &stdout) err := runDwim(t.Context(), missing, &stdout)
if err == nil { if err == nil {
t.Fatal("runDwim on a missing root: error = nil, want it fatal") t.Fatal("runDwim on a missing root: error = nil, want it fatal")
} }
@@ -84,127 +81,3 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
t.Errorf("stderr = %q, want nothing from trees or dwim", got) t.Errorf("stderr = %q, want nothing from trees or dwim", got)
} }
} }
func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
// dwim passes its --exclude paths to the scan: an excluded copy is
// not recorded, so its pair is not reported.
t.Setenv(databaseEnv, testDBPath(t))
root := realTempDir(t)
kept := []string{
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
}
writeFile(t, root, "b1/f.bin", pattern(2, 5000))
excluded := filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000)))
var stdout bytes.Buffer
stderr := captureStderr(t)
opts := scanOptions{workers: 1, excludePaths: []string{excluded}}
err := runDwim(t.Context(), root, opts, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
if len(lines) != 2 {
t.Fatalf("stdout = %q, want a header and one group",
stdout.String())
}
want := kept[0] + "\t" + kept[1] + "\t1\t300\t"
if !strings.HasPrefix(lines[1], want) {
t.Errorf("group = %q, want it to start %q", lines[1], want)
}
skipped := "scan: skipped " + excluded + " (--exclude)\n"
if got := stderr(); !strings.Contains(got, skipped) {
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
}
}
func TestRunDwimTakesScanFlags(t *testing.T) {
// The dwim command parses and checks the scan flags as scan does.
// Each case stops before the walk of / begins, and before the
// database is created. "dwim --exclude /" excludes its own operand.
cases := map[string]struct {
args []string
code int
want string
}{
"--exclude /": {
[]string{cmdDwim, excludeFlag, "/"},
exitFatal,
"sfdupes: scan /: operand excluded by --exclude /\n",
},
"--exclude empty": {
[]string{cmdDwim, excludeFlag, ""},
exitUsage,
"Error: --exclude must not be empty\n",
},
"--workers 0": {
[]string{cmdDwim, workersFlag, "0", "-x", "--no-default-excludes"},
exitUsage,
"Error: --workers must be at least 1, got 0\n",
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
stdout := captureStdout(t)
stderr := captureStderr(t)
code := run(tc.args, os.Stdout, os.Stderr)
if code != tc.code {
t.Errorf("run(%q) = %d, want %d", tc.args, code, tc.code)
}
if !strings.HasSuffix(stderr(), tc.want) || stdout() != "" {
t.Errorf("stdout, stderr = %q, %q, want \"\" and stderr "+
"ending %q", stdout(), stderr(), tc.want)
}
_, err := os.Lstat(path)
if !errors.Is(err, os.ErrNotExist) {
t.Errorf("lstat %s: %v, want no database", path, err)
}
})
}
}
func TestRunDwimHelpListsScanFlags(t *testing.T) {
t.Parallel()
// README "Subcommands" shows this usage line for dwim.
const want = "\n sfdupes dwim [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]...\n"
var stdout, stderr bytes.Buffer
run([]string{cmdDwim, "-h"}, &stdout, &stderr)
if !strings.Contains(stderr.String(), want) {
t.Errorf("dwim -h stderr = %q, want the usage line %q",
stderr.String(), want)
}
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
t.Errorf("dwim -h stderr = %q, want %s PATH", stderr.String(),
excludeFlag)
}
// README "Settings": the help lists the default excludes and the
// filesystem types the walk never records anything on.
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
if !strings.Contains(stderr.String(), help) {
t.Errorf("dwim -h stderr = %q, want it to hold %q", stderr.String(),
help)
}
}
+1 -1
View File
@@ -254,7 +254,7 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
return devices, lines.Err() return devices, lines.Err()
} }
// defaultExcludesHelp describes, for the scan and dwim help, the // defaultExcludesHelp describes, for the scan command's help, the
// default excludes of the system goos names and, where the scan lists // default excludes of the system goos names and, where the scan lists
// the mounted filesystems, the types it never records anything on. // the mounted filesystems, the types it never records anything on.
func defaultExcludesHelp(goos string) string { func defaultExcludesHelp(goos string) string {
+14 -13
View File
@@ -98,16 +98,17 @@ func scanPhase(name string) string {
// runs, and the lock is released after it. When ctx is cancelled, as by // runs, and the lock is released after it. When ctx is cancelled, as by
// the SIGINT or SIGTERM that interruptContext catches, the scan keeps // the SIGINT or SIGTERM that interruptContext catches, the scan keeps
// what it has hashed (see syncScan), prints how many files its walk // what it has hashed (see syncScan), prints how many files its walk
// reached, and returns errInterrupted. opts.workers must be at least 1; // reached, and returns errInterrupted. workers must be at least 1; the
// the scan and dwim commands reject anything less. Before the lock is // scan command rejects anything less. Before the lock is taken, the
// taken, the scan lists the filesystems that are not persistent, on // scan lists the filesystems that are not persistent, on which the walk
// which the walk records nothing; unless opts.noDefaultExcludes is set, // records nothing; unless noDefaultExcludes is set, the walk does not
// the walk does not enter the default excludes either. The walk neither // enter the default excludes either. The walk neither enters nor
// enters nor records the paths in opts.excludePaths, the --exclude // records the paths in excludePaths, the --exclude values; an operand
// values; an operand at or below one is a fatal error. Once the // at or below one is a fatal error. Once the database is open, the scan
// database is open, the scan notes its files, which the walk passes // notes its files, which the walk passes over.
// over. func runScan(ctx context.Context, roots []string, workers int,
func runScan(ctx context.Context, roots []string, opts scanOptions) error { oneFS, noDefaultExcludes bool, excludePaths []string,
) error {
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
return err return err
@@ -122,7 +123,7 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
var rules walkRules var rules walkRules
rules.excludePaths, err = excludePathInodes(opts.excludePaths) rules.excludePaths, err = excludePathInodes(excludePaths)
if err != nil { if err != nil {
return fmt.Errorf("--exclude: %w", err) return fmt.Errorf("--exclude: %w", err)
} }
@@ -137,7 +138,7 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
return fmt.Errorf("list the mounted filesystems: %w", err) return fmt.Errorf("list the mounted filesystems: %w", err)
} }
if !opts.noDefaultExcludes { if !noDefaultExcludes {
rules.defaultExcludes = defaultExcludes(runtime.GOOS) rules.defaultExcludes = defaultExcludes(runtime.GOOS)
} }
@@ -166,7 +167,7 @@ func runScan(ctx context.Context, roots []string, opts scanOptions) error {
return err return err
} }
st, err := syncScan(ctx, db, roots, opts.workers, opts.oneFS, rules) st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
if errors.Is(err, context.Canceled) { if errors.Is(err, context.Canceled) {
return interrupted(st.walked) return interrupted(st.walked)
} }
+1 -1
View File
@@ -2196,7 +2196,7 @@ func TestScanHashWriteFailureUnwindsPool(t *testing.T) {
var stderr bytes.Buffer var stderr bytes.Buffer
code := run([]string{ code := run([]string{
cmdScan, workersFlag, strconv.Itoa(hashLeakWorkers), dir, cmdScan, "--workers", strconv.Itoa(hashLeakWorkers), dir,
}, io.Discard, &stderr) }, io.Discard, &stderr)
if code != exitFatal { if code != exitFatal {
t.Fatalf("run(scan) = %d, want %d; stderr: %s", t.Fatalf("run(scan) = %d, want %d; stderr: %s",