Compare commits

...
Author SHA1 Message Date
clawbot a595dd96ae Show the walk's path on the line below its display (closes #160)
check / check (push) Canceled after 0s
On a terminal, each draw of the walk shows the path of the last file
it found on the line below, cut short before the last column with
wide characters counted as two columns. The progressbar library's
detail row draws it. Control characters in the path are written out,
as \t or \x1b, so that the terminal does not act on them. The line is
erased before a warning and when the walk ends, so the next phase is
drawn on it. Nothing changes when stderr is not a terminal.

github.com/rivo/uniseg, already in the module graph through the
progressbar library, becomes a direct dependency for the column count.

Model: opus-5-5
2026-11-11 07:59:14 +00:00
clawbot a9ab6124ae dwim: take the scan flags (closes #162) (#193)
check / check (push) Canceled after 0s
2026-11-11 08:37:18 +01:00
clawbot db56fd0e4c Say in --help where the database is and what each report prints (closes #177) (#189)
check / check (push) Canceled after 0s
2026-11-11 07:22:28 +01:00
13 changed files with 721 additions and 126 deletions

No files matched your search

+45 -26
View File
@@ -22,9 +22,8 @@ records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` work from the database and system's job scheduler. `report`, `trees` and `info` work from the database and
can run at any time. `report` and `trees` leave out a copy that was deleted can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first first look at a new machine with one command.
look at a new machine with one command.
## Getting Started ## Getting Started
@@ -255,15 +254,14 @@ In order:
- `report` prints the file-level duplicate report from the database. - `report` prints the file-level duplicate report from the database.
- `trees` prints the duplicate-tree report from the database. - `trees` prints the duplicate-tree report from the database.
- `info` prints the database's details. - `info` prints the database's details.
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree - `dwim` scans `/`, then prints the duplicate-tree report.
report.
``` ```
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH... sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
sfdupes report > dupes.tsv sfdupes report > dupes.tsv
sfdupes trees > dupetrees.tsv sfdupes trees > dupetrees.tsv
sfdupes info sfdupes info
sfdupes dwim sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]...
sfdupes --version sfdupes --version
sfdupes [command] --help sfdupes [command] --help
``` ```
@@ -293,20 +291,20 @@ The reference below is also the program's specification.
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
it. The path names the file exactly, whatever characters it holds. A relative it. The path names the file exactly, whatever characters it holds. A relative
path is relative to the working directory. path is relative to the working directory.
- `--workers N` (`scan`) sets how many directories `scan` reads at once while it - `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once
lists the trees. It also sets how many files `scan` reads at once while it while it lists the trees. It also sets how many files `scan` reads at once
hashes them. The default is the number of CPUs. `N` must be at least 1. A while it hashes them. The default is the number of CPUs. `N` must be at
smaller value is a usage error, reported in one line on stderr with exit 2 least 1. A smaller value is a usage error, reported in one line on stderr with
before anything is scanned. exit 2 before anything is scanned.
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of - `-x`, `--one-file-system` (`scan`, `dwim`) keeps the walk of each operand out
directories on other filesystems, as the same flag does for `du` and `rsync`. of directories on other filesystems, as the same flag does for `du` and
By default the walk also enters directories on other filesystems. `rsync`. By default the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes. - `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
Those are the system and temporary directories the walk otherwise leaves out, excludes. Those are the system and temporary directories the walk otherwise
such as `/var/tmp` on Linux (see "Rules for the walk"). leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
- `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below - `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
it. `PATH` may be a directory or a file. The flag may be given more than once. below it. `PATH` may be a directory or a file. The flag may be given more than
The walk knows `PATH` by its device ID and inode number, as it knows the once. The walk knows `PATH` by its device ID and inode number, as it knows the
database files (see "Rules for the walk"). So the walk leaves `PATH` out database files (see "Rules for the walk"). So the walk leaves `PATH` out
whatever path leads to it. That includes a hard link to a file given as whatever path leads to it. That includes a hard link to a file given as
`PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in `PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in
@@ -317,8 +315,10 @@ The reference below is also the program's specification.
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and - `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
exits 0. It writes nothing to stderr. exits 0. It writes nothing to stderr.
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr - `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
default excludes of the system it runs on, and the filesystem types the walk names the database. The help of `report`, `trees`, `info` and `dwim` says what
each prints. `sfdupes scan --help` and `sfdupes dwim --help` also list the
default excludes of the system they run on, and the filesystem types the walk
never records anything on. never records anything on.
### Duplicate detection ### Duplicate detection
@@ -857,14 +857,25 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
### `dwim` mode ### `dwim` mode
`dwim`, short for "do what I mean", scans a new machine and reports its `dwim`, short for "do what I mean", scans a new machine and reports its
duplicate trees in one command. It takes no positional arguments and no flags of duplicate trees in one command. It takes no positional arguments. Like `scan`,
its own. Like `scan`, it should run as root, so that it can read every file. it should run as root, so that it can read every file.
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default `dwim` takes these flags of `scan`, with the same meaning (see "Settings"):
excludes and crosses into other filesystems. `--workers` keeps its default. The
- `--workers N`;
- `-x`, `--one-file-system`;
- `--no-default-excludes`;
- `--exclude PATH`, which may be given more than once.
`dwim` runs what `sfdupes scan /` runs with the same flags. With no flags, the
walk skips the default excludes and crosses into other filesystems. The
database, its creation and the scan lock are those of `scan`. Then `dwim` runs database, its creation and the scan lock are those of `scan`. Then `dwim` runs
what `sfdupes trees` runs. The largest trees come first. what `sfdupes trees` runs. The largest trees come first.
```
sfdupes dwim -x --exclude /srv/cache
```
The report goes to stdout, exactly as `trees` prints it. On stderr comes what The report goes to stdout, exactly as `trees` prints it. On stderr comes what
`scan` prints, then what `trees` prints, then one last line: `scan` prints, then what `trees` prints, then one last line:
@@ -1190,6 +1201,14 @@ On a terminal:
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ | ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
``` ```
- During `scan`'s `walk`, the line below the display shows the path of the last
file the walk found. Each draw shows the path of that moment, cut short before
the terminal's last column. The path is escaped as in the reports (see "Report
output format"). Each control character left after that is written as `\x` and
its code in two hexadecimal digits, such as `\x1b` for the escape character. A
control character is one that a terminal acts on instead of showing. The line
is erased when the walk ends, and the next phase is drawn on it.
- No line reaches the terminal's last column. A line too wide for the terminal - No line reaches the terminal's last column. A line too wide for the terminal
first leaves out its rate, the words of a stalled phase, or what the phase is first leaves out its rate, the words of a stalled phase, or what the phase is
waiting for. It leaves out its bar next. It is cut short at the edge only when waiting for. It leaves out its bar next. It is cut short at the edge only when
+15
View File
@@ -29,6 +29,21 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- On a terminal, the line below the `walk` display shows the path of the last
file the walk found, at each draw. Its control characters are written out, as
`\t` or `\x1b`, so that the terminal does not act on them. The line is erased
when the walk ends, and the next phase is drawn on it (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/160)
- `dwim` takes the `scan` flags `--workers`, `-x`, `--no-default-excludes` and
`--exclude`, with the same meaning. Both commands register and check them with
the same code (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/162)
- The help of `sfdupes` and of `scan` names the database, its default path and
`SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
order of the rows and the escapes. The help of `info` lists its lines. The
help of `dwim` says that it scans `/`, should run as root and keeps the scan
in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither - New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
enters nor records an excluded path, and keeps the records below it. An enters nor records an excluded path, and keeps the records below it. An
operand at or below one is a fatal error (2026-11-11, operand at or below one is a fatal error (2026-11-11,
+1 -1
View File
@@ -3,6 +3,7 @@ module sneak.berlin/go/sfdupes
go 1.25.7 go 1.25.7
require ( require (
github.com/rivo/uniseg v0.4.7
github.com/schollz/progressbar/v3 v3.19.1 github.com/schollz/progressbar/v3 v3.19.1
github.com/spf13/cobra v1.10.2 github.com/spf13/cobra v1.10.2
golang.org/x/sys v0.46.0 golang.org/x/sys v0.46.0
@@ -18,7 +19,6 @@ require (
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/spf13/pflag v1.0.9 // indirect github.com/spf13/pflag v1.0.9 // indirect
modernc.org/libc v1.74.1 // indirect modernc.org/libc v1.74.1 // indirect
modernc.org/mathutil v1.7.1 // indirect modernc.org/mathutil v1.7.1 // indirect
+6 -7
View File
@@ -358,8 +358,9 @@ func TestRunScanInterrupted(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
dir := buildSmokeTree(t) dir := buildSmokeTree(t)
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false, opts := scanOptions{workers: walkCancelWorkers}
false, nil)
err := runScan(t.Context(), []string{dir}, opts)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -382,8 +383,7 @@ func TestRunScanInterrupted(t *testing.T) {
go func() { go func() {
defer close(done) defer close(done)
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers, err = runScan(cancelledContext(t), []string{dir}, opts)
false, false, nil)
}() }()
awaitReturn(t, done, "runScan") awaitReturn(t, done, "runScan")
@@ -410,8 +410,7 @@ func TestRunScanInterrupted(t *testing.T) {
got, before) got, before)
} }
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false, err = runScan(t.Context(), []string{dir}, opts)
false, nil)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -437,7 +436,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
dir := buildWalkCancelTree(t) dir := buildWalkCancelTree(t)
err := runScan(newWalkClock(hashCancelAtDone), []string{dir}, err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
walkCancelWorkers, false, false, nil) scanOptions{workers: walkCancelWorkers})
if !errors.Is(err, errInterrupted) { if !errors.Is(err, errInterrupted) {
t.Fatalf("runScan interrupted mid-hash = %v, want %v", t.Fatalf("runScan interrupted mid-hash = %v, want %v",
err, errInterrupted) err, errInterrupted)
+153 -51
View File
@@ -32,17 +32,79 @@ const (
cmdDwim = "dwim" cmdDwim = "dwim"
) )
// databaseHelp is the paragraph of the root and scan help that names
// the database (README "Location").
const databaseHelp = "scan and dwim store what they find in a database " +
"file. report, trees\nand info read it. The file is $" + databaseEnv +
" when that is set and\nnot empty, and otherwise " +
defaultDatabasePath + ", where creating it\nneeds root.\n"
// escapesHelp is the paragraph of the report and trees help on how
// paths are escaped (README "Report output format").
const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
\\ a backslash
\t a tab
\n a newline
\r a carriage return
`
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
// summary in the help of their subcommands.
const (
reportHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate file. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that file
size the file's size in bytes
Rows are ordered by size, largest first, then by path.
` + escapesHelp
treesHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate directory tree. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that tree
files the number of regular files in one copy
size their total size in bytes
disk_size the bytes the dupe copy takes on disk
Rows are ordered by size, largest first, then by path.
` + escapesHelp
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
value, separated by a tab:
database the path, and where it came from
database file size in bytes, then rounded, as 2147483648 (2.0 GiB)
-wal file size in bytes, then rounded; only if a -wal file exists
modified when the database file last changed, in UTC
schema version the version of the database's layout
records the number of records
size of recorded files their total size, in bytes, then rounded
`
dwimHelp = `dwim runs what "sfdupes scan /" runs with the same flags, then what
"sfdupes trees" runs. Run it as root, so that the scan can read every
file. The scan is stored in the database, as scan stores it, so
"sfdupes trees" prints the report again later.
`
)
// errNoSubcommand is returned by the root command when it is invoked // errNoSubcommand is returned by the root command when it is invoked
// without a subcommand. That is a usage error, and the usage text // without a subcommand. That is a usage error, and the usage text
// cobra prints for it is the whole message. // cobra prints for it is the whole message.
var errNoSubcommand = errors.New("no subcommand") var errNoSubcommand = errors.New("no subcommand")
// errWorkersBelowOne is the usage error for a scan --workers value // errWorkersBelowOne is the usage error for a --workers value below 1.
// below 1.
var errWorkersBelowOne = errors.New("--workers must be at least 1") var errWorkersBelowOne = errors.New("--workers must be at least 1")
// errEmptyExcludeValue is the usage error for an empty scan --exclude // errEmptyExcludeValue is the usage error for an empty --exclude value,
// value, as from --exclude "$DIR" with DIR unset. Taken as a path, it // as from --exclude "$DIR" with DIR unset. Taken as a path, it
// would name the working directory. // would name the working directory.
var errEmptyExcludeValue = errors.New("--exclude must not be empty") var errEmptyExcludeValue = errors.New("--exclude must not be empty")
@@ -193,76 +255,116 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}), }),
} }
dwimCmd := &cobra.Command{ root.Long = root.Short + ".\n\n" + databaseHelp
Use: cmdDwim, reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
Short: "Scan / with the default excludes, then print the duplicate-tree report", treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
Args: cobra.NoArgs, infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", stdout)
}),
}
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd) root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd,
newDwimCommand(stdout))
return root return root
} }
// newScanCommand builds the scan subcommand, with its --workers, -x, // scanOptions holds the values of the flags that scan and dwim share:
// --no-default-excludes and --exclude flags. Its help lists the default // --workers, -x, --no-default-excludes and --exclude.
// excludes of the system it runs on, and the filesystem types it never type scanOptions struct {
// records anything on. workers int
oneFS bool
noDefaultExcludes bool
excludePaths []string
}
// scanFlagsUsage names the flags that scan and dwim share, for their
// usage lines.
const scanFlagsUsage = "[--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]..."
// addScanFlags adds the flags that scan and dwim share to cmd, which
// stores their values in opts.
func addScanFlags(cmd *cobra.Command, opts *scanOptions) {
cmd.Flags().IntVar(&opts.workers, "workers", runtime.NumCPU(),
"concurrent workers for the walk and hash phases")
cmd.Flags().BoolVarP(&opts.oneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
cmd.Flags().BoolVar(&opts.noDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
// A string array, not a slice, so a comma in a path is not a
// separator.
cmd.Flags().StringArrayVar(&opts.excludePaths, "exclude", nil,
"never enter or record `PATH` or anything below it; repeatable")
}
// check rejects the flag values that are usage errors, before the
// command runs. It is the PreRunE of scan and dwim.
func (o *scanOptions) check(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, o.workers)
if err != nil {
return err
}
return rejectEmptyExcludeValue(cmd, o.excludePaths)
}
// newScanCommand builds the scan subcommand. Its help names the
// database, and lists the default excludes of the system it runs on and
// the filesystem types it never records anything on.
func newScanCommand() *cobra.Command { func newScanCommand() *cobra.Command {
var ( var opts scanOptions
scanWorkers int
scanOneFS bool
scanNoDefaultExcludes bool
scanExclude []string
)
short := "Walk trees and synchronize the scan database" short := "Walk trees and synchronize the scan database"
scanCmd := &cobra.Command{ scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " + Use: cmdScan + " " + scanFlagsUsage + " PATH...",
"[--exclude PATH]... PATH...",
// The Use line already names every flag, so cobra must not // The Use line already names every flag, so cobra must not
// append its generic "[flags]" to it. // append its generic "[flags]" to it.
DisableFlagsInUseLine: true, DisableFlagsInUseLine: true,
Short: short, Short: short,
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS), Long: short + ".\n\n" + databaseHelp + "\n" +
Args: cobra.MinimumNArgs(1), defaultExcludesHelp(runtime.GOOS),
PreRunE: func(cmd *cobra.Command, _ []string) error { Args: cobra.MinimumNArgs(1),
err := checkScanWorkers(cmd, scanWorkers) PreRunE: opts.check,
if err != nil {
return err
}
return rejectEmptyExcludeValue(cmd, scanExclude)
},
RunE: runE(func(ctx context.Context, args []string) error { RunE: runE(func(ctx context.Context, args []string) error {
ctx, stop := interruptContext(ctx) ctx, stop := interruptContext(ctx)
defer stop() defer stop()
return runScan(ctx, args, scanWorkers, scanOneFS, return runScan(ctx, args, opts)
scanNoDefaultExcludes, scanExclude)
}), }),
} }
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(), addScanFlags(scanCmd, &opts)
"concurrent workers for the walk and hash phases")
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
"do not cross filesystem boundaries")
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
false, "enter the default excludes, the paths listed above")
// A string array, not a slice, so a comma in a path is not a
// separator.
scanCmd.Flags().StringArrayVar(&scanExclude, "exclude", nil,
"never enter or record `PATH` or anything below it; repeatable")
return scanCmd return scanCmd
} }
// checkScanWorkers rejects a scan --workers value below 1. That is a // newDwimCommand builds the dwim subcommand, which takes the scan flags
// usage error reported in one line: cobra prints the returned message // with the meaning they have for scan. Its help says what dwim runs,
// without the usage text, and run exits 2. // then lists the default excludes and the filesystem types the scan
// never records anything on.
func newDwimCommand(stdout io.Writer) *cobra.Command {
var opts scanOptions
short := "Scan /, then print the duplicate-tree report"
dwimCmd := &cobra.Command{
Use: cmdDwim + " " + scanFlagsUsage,
DisableFlagsInUseLine: true,
Short: short,
Long: short + ".\n\n" + dwimHelp + "\n" +
defaultExcludesHelp(runtime.GOOS),
Args: cobra.NoArgs,
PreRunE: opts.check,
RunE: runE(func(ctx context.Context, _ []string) error {
return runDwim(ctx, "/", opts, stdout)
}),
}
addScanFlags(dwimCmd, &opts)
return dwimCmd
}
// checkScanWorkers rejects a --workers value below 1. That is a usage
// error reported in one line: cobra prints the returned message without
// the usage text, and run exits 2.
func checkScanWorkers(cmd *cobra.Command, workers int) error { func checkScanWorkers(cmd *cobra.Command, workers int) error {
if workers >= 1 { if workers >= 1 {
return nil return nil
@@ -273,7 +375,7 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers) return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
} }
// rejectEmptyExcludeValue rejects an empty scan --exclude value. That // rejectEmptyExcludeValue rejects an empty --exclude value. That
// is a usage error reported in one line, as for --workers below 1. // is a usage error reported in one line, as for --workers below 1.
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error { func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
if !slices.Contains(excludePaths, "") { if !slices.Contains(excludePaths, "") {
+77 -4
View File
@@ -22,6 +22,12 @@ import (
// error prints and a runtime failure must not. // error prints and a runtime failure must not.
const usageMarker = "Usage:" const usageMarker = "Usage:"
// helpFlag asks for the help text.
const helpFlag = "--help"
// workersFlag is the scan and dwim flag that sets the worker count.
const workersFlag = "--workers"
// TestMain stops the tests at once, saying why, when their temporary // TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing // directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail // there (README "Rules for the walk"), so most scan tests would fail
@@ -570,8 +576,8 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
}{ }{
cmdScan: { cmdScan: {
run: func() error { run: func() error {
return runScan(t.Context(), []string{missing}, 1, false, false, return runScan(t.Context(), []string{missing},
nil) scanOptions{workers: 1})
}, },
want: scanWant, want: scanWant,
}, },
@@ -662,7 +668,7 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
args := []string{cmdScan, "--workers", workers, t.TempDir()} args := []string{cmdScan, workersFlag, workers, t.TempDir()}
code := run(args, &stdout, &stderr) code := run(args, &stdout, &stderr)
if code != exitUsage { if code != exitUsage {
@@ -693,7 +699,7 @@ func TestRunHelp(t *testing.T) {
// README §Settings: help goes to stderr, exits 0, and leaves stdout // README §Settings: help goes to stderr, exits 0, and leaves stdout
// empty. // empty.
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}} cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
for _, args := range cases { for _, args := range cases {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -754,6 +760,73 @@ func TestRunScanUsageLine(t *testing.T) {
} }
} }
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
t.Parallel()
// README §Settings: the help names the database, and says what
// report, trees, info and dwim print.
database := []string{
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
}
escapes := []string{`\\`, `\t`, `\n`, `\r`}
// The subcommand whose help is printed; none for the root.
cases := []struct {
subcommand []string
want []string
}{
{nil, database},
{[]string{cmdScan}, database},
{
[]string{cmdReport},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n size ",
"largest first",
}, escapes...),
},
{
[]string{cmdTrees},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n files ",
"\n size ", "\n disk_size ", "largest first",
}, escapes...),
},
{
[]string{cmdInfo},
[]string{
"\n database ", "\n database file size ",
"\n -wal file size ", "\n modified ",
"\n schema version ", "\n records ",
"\n size of recorded files ",
},
},
{
[]string{cmdDwim},
[]string{
"sfdupes scan /", "as root", "sfdupes trees", "database",
},
},
}
for _, tc := range cases {
var stdout, stderr bytes.Buffer
args := slices.Concat(tc.subcommand, []string{helpFlag})
code := run(args, &stdout, &stderr)
if code != exitOK {
t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
}
for _, want := range tc.want {
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) help lacks %q:\n%s",
args, want, stderr.String())
}
}
}
}
func TestRunVersion(t *testing.T) { func TestRunVersion(t *testing.T) {
t.Parallel() t.Parallel()
+10 -10
View File
@@ -5,20 +5,20 @@ import (
"fmt" "fmt"
"io" "io"
"os" "os"
"runtime"
) )
// runDwim implements the dwim subcommand: it runs what "sfdupes scan // runDwim implements the dwim subcommand: it runs what "sfdupes scan
// root" runs with no flags, then what "sfdupes trees" runs, and ends // root" runs with the scan flags in opts, then what "sfdupes trees"
// with a line on stderr saying how to print the report again. The dwim // runs, and ends with a line on stderr saying how to print the report
// command passes "/" as root. It catches SIGINT and SIGTERM as scan // again. The dwim command passes "/" as root. It catches SIGINT and
// does, but only during the scan. An error from the scan, an interrupt // SIGTERM as scan does, but only during the scan. An error from the
// included, ends it before the report. During the report, a signal // scan, an interrupt included, ends it before the report. During the
// ends it at once, as it ends trees. // report, a signal ends it at once, as it ends trees.
func runDwim(ctx context.Context, root string, stdout io.Writer) error { func runDwim(
ctx context.Context, root string, opts scanOptions, stdout io.Writer,
) error {
scanCtx, stop := interruptContext(ctx) scanCtx, stop := interruptContext(ctx)
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false, err := runScan(scanCtx, []string{root}, opts)
nil)
stop() stop()
+129 -2
View File
@@ -2,7 +2,10 @@ package sfdupes //nolint:testpackage // calls the unexported runDwim
import ( import (
"bytes" "bytes"
"errors"
"os"
"path/filepath" "path/filepath"
"runtime"
"strings" "strings"
"testing" "testing"
) )
@@ -28,7 +31,7 @@ func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
err := runDwim(t.Context(), root, &stdout) err := runDwim(t.Context(), root, scanOptions{workers: 1}, &stdout)
if err != nil { if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr()) t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
} }
@@ -67,7 +70,7 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
stderr := captureStderr(t) stderr := captureStderr(t)
err := runDwim(t.Context(), missing, &stdout) err := runDwim(t.Context(), missing, scanOptions{workers: 1}, &stdout)
if err == nil { if err == nil {
t.Fatal("runDwim on a missing root: error = nil, want it fatal") t.Fatal("runDwim on a missing root: error = nil, want it fatal")
} }
@@ -81,3 +84,127 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
t.Errorf("stderr = %q, want nothing from trees or dwim", got) t.Errorf("stderr = %q, want nothing from trees or dwim", got)
} }
} }
func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
// dwim passes its --exclude paths to the scan: an excluded copy is
// not recorded, so its pair is not reported.
t.Setenv(databaseEnv, testDBPath(t))
root := realTempDir(t)
kept := []string{
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
}
writeFile(t, root, "b1/f.bin", pattern(2, 5000))
excluded := filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000)))
var stdout bytes.Buffer
stderr := captureStderr(t)
opts := scanOptions{workers: 1, excludePaths: []string{excluded}}
err := runDwim(t.Context(), root, opts, &stdout)
if err != nil {
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
}
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
if len(lines) != 2 {
t.Fatalf("stdout = %q, want a header and one group",
stdout.String())
}
want := kept[0] + "\t" + kept[1] + "\t1\t300\t"
if !strings.HasPrefix(lines[1], want) {
t.Errorf("group = %q, want it to start %q", lines[1], want)
}
skipped := "scan: skipped " + excluded + " (--exclude)\n"
if got := stderr(); !strings.Contains(got, skipped) {
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
}
}
func TestRunDwimTakesScanFlags(t *testing.T) {
// The dwim command parses and checks the scan flags as scan does.
// Each case stops before the walk of / begins, and before the
// database is created. "dwim --exclude /" excludes its own operand.
cases := map[string]struct {
args []string
code int
want string
}{
"--exclude /": {
[]string{cmdDwim, excludeFlag, "/"},
exitFatal,
"sfdupes: scan /: operand excluded by --exclude /\n",
},
"--exclude empty": {
[]string{cmdDwim, excludeFlag, ""},
exitUsage,
"Error: --exclude must not be empty\n",
},
"--workers 0": {
[]string{cmdDwim, workersFlag, "0", "-x", "--no-default-excludes"},
exitUsage,
"Error: --workers must be at least 1, got 0\n",
},
}
for name, tc := range cases {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
stdout := captureStdout(t)
stderr := captureStderr(t)
code := run(tc.args, os.Stdout, os.Stderr)
if code != tc.code {
t.Errorf("run(%q) = %d, want %d", tc.args, code, tc.code)
}
if !strings.HasSuffix(stderr(), tc.want) || stdout() != "" {
t.Errorf("stdout, stderr = %q, %q, want \"\" and stderr "+
"ending %q", stdout(), stderr(), tc.want)
}
_, err := os.Lstat(path)
if !errors.Is(err, os.ErrNotExist) {
t.Errorf("lstat %s: %v, want no database", path, err)
}
})
}
}
func TestRunDwimHelpListsScanFlags(t *testing.T) {
t.Parallel()
// README "Subcommands" shows this usage line for dwim.
const want = "\n sfdupes dwim [--workers N] [-x] [--no-default-excludes] " +
"[--exclude PATH]...\n"
var stdout, stderr bytes.Buffer
run([]string{cmdDwim, "-h"}, &stdout, &stderr)
if !strings.Contains(stderr.String(), want) {
t.Errorf("dwim -h stderr = %q, want the usage line %q",
stderr.String(), want)
}
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
t.Errorf("dwim -h stderr = %q, want %s PATH", stderr.String(),
excludeFlag)
}
// README "Settings": the help lists the default excludes and the
// filesystem types the walk never records anything on.
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
if !strings.Contains(stderr.String(), help) {
t.Errorf("dwim -h stderr = %q, want it to hold %q", stderr.String(),
help)
}
}
+1 -1
View File
@@ -254,7 +254,7 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
return devices, lines.Err() return devices, lines.Err()
} }
// defaultExcludesHelp describes, for the scan command's help, the // defaultExcludesHelp describes, for the scan and dwim help, the
// default excludes of the system goos names and, where the scan lists // default excludes of the system goos names and, where the scan lists
// the mounted filesystems, the types it never records anything on. // the mounted filesystems, the types it never records anything on.
func defaultExcludesHelp(goos string) string { func defaultExcludesHelp(goos string) string {
+107 -9
View File
@@ -9,8 +9,10 @@ import (
"strings" "strings"
"sync" "sync"
"time" "time"
"unicode"
"unicode/utf8" "unicode/utf8"
"github.com/rivo/uniseg"
"github.com/schollz/progressbar/v3" "github.com/schollz/progressbar/v3"
"golang.org/x/term" "golang.org/x/term"
) )
@@ -116,7 +118,8 @@ type drawnCount struct {
// terminal's width. A phase with an exact total ends the line with its // terminal's width. A phase with an exact total ends the line with its
// percent and bar, at the terminal's right edge, when the terminal has // percent and bar, at the terminal's right edge, when the terminal has
// room for them. It is drawn again every redrawInterval, so its elapsed // room for them. It is drawn again every redrawInterval, so its elapsed
// time keeps moving while the phase waits for its next item. When // time keeps moving while the phase waits for its next item. Each draw
// of the walk shows the path it reached last on the line below. When
// stderr is not a TTY it emits no ANSI redraws: it prints line as the // stderr is not a TTY it emits no ANSI redraws: it prints line as the
// phase starts, every plainInterval while it runs, and as it ends. // phase starts, every plainInterval while it runs, and as it ends.
// //
@@ -141,6 +144,7 @@ type progress struct {
estimate bool estimate bool
waitingFor string waitingFor string
stalledOn string stalledOn string
path string // the path the walk reached last; "" for none
count int64 count int64
start time.Time // when the phase started, for the elapsed time start time.Time // when the phase started, for the elapsed time
rateStart time.Time // when the count started, for the time remaining rateStart time.Time // when the count started, for the time remaining
@@ -239,6 +243,82 @@ func (p *progress) draw() {
// less the terminal's last column, which spinnerMargin keeps clear. A // less the terminal's last column, which spinnerMargin keeps clear. A
// bar's cells take one column each but more than one byte. // bar's cells take one column each but more than one byte.
p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1) p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1)
if p.path != "" {
// The library draws the path on the line below and returns to
// the phase's line. The path is escaped as warnings are, and its
// other control characters are written out, so that the terminal
// shows them instead of acting on them. It stays clear of the
// last column, so that no terminal wraps it.
shown := escapeControlChars(escapePath(p.path))
_ = p.bar.AddDetail(cutToColumns(shown, width-1))
}
}
// escapeControlChars returns s with each control character written as
// \x and its code in two hex digits, as \x1b for the escape character.
// The control characters are U+0000 to U+001F, U+007F and U+0080 to
// U+009F, as unicode.IsControl reports them. A byte from 0x80 to 0x9F
// that is not part of a valid UTF-8 character is written the same way,
// as \x9b, since a terminal not in UTF-8 mode acts on it. Every other
// byte is kept as it is.
func escapeControlChars(s string) string {
var b strings.Builder
for rest := s; rest != ""; {
char, size := utf8.DecodeRuneInString(rest)
if char == utf8.RuneError && size == 1 {
// Not part of a valid UTF-8 character. A terminal not in
// UTF-8 mode reads the byte as the character with its code,
// so that 0x9b is a control character there and 0xe9 is not.
char = rune(rest[0])
}
if unicode.IsControl(char) {
fmt.Fprintf(&b, `\x%02x`, char)
} else {
b.WriteString(rest[:size])
}
rest = rest[size:]
}
return b.String()
}
// cutToColumns returns s cut to at most columns terminal columns, at a
// character boundary. A wide character, as in Chinese, takes two
// columns.
func cutToColumns(s string, columns int) string {
used, end, state := 0, 0, -1
for rest := s; rest != ""; {
char, next, width, nextState := uniseg.FirstGraphemeClusterInString(
rest, state)
if used+width > columns {
break
}
used += width
end += len(char)
rest, state = next, nextState
}
return s[:end]
}
// clear erases the display from the terminal: its line, and the path on
// the line below when one is shown. p.mu must be held.
func (p *progress) clear() {
if p.bar == nil {
return
}
if p.path != "" {
_ = p.bar.AddDetail("")
}
_ = p.bar.Clear()
} }
// newBar has the library make what draw draws: its spinner, followed by // newBar has the library make what draw draws: its spinner, followed by
@@ -263,6 +343,8 @@ func (p *progress) newBar(width int) {
// The text carries the elapsed time and the time remaining. // The text carries the elapsed time and the time remaining.
progressbar.OptionSetElapsedTime(false), progressbar.OptionSetElapsedTime(false),
progressbar.OptionSetPredictTime(false), progressbar.OptionSetPredictTime(false),
// The line below the phase's line, for the walk's path.
progressbar.OptionSetMaxDetailRow(1),
) )
p.widest = 0 p.widest = 0
} }
@@ -314,6 +396,19 @@ func (p *progress) increment() {
p.lastItem = now p.lastItem = now
} }
// setPath records the path the walk reached last, which a display on a
// terminal shows at its next draw.
func (p *progress) setPath(path string) {
if p == nil {
return
}
p.mu.Lock()
defer p.mu.Unlock()
p.path = path
}
// warnf prints a one-line warning to stderr without corrupting the // warnf prints a one-line warning to stderr without corrupting the
// display. The whole message is escaped like a report's path columns, // display. The whole message is escaped like a report's path columns,
// so a path holding a newline cannot split the warning. // so a path holding a newline cannot split the warning.
@@ -327,11 +422,9 @@ func (p *progress) warnf(format string, args ...any) {
p.mu.Lock() p.mu.Lock()
defer p.mu.Unlock() defer p.mu.Unlock()
if p.bar != nil { // The warning takes the display's line; the next draw shows the
// The warning takes the display's line; the next draw shows // display again, on the line below.
// the display again, on the line below. p.clear()
_ = p.bar.Clear()
}
fmt.Fprintln(os.Stderr, msg) fmt.Fprintln(os.Stderr, msg)
} }
@@ -392,9 +485,7 @@ func (w *progressWriter) Flush() error {
w.p.mu.Lock() w.p.mu.Lock()
defer w.p.mu.Unlock() defer w.p.mu.Unlock()
if w.p.bar != nil { w.p.clear()
_ = w.p.bar.Clear()
}
} }
_, err := w.out.Write(w.buf) _, err := w.out.Write(w.buf)
@@ -414,6 +505,13 @@ func (p *progress) finish() {
close(p.stop) close(p.stop)
<-p.done <-p.done
// The walk's path is only for someone watching the walk. Its line is
// erased, and the next phase is drawn on it.
p.mu.Lock()
p.clear()
p.path = ""
p.mu.Unlock()
p.draw() p.draw()
if p.width != nil { if p.width != nil {
+162
View File
@@ -6,6 +6,7 @@ import (
"math" "math"
"os" "os"
"path/filepath" "path/filepath"
"regexp"
"slices" "slices"
"strings" "strings"
"sync" "sync"
@@ -1138,3 +1139,164 @@ func TestWarningAfterLastItemKeepsBar(t *testing.T) {
t.Errorf("terminal shows %q, want a count of 2 of 2", shown) t.Errorf("terminal shows %q, want a count of 2 of 2", shown)
} }
} }
func TestCutToColumns(t *testing.T) {
t.Parallel()
cases := []struct {
s string
columns int
want string
}{
{"/srv/a", 10, "/srv/a"},
{"/srv/photos", 8, "/srv/pho"},
// Each of these two fullwidth letters takes two columns.
{"/AB/a", 4, "/A"},
{"/AB/a", 5, "/AB"},
// An accent that combines with the letter before stays with it.
{"/café/a", 5, "/café"},
{"/srv", 0, ""},
}
for _, tc := range cases {
if got := cutToColumns(tc.s, tc.columns); got != tc.want {
t.Errorf("cutToColumns(%q, %d) = %q, want %q",
tc.s, tc.columns, got, tc.want)
}
}
}
func TestEscapeControlChars(t *testing.T) {
t.Parallel()
cases := map[string]string{
"/srv/music": "/srv/music",
"/a\x1b[2Jb": `/a\x1b[2Jb`,
"/a\x00\a\x7fb": `/a\x00\x07\x7fb`,
// U+009B takes two bytes in UTF-8.
"/a\u009b2Jb": `/a\x9b2Jb`,
// Byte 0x9b alone is not valid UTF-8. A terminal not in UTF-8
// mode acts on it, as on every byte from 0x80 to 0x9f.
"/a\x9b2Jb": `/a\x9b2Jb`,
"/a\x80\x9f\xa0b": `/a\x80\x9f` + "\xa0b",
// In ě, which is c4 9b in UTF-8, byte 0x9b is part of a valid
// character.
"/aě2Jb": "/aě2Jb",
"/café/AB": "/café/AB",
"/latin1-caf\xe9": "/latin1-caf\xe9",
}
for in, want := range cases {
if got := escapeControlChars(in); got != want {
t.Errorf("escapeControlChars(%q) = %q, want %q", in, got, want)
}
}
}
// pathLines returns the text of each line the library wrote below a
// display's line. It writes one as an erase to the end of the line, a
// carriage return, the text and a newline.
func pathLines(out string) []string {
matches := regexp.MustCompile("\x1b\\[K\r([^\n]*)\n").
FindAllStringSubmatch(out, -1)
lines := make([]string, 0, len(matches))
for _, m := range matches {
lines = append(lines, m[1])
}
return lines
}
// TestTerminalWalkShowsPathBelow checks that the walk's display on a
// terminal 30 columns wide shows the path it reached last on the line
// below, cut to 29 columns. That line is erased before a warning is
// printed, and again when the walk ends, so the next phase is drawn on
// an empty line.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalWalkShowsPathBelow(t *testing.T) {
stderr := captureStderr(t)
p := startProgress(progressParams{
label: scanPhase("walk"), unit: scanUnit, total: -1,
}, func() int { return 30 }, testRedraw, time.Now)
const shown = "/srv/photos/2026/holiday/beac"
p.setPath("/srv/photos/2026/holiday/beach.jpg")
p.increment()
awaitOutput(t, stderr, shown, 1)
p.warnf("warning")
before, _, _ := strings.Cut(stderr(), "warning")
if lines := pathLines(before); lines[len(lines)-1] != "" {
t.Errorf("line below shows %q as the warning is printed, want it "+
"empty", lines[len(lines)-1])
}
// Drawn again below the warning.
awaitOutput(t, stderr, shown, strings.Count(before, shown)+1)
p.finish()
lines := pathLines(stderr())
if !slices.Contains(lines, shown) {
t.Errorf("lines below the display = %q, want %q among them",
lines, shown)
}
if last := lines[len(lines)-1]; last != "" {
t.Errorf("line below the display shows %q after the walk, want it "+
"empty", last)
}
}
// TestTerminalWalkEscapesControlChars checks that a walked path holding
// escape characters, which would clear the screen and set the window
// title, puts none on the line below the walk's display. The line shows
// each control character written out, as \x1b.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestTerminalWalkEscapesControlChars(t *testing.T) {
stderr := captureStderr(t)
p := startProgress(progressParams{
label: scanPhase("walk"), unit: scanUnit, total: -1,
}, func() int { return 80 }, testRedraw, time.Now)
p.setPath("/srv/\x1b[2J\x1b]0;title\a/beach.jpg")
p.increment()
awaitOutput(t, stderr, "beach.jpg", 1)
p.finish()
lines := pathLines(stderr())
for _, line := range lines {
if strings.Contains(line, "\x1b") {
t.Errorf("line below the display = %q, want no escape "+
"character", line)
}
}
const shown = `/srv/\x1b[2J\x1b]0;title\x07/beach.jpg`
if !slices.Contains(lines, shown) {
t.Errorf("lines below the display = %q, want %q among them",
lines, shown)
}
}
// TestPlainWalkShowsNoPath checks that the walk's display prints no path
// when stderr is not a terminal.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestPlainWalkShowsNoPath(t *testing.T) {
stderr := captureStderr(t)
p := startProgress(progressParams{
label: scanPhase("walk"), unit: scanUnit, total: -1,
}, nil, testRedraw, time.Now)
p.setPath("/srv/a")
p.increment()
p.finish()
if strings.Contains(stderr(), "/srv/a") {
t.Errorf("stderr = %q, want no path", stderr())
}
}
+14 -14
View File
@@ -98,17 +98,16 @@ func scanPhase(name string) string {
// runs, and the lock is released after it. When ctx is cancelled, as by // runs, and the lock is released after it. When ctx is cancelled, as by
// the SIGINT or SIGTERM that interruptContext catches, the scan keeps // the SIGINT or SIGTERM that interruptContext catches, the scan keeps
// what it has hashed (see syncScan), prints how many files its walk // what it has hashed (see syncScan), prints how many files its walk
// reached, and returns errInterrupted. workers must be at least 1; the // reached, and returns errInterrupted. opts.workers must be at least 1;
// scan command rejects anything less. Before the lock is taken, the // the scan and dwim commands reject anything less. Before the lock is
// scan lists the filesystems that are not persistent, on which the walk // taken, the scan lists the filesystems that are not persistent, on
// records nothing; unless noDefaultExcludes is set, the walk does not // which the walk records nothing; unless opts.noDefaultExcludes is set,
// enter the default excludes either. The walk neither enters nor // the walk does not enter the default excludes either. The walk neither
// records the paths in excludePaths, the --exclude values; an operand // enters nor records the paths in opts.excludePaths, the --exclude
// at or below one is a fatal error. Once the database is open, the scan // values; an operand at or below one is a fatal error. Once the
// notes its files, which the walk passes over. // database is open, the scan notes its files, which the walk passes
func runScan(ctx context.Context, roots []string, workers int, // over.
oneFS, noDefaultExcludes bool, excludePaths []string, func runScan(ctx context.Context, roots []string, opts scanOptions) error {
) error {
dbPath, source, err := databasePath() dbPath, source, err := databasePath()
if err != nil { if err != nil {
return err return err
@@ -123,7 +122,7 @@ func runScan(ctx context.Context, roots []string, workers int,
var rules walkRules var rules walkRules
rules.excludePaths, err = excludePathInodes(excludePaths) rules.excludePaths, err = excludePathInodes(opts.excludePaths)
if err != nil { if err != nil {
return fmt.Errorf("--exclude: %w", err) return fmt.Errorf("--exclude: %w", err)
} }
@@ -138,7 +137,7 @@ func runScan(ctx context.Context, roots []string, workers int,
return fmt.Errorf("list the mounted filesystems: %w", err) return fmt.Errorf("list the mounted filesystems: %w", err)
} }
if !noDefaultExcludes { if !opts.noDefaultExcludes {
rules.defaultExcludes = defaultExcludes(runtime.GOOS) rules.defaultExcludes = defaultExcludes(runtime.GOOS)
} }
@@ -167,7 +166,7 @@ func runScan(ctx context.Context, roots []string, workers int,
return err return err
} }
st, err := syncScan(ctx, db, roots, workers, oneFS, rules) st, err := syncScan(ctx, db, roots, opts.workers, opts.oneFS, rules)
if errors.Is(err, context.Canceled) { if errors.Is(err, context.Canceled) {
return interrupted(st.walked) return interrupted(st.walked)
} }
@@ -563,6 +562,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
s.st.walked++ s.st.walked++
prog.setPath(ev.rec.path)
prog.increment() prog.increment()
old, ok := s.existing[ev.rec.path] old, ok := s.existing[ev.rec.path]
+1 -1
View File
@@ -2196,7 +2196,7 @@ func TestScanHashWriteFailureUnwindsPool(t *testing.T) {
var stderr bytes.Buffer var stderr bytes.Buffer
code := run([]string{ code := run([]string{
cmdScan, "--workers", strconv.Itoa(hashLeakWorkers), dir, cmdScan, workersFlag, strconv.Itoa(hashLeakWorkers), dir,
}, io.Discard, &stderr) }, io.Discard, &stderr)
if code != exitFatal { if code != exitFatal {
t.Fatalf("run(scan) = %d, want %d; stderr: %s", t.Fatalf("run(scan) = %d, want %d; stderr: %s",