Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
dc82531aec | ||
 clawbot
|
db56fd0e4c |
No files matched your search
@@ -22,9 +22,8 @@ records the files it finds in an SQLite database. SQLite keeps a whole database
|
||||
in one file. The database is kept between runs. A rescan reads only new and
|
||||
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
||||
system's job scheduler. `report`, `trees` and `info` work from the database and
|
||||
can run at any time. `report` and `trees` leave out a copy that was deleted
|
||||
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first
|
||||
look at a new machine with one command.
|
||||
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
|
||||
first look at a new machine with one command.
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -44,9 +43,7 @@ as opposed to a directory, a symlink or a device node. A symlink is a file that
|
||||
holds the path of another file and stands in for it. A device node is a file
|
||||
that stands for a piece of hardware, such as a disk or a terminal. `report`
|
||||
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
||||
Both cover the whole database, or only the paths given after them, as in
|
||||
`sfdupes report /srv/photos`. `info` prints the database's path, size and record
|
||||
count.
|
||||
`info` prints the database's path, size and record count.
|
||||
|
||||
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
||||
another file. The first `scan` creates it.
|
||||
@@ -257,15 +254,14 @@ In order:
|
||||
- `report` prints the file-level duplicate report from the database.
|
||||
- `trees` prints the duplicate-tree report from the database.
|
||||
- `info` prints the database's details.
|
||||
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
|
||||
report.
|
||||
- `dwim` scans `/`, then prints the duplicate-tree report.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] [--exclude PATH]... PATH...
|
||||
sfdupes report [PATH...] > dupes.tsv
|
||||
sfdupes trees [PATH...] > dupetrees.tsv
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
sfdupes dwim
|
||||
sfdupes dwim [--workers N] [-x] [--no-default-excludes] [--exclude PATH]...
|
||||
sfdupes --version
|
||||
sfdupes [command] --help
|
||||
```
|
||||
@@ -295,20 +291,20 @@ The reference below is also the program's specification.
|
||||
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
|
||||
it. The path names the file exactly, whatever characters it holds. A relative
|
||||
path is relative to the working directory.
|
||||
- `--workers N` (`scan`) sets how many directories `scan` reads at once while it
|
||||
lists the trees. It also sets how many files `scan` reads at once while it
|
||||
hashes them. The default is the number of CPUs. `N` must be at least 1. A
|
||||
smaller value is a usage error, reported in one line on stderr with exit 2
|
||||
before anything is scanned.
|
||||
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
|
||||
directories on other filesystems, as the same flag does for `du` and `rsync`.
|
||||
By default the walk also enters directories on other filesystems.
|
||||
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
|
||||
Those are the system and temporary directories the walk otherwise leaves out,
|
||||
such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||
- `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
|
||||
it. `PATH` may be a directory or a file. The flag may be given more than once.
|
||||
The walk knows `PATH` by its device ID and inode number, as it knows the
|
||||
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once
|
||||
while it lists the trees. It also sets how many files `scan` reads at once
|
||||
while it hashes them. The default is the number of CPUs. `N` must be at
|
||||
least 1. A smaller value is a usage error, reported in one line on stderr with
|
||||
exit 2 before anything is scanned.
|
||||
- `-x`, `--one-file-system` (`scan`, `dwim`) keeps the walk of each operand out
|
||||
of directories on other filesystems, as the same flag does for `du` and
|
||||
`rsync`. By default the walk also enters directories on other filesystems.
|
||||
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
|
||||
excludes. Those are the system and temporary directories the walk otherwise
|
||||
leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
|
||||
below it. `PATH` may be a directory or a file. The flag may be given more than
|
||||
once. The walk knows `PATH` by its device ID and inode number, as it knows the
|
||||
database files (see "Rules for the walk"). So the walk leaves `PATH` out
|
||||
whatever path leads to it. That includes a hard link to a file given as
|
||||
`PATH`. A symlink given as `PATH` excludes its target. An empty `PATH`, as in
|
||||
@@ -319,8 +315,10 @@ The reference below is also the program's specification.
|
||||
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
|
||||
exits 0. It writes nothing to stderr.
|
||||
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
|
||||
and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the
|
||||
default excludes of the system it runs on, and the filesystem types the walk
|
||||
and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
|
||||
names the database. The help of `report`, `trees`, `info` and `dwim` says what
|
||||
each prints. `sfdupes scan --help` and `sfdupes dwim --help` also list the
|
||||
default excludes of the system they run on, and the filesystem types the walk
|
||||
never records anything on.
|
||||
|
||||
### Duplicate detection
|
||||
@@ -688,12 +686,11 @@ and `scan` leaves it ignored.
|
||||
|
||||
### `report` mode
|
||||
|
||||
`report` reads every record from the database. It takes optional `PATH`
|
||||
operands. With none, it uses every record. With one or more, it uses only the
|
||||
records at or below them (see "Operands of `report` and `trees`"). `report`
|
||||
never opens a scanned file. It checks that each path in a group still exists,
|
||||
and looks at no other path. It never looks up an operand. The check is `lstat`,
|
||||
the system call that looks up a path without following a symlink:
|
||||
`report` reads every record from the database. It takes no positional arguments.
|
||||
A positional argument is anything on the command line that is not a flag, such
|
||||
as the paths `scan` takes. `report` never opens a scanned file. It checks that
|
||||
each path in a group still exists, and looks at no other path. The check is
|
||||
`lstat`, the system call that looks up a path without following a symlink:
|
||||
|
||||
- A path is gone when `lstat` finds that it, or a directory above it, does not
|
||||
exist.
|
||||
@@ -725,27 +722,6 @@ with one signature come together, and so do the records of one size. So `report`
|
||||
holds only the duplicate groups of the size it is reading. Once it reaches a
|
||||
smaller size, it writes them, ordered by `first` path.
|
||||
|
||||
#### Operands of `report` and `trees`
|
||||
|
||||
`report` and `trees` take the same `PATH` operands:
|
||||
|
||||
- A record is used when its path is an operand or lies below one. `/srv/photos`
|
||||
covers `/srv/photos/x` but not `/srv/photos2/x`.
|
||||
- The records used are grouped, ordered and totalled as they are without
|
||||
operands. A copy outside the operands is not reported. With copies `/a/f`,
|
||||
`/srv/photos/x` and `/srv/photos/y`, `sfdupes report /srv/photos` prints one
|
||||
row, `/srv/photos/x` and `/srv/photos/y`.
|
||||
- A relative operand is joined to the working directory as `pwd -P` prints it.
|
||||
- Each operand is then cleaned as text, as `realpath -ms` prints it. Its `.`
|
||||
elements and repeated or trailing slashes are dropped. Each `..` drops the
|
||||
element before it.
|
||||
- No operand is looked up on the filesystem. A symlink in an operand is not
|
||||
followed, and an operand need not exist. The records hold real paths (see
|
||||
"Operands" under "`scan` mode"), so an operand that passes through a symlink
|
||||
matches no record.
|
||||
- An operand that matches no record is not an error. The report then has no
|
||||
rows.
|
||||
|
||||
#### Report output format
|
||||
|
||||
The report is TSV (tab-separated values) on stdout. It holds a header line, then
|
||||
@@ -773,8 +749,7 @@ Grouping and ordering use the stored path. The warnings about skipped paths that
|
||||
|
||||
The summary line on stderr gives:
|
||||
|
||||
- the records read, which are all the records in the database, with or without
|
||||
operands;
|
||||
- the records read;
|
||||
- the duplicate groups;
|
||||
- the dupe files, which is the number of rows;
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
@@ -785,12 +760,11 @@ report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
||||
|
||||
### `trees` mode
|
||||
|
||||
`trees` reads the same database as `report` and takes the same `PATH` operands
|
||||
(see "Operands of `report` and `trees`"). It reports whole duplicate directory
|
||||
trees. Two directories are duplicate trees when they hold the same relative
|
||||
paths, and each path holds a file with the same signature in both. It rebuilds
|
||||
the directory tree from the paths in the records, split on `/`. Like `report`,
|
||||
it never opens a scanned file and never looks up an operand. It checks only that
|
||||
`trees` reads the same database as `report` and takes no positional arguments.
|
||||
It reports whole duplicate directory trees. Two directories are duplicate trees
|
||||
when they hold the same relative paths, and each path holds a file with the same
|
||||
signature in both. It rebuilds the directory tree from the paths in the records,
|
||||
split on `/`. Like `report`, it never opens a scanned file. It checks only that
|
||||
the directory at the top of each copy still exists, as `report` checks a path.
|
||||
|
||||
Definitions:
|
||||
@@ -818,11 +792,6 @@ Processing:
|
||||
directories that all have the same digest. The parents' group, or an
|
||||
ancestor's, already implies it. A group is always reported when it holds
|
||||
sibling directories, or when its members' parents have different digests.
|
||||
- With operands, `trees` builds the tree from the records it uses. It reports
|
||||
only directories at or below an operand. Any other directory lies above an
|
||||
operand and can hold files that are not used, so its digest does not describe
|
||||
it. Such a directory is never reported and never implies a group of its
|
||||
children.
|
||||
- A directory that is gone is dropped from its group. A group left with fewer
|
||||
than two directories is not reported.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
@@ -876,8 +845,7 @@ directory's path is `/`.
|
||||
|
||||
The summary line on stderr gives:
|
||||
|
||||
- the records read, which are all the records in the database, with or without
|
||||
operands;
|
||||
- the records read;
|
||||
- the duplicate-tree groups;
|
||||
- the dupe trees, which is the number of rows;
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
@@ -889,16 +857,25 @@ trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimab
|
||||
### `dwim` mode
|
||||
|
||||
`dwim`, short for "do what I mean", scans a new machine and reports its
|
||||
duplicate trees in one command. It takes no positional arguments and no flags of
|
||||
its own. A positional argument is anything on the command line that is not a
|
||||
flag, such as the paths `scan` takes. Like `scan`, it should run as root, so
|
||||
that it can read every file.
|
||||
duplicate trees in one command. It takes no positional arguments. Like `scan`,
|
||||
it should run as root, so that it can read every file.
|
||||
|
||||
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
|
||||
excludes and crosses into other filesystems. `--workers` keeps its default. The
|
||||
`dwim` takes these flags of `scan`, with the same meaning (see "Settings"):
|
||||
|
||||
- `--workers N`;
|
||||
- `-x`, `--one-file-system`;
|
||||
- `--no-default-excludes`;
|
||||
- `--exclude PATH`, which may be given more than once.
|
||||
|
||||
`dwim` runs what `sfdupes scan /` runs with the same flags. With no flags, the
|
||||
walk skips the default excludes and crosses into other filesystems. The
|
||||
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
|
||||
what `sfdupes trees` runs. The largest trees come first.
|
||||
|
||||
```
|
||||
sfdupes dwim -x --exclude /srv/cache
|
||||
```
|
||||
|
||||
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
|
||||
`scan` prints, then what `trees` prints, then one last line:
|
||||
|
||||
@@ -1283,7 +1260,7 @@ showing the count it stopped at.
|
||||
- `scan` with no operand;
|
||||
- `--workers` below 1;
|
||||
- an empty `--exclude` value, as in `--exclude ""`;
|
||||
- `info` or `dwim` with a positional argument.
|
||||
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
||||
|
||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||
stdout, such as a full disk, is reported the same way. Two cases never reach
|
||||
|
||||
@@ -29,11 +29,15 @@
|
||||
Newest first. An entry ends with its date and its issue or branch, where those
|
||||
are known.
|
||||
|
||||
- `report` and `trees` take optional `PATH` operands. With operands, they use
|
||||
only the records at or below them. `trees` then reports only directories at or
|
||||
below an operand. Operands are made absolute and cleaned as text, never looked
|
||||
up on the filesystem (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/174)
|
||||
- `dwim` takes the `scan` flags `--workers`, `-x`, `--no-default-excludes` and
|
||||
`--exclude`, with the same meaning. Both commands register and check them with
|
||||
the same code (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/162)
|
||||
|
||||
- The help of `sfdupes` and of `scan` names the database, its default path and
|
||||
`SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
|
||||
order of the rows and the escapes. The help of `info` lists its lines. The
|
||||
help of `dwim` says that it scans `/`, should run as root and keeps the scan
|
||||
in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
|
||||
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
|
||||
enters nor records an excluded path, and keeps the records below it. An
|
||||
operand at or below one is a fatal error (2026-11-11,
|
||||
|
||||
@@ -358,8 +358,9 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
dir := buildSmokeTree(t)
|
||||
|
||||
err := runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
||||
false, nil)
|
||||
opts := scanOptions{workers: walkCancelWorkers}
|
||||
|
||||
err := runScan(t.Context(), []string{dir}, opts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -382,8 +383,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
go func() {
|
||||
defer close(done)
|
||||
|
||||
err = runScan(cancelledContext(t), []string{dir}, walkCancelWorkers,
|
||||
false, false, nil)
|
||||
err = runScan(cancelledContext(t), []string{dir}, opts)
|
||||
}()
|
||||
|
||||
awaitReturn(t, done, "runScan")
|
||||
@@ -410,8 +410,7 @@ func TestRunScanInterrupted(t *testing.T) {
|
||||
got, before)
|
||||
}
|
||||
|
||||
err = runScan(t.Context(), []string{dir}, walkCancelWorkers, false,
|
||||
false, nil)
|
||||
err = runScan(t.Context(), []string{dir}, opts)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -437,7 +436,7 @@ func TestRunScanInterruptedMidHash(t *testing.T) {
|
||||
dir := buildWalkCancelTree(t)
|
||||
|
||||
err := runScan(newWalkClock(hashCancelAtDone), []string{dir},
|
||||
walkCancelWorkers, false, false, nil)
|
||||
scanOptions{workers: walkCancelWorkers})
|
||||
if !errors.Is(err, errInterrupted) {
|
||||
t.Fatalf("runScan interrupted mid-hash = %v, want %v",
|
||||
err, errInterrupted)
|
||||
|
||||
+160
-69
@@ -32,24 +32,79 @@ const (
|
||||
cmdDwim = "dwim"
|
||||
)
|
||||
|
||||
// operandsHelp explains the PATH operands of report and trees in their
|
||||
// help.
|
||||
const operandsHelp = "With PATH operands, only the records at or " +
|
||||
"below them are used. A relative PATH is joined to the working " +
|
||||
"directory. No PATH is looked up on the filesystem, so a symlink in " +
|
||||
"one is not followed."
|
||||
// databaseHelp is the paragraph of the root and scan help that names
|
||||
// the database (README "Location").
|
||||
const databaseHelp = "scan and dwim store what they find in a database " +
|
||||
"file. report, trees\nand info read it. The file is $" + databaseEnv +
|
||||
" when that is set and\nnot empty, and otherwise " +
|
||||
defaultDatabasePath + ", where creating it\nneeds root.\n"
|
||||
|
||||
// escapesHelp is the paragraph of the report and trees help on how
|
||||
// paths are escaped (README "Report output format").
|
||||
const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
|
||||
|
||||
\\ a backslash
|
||||
\t a tab
|
||||
\n a newline
|
||||
\r a carriage return
|
||||
`
|
||||
|
||||
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
|
||||
// summary in the help of their subcommands.
|
||||
const (
|
||||
reportHelp = `The report is tab-separated text on stdout: a header line, then one
|
||||
row per duplicate file. The columns are:
|
||||
|
||||
first the path that sorts first, byte by byte, in its group of copies
|
||||
dupe another copy of that file
|
||||
size the file's size in bytes
|
||||
|
||||
Rows are ordered by size, largest first, then by path.
|
||||
|
||||
` + escapesHelp
|
||||
|
||||
treesHelp = `The report is tab-separated text on stdout: a header line, then one
|
||||
row per duplicate directory tree. The columns are:
|
||||
|
||||
first the path that sorts first, byte by byte, in its group of copies
|
||||
dupe another copy of that tree
|
||||
files the number of regular files in one copy
|
||||
size their total size in bytes
|
||||
disk_size the bytes the dupe copy takes on disk
|
||||
|
||||
Rows are ordered by size, largest first, then by path.
|
||||
|
||||
` + escapesHelp
|
||||
|
||||
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
|
||||
value, separated by a tab:
|
||||
|
||||
database the path, and where it came from
|
||||
database file size in bytes, then rounded, as 2147483648 (2.0 GiB)
|
||||
-wal file size in bytes, then rounded; only if a -wal file exists
|
||||
modified when the database file last changed, in UTC
|
||||
schema version the version of the database's layout
|
||||
records the number of records
|
||||
size of recorded files their total size, in bytes, then rounded
|
||||
`
|
||||
|
||||
dwimHelp = `dwim runs what "sfdupes scan /" runs with the same flags, then what
|
||||
"sfdupes trees" runs. Run it as root, so that the scan can read every
|
||||
file. The scan is stored in the database, as scan stores it, so
|
||||
"sfdupes trees" prints the report again later.
|
||||
`
|
||||
)
|
||||
|
||||
// errNoSubcommand is returned by the root command when it is invoked
|
||||
// without a subcommand. That is a usage error, and the usage text
|
||||
// cobra prints for it is the whole message.
|
||||
var errNoSubcommand = errors.New("no subcommand")
|
||||
|
||||
// errWorkersBelowOne is the usage error for a scan --workers value
|
||||
// below 1.
|
||||
// errWorkersBelowOne is the usage error for a --workers value below 1.
|
||||
var errWorkersBelowOne = errors.New("--workers must be at least 1")
|
||||
|
||||
// errEmptyExcludeValue is the usage error for an empty scan --exclude
|
||||
// value, as from --exclude "$DIR" with DIR unset. Taken as a path, it
|
||||
// errEmptyExcludeValue is the usage error for an empty --exclude value,
|
||||
// as from --exclude "$DIR" with DIR unset. Taken as a path, it
|
||||
// would name the working directory.
|
||||
var errEmptyExcludeValue = errors.New("--exclude must not be empty")
|
||||
|
||||
@@ -174,24 +229,20 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
"print the version to stdout")
|
||||
|
||||
reportCmd := &cobra.Command{
|
||||
Use: cmdReport + " [PATH...]",
|
||||
Use: cmdReport,
|
||||
Short: "Read the scan database and print the file-level duplicates report",
|
||||
Long: "Read the scan database and print the file-level duplicates " +
|
||||
"report.\n\n" + operandsHelp,
|
||||
Args: cobra.ArbitraryArgs,
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
return runReport(ctx, args, stdout)
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runReport(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
treesCmd := &cobra.Command{
|
||||
Use: cmdTrees + " [PATH...]",
|
||||
Use: cmdTrees,
|
||||
Short: "Read the scan database and print the duplicate-tree report",
|
||||
Long: "Read the scan database and print the duplicate-tree " +
|
||||
"report.\n\n" + operandsHelp,
|
||||
Args: cobra.ArbitraryArgs,
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
return runTrees(ctx, args, stdout)
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runTrees(ctx, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
@@ -204,76 +255,116 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
}),
|
||||
}
|
||||
|
||||
dwimCmd := &cobra.Command{
|
||||
Use: cmdDwim,
|
||||
Short: "Scan / with the default excludes, then print the duplicate-tree report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runDwim(ctx, "/", stdout)
|
||||
}),
|
||||
}
|
||||
root.Long = root.Short + ".\n\n" + databaseHelp
|
||||
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
|
||||
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
|
||||
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
|
||||
|
||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
|
||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd,
|
||||
newDwimCommand(stdout))
|
||||
|
||||
return root
|
||||
}
|
||||
|
||||
// newScanCommand builds the scan subcommand, with its --workers, -x,
|
||||
// --no-default-excludes and --exclude flags. Its help lists the default
|
||||
// excludes of the system it runs on, and the filesystem types it never
|
||||
// records anything on.
|
||||
// scanOptions holds the values of the flags that scan and dwim share:
|
||||
// --workers, -x, --no-default-excludes and --exclude.
|
||||
type scanOptions struct {
|
||||
workers int
|
||||
oneFS bool
|
||||
noDefaultExcludes bool
|
||||
excludePaths []string
|
||||
}
|
||||
|
||||
// scanFlagsUsage names the flags that scan and dwim share, for their
|
||||
// usage lines.
|
||||
const scanFlagsUsage = "[--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]..."
|
||||
|
||||
// addScanFlags adds the flags that scan and dwim share to cmd, which
|
||||
// stores their values in opts.
|
||||
func addScanFlags(cmd *cobra.Command, opts *scanOptions) {
|
||||
cmd.Flags().IntVar(&opts.workers, "workers", runtime.NumCPU(),
|
||||
"concurrent workers for the walk and hash phases")
|
||||
cmd.Flags().BoolVarP(&opts.oneFS, "one-file-system", "x", false,
|
||||
"do not cross filesystem boundaries")
|
||||
cmd.Flags().BoolVar(&opts.noDefaultExcludes, "no-default-excludes",
|
||||
false, "enter the default excludes, the paths listed above")
|
||||
// A string array, not a slice, so a comma in a path is not a
|
||||
// separator.
|
||||
cmd.Flags().StringArrayVar(&opts.excludePaths, "exclude", nil,
|
||||
"never enter or record `PATH` or anything below it; repeatable")
|
||||
}
|
||||
|
||||
// check rejects the flag values that are usage errors, before the
|
||||
// command runs. It is the PreRunE of scan and dwim.
|
||||
func (o *scanOptions) check(cmd *cobra.Command, _ []string) error {
|
||||
err := checkScanWorkers(cmd, o.workers)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return rejectEmptyExcludeValue(cmd, o.excludePaths)
|
||||
}
|
||||
|
||||
// newScanCommand builds the scan subcommand. Its help names the
|
||||
// database, and lists the default excludes of the system it runs on and
|
||||
// the filesystem types it never records anything on.
|
||||
func newScanCommand() *cobra.Command {
|
||||
var (
|
||||
scanWorkers int
|
||||
scanOneFS bool
|
||||
scanNoDefaultExcludes bool
|
||||
scanExclude []string
|
||||
)
|
||||
var opts scanOptions
|
||||
|
||||
short := "Walk trees and synchronize the scan database"
|
||||
|
||||
scanCmd := &cobra.Command{
|
||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]... PATH...",
|
||||
Use: cmdScan + " " + scanFlagsUsage + " PATH...",
|
||||
// The Use line already names every flag, so cobra must not
|
||||
// append its generic "[flags]" to it.
|
||||
DisableFlagsInUseLine: true,
|
||||
Short: short,
|
||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
||||
err := checkScanWorkers(cmd, scanWorkers)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return rejectEmptyExcludeValue(cmd, scanExclude)
|
||||
},
|
||||
Long: short + ".\n\n" + databaseHelp + "\n" +
|
||||
defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
PreRunE: opts.check,
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
ctx, stop := interruptContext(ctx)
|
||||
defer stop()
|
||||
|
||||
return runScan(ctx, args, scanWorkers, scanOneFS,
|
||||
scanNoDefaultExcludes, scanExclude)
|
||||
return runScan(ctx, args, opts)
|
||||
}),
|
||||
}
|
||||
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
|
||||
"concurrent workers for the walk and hash phases")
|
||||
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
|
||||
"do not cross filesystem boundaries")
|
||||
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
|
||||
false, "enter the default excludes, the paths listed above")
|
||||
// A string array, not a slice, so a comma in a path is not a
|
||||
// separator.
|
||||
scanCmd.Flags().StringArrayVar(&scanExclude, "exclude", nil,
|
||||
"never enter or record `PATH` or anything below it; repeatable")
|
||||
addScanFlags(scanCmd, &opts)
|
||||
|
||||
return scanCmd
|
||||
}
|
||||
|
||||
// checkScanWorkers rejects a scan --workers value below 1. That is a
|
||||
// usage error reported in one line: cobra prints the returned message
|
||||
// without the usage text, and run exits 2.
|
||||
// newDwimCommand builds the dwim subcommand, which takes the scan flags
|
||||
// with the meaning they have for scan. Its help says what dwim runs,
|
||||
// then lists the default excludes and the filesystem types the scan
|
||||
// never records anything on.
|
||||
func newDwimCommand(stdout io.Writer) *cobra.Command {
|
||||
var opts scanOptions
|
||||
|
||||
short := "Scan /, then print the duplicate-tree report"
|
||||
|
||||
dwimCmd := &cobra.Command{
|
||||
Use: cmdDwim + " " + scanFlagsUsage,
|
||||
DisableFlagsInUseLine: true,
|
||||
Short: short,
|
||||
Long: short + ".\n\n" + dwimHelp + "\n" +
|
||||
defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.NoArgs,
|
||||
PreRunE: opts.check,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runDwim(ctx, "/", opts, stdout)
|
||||
}),
|
||||
}
|
||||
addScanFlags(dwimCmd, &opts)
|
||||
|
||||
return dwimCmd
|
||||
}
|
||||
|
||||
// checkScanWorkers rejects a --workers value below 1. That is a usage
|
||||
// error reported in one line: cobra prints the returned message without
|
||||
// the usage text, and run exits 2.
|
||||
func checkScanWorkers(cmd *cobra.Command, workers int) error {
|
||||
if workers >= 1 {
|
||||
return nil
|
||||
@@ -284,7 +375,7 @@ func checkScanWorkers(cmd *cobra.Command, workers int) error {
|
||||
return fmt.Errorf("%w, got %d", errWorkersBelowOne, workers)
|
||||
}
|
||||
|
||||
// rejectEmptyExcludeValue rejects an empty scan --exclude value. That
|
||||
// rejectEmptyExcludeValue rejects an empty --exclude value. That
|
||||
// is a usage error reported in one line, as for --workers below 1.
|
||||
func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
|
||||
if !slices.Contains(excludePaths, "") {
|
||||
|
||||
@@ -22,6 +22,12 @@ import (
|
||||
// error prints and a runtime failure must not.
|
||||
const usageMarker = "Usage:"
|
||||
|
||||
// helpFlag asks for the help text.
|
||||
const helpFlag = "--help"
|
||||
|
||||
// workersFlag is the scan and dwim flag that sets the worker count.
|
||||
const workersFlag = "--workers"
|
||||
|
||||
// TestMain stops the tests at once, saying why, when their temporary
|
||||
// directory is not on a persistent filesystem: scan records nothing
|
||||
// there (README "Rules for the walk"), so most scan tests would fail
|
||||
@@ -570,21 +576,21 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
}{
|
||||
cmdScan: {
|
||||
run: func() error {
|
||||
return runScan(t.Context(), []string{missing}, 1, false, false,
|
||||
nil)
|
||||
return runScan(t.Context(), []string{missing},
|
||||
scanOptions{workers: 1})
|
||||
},
|
||||
want: scanWant,
|
||||
},
|
||||
cmdReport: {
|
||||
run: func() error {
|
||||
return runReport(cancelledContext(t), nil, io.Discard)
|
||||
return runReport(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "report: database " + defaultDatabasePath + " (default)\n" +
|
||||
reportOpenLine + "\n" + reportOpenLine + "\n",
|
||||
},
|
||||
cmdTrees: {
|
||||
run: func() error {
|
||||
return runTrees(cancelledContext(t), nil, io.Discard)
|
||||
return runTrees(cancelledContext(t), io.Discard)
|
||||
},
|
||||
want: "trees: database " + defaultDatabasePath + " (default)\n" +
|
||||
treesOpenLine + "\n" + treesOpenLine + "\n",
|
||||
@@ -620,6 +626,8 @@ func TestRunUsageErrors(t *testing.T) {
|
||||
}{
|
||||
"no subcommand": {[]string{}, usageMarker},
|
||||
"scan without paths": {[]string{cmdScan}, usageMarker},
|
||||
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
||||
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
||||
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
||||
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
|
||||
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
||||
@@ -660,7 +668,7 @@ func TestRunScanRejectsWorkersBelowOne(t *testing.T) {
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
args := []string{cmdScan, "--workers", workers, t.TempDir()}
|
||||
args := []string{cmdScan, workersFlag, workers, t.TempDir()}
|
||||
|
||||
code := run(args, &stdout, &stderr)
|
||||
if code != exitUsage {
|
||||
@@ -691,7 +699,7 @@ func TestRunHelp(t *testing.T) {
|
||||
|
||||
// README §Settings: help goes to stderr, exits 0, and leaves stdout
|
||||
// empty.
|
||||
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
|
||||
cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
|
||||
|
||||
for _, args := range cases {
|
||||
var stdout, stderr bytes.Buffer
|
||||
@@ -752,6 +760,73 @@ func TestRunScanUsageLine(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// README §Settings: the help names the database, and says what
|
||||
// report, trees, info and dwim print.
|
||||
database := []string{
|
||||
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
|
||||
}
|
||||
escapes := []string{`\\`, `\t`, `\n`, `\r`}
|
||||
|
||||
// The subcommand whose help is printed; none for the root.
|
||||
cases := []struct {
|
||||
subcommand []string
|
||||
want []string
|
||||
}{
|
||||
{nil, database},
|
||||
{[]string{cmdScan}, database},
|
||||
{
|
||||
[]string{cmdReport},
|
||||
append([]string{
|
||||
"tab-separated", "\n first ", "\n dupe ", "\n size ",
|
||||
"largest first",
|
||||
}, escapes...),
|
||||
},
|
||||
{
|
||||
[]string{cmdTrees},
|
||||
append([]string{
|
||||
"tab-separated", "\n first ", "\n dupe ", "\n files ",
|
||||
"\n size ", "\n disk_size ", "largest first",
|
||||
}, escapes...),
|
||||
},
|
||||
{
|
||||
[]string{cmdInfo},
|
||||
[]string{
|
||||
"\n database ", "\n database file size ",
|
||||
"\n -wal file size ", "\n modified ",
|
||||
"\n schema version ", "\n records ",
|
||||
"\n size of recorded files ",
|
||||
},
|
||||
},
|
||||
{
|
||||
[]string{cmdDwim},
|
||||
[]string{
|
||||
"sfdupes scan /", "as root", "sfdupes trees", "database",
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
args := slices.Concat(tc.subcommand, []string{helpFlag})
|
||||
|
||||
code := run(args, &stdout, &stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
|
||||
}
|
||||
|
||||
for _, want := range tc.want {
|
||||
if !strings.Contains(stderr.String(), want) {
|
||||
t.Errorf("run(%v) help lacks %q:\n%s",
|
||||
args, want, stderr.String())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunVersion(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1764,13 +1839,9 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
|
||||
scanFixture(t)
|
||||
|
||||
cases := map[string]func(context.Context, io.Writer) error{
|
||||
cmdReport: func(ctx context.Context, w io.Writer) error {
|
||||
return runReport(ctx, nil, w)
|
||||
},
|
||||
cmdTrees: func(ctx context.Context, w io.Writer) error {
|
||||
return runTrees(ctx, nil, w)
|
||||
},
|
||||
cmdInfo: runInfo,
|
||||
cmdReport: runReport,
|
||||
cmdTrees: runTrees,
|
||||
cmdInfo: runInfo,
|
||||
}
|
||||
|
||||
for name, fn := range cases {
|
||||
|
||||
+11
-11
@@ -5,20 +5,20 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"runtime"
|
||||
)
|
||||
|
||||
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
|
||||
// root" runs with no flags, then what "sfdupes trees" runs, and ends
|
||||
// with a line on stderr saying how to print the report again. The dwim
|
||||
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
|
||||
// does, but only during the scan. An error from the scan, an interrupt
|
||||
// included, ends it before the report. During the report, a signal
|
||||
// ends it at once, as it ends trees.
|
||||
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
||||
// root" runs with the scan flags in opts, then what "sfdupes trees"
|
||||
// runs, and ends with a line on stderr saying how to print the report
|
||||
// again. The dwim command passes "/" as root. It catches SIGINT and
|
||||
// SIGTERM as scan does, but only during the scan. An error from the
|
||||
// scan, an interrupt included, ends it before the report. During the
|
||||
// report, a signal ends it at once, as it ends trees.
|
||||
func runDwim(
|
||||
ctx context.Context, root string, opts scanOptions, stdout io.Writer,
|
||||
) error {
|
||||
scanCtx, stop := interruptContext(ctx)
|
||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false,
|
||||
nil)
|
||||
err := runScan(scanCtx, []string{root}, opts)
|
||||
|
||||
stop()
|
||||
|
||||
@@ -26,7 +26,7 @@ func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
||||
return err
|
||||
}
|
||||
|
||||
err = runTrees(ctx, nil, stdout)
|
||||
err = runTrees(ctx, stdout)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -2,7 +2,10 @@ package sfdupes //nolint:testpackage // calls the unexported runDwim
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"errors"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"runtime"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
@@ -28,7 +31,7 @@ func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
err := runDwim(t.Context(), root, &stdout)
|
||||
err := runDwim(t.Context(), root, scanOptions{workers: 1}, &stdout)
|
||||
if err != nil {
|
||||
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
||||
}
|
||||
@@ -67,7 +70,7 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
err := runDwim(t.Context(), missing, &stdout)
|
||||
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, &stdout)
|
||||
if err == nil {
|
||||
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
|
||||
}
|
||||
@@ -81,3 +84,127 @@ func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
||||
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
|
||||
// dwim passes its --exclude paths to the scan: an excluded copy is
|
||||
// not recorded, so its pair is not reported.
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
root := realTempDir(t)
|
||||
|
||||
kept := []string{
|
||||
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
|
||||
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
|
||||
}
|
||||
writeFile(t, root, "b1/f.bin", pattern(2, 5000))
|
||||
excluded := filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000)))
|
||||
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
opts := scanOptions{workers: 1, excludePaths: []string{excluded}}
|
||||
|
||||
err := runDwim(t.Context(), root, opts, &stdout)
|
||||
if err != nil {
|
||||
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
||||
}
|
||||
|
||||
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
|
||||
if len(lines) != 2 {
|
||||
t.Fatalf("stdout = %q, want a header and one group",
|
||||
stdout.String())
|
||||
}
|
||||
|
||||
want := kept[0] + "\t" + kept[1] + "\t1\t300\t"
|
||||
if !strings.HasPrefix(lines[1], want) {
|
||||
t.Errorf("group = %q, want it to start %q", lines[1], want)
|
||||
}
|
||||
|
||||
skipped := "scan: skipped " + excluded + " (--exclude)\n"
|
||||
if got := stderr(); !strings.Contains(got, skipped) {
|
||||
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunDwimTakesScanFlags(t *testing.T) {
|
||||
// The dwim command parses and checks the scan flags as scan does.
|
||||
// Each case stops before the walk of / begins, and before the
|
||||
// database is created. "dwim --exclude /" excludes its own operand.
|
||||
cases := map[string]struct {
|
||||
args []string
|
||||
code int
|
||||
want string
|
||||
}{
|
||||
"--exclude /": {
|
||||
[]string{cmdDwim, excludeFlag, "/"},
|
||||
exitFatal,
|
||||
"sfdupes: scan /: operand excluded by --exclude /\n",
|
||||
},
|
||||
"--exclude empty": {
|
||||
[]string{cmdDwim, excludeFlag, ""},
|
||||
exitUsage,
|
||||
"Error: --exclude must not be empty\n",
|
||||
},
|
||||
"--workers 0": {
|
||||
[]string{cmdDwim, workersFlag, "0", "-x", "--no-default-excludes"},
|
||||
exitUsage,
|
||||
"Error: --workers must be at least 1, got 0\n",
|
||||
},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
stdout := captureStdout(t)
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run(tc.args, os.Stdout, os.Stderr)
|
||||
if code != tc.code {
|
||||
t.Errorf("run(%q) = %d, want %d", tc.args, code, tc.code)
|
||||
}
|
||||
|
||||
if !strings.HasSuffix(stderr(), tc.want) || stdout() != "" {
|
||||
t.Errorf("stdout, stderr = %q, %q, want \"\" and stderr "+
|
||||
"ending %q", stdout(), stderr(), tc.want)
|
||||
}
|
||||
|
||||
_, err := os.Lstat(path)
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Errorf("lstat %s: %v, want no database", path, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunDwimHelpListsScanFlags(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// README "Subcommands" shows this usage line for dwim.
|
||||
const want = "\n sfdupes dwim [--workers N] [-x] [--no-default-excludes] " +
|
||||
"[--exclude PATH]...\n"
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
run([]string{cmdDwim, "-h"}, &stdout, &stderr)
|
||||
|
||||
if !strings.Contains(stderr.String(), want) {
|
||||
t.Errorf("dwim -h stderr = %q, want the usage line %q",
|
||||
stderr.String(), want)
|
||||
}
|
||||
|
||||
if !strings.Contains(stderr.String(), excludeFlag+" PATH") {
|
||||
t.Errorf("dwim -h stderr = %q, want %s PATH", stderr.String(),
|
||||
excludeFlag)
|
||||
}
|
||||
|
||||
// README "Settings": the help lists the default excludes and the
|
||||
// filesystem types the walk never records anything on.
|
||||
help := strings.TrimSpace(defaultExcludesHelp(runtime.GOOS))
|
||||
if !strings.Contains(stderr.String(), help) {
|
||||
t.Errorf("dwim -h stderr = %q, want it to hold %q", stderr.String(),
|
||||
help)
|
||||
}
|
||||
}
|
||||
@@ -254,7 +254,7 @@ func readMountInfo(mountinfo io.Reader) (map[uint64]string, error) {
|
||||
return devices, lines.Err()
|
||||
}
|
||||
|
||||
// defaultExcludesHelp describes, for the scan command's help, the
|
||||
// defaultExcludesHelp describes, for the scan and dwim help, the
|
||||
// default excludes of the system goos names and, where the scan lists
|
||||
// the mounted filesystems, the types it never records anything on.
|
||||
func defaultExcludesHelp(goos string) string {
|
||||
|
||||
+14
-69
@@ -9,7 +9,6 @@ import (
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"syscall"
|
||||
@@ -45,12 +44,10 @@ func reportPhase(name string) string {
|
||||
}
|
||||
|
||||
// runReport implements the report subcommand: it prints the file-level
|
||||
// duplicates report as TSV on stdout, from the records at or below the
|
||||
// operands in args, or from every record when args is empty. Of the
|
||||
// scanned filesystem it only checks, with pathExists, that each path in
|
||||
// a duplicate group is still there; it never looks up an operand. It
|
||||
// names the database on stderr before it opens it. Any database
|
||||
// problem, including a missing database, is fatal.
|
||||
// duplicates report as TSV on stdout. Of the scanned filesystem it
|
||||
// only checks, with pathExists, that each path in a duplicate group is
|
||||
// still there. It names the database on stderr before it opens it. Any
|
||||
// database problem, including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in two phases, each under a progress display of its
|
||||
// own on stderr: open (open the database) and read (read the number of
|
||||
@@ -58,12 +55,7 @@ func reportPhase(name string) string {
|
||||
// order, writing each size's duplicate groups to stdout once that size
|
||||
// is read; see readReport). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
operands, err := cleanOperands(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -95,8 +87,7 @@ func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
label: reportPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db/disk/out",
|
||||
})
|
||||
report, err := readReport(ctx, db, dbPath, operands, stdout, pathExists,
|
||||
prog)
|
||||
report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -117,14 +108,12 @@ func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
// dbPath, as its total, then reads every record in signature order,
|
||||
// counting each on prog, and writes the report to stdout: the header
|
||||
// line, then the rows of each size's duplicate groups once that size is
|
||||
// read. Only the records that underOperand finds under operands join a
|
||||
// group, and a path for which exists returns false is left out of its
|
||||
// group. stdout may be the terminal prog is drawn on, so the lines go
|
||||
// through prog's writer. It returns what it wrote, with the totals for
|
||||
// the summary line.
|
||||
// read. A path for which exists returns false is left out of its group.
|
||||
// stdout may be the terminal prog is drawn on, so the lines go through
|
||||
// prog's writer. It returns what it wrote, with the totals for the
|
||||
// summary line.
|
||||
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
operands []string, stdout io.Writer, exists func(path string) bool,
|
||||
prog *progress,
|
||||
stdout io.Writer, exists func(path string) bool, prog *progress,
|
||||
) (*reportWriter, error) {
|
||||
out := prog.writer(stdout)
|
||||
report := &reportWriter{out: out, exists: exists}
|
||||
@@ -139,12 +128,6 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
|
||||
prog.increment()
|
||||
|
||||
report.records++
|
||||
|
||||
if !underOperand(r.path, operands) {
|
||||
return nil
|
||||
}
|
||||
|
||||
writeErr = report.add(r)
|
||||
|
||||
return writeErr
|
||||
@@ -176,7 +159,7 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
// records of one size, so it holds only the duplicate groups of the
|
||||
// size being read, and writes them, ordered by first path, once a
|
||||
// record of a smaller size comes. It counts what the summary line
|
||||
// reports; readReport counts the records read.
|
||||
// reports.
|
||||
type reportWriter struct {
|
||||
out io.Writer
|
||||
|
||||
@@ -196,6 +179,8 @@ type reportWriter struct {
|
||||
// its paths; any other ends them, and one of a smaller size first
|
||||
// writes the groups of the size before.
|
||||
func (w *reportWriter) add(r scanRec) error {
|
||||
w.records++
|
||||
|
||||
if r.size == w.sig.size && r.head == w.sig.head &&
|
||||
r.tail == w.sig.tail && r.content == w.sig.content {
|
||||
w.paths = append(w.paths, r.path)
|
||||
@@ -288,46 +273,6 @@ func pathExists(path string) bool {
|
||||
return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
|
||||
}
|
||||
|
||||
// cleanOperands returns the operands of report or trees, each made
|
||||
// absolute by absolutePath and then cleaned. The records hold real
|
||||
// paths, with no "." or ".." element and no repeated or trailing
|
||||
// slash, so an operand is cleaned to match them. No operand is looked
|
||||
// up on the filesystem.
|
||||
func cleanOperands(args []string) ([]string, error) {
|
||||
operands := make([]string, 0, len(args))
|
||||
|
||||
for _, a := range args {
|
||||
abs, err := absolutePath(a)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve path %s: %w", escapePath(a), err)
|
||||
}
|
||||
|
||||
operands = append(operands, filepath.Clean(abs))
|
||||
}
|
||||
|
||||
return operands, nil
|
||||
}
|
||||
|
||||
// underOperand reports whether path is one of operands or lies below
|
||||
// one of them. Each operand is clean and absolute, as cleanOperands
|
||||
// returns it. With no operands, a report covers the whole database, so
|
||||
// underOperand is true for every path.
|
||||
func underOperand(path string, operands []string) bool {
|
||||
if len(operands) == 0 {
|
||||
return true
|
||||
}
|
||||
|
||||
for _, o := range operands {
|
||||
// Only the root directory's clean path ends in "/".
|
||||
dir := strings.TrimSuffix(o, "/") + "/"
|
||||
if path == o || strings.HasPrefix(path, dir) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// escapePath returns a path as it is written in a report column (README
|
||||
// "Report output format"): a backslash, tab, newline or carriage return
|
||||
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
||||
|
||||
@@ -102,8 +102,7 @@ func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
_, err := readReport(t.Context(), db, "", nil, &out, everyPathExists,
|
||||
nil)
|
||||
_, err := readReport(t.Context(), db, "", &out, everyPathExists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -149,103 +148,6 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
|
||||
return dupeGroups(t, db)
|
||||
}
|
||||
|
||||
// TestRunReportOperands runs report with operands on three copies of
|
||||
// one file, two of them under srv/photos, and a fourth copy in
|
||||
// srv/photos2, all in a new directory. Only the records at or below the
|
||||
// operands form the groups and the totals, however each operand is
|
||||
// written.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestRunReportOperands(t *testing.T) {
|
||||
h := testHash("h")
|
||||
|
||||
recs := make([]scanRec, 0, 4)
|
||||
for _, p := range []string{
|
||||
"/a/copy", "/srv/photos/x", "/srv/photos/y", "/srv/photos2/z",
|
||||
} {
|
||||
recs = append(recs,
|
||||
scanRec{size: 7, head: h, tail: h, content: h, path: p})
|
||||
}
|
||||
|
||||
dir, recs := recsOnDisk(t, recs)
|
||||
a, x, y, z := recs[0].path, recs[1].path, recs[2].path, recs[3].path
|
||||
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
header := "first\tdupe\tsize\n"
|
||||
row := func(first, dupe string) string {
|
||||
return first + "\t" + dupe + "\t7\n"
|
||||
}
|
||||
photos := header + row(x, y)
|
||||
|
||||
cases := map[string]struct {
|
||||
operands []string
|
||||
want string
|
||||
summary string
|
||||
}{
|
||||
"directory": {[]string{dir + "/srv/photos"}, photos,
|
||||
"1 duplicate groups, 1 dupe files, 7 B reclaimable"},
|
||||
"trailing slash": {[]string{dir + "/srv/photos/"}, photos, ""},
|
||||
"dot and dot-dot": {[]string{dir + "/srv/./a/../photos"}, photos, ""},
|
||||
"files": {[]string{a, z}, header + row(a, z), ""},
|
||||
"overlapping": {[]string{dir + "/srv", dir + "/srv/photos"},
|
||||
header + row(x, y) + row(x, z), ""},
|
||||
"root": {[]string{"/"},
|
||||
header + row(a, x) + row(a, y) + row(a, z), ""},
|
||||
"no record": {[]string{dir + "/srv/photo"}, header,
|
||||
"0 duplicate groups, 0 dupe files, 0 B reclaimable"},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run(append([]string{cmdReport}, tc.operands...),
|
||||
&stdout, os.Stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(report %q) = %d, want %d; stderr: %s",
|
||||
tc.operands, code, exitOK, stderr())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != tc.want {
|
||||
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||
}
|
||||
|
||||
// The records read are all four, whatever the operands.
|
||||
summary := "report: 4 records read, " + tc.summary + "\n"
|
||||
if got := stderr(); tc.summary != "" &&
|
||||
!strings.HasSuffix(got, summary) {
|
||||
t.Errorf("stderr = %q, want it to end %q", got, summary)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
//nolint:paralleltest // t.Chdir changes the process-wide working directory
|
||||
func TestCleanOperandsMakesRelativePathsAbsolute(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Chdir(dir)
|
||||
|
||||
wd, err := absolutePath(".")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got, err := cleanOperands([]string{"photos/", "a/../b", "/c//d"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
want := []string{
|
||||
filepath.Join(wd, "photos"), filepath.Join(wd, "b"), "/c/d",
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("cleanOperands = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReportEscapesPaths(t *testing.T) {
|
||||
dir, recs := recsOnDisk(t, awkwardPairRecs())
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
@@ -293,7 +195,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
|
||||
// Counts the records read, and draws nothing.
|
||||
prog := &progress{now: time.Now}
|
||||
|
||||
_, err = readReport(t.Context(), db, "", nil, failingWriter{},
|
||||
_, err = readReport(t.Context(), db, "", failingWriter{},
|
||||
everyPathExists, prog)
|
||||
if !errors.Is(err, errWriteFailed) ||
|
||||
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
||||
@@ -390,8 +292,7 @@ func TestReadReportOnTerminal(t *testing.T) {
|
||||
waitingFor: countingRecords,
|
||||
}, terminalWidth, time.Hour, time.Now)
|
||||
|
||||
_, err = readReport(t.Context(), db, "", nil, os.Stderr,
|
||||
everyPathExists, p)
|
||||
_, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p)
|
||||
|
||||
p.finish()
|
||||
|
||||
@@ -613,7 +514,7 @@ func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
|
||||
return true
|
||||
}
|
||||
|
||||
_, err = readReport(t.Context(), db, "", nil, io.Discard, exists, nil)
|
||||
_, err = readReport(t.Context(), db, "", io.Discard, exists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -647,8 +548,7 @@ func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
|
||||
|
||||
exists := func(path string) bool { return path != "/a/gone" }
|
||||
|
||||
report, err := readReport(t.Context(), db, "", nil, io.Discard, exists,
|
||||
nil)
|
||||
report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -681,7 +581,7 @@ func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
report, err := readReport(t.Context(), db, "", nil, &out, exists, nil)
|
||||
report, err := readReport(t.Context(), db, "", &out, exists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
+13
-14
@@ -98,17 +98,16 @@ func scanPhase(name string) string {
|
||||
// runs, and the lock is released after it. When ctx is cancelled, as by
|
||||
// the SIGINT or SIGTERM that interruptContext catches, the scan keeps
|
||||
// what it has hashed (see syncScan), prints how many files its walk
|
||||
// reached, and returns errInterrupted. workers must be at least 1; the
|
||||
// scan command rejects anything less. Before the lock is taken, the
|
||||
// scan lists the filesystems that are not persistent, on which the walk
|
||||
// records nothing; unless noDefaultExcludes is set, the walk does not
|
||||
// enter the default excludes either. The walk neither enters nor
|
||||
// records the paths in excludePaths, the --exclude values; an operand
|
||||
// at or below one is a fatal error. Once the database is open, the scan
|
||||
// notes its files, which the walk passes over.
|
||||
func runScan(ctx context.Context, roots []string, workers int,
|
||||
oneFS, noDefaultExcludes bool, excludePaths []string,
|
||||
) error {
|
||||
// reached, and returns errInterrupted. opts.workers must be at least 1;
|
||||
// the scan and dwim commands reject anything less. Before the lock is
|
||||
// taken, the scan lists the filesystems that are not persistent, on
|
||||
// which the walk records nothing; unless opts.noDefaultExcludes is set,
|
||||
// the walk does not enter the default excludes either. The walk neither
|
||||
// enters nor records the paths in opts.excludePaths, the --exclude
|
||||
// values; an operand at or below one is a fatal error. Once the
|
||||
// database is open, the scan notes its files, which the walk passes
|
||||
// over.
|
||||
func runScan(ctx context.Context, roots []string, opts scanOptions) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -123,7 +122,7 @@ func runScan(ctx context.Context, roots []string, workers int,
|
||||
|
||||
var rules walkRules
|
||||
|
||||
rules.excludePaths, err = excludePathInodes(excludePaths)
|
||||
rules.excludePaths, err = excludePathInodes(opts.excludePaths)
|
||||
if err != nil {
|
||||
return fmt.Errorf("--exclude: %w", err)
|
||||
}
|
||||
@@ -138,7 +137,7 @@ func runScan(ctx context.Context, roots []string, workers int,
|
||||
return fmt.Errorf("list the mounted filesystems: %w", err)
|
||||
}
|
||||
|
||||
if !noDefaultExcludes {
|
||||
if !opts.noDefaultExcludes {
|
||||
rules.defaultExcludes = defaultExcludes(runtime.GOOS)
|
||||
}
|
||||
|
||||
@@ -167,7 +166,7 @@ func runScan(ctx context.Context, roots []string, workers int,
|
||||
return err
|
||||
}
|
||||
|
||||
st, err := syncScan(ctx, db, roots, workers, oneFS, rules)
|
||||
st, err := syncScan(ctx, db, roots, opts.workers, opts.oneFS, rules)
|
||||
if errors.Is(err, context.Canceled) {
|
||||
return interrupted(st.walked)
|
||||
}
|
||||
|
||||
@@ -2196,7 +2196,7 @@ func TestScanHashWriteFailureUnwindsPool(t *testing.T) {
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := run([]string{
|
||||
cmdScan, "--workers", strconv.Itoa(hashLeakWorkers), dir,
|
||||
cmdScan, workersFlag, strconv.Itoa(hashLeakWorkers), dir,
|
||||
}, io.Discard, &stderr)
|
||||
if code != exitFatal {
|
||||
t.Fatalf("run(scan) = %d, want %d; stderr: %s",
|
||||
|
||||
+14
-45
@@ -37,14 +37,12 @@ func treesPhase(name string) string {
|
||||
|
||||
// runTrees implements the trees subcommand: it reads every record from
|
||||
// the database in path order, reconstructs the directory hierarchy from
|
||||
// the paths of the records at or below the operands in args (of every
|
||||
// record when args is empty), computes a Merkle-style digest per
|
||||
// directory, and prints maximal duplicate-tree groups as TSV on stdout.
|
||||
// Of the scanned filesystem it only checks, with pathExists, that the
|
||||
// root of each duplicate tree is still there, never the files below it;
|
||||
// it never looks up an operand. It names the database on stderr before
|
||||
// it opens it. Any database problem, including a missing database, is
|
||||
// fatal.
|
||||
// the record paths, computes a Merkle-style digest per directory, and
|
||||
// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned
|
||||
// filesystem it only checks, with pathExists, that the root of each
|
||||
// duplicate tree is still there, never the files below it. It names the
|
||||
// database on stderr before it opens it. Any database problem,
|
||||
// including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in four phases, each under a progress display of its
|
||||
// own on stderr: open (open the database), read (read the number of
|
||||
@@ -52,12 +50,7 @@ func treesPhase(name string) string {
|
||||
// hierarchy), group (group the directories by digest) and write (write
|
||||
// the groups to stdout). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
operands, err := cleanOperands(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -86,7 +79,7 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
label: treesPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db",
|
||||
})
|
||||
records, super, allDirs, err := readTree(ctx, db, operands, prog)
|
||||
records, super, allDirs, err := readTree(ctx, db, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -125,17 +118,10 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
}
|
||||
|
||||
// readTree gives prog the number of records in db as its total, then
|
||||
// reads every record in path order, counting each on prog. It adds the
|
||||
// records that underOperand finds under operands to the directory
|
||||
// hierarchy. It returns the number of records read, the super-root and
|
||||
// every directory at or below one of operands.
|
||||
//
|
||||
// Any other directory lies above an operand and can hold files that are
|
||||
// not used, so its digest does not describe it. It is left out of the
|
||||
// directories returned. A directory whose parent is left out gets the
|
||||
// super-root as its parent, so suppressed never judges its group by
|
||||
// that parent's digest.
|
||||
func readTree(ctx context.Context, db *sql.DB, operands []string,
|
||||
// reads every record in path order into the directory hierarchy,
|
||||
// counting each on prog. It returns the number of records read, the
|
||||
// super-root and every directory.
|
||||
func readTree(ctx context.Context, db *sql.DB,
|
||||
prog *progress,
|
||||
) (int, *treeNode, []*treeNode, error) {
|
||||
records := 0
|
||||
@@ -145,10 +131,7 @@ func readTree(ctx context.Context, db *sql.DB, operands []string,
|
||||
records++
|
||||
|
||||
prog.increment()
|
||||
|
||||
if underOperand(r.path, operands) {
|
||||
tree.add(r)
|
||||
}
|
||||
tree.add(r)
|
||||
})
|
||||
if err != nil {
|
||||
return 0, nil, nil, err
|
||||
@@ -156,21 +139,7 @@ func readTree(ctx context.Context, db *sql.DB, operands []string,
|
||||
|
||||
super, allDirs := tree.finish()
|
||||
|
||||
dirs := allDirs[:0]
|
||||
|
||||
for _, d := range allDirs {
|
||||
if !underOperand(d.path, operands) {
|
||||
continue
|
||||
}
|
||||
|
||||
if !underOperand(d.parent.path, operands) {
|
||||
d.parent = super
|
||||
}
|
||||
|
||||
dirs = append(dirs, d)
|
||||
}
|
||||
|
||||
return records, super, dirs, nil
|
||||
return records, super, allDirs, nil
|
||||
}
|
||||
|
||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||
|
||||
@@ -214,57 +214,6 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesOperands runs trees with operands on records in a new
|
||||
// directory. x/a and y/a are copies, and so are y/a and srv/b. x and y
|
||||
// also hold files that differ. Given x/a and y/a, trees reads x and y
|
||||
// as holding one copy each of a, yet neither is reported: only the
|
||||
// directories at or below an operand are.
|
||||
func TestRunTreesOperands(t *testing.T) {
|
||||
f, g, k := testHash("f"), testHash("g"), testHash("k")
|
||||
dir, recs := recsOnDisk(t, []scanRec{
|
||||
{size: 10, head: f, tail: f, content: f, path: "/srv/b/f"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/x/a/f"},
|
||||
{size: 20, head: g, tail: g, content: g, path: "/x/g"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/y/a/f"},
|
||||
{size: 30, head: k, tail: k, content: k, path: "/y/k"},
|
||||
})
|
||||
b, xa, ya := dir+"/srv/b", dir+"/x/a", dir+"/y/a"
|
||||
|
||||
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
|
||||
row := func(first, dupe string) string {
|
||||
return first + "\t" + dupe + "\t1\t10\t0\n"
|
||||
}
|
||||
|
||||
cases := map[string]struct {
|
||||
operands []string
|
||||
want string
|
||||
}{
|
||||
"none": {nil, header + row(b, xa) + row(b, ya)},
|
||||
"two copies": {[]string{xa, ya}, header + row(xa, ya)},
|
||||
"parents": {[]string{dir + "/x", dir + "/y"}, header + row(xa, ya)},
|
||||
"one copy": {[]string{b}, header},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := run(append([]string{cmdTrees}, tc.operands...),
|
||||
&stdout, &stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(trees %q) = %d, want %d; stderr: %s",
|
||||
tc.operands, code, exitOK, stderr.String())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != tc.want {
|
||||
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
||||
// take different space on disk. The row's disk_size is the sum over the
|
||||
// dupe copy's files, those in its subdirectory included.
|
||||
|
||||
Reference in new issue
Block a user