Compare commits

..
1 Commits
Author SHA1 Message Date
sneak 784a7b7a74 Limit report and trees to given paths (closes #174)
check / check (push) Canceled after 0s
report and trees take optional PATH operands and use only the records
at or below them. Operands are made absolute and cleaned as text, never
looked up on the filesystem; each reported copy is still checked on
disk. With operands, trees reports only directories at or below an
operand: any other directory lies above an operand and can hold unused
files, so its digest neither gets reported nor suppresses its
children's group. The summary still counts every record read.

Model: opus-5-5
2026-11-11 05:46:52 +00:00
6 changed files with 28 additions and 227 deletions

No files matched your search

+8 -10
View File
@@ -22,8 +22,9 @@ records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` work from the database and
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
first look at a new machine with one command.
can run at any time. `report` and `trees` leave out a copy that was deleted
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first
look at a new machine with one command.
## Getting Started
@@ -318,11 +319,9 @@ The reference below is also the program's specification.
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
exits 0. It writes nothing to stderr.
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
names the database. The help of `report`, `trees`, `info` and `dwim` says what
each prints. `sfdupes scan --help` also lists the default excludes of the
system it runs on, and the filesystem types the walk never records anything
on.
and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the
default excludes of the system it runs on, and the filesystem types the walk
never records anything on.
### Duplicate detection
@@ -744,8 +743,8 @@ smaller size, it writes them, ordered by `first` path.
followed, and an operand need not exist. The records hold real paths (see
"Operands" under "`scan` mode"), so an operand that passes through a symlink
matches no record.
- An operand that matches no record is not an error; it adds nothing to the
report.
- An operand that matches no record is not an error. The report then has no
rows.
#### Report output format
@@ -1284,7 +1283,6 @@ showing the count it stopped at.
- `scan` with no operand;
- `--workers` below 1;
- an empty `--exclude` value, as in `--exclude ""`;
- an empty `PATH` for `report` or `trees`, as in `sfdupes report ""`;
- `info` or `dwim` with a positional argument.
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
+1 -6
View File
@@ -32,13 +32,8 @@ are known.
- `report` and `trees` take optional `PATH` operands. With operands, they use
only the records at or below them. `trees` then reports only directories at or
below an operand. Operands are made absolute and cleaned as text, never looked
up on the filesystem. An empty operand is a usage error (2026-11-11,
up on the filesystem (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/174)
- The help of `sfdupes` and of `scan` names the database, its default path and
`SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
order of the rows and the escapes. The help of `info` lists its lines. The
help of `dwim` says that it scans `/`, should run as root and keeps the scan
in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
enters nor records an excluded path, and keeps the records below it. An
operand at or below one is a fatal error (2026-11-11,
+17 -100
View File
@@ -32,75 +32,12 @@ const (
cmdDwim = "dwim"
)
// databaseHelp is the paragraph of the root and scan help that names
// the database (README "Location").
const databaseHelp = "scan and dwim store what they find in a database " +
"file. report, trees\nand info read it. The file is $" + databaseEnv +
" when that is set and\nnot empty, and otherwise " +
defaultDatabasePath + ", where creating it\nneeds root.\n"
// escapesHelp is the paragraph of the report and trees help on how
// paths are escaped (README "Report output format").
const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
\\ a backslash
\t a tab
\n a newline
\r a carriage return
`
// operandsHelp is the paragraph of the report and trees help on their
// PATH operands (README "Operands of `report` and `trees`").
const operandsHelp = `With PATH operands, only the records at or below them are used. A
relative PATH is joined to the working directory. No PATH is looked up
on the filesystem, so a symlink in one is not followed.
`
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
// summary in the help of their subcommands.
const (
reportHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate file. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that file
size the file's size in bytes
Rows are ordered by size, largest first, then by path.
` + escapesHelp + "\n" + operandsHelp
treesHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate directory tree. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that tree
files the number of regular files in one copy
size their total size in bytes
disk_size the bytes the dupe copy takes on disk
Rows are ordered by size, largest first, then by path.
` + escapesHelp + "\n" + operandsHelp
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
value, separated by a tab:
database the path, and where it came from
database file size in bytes, then rounded, as 2147483648 (2.0 GiB)
-wal file size in bytes, then rounded; only if a -wal file exists
modified when the database file last changed, in UTC
schema version the version of the database's layout
records the number of records
size of recorded files their total size, in bytes, then rounded
`
dwimHelp = `dwim runs what "sfdupes scan /" runs, then what "sfdupes trees" runs.
Run it as root, so that the scan can read every file. The scan is stored
in the database, as scan stores it, so "sfdupes trees" prints the report
again later.
`
)
// operandsHelp explains the PATH operands of report and trees in their
// help.
const operandsHelp = "With PATH operands, only the records at or " +
"below them are used. A relative PATH is joined to the working " +
"directory. No PATH is looked up on the filesystem, so a symlink in " +
"one is not followed."
// errNoSubcommand is returned by the root command when it is invoked
// without a subcommand. That is a usage error, and the usage text
@@ -116,11 +53,6 @@ var errWorkersBelowOne = errors.New("--workers must be at least 1")
// would name the working directory.
var errEmptyExcludeValue = errors.New("--exclude must not be empty")
// errEmptyOperand is the usage error for an empty PATH operand of
// report or trees, as from "$DIR" with DIR unset. Taken as a path, it
// would name the working directory.
var errEmptyOperand = errors.New("PATH must not be empty")
// Version is the build version, injected at link time via -ldflags
// (see the Makefile); empty when the linker set none, as for a binary
// built by go install. buildVersion turns it into what --version prints.
@@ -244,7 +176,9 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
reportCmd := &cobra.Command{
Use: cmdReport + " [PATH...]",
Short: "Read the scan database and print the file-level duplicates report",
Args: rejectEmptyOperand,
Long: "Read the scan database and print the file-level duplicates " +
"report.\n\n" + operandsHelp,
Args: cobra.ArbitraryArgs,
RunE: runE(func(ctx context.Context, args []string) error {
return runReport(ctx, args, stdout)
}),
@@ -253,7 +187,9 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
treesCmd := &cobra.Command{
Use: cmdTrees + " [PATH...]",
Short: "Read the scan database and print the duplicate-tree report",
Args: rejectEmptyOperand,
Long: "Read the scan database and print the duplicate-tree " +
"report.\n\n" + operandsHelp,
Args: cobra.ArbitraryArgs,
RunE: runE(func(ctx context.Context, args []string) error {
return runTrees(ctx, args, stdout)
}),
@@ -277,21 +213,15 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}),
}
root.Long = root.Short + ".\n\n" + databaseHelp
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
dwimCmd.Long = dwimCmd.Short + ".\n\n" + dwimHelp
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
return root
}
// newScanCommand builds the scan subcommand, with its --workers, -x,
// --no-default-excludes and --exclude flags. Its help names the
// database, and lists the default excludes of the system it runs on and
// the filesystem types it never records anything on.
// --no-default-excludes and --exclude flags. Its help lists the default
// excludes of the system it runs on, and the filesystem types it never
// records anything on.
func newScanCommand() *cobra.Command {
var (
scanWorkers int
@@ -309,9 +239,8 @@ func newScanCommand() *cobra.Command {
// append its generic "[flags]" to it.
DisableFlagsInUseLine: true,
Short: short,
Long: short + ".\n\n" + databaseHelp + "\n" +
defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1),
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1),
PreRunE: func(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, scanWorkers)
if err != nil {
@@ -367,18 +296,6 @@ func rejectEmptyExcludeValue(cmd *cobra.Command, excludePaths []string) error {
return errEmptyExcludeValue
}
// rejectEmptyOperand rejects an empty PATH operand of report or trees.
// That is a usage error reported in one line, as for --workers below 1.
func rejectEmptyOperand(cmd *cobra.Command, args []string) error {
if !slices.Contains(args, "") {
return nil
}
cmd.SilenceUsage = true
return errEmptyOperand
}
// runE adapts a subcommand implementation, or the version print, to
// cobra's RunE. Cobra prints the error and the command's usage text for
// every error RunE returns, but a subcommand that ran and failed has no
+1 -76
View File
@@ -22,9 +22,6 @@ import (
// error prints and a runtime failure must not.
const usageMarker = "Usage:"
// helpFlag asks for the help text.
const helpFlag = "--help"
// TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail
@@ -694,7 +691,7 @@ func TestRunHelp(t *testing.T) {
// README §Settings: help goes to stderr, exits 0, and leaves stdout
// empty.
cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
for _, args := range cases {
var stdout, stderr bytes.Buffer
@@ -755,78 +752,6 @@ func TestRunScanUsageLine(t *testing.T) {
}
}
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
t.Parallel()
// README §Settings: the help names the database, and says what
// report, trees, info and dwim print. The help of report and trees
// also explains their PATH operands.
database := []string{
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
}
escapes := []string{`\\`, `\t`, `\n`, `\r`}
operands := []string{
"With PATH operands", "at or below them", "working directory",
"No PATH is looked up", "not followed",
}
// The subcommand whose help is printed; none for the root.
cases := []struct {
subcommand []string
want []string
}{
{nil, database},
{[]string{cmdScan}, database},
{
[]string{cmdReport},
slices.Concat([]string{
"tab-separated", "\n first ", "\n dupe ", "\n size ",
"largest first",
}, escapes, operands),
},
{
[]string{cmdTrees},
slices.Concat([]string{
"tab-separated", "\n first ", "\n dupe ", "\n files ",
"\n size ", "\n disk_size ", "largest first",
}, escapes, operands),
},
{
[]string{cmdInfo},
[]string{
"\n database ", "\n database file size ",
"\n -wal file size ", "\n modified ",
"\n schema version ", "\n records ",
"\n size of recorded files ",
},
},
{
[]string{cmdDwim},
[]string{
"sfdupes scan /", "as root", "sfdupes trees", "database",
},
},
}
for _, tc := range cases {
var stdout, stderr bytes.Buffer
args := slices.Concat(tc.subcommand, []string{helpFlag})
code := run(args, &stdout, &stderr)
if code != exitOK {
t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
}
for _, want := range tc.want {
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) help lacks %q:\n%s",
args, want, stderr.String())
}
}
}
}
func TestRunVersion(t *testing.T) {
t.Parallel()
+1 -2
View File
@@ -292,8 +292,7 @@ func pathExists(path string) bool {
// absolute by absolutePath and then cleaned. The records hold real
// paths, with no "." or ".." element and no repeated or trailing
// slash, so an operand is cleaned to match them. No operand is looked
// up on the filesystem. None is empty: rejectEmptyOperand refuses an
// empty operand on the command line.
// up on the filesystem.
func cleanOperands(args []string) ([]string, error) {
operands := make([]string, 0, len(args))
-33
View File
@@ -246,39 +246,6 @@ func TestCleanOperandsMakesRelativePathsAbsolute(t *testing.T) {
}
}
// TestRunReportsRejectEmptyOperand checks that an empty PATH operand of
// report or trees, alone or after another operand, is a usage error
// reported in one line before the database is opened (README "Error
// handling and exit codes").
func TestRunReportsRejectEmptyOperand(t *testing.T) {
cases := map[string][]string{
"report": {cmdReport, ""},
"trees": {cmdTrees, ""},
"after an operand": {cmdTrees, t.TempDir(), ""},
}
for name, args := range cases {
t.Run(name, func(t *testing.T) {
// No database exists there, so opening one would be a fatal
// error, exit 1.
t.Setenv(databaseEnv, testDBPath(t))
var stdout, stderr bytes.Buffer
code := run(args, &stdout, &stderr)
if code != exitUsage {
t.Errorf("run(%q) = %d, want %d", args, code, exitUsage)
}
want := "Error: PATH must not be empty\n"
if stdout.String() != "" || stderr.String() != want {
t.Errorf("stdout, stderr = %q, %q, want \"\", %q",
stdout.String(), stderr.String(), want)
}
})
}
}
func TestRunReportEscapesPaths(t *testing.T) {
dir, recs := recsOnDisk(t, awkwardPairRecs())
t.Setenv(databaseEnv, seedDatabase(t, recs))