Compare commits

..
Author SHA1 Message Date
sneak 9e6e3b9287 Say in --help where the database is and what each report prints (closes #177)
check / check (push) Successful in 3m4s
The help of sfdupes and of scan names the database, its default path
and SFDUPES_DATABASE. The help of report and trees lists the columns,
the order of the rows and the four escapes. The help of info lists its
lines. The help of dwim says that it scans /, should run as root and
keeps the scan in the database.

The README's opening drops its sentence on copies deleted after the
scan, an edge case that its report and trees sections already cover.

Model: opus-5-5
2026-11-11 05:57:25 +00:00
6 changed files with 173 additions and 87 deletions

No files matched your search

+19 -28
View File
@@ -22,9 +22,8 @@ records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` work from the database and system's job scheduler. `report`, `trees` and `info` work from the database and
can run at any time. `report` and `trees` leave out a copy that was deleted can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first first look at a new machine with one command.
look at a new machine with one command.
## Getting Started ## Getting Started
@@ -129,8 +128,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
- one line naming the database; - one line naming the database;
- progress lines, as described under "Progress"; - progress lines, as described under "Progress";
- a warning for each path it skips; - a warning for each path it skips;
- a line for each system, temporary or cache directory the walk leaves out - a line for each system or temporary directory the walk leaves out (see
(see "Rules for the walk"); "Rules for the walk");
- a line for each path left out because its filesystem is one the kernel, - a line for each path left out because its filesystem is one the kernel,
the core of the operating system, makes up or keeps in memory, such as the core of the operating system, makes up or keeps in memory, such as
`tmpfs`; `tmpfs`;
@@ -302,9 +301,8 @@ The reference below is also the program's specification.
directories on other filesystems, as the same flag does for `du` and `rsync`. directories on other filesystems, as the same flag does for `du` and `rsync`.
By default the walk also enters directories on other filesystems. By default the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes. - `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
Those are the system, temporary and cache directories the walk otherwise Those are the system and temporary directories the walk otherwise leaves out,
leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules for the such as `/var/tmp` on Linux (see "Rules for the walk").
walk").
- `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below - `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
it. `PATH` may be a directory or a file. The flag may be given more than once. it. `PATH` may be a directory or a file. The flag may be given more than once.
The walk knows `PATH` by its device ID and inode number, as it knows the The walk knows `PATH` by its device ID and inode number, as it knows the
@@ -318,9 +316,11 @@ The reference below is also the program's specification.
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and - `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
exits 0. It writes nothing to stderr. exits 0. It writes nothing to stderr.
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr - `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
default excludes of the system it runs on, and the filesystem types the walk names the database. The help of `report`, `trees`, `info` and `dwim` says what
never records anything on. each prints. `sfdupes scan --help` also lists the default excludes of the
system it runs on, and the filesystem types the walk never records anything
on.
### Duplicate detection ### Duplicate detection
@@ -571,18 +571,13 @@ walk practical on large or busy storage pools.
it, as described under "What a scan does to the database". it, as described under "What a scan does to the database".
`--no-default-excludes` does not change this rule. `--no-default-excludes` does not change this rule.
- Below an operand, the walk does not enter a default exclude. The default - Below an operand, the walk does not enter a default exclude. On Linux, the
excludes hold the kernel's view of the running system, temporary files, and default excludes hold temporary files and the kernel's view of the running
caches. A cache here is data that programs and package managers download or system. They depend on the system, and `sfdupes scan --help` lists those of
build again on their own. The default excludes depend on the system, and the system it runs on. On Linux they are:
`sfdupes scan --help` lists those of the system it runs on. In the lists, `*`
stands for any one directory name, so `/home/*/.cache` covers each user's
cache. On Linux they are:
``` ```
/proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache /proc, /sys, /dev, /run, /tmp, /var/tmp
/root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
/home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
``` ```
On macOS they are: On macOS they are:
@@ -590,15 +585,11 @@ walk practical on large or busy storage pools.
``` ```
/dev, /private/tmp, /private/var/tmp, /private/var/vm /dev, /private/tmp, /private/var/tmp, /private/var/vm
/System/Volumes/VM, /System/Volumes/Data /System/Volumes/VM, /System/Volumes/Data
/private/var/folders, /Library/Caches
/Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
/Users/*/.cargo/registry, /Users/*/go/pkg/mod
``` ```
`/private/var/folders` holds each user's temporary files and caches on A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already,
macOS. A walk of `/` on macOS reaches everything in `/System/Volumes/Data` through firmlinks. Firmlinks are the links macOS uses to show the data
already, through firmlinks. Firmlinks are the links macOS uses to show the volume inside the system volume. Walking `/System/Volumes/Data` as well
data volume inside the system volume. Walking `/System/Volumes/Data` as well
would record each of those files twice. Any other system has no default would record each of those files twice. Any other system has no default
excludes. excludes.
+5 -6
View File
@@ -29,12 +29,11 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- The default excludes cover caches as well: `/var/cache`, and each user's - The help of `sfdupes` and of `scan` names the database, its default path and
`.cache`, `.npm`, `.cargo/registry` and `go/pkg/mod`, root's included. On `SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
macOS they add `/private/var/folders`, `/Library/Caches` and each user's order of the rows and the escapes. The help of `info` lists its lines. The
`Library/Caches` and the same four. `--no-default-excludes` still turns them help of `dwim` says that it scans `/`, should run as root and keeps the scan
all off (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/181) in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither - New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
enters nor records an excluded path, and keeps the records below it. An enters nor records an excluded path, and keeps the records below it. An
operand at or below one is a fatal error (2026-11-11, operand at or below one is a fatal error (2026-11-11,
+75 -5
View File
@@ -32,6 +32,69 @@ const (
cmdDwim = "dwim" cmdDwim = "dwim"
) )
// databaseHelp is the paragraph of the root and scan help that names
// the database (README "Location").
const databaseHelp = "scan and dwim store what they find in a database " +
"file. report, trees\nand info read it. The file is $" + databaseEnv +
" when that is set and\nnot empty, and otherwise " +
defaultDatabasePath + ", where creating it\nneeds root.\n"
// escapesHelp is the paragraph of the report and trees help on how
// paths are escaped (README "Report output format").
const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
\\ a backslash
\t a tab
\n a newline
\r a carriage return
`
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
// summary in the help of their subcommands.
const (
reportHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate file. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that file
size the file's size in bytes
Rows are ordered by size, largest first, then by path.
` + escapesHelp
treesHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate directory tree. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that tree
files the number of regular files in one copy
size their total size in bytes
disk_size the bytes the dupe copy takes on disk
Rows are ordered by size, largest first, then by path.
` + escapesHelp
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
value, separated by a tab:
database the path, and where it came from
database file size in bytes, then rounded, as 2147483648 (2.0 GiB)
-wal file size in bytes, then rounded; only if a -wal file exists
modified when the database file last changed, in UTC
schema version the version of the database's layout
records the number of records
size of recorded files their total size, in bytes, then rounded
`
dwimHelp = `dwim runs what "sfdupes scan /" runs, then what "sfdupes trees" runs.
Run it as root, so that the scan can read every file. The scan is stored
in the database, as scan stores it, so "sfdupes trees" prints the report
again later.
`
)
// errNoSubcommand is returned by the root command when it is invoked // errNoSubcommand is returned by the root command when it is invoked
// without a subcommand. That is a usage error, and the usage text // without a subcommand. That is a usage error, and the usage text
// cobra prints for it is the whole message. // cobra prints for it is the whole message.
@@ -202,15 +265,21 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}), }),
} }
root.Long = root.Short + ".\n\n" + databaseHelp
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
dwimCmd.Long = dwimCmd.Short + ".\n\n" + dwimHelp
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd) root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
return root return root
} }
// newScanCommand builds the scan subcommand, with its --workers, -x, // newScanCommand builds the scan subcommand, with its --workers, -x,
// --no-default-excludes and --exclude flags. Its help lists the default // --no-default-excludes and --exclude flags. Its help names the
// excludes of the system it runs on, and the filesystem types it never // database, and lists the default excludes of the system it runs on and
// records anything on. // the filesystem types it never records anything on.
func newScanCommand() *cobra.Command { func newScanCommand() *cobra.Command {
var ( var (
scanWorkers int scanWorkers int
@@ -228,8 +297,9 @@ func newScanCommand() *cobra.Command {
// append its generic "[flags]" to it. // append its generic "[flags]" to it.
DisableFlagsInUseLine: true, DisableFlagsInUseLine: true,
Short: short, Short: short,
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS), Long: short + ".\n\n" + databaseHelp + "\n" +
Args: cobra.MinimumNArgs(1), defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1),
PreRunE: func(cmd *cobra.Command, _ []string) error { PreRunE: func(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, scanWorkers) err := checkScanWorkers(cmd, scanWorkers)
if err != nil { if err != nil {
+71 -1
View File
@@ -22,6 +22,9 @@ import (
// error prints and a runtime failure must not. // error prints and a runtime failure must not.
const usageMarker = "Usage:" const usageMarker = "Usage:"
// helpFlag asks for the help text.
const helpFlag = "--help"
// TestMain stops the tests at once, saying why, when their temporary // TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing // directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail // there (README "Rules for the walk"), so most scan tests would fail
@@ -693,7 +696,7 @@ func TestRunHelp(t *testing.T) {
// README §Settings: help goes to stderr, exits 0, and leaves stdout // README §Settings: help goes to stderr, exits 0, and leaves stdout
// empty. // empty.
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}} cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
for _, args := range cases { for _, args := range cases {
var stdout, stderr bytes.Buffer var stdout, stderr bytes.Buffer
@@ -754,6 +757,73 @@ func TestRunScanUsageLine(t *testing.T) {
} }
} }
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
t.Parallel()
// README §Settings: the help names the database, and says what
// report, trees, info and dwim print.
database := []string{
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
}
escapes := []string{`\\`, `\t`, `\n`, `\r`}
// The subcommand whose help is printed; none for the root.
cases := []struct {
subcommand []string
want []string
}{
{nil, database},
{[]string{cmdScan}, database},
{
[]string{cmdReport},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n size ",
"largest first",
}, escapes...),
},
{
[]string{cmdTrees},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n files ",
"\n size ", "\n disk_size ", "largest first",
}, escapes...),
},
{
[]string{cmdInfo},
[]string{
"\n database ", "\n database file size ",
"\n -wal file size ", "\n modified ",
"\n schema version ", "\n records ",
"\n size of recorded files ",
},
},
{
[]string{cmdDwim},
[]string{
"sfdupes scan /", "as root", "sfdupes trees", "database",
},
},
}
for _, tc := range cases {
var stdout, stderr bytes.Buffer
args := slices.Concat(tc.subcommand, []string{helpFlag})
code := run(args, &stdout, &stderr)
if code != exitOK {
t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
}
for _, want := range tc.want {
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) help lacks %q:\n%s",
args, want, stderr.String())
}
}
}
}
func TestRunVersion(t *testing.T) { func TestRunVersion(t *testing.T) {
t.Parallel() t.Parallel()
+2 -15
View File
@@ -72,27 +72,14 @@ type walkRules struct {
func defaultExcludes(goos string) []string { func defaultExcludes(goos string) []string {
switch goos { switch goos {
case goosLinux: case goosLinux:
// The kernel's view of the running system, temporary files, and return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
// caches that programs and package managers fill again on their
// own: the system's, root's and each user's.
return []string{
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
"/root/go/pkg/mod",
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
"/home/*/go/pkg/mod",
}
case goosDarwin: case goosDarwin:
// A walk of / reaches everything in /System/Volumes/Data through // A walk of / reaches everything in /System/Volumes/Data through
// the firmlinks, so walking it as well would record each of // the firmlinks, so walking it as well would record each of
// those files twice. /private/var/folders holds each user's // those files twice.
// temporary files and caches.
return []string{ return []string{
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm", "/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data", "/System/Volumes/VM", "/System/Volumes/Data",
"/private/var/folders", "/Library/Caches",
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
} }
case goosWindows: case goosWindows:
// Relative to the root of the drive an operand lies on. The // Relative to the root of the drive an operand lies on. The
+1 -32
View File
@@ -24,19 +24,10 @@ func TestDefaultExcludes(t *testing.T) {
// on any other system. The Windows list exists in the code only: the // on any other system. The Windows list exists in the code only: the
// binary does not build for Windows, so the README leaves it out. // binary does not build for Windows, so the README leaves it out.
cases := map[string][]string{ cases := map[string][]string{
goosLinux: { goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
"/root/go/pkg/mod",
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
"/home/*/go/pkg/mod",
},
goosDarwin: { goosDarwin: {
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm", "/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data", "/System/Volumes/VM", "/System/Volumes/Data",
"/private/var/folders", "/Library/Caches",
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
}, },
goosWindows: { goosWindows: {
"pagefile.sys", "swapfile.sys", "hiberfil.sys", "pagefile.sys", "swapfile.sys", "hiberfil.sys",
@@ -72,28 +63,6 @@ func TestIsDefaultExclude(t *testing.T) {
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want) t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
} }
} }
// The Linux list matches each user's cache directory itself, and no
// .cache directory elsewhere.
linux := map[string]bool{
"/home/u/.cache": true,
"/home/u/go/pkg/mod": true,
"/root/.cache": true,
"/var/cache": true,
"/home/u/.cache/pip": false,
"/home/u/src/.cache": false,
"/srv/home/u/.cache": false,
"/home/u/.cargo/bin": false,
"/home/u/go/pkg/mod/cache": false,
}
for path, want := range linux {
got := isDefaultExclude(path, defaultExcludes(goosLinux))
if got != want {
t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
got, want)
}
}
} }
func TestReadMountInfo(t *testing.T) { func TestReadMountInfo(t *testing.T) {