Compare commits

..
Author SHA1 Message Date
sneak 9e6e3b9287 Say in --help where the database is and what each report prints (closes #177)
check / check (push) Successful in 3m4s
The help of sfdupes and of scan names the database, its default path
and SFDUPES_DATABASE. The help of report and trees lists the columns,
the order of the rows and the four escapes. The help of info lists its
lines. The help of dwim says that it scans /, should run as root and
keeps the scan in the database.

The README's opening drops its sentence on copies deleted after the
scan, an edge case that its report and trees sections already cover.

Model: opus-5-5
2026-11-11 05:57:25 +00:00
6 changed files with 173 additions and 87 deletions

No files matched your search

+19 -28
View File
@@ -22,9 +22,8 @@ records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` work from the database and
can run at any time. `report` and `trees` leave out a copy that was deleted
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first
look at a new machine with one command.
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
first look at a new machine with one command.
## Getting Started
@@ -129,8 +128,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
- one line naming the database;
- progress lines, as described under "Progress";
- a warning for each path it skips;
- a line for each system, temporary or cache directory the walk leaves out
(see "Rules for the walk");
- a line for each system or temporary directory the walk leaves out (see
"Rules for the walk");
- a line for each path left out because its filesystem is one the kernel,
the core of the operating system, makes up or keeps in memory, such as
`tmpfs`;
@@ -302,9 +301,8 @@ The reference below is also the program's specification.
directories on other filesystems, as the same flag does for `du` and `rsync`.
By default the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
Those are the system, temporary and cache directories the walk otherwise
leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules for the
walk").
Those are the system and temporary directories the walk otherwise leaves out,
such as `/var/tmp` on Linux (see "Rules for the walk").
- `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
it. `PATH` may be a directory or a file. The flag may be given more than once.
The walk knows `PATH` by its device ID and inode number, as it knows the
@@ -318,9 +316,11 @@ The reference below is also the program's specification.
- `--version`, `-v` prints `sfdupes VERSION` (see "Install") to stdout and
exits 0. It writes nothing to stderr.
- `--help`, `-h`, alone or after a subcommand, prints the help text to stderr
and exits 0. It writes nothing to stdout. `sfdupes scan --help` also lists the
default excludes of the system it runs on, and the filesystem types the walk
never records anything on.
and exits 0. It writes nothing to stdout. The help of `sfdupes` and of `scan`
names the database. The help of `report`, `trees`, `info` and `dwim` says what
each prints. `sfdupes scan --help` also lists the default excludes of the
system it runs on, and the filesystem types the walk never records anything
on.
### Duplicate detection
@@ -571,18 +571,13 @@ walk practical on large or busy storage pools.
it, as described under "What a scan does to the database".
`--no-default-excludes` does not change this rule.
- Below an operand, the walk does not enter a default exclude. The default
excludes hold the kernel's view of the running system, temporary files, and
caches. A cache here is data that programs and package managers download or
build again on their own. The default excludes depend on the system, and
`sfdupes scan --help` lists those of the system it runs on. In the lists, `*`
stands for any one directory name, so `/home/*/.cache` covers each user's
cache. On Linux they are:
- Below an operand, the walk does not enter a default exclude. On Linux, the
default excludes hold temporary files and the kernel's view of the running
system. They depend on the system, and `sfdupes scan --help` lists those of
the system it runs on. On Linux they are:
```
/proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache
/root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
/home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
/proc, /sys, /dev, /run, /tmp, /var/tmp
```
On macOS they are:
@@ -590,15 +585,11 @@ walk practical on large or busy storage pools.
```
/dev, /private/tmp, /private/var/tmp, /private/var/vm
/System/Volumes/VM, /System/Volumes/Data
/private/var/folders, /Library/Caches
/Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
/Users/*/.cargo/registry, /Users/*/go/pkg/mod
```
`/private/var/folders` holds each user's temporary files and caches on
macOS. A walk of `/` on macOS reaches everything in `/System/Volumes/Data`
already, through firmlinks. Firmlinks are the links macOS uses to show the
data volume inside the system volume. Walking `/System/Volumes/Data` as well
A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already,
through firmlinks. Firmlinks are the links macOS uses to show the data
volume inside the system volume. Walking `/System/Volumes/Data` as well
would record each of those files twice. Any other system has no default
excludes.
+5 -6
View File
@@ -29,12 +29,11 @@
Newest first. An entry ends with its date and its issue or branch, where those
are known.
- The default excludes cover caches as well: `/var/cache`, and each user's
`.cache`, `.npm`, `.cargo/registry` and `go/pkg/mod`, root's included. On
macOS they add `/private/var/folders`, `/Library/Caches` and each user's
`Library/Caches` and the same four. `--no-default-excludes` still turns them
all off (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/181)
- The help of `sfdupes` and of `scan` names the database, its default path and
`SFDUPES_DATABASE`. The help of `report` and `trees` lists the columns, the
order of the rows and the escapes. The help of `info` lists its lines. The
help of `dwim` says that it scans `/`, should run as root and keeps the scan
in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
enters nor records an excluded path, and keeps the records below it. An
operand at or below one is a fatal error (2026-11-11,
+75 -5
View File
@@ -32,6 +32,69 @@ const (
cmdDwim = "dwim"
)
// databaseHelp is the paragraph of the root and scan help that names
// the database (README "Location").
const databaseHelp = "scan and dwim store what they find in a database " +
"file. report, trees\nand info read it. The file is $" + databaseEnv +
" when that is set and\nnot empty, and otherwise " +
defaultDatabasePath + ", where creating it\nneeds root.\n"
// escapesHelp is the paragraph of the report and trees help on how
// paths are escaped (README "Report output format").
const escapesHelp = `Four characters in a path are escaped, so each row stays one line:
\\ a backslash
\t a tab
\n a newline
\r a carriage return
`
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
// summary in the help of their subcommands.
const (
reportHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate file. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that file
size the file's size in bytes
Rows are ordered by size, largest first, then by path.
` + escapesHelp
treesHelp = `The report is tab-separated text on stdout: a header line, then one
row per duplicate directory tree. The columns are:
first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that tree
files the number of regular files in one copy
size their total size in bytes
disk_size the bytes the dupe copy takes on disk
Rows are ordered by size, largest first, then by path.
` + escapesHelp
infoHelp = `The details go to stdout, one per line. Each line holds a name and its
value, separated by a tab:
database the path, and where it came from
database file size in bytes, then rounded, as 2147483648 (2.0 GiB)
-wal file size in bytes, then rounded; only if a -wal file exists
modified when the database file last changed, in UTC
schema version the version of the database's layout
records the number of records
size of recorded files their total size, in bytes, then rounded
`
dwimHelp = `dwim runs what "sfdupes scan /" runs, then what "sfdupes trees" runs.
Run it as root, so that the scan can read every file. The scan is stored
in the database, as scan stores it, so "sfdupes trees" prints the report
again later.
`
)
// errNoSubcommand is returned by the root command when it is invoked
// without a subcommand. That is a usage error, and the usage text
// cobra prints for it is the whole message.
@@ -202,15 +265,21 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
}),
}
root.Long = root.Short + ".\n\n" + databaseHelp
reportCmd.Long = reportCmd.Short + ".\n\n" + reportHelp
treesCmd.Long = treesCmd.Short + ".\n\n" + treesHelp
infoCmd.Long = infoCmd.Short + ".\n\n" + infoHelp
dwimCmd.Long = dwimCmd.Short + ".\n\n" + dwimHelp
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
return root
}
// newScanCommand builds the scan subcommand, with its --workers, -x,
// --no-default-excludes and --exclude flags. Its help lists the default
// excludes of the system it runs on, and the filesystem types it never
// records anything on.
// --no-default-excludes and --exclude flags. Its help names the
// database, and lists the default excludes of the system it runs on and
// the filesystem types it never records anything on.
func newScanCommand() *cobra.Command {
var (
scanWorkers int
@@ -228,8 +297,9 @@ func newScanCommand() *cobra.Command {
// append its generic "[flags]" to it.
DisableFlagsInUseLine: true,
Short: short,
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1),
Long: short + ".\n\n" + databaseHelp + "\n" +
defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1),
PreRunE: func(cmd *cobra.Command, _ []string) error {
err := checkScanWorkers(cmd, scanWorkers)
if err != nil {
+71 -1
View File
@@ -22,6 +22,9 @@ import (
// error prints and a runtime failure must not.
const usageMarker = "Usage:"
// helpFlag asks for the help text.
const helpFlag = "--help"
// TestMain stops the tests at once, saying why, when their temporary
// directory is not on a persistent filesystem: scan records nothing
// there (README "Rules for the walk"), so most scan tests would fail
@@ -693,7 +696,7 @@ func TestRunHelp(t *testing.T) {
// README §Settings: help goes to stderr, exits 0, and leaves stdout
// empty.
cases := [][]string{{"--help"}, {"-h"}, {cmdScan, "--help"}}
cases := [][]string{{helpFlag}, {"-h"}, {cmdScan, helpFlag}}
for _, args := range cases {
var stdout, stderr bytes.Buffer
@@ -754,6 +757,73 @@ func TestRunScanUsageLine(t *testing.T) {
}
}
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
t.Parallel()
// README §Settings: the help names the database, and says what
// report, trees, info and dwim print.
database := []string{
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
}
escapes := []string{`\\`, `\t`, `\n`, `\r`}
// The subcommand whose help is printed; none for the root.
cases := []struct {
subcommand []string
want []string
}{
{nil, database},
{[]string{cmdScan}, database},
{
[]string{cmdReport},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n size ",
"largest first",
}, escapes...),
},
{
[]string{cmdTrees},
append([]string{
"tab-separated", "\n first ", "\n dupe ", "\n files ",
"\n size ", "\n disk_size ", "largest first",
}, escapes...),
},
{
[]string{cmdInfo},
[]string{
"\n database ", "\n database file size ",
"\n -wal file size ", "\n modified ",
"\n schema version ", "\n records ",
"\n size of recorded files ",
},
},
{
[]string{cmdDwim},
[]string{
"sfdupes scan /", "as root", "sfdupes trees", "database",
},
},
}
for _, tc := range cases {
var stdout, stderr bytes.Buffer
args := slices.Concat(tc.subcommand, []string{helpFlag})
code := run(args, &stdout, &stderr)
if code != exitOK {
t.Fatalf("run(%v) = %d, want %d", args, code, exitOK)
}
for _, want := range tc.want {
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) help lacks %q:\n%s",
args, want, stderr.String())
}
}
}
}
func TestRunVersion(t *testing.T) {
t.Parallel()
+2 -15
View File
@@ -72,27 +72,14 @@ type walkRules struct {
func defaultExcludes(goos string) []string {
switch goos {
case goosLinux:
// The kernel's view of the running system, temporary files, and
// caches that programs and package managers fill again on their
// own: the system's, root's and each user's.
return []string{
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
"/root/go/pkg/mod",
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
"/home/*/go/pkg/mod",
}
return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
case goosDarwin:
// A walk of / reaches everything in /System/Volumes/Data through
// the firmlinks, so walking it as well would record each of
// those files twice. /private/var/folders holds each user's
// temporary files and caches.
// those files twice.
return []string{
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data",
"/private/var/folders", "/Library/Caches",
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
}
case goosWindows:
// Relative to the root of the drive an operand lies on. The
+1 -32
View File
@@ -24,19 +24,10 @@ func TestDefaultExcludes(t *testing.T) {
// on any other system. The Windows list exists in the code only: the
// binary does not build for Windows, so the README leaves it out.
cases := map[string][]string{
goosLinux: {
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
"/root/go/pkg/mod",
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
"/home/*/go/pkg/mod",
},
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
goosDarwin: {
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data",
"/private/var/folders", "/Library/Caches",
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
},
goosWindows: {
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
@@ -72,28 +63,6 @@ func TestIsDefaultExclude(t *testing.T) {
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
}
}
// The Linux list matches each user's cache directory itself, and no
// .cache directory elsewhere.
linux := map[string]bool{
"/home/u/.cache": true,
"/home/u/go/pkg/mod": true,
"/root/.cache": true,
"/var/cache": true,
"/home/u/.cache/pip": false,
"/home/u/src/.cache": false,
"/srv/home/u/.cache": false,
"/home/u/.cargo/bin": false,
"/home/u/go/pkg/mod/cache": false,
}
for path, want := range linux {
got := isDefaultExclude(path, defaultExcludes(goosLinux))
if got != want {
t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
got, want)
}
}
}
func TestReadMountInfo(t *testing.T) {