Leave caches out of the walk by default (closes #181)
check / check (push) Canceled after 0s

The default excludes now cover caches that programs and package
managers fill again on their own. On Linux: /var/cache and, for root
and each user under /home, .cache, .npm, .cargo/registry and
go/pkg/mod. On macOS: /private/var/folders, /Library/Caches and, for
each user, Library/Caches plus the same four. README and scan --help
list them; --no-default-excludes still turns them all off.

Model: opus-5-5
This commit is contained in:
sneak committed 2026-11-11 05:41:30 +00:00
1 parent 6caaff935a
commit 8ff9353da0
4 files changed
+75 -15

No files matched your search

+22 -12
View File
@@ -129,8 +129,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
- one line naming the database; - one line naming the database;
- progress lines, as described under "Progress"; - progress lines, as described under "Progress";
- a warning for each path it skips; - a warning for each path it skips;
- a line for each system or temporary directory the walk leaves out (see - a line for each system, temporary or cache directory the walk leaves out
"Rules for the walk"); (see "Rules for the walk");
- a line for each path left out because its filesystem is one the kernel, - a line for each path left out because its filesystem is one the kernel,
the core of the operating system, makes up or keeps in memory, such as the core of the operating system, makes up or keeps in memory, such as
`tmpfs`; `tmpfs`;
@@ -302,8 +302,9 @@ The reference below is also the program's specification.
directories on other filesystems, as the same flag does for `du` and `rsync`. directories on other filesystems, as the same flag does for `du` and `rsync`.
By default the walk also enters directories on other filesystems. By default the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes. - `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
Those are the system and temporary directories the walk otherwise leaves out, Those are the system, temporary and cache directories the walk otherwise
such as `/var/tmp` on Linux (see "Rules for the walk"). leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules for the
walk").
- `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below - `--exclude PATH` (`scan`) keeps the walk out of `PATH` and everything below
it. `PATH` may be a directory or a file. The flag may be given more than once. it. `PATH` may be a directory or a file. The flag may be given more than once.
The walk knows `PATH` by its device ID and inode number, as it knows the The walk knows `PATH` by its device ID and inode number, as it knows the
@@ -570,13 +571,18 @@ walk practical on large or busy storage pools.
it, as described under "What a scan does to the database". it, as described under "What a scan does to the database".
`--no-default-excludes` does not change this rule. `--no-default-excludes` does not change this rule.
- Below an operand, the walk does not enter a default exclude. On Linux, the - Below an operand, the walk does not enter a default exclude. The default
default excludes hold temporary files and the kernel's view of the running excludes hold the kernel's view of the running system, temporary files, and
system. They depend on the system, and `sfdupes scan --help` lists those of caches. A cache here is data that programs and package managers download or
the system it runs on. On Linux they are: build again on their own. The default excludes depend on the system, and
`sfdupes scan --help` lists those of the system it runs on. In the lists, `*`
stands for any one directory name, so `/home/*/.cache` covers each user's
cache. On Linux they are:
``` ```
/proc, /sys, /dev, /run, /tmp, /var/tmp /proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache
/root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
/home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
``` ```
On macOS they are: On macOS they are:
@@ -584,11 +590,15 @@ walk practical on large or busy storage pools.
``` ```
/dev, /private/tmp, /private/var/tmp, /private/var/vm /dev, /private/tmp, /private/var/tmp, /private/var/vm
/System/Volumes/VM, /System/Volumes/Data /System/Volumes/VM, /System/Volumes/Data
/private/var/folders, /Library/Caches
/Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
/Users/*/.cargo/registry, /Users/*/go/pkg/mod
``` ```
A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already, `/private/var/folders` holds each user's temporary files and caches on
through firmlinks. Firmlinks are the links macOS uses to show the data macOS. A walk of `/` on macOS reaches everything in `/System/Volumes/Data`
volume inside the system volume. Walking `/System/Volumes/Data` as well already, through firmlinks. Firmlinks are the links macOS uses to show the
data volume inside the system volume. Walking `/System/Volumes/Data` as well
would record each of those files twice. Any other system has no default would record each of those files twice. Any other system has no default
excludes. excludes.
+6
View File
@@ -29,6 +29,12 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- The default excludes cover caches as well: `/var/cache`, and each user's
`.cache`, `.npm`, `.cargo/registry` and `go/pkg/mod`, root's included. On
macOS they add `/private/var/folders`, `/Library/Caches` and each user's
`Library/Caches` and the same four. `--no-default-excludes` still turns them
all off (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/181)
- New `scan` flag `--exclude PATH`, which may be repeated. The walk neither - New `scan` flag `--exclude PATH`, which may be repeated. The walk neither
enters nor records an excluded path, and keeps the records below it. An enters nor records an excluded path, and keeps the records below it. An
operand at or below one is a fatal error (2026-11-11, operand at or below one is a fatal error (2026-11-11,
+15 -2
View File
@@ -72,14 +72,27 @@ type walkRules struct {
func defaultExcludes(goos string) []string { func defaultExcludes(goos string) []string {
switch goos { switch goos {
case goosLinux: case goosLinux:
return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"} // The kernel's view of the running system, temporary files, and
// caches that programs and package managers fill again on their
// own: the system's, root's and each user's.
return []string{
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
"/root/go/pkg/mod",
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
"/home/*/go/pkg/mod",
}
case goosDarwin: case goosDarwin:
// A walk of / reaches everything in /System/Volumes/Data through // A walk of / reaches everything in /System/Volumes/Data through
// the firmlinks, so walking it as well would record each of // the firmlinks, so walking it as well would record each of
// those files twice. // those files twice. /private/var/folders holds each user's
// temporary files and caches.
return []string{ return []string{
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm", "/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data", "/System/Volumes/VM", "/System/Volumes/Data",
"/private/var/folders", "/Library/Caches",
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
} }
case goosWindows: case goosWindows:
// Relative to the root of the drive an operand lies on. The // Relative to the root of the drive an operand lies on. The
+32 -1
View File
@@ -24,10 +24,19 @@ func TestDefaultExcludes(t *testing.T) {
// on any other system. The Windows list exists in the code only: the // on any other system. The Windows list exists in the code only: the
// binary does not build for Windows, so the README leaves it out. // binary does not build for Windows, so the README leaves it out.
cases := map[string][]string{ cases := map[string][]string{
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}, goosLinux: {
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
"/root/go/pkg/mod",
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
"/home/*/go/pkg/mod",
},
goosDarwin: { goosDarwin: {
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm", "/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
"/System/Volumes/VM", "/System/Volumes/Data", "/System/Volumes/VM", "/System/Volumes/Data",
"/private/var/folders", "/Library/Caches",
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
}, },
goosWindows: { goosWindows: {
"pagefile.sys", "swapfile.sys", "hiberfil.sys", "pagefile.sys", "swapfile.sys", "hiberfil.sys",
@@ -63,6 +72,28 @@ func TestIsDefaultExclude(t *testing.T) {
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want) t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
} }
} }
// The Linux list matches each user's cache directory itself, and no
// .cache directory elsewhere.
linux := map[string]bool{
"/home/u/.cache": true,
"/home/u/go/pkg/mod": true,
"/root/.cache": true,
"/var/cache": true,
"/home/u/.cache/pip": false,
"/home/u/src/.cache": false,
"/srv/home/u/.cache": false,
"/home/u/.cargo/bin": false,
"/home/u/go/pkg/mod/cache": false,
}
for path, want := range linux {
got := isDefaultExclude(path, defaultExcludes(goosLinux))
if got != want {
t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
got, want)
}
}
} }
func TestReadMountInfo(t *testing.T) { func TestReadMountInfo(t *testing.T) {