Compare commits

...
2 Commits
Author SHA1 Message Date
sneak 6ded88234a Keep a non-root user's database in the XDG state directory (closes #171)
check / check (push) Canceled after 0s
Without SFDUPES_DATABASE, a user other than root now uses
sfdupes/db.sqlite under $XDG_STATE_HOME (~/.local/state by default),
which scan creates, instead of failing on /var/lib/sfdupes. Root keeps
/var/lib/sfdupes/db.sqlite. sfdupes --help and README say where the
database is. Adds github.com/adrg/xdg, the decided XDG library.

Model: opus-5-5
2026-11-11 04:00:23 +00:00
clawbot 6c529fcc6d Leave empty files out of report and trees (closes #178) (#183)
check / check (push) Canceled after 0s
2026-11-11 03:52:45 +01:00
14 changed files with 224 additions and 43 deletions

No files matched your search

+29 -16
View File
@@ -29,7 +29,6 @@ first look at a new machine with one command.
```sh ```sh
make build make build
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
./sfdupes scan /srv ./sfdupes scan /srv
./sfdupes report > dupes.tsv ./sfdupes report > dupes.tsv
./sfdupes trees > dupetrees.tsv ./sfdupes trees > dupetrees.tsv
@@ -45,8 +44,9 @@ that stands for a piece of hardware, such as a disk or a terminal. `report`
prints the file-level duplicates. `trees` prints the duplicate directory trees. prints the file-level duplicates. `trees` prints the duplicate directory trees.
`info` prints the database's path, size and record count. `info` prints the database's path, size and record count.
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names Root keeps the database in `/var/lib/sfdupes/db.sqlite`. On Linux, any other
another file. The first `scan` creates it. user keeps it in `~/.local/state/sfdupes/db.sqlite`. `SFDUPES_DATABASE` names
another file (see "Location"). The first `scan` creates the database.
`report`, `trees` and `info` write their data to stdout. Progress, warnings and `report`, `trees` and `info` write their data to stdout. Progress, warnings and
errors go to stderr. stdout and stderr are a program's two output streams. The errors go to stderr. stdout and stderr are a program's two output streams. The
@@ -146,8 +146,9 @@ walk"). As the file `/etc/cron.d/sfdupes`:
- `report`, `trees` and `info` need only read access to the database. The umask - `report`, `trees` and `info` need only read access to the database. The umask
is the setting that removes permissions from the files a process creates. is the setting that removes permissions from the files a process creates.
Under the usual umask of `022`, the first scan creates the database readable Under the usual umask of `022`, the first scan creates the database readable
by every user. An unprivileged user can then run those three commands against by every user. Another user can then run those three commands against root's
root's database. database, with `SFDUPES_DATABASE=/var/lib/sfdupes/db.sqlite` before the
command.
### Reading the reports ### Reading the reports
@@ -283,9 +284,9 @@ The reference below is also the program's specification.
### Settings ### Settings
- `SFDUPES_DATABASE` names the database file. When it is unset or empty, the - `SFDUPES_DATABASE` names the database file. When it is unset or empty, the
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for database is at its default path (see "Location"). There is no command-line
it. The path names the file exactly, whatever characters it holds. A relative flag for it. The path names the file exactly, whatever characters it holds. A
path is relative to the working directory. relative path is relative to the working directory.
- `--workers N` (`scan`) sets how many directories `scan` reads at once while it - `--workers N` (`scan`) sets how many directories `scan` reads at once while it
lists the trees. It also sets how many files `scan` reads at once while it lists the trees. It also sets how many files `scan` reads at once while it
hashes them. The default is the number of CPUs. `N` must be at least 1. A hashes them. The default is the number of CPUs. `N` must be at least 1. A
@@ -665,6 +666,7 @@ Processing:
- Records are grouped by signature, the four values - Records are grouped by signature, the four values
`(size, head, tail, content)`. `(size, head, tail, content)`.
- Every group with two or more paths is a duplicate group. - Every group with two or more paths is a duplicate group.
- Empty files form no group, because removing one frees nothing.
- Within a group, paths are sorted in byte order. The first is `first`, and - Within a group, paths are sorted in byte order. The first is `first`, and
every other path is a `dupe`. every other path is a `dupe`.
- Groups are ordered by size, largest first, then by `first` path. The output is - Groups are ordered by size, largest first, then by `first` path. The output is
@@ -713,7 +715,7 @@ The summary line on stderr gives:
- the reclaimable bytes, the sum of `size` over all rows. - the reclaimable bytes, the sum of `size` over all rows.
``` ```
report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
``` ```
### `trees` mode ### `trees` mode
@@ -741,6 +743,9 @@ Processing:
- `trees` builds the directory tree and computes every directory's digest. It - `trees` builds the directory tree and computes every directory's digest. It
then groups the directories by digest. Every group with two or more then groups the directories by digest. Every group with two or more
directories is a duplicate-tree group. directories is a duplicate-tree group.
- `trees` leaves out a group whose trees hold 0 bytes, because removing one
frees nothing. An empty file still counts toward the digest of a tree that
holds other files.
- `trees` leaves out a group that a group of its members' parents already - `trees` leaves out a group that a group of its members' parents already
implies. A group is left out when its members' parents are distinct implies. A group is left out when its members' parents are distinct
directories that all have the same digest. The parents' group, or an directories that all have the same digest. The parents' group, or an
@@ -876,7 +881,15 @@ All five subcommands work on one SQLite database file.
#### Location #### Location
The database is the file named by `SFDUPES_DATABASE`, when that is set and not The database is the file named by `SFDUPES_DATABASE`, when that is set and not
empty, and otherwise `/var/lib/sfdupes/db.sqlite`. empty. Otherwise it is at its default path, which depends on the user:
- For root, it is `/var/lib/sfdupes/db.sqlite`.
- For any other user, it is `sfdupes/db.sqlite` in the XDG state directory. That
directory is `$XDG_STATE_HOME` when that is set to an absolute path. Otherwise
it is `~/.local/state` on Linux and `~/Library/Application Support` on macOS.
All five subcommands use the same default, so a user's `report` reads the
database that user's `scan` wrote.
Each subcommand prints one line on stderr naming the database it uses, before Each subcommand prints one line on stderr naming the database it uses, before
anything else and whether or not stderr is a terminal. The line gives the path anything else and whether or not stderr is a terminal. The line gives the path
@@ -916,9 +929,9 @@ Only one `scan` at a time can hold the lock.
- For its whole run, `scan` holds a lock on a lock file beside the database. The - For its whole run, `scan` holds a lock on a lock file beside the database. The
lock is taken with the `flock(2)` system call, a request a program makes to lock is taken with the `flock(2)` system call, a request a program makes to
the kernel, and only one process can hold it. The lock file's name is the the kernel, and only one process can hold it. The lock file's name is the
database's path, with symlinks resolved, plus `.lock`. By default that is database's path, with symlinks resolved, plus `.lock`. For root, by default,
`/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same that is `/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the
lock file. same lock file.
- `scan` takes the lock before it walks the filesystem or opens the database. - `scan` takes the lock before it walks the filesystem or opens the database.
Just before that, it creates the database file, empty, if it is missing. So Just before that, it creates the database file, empty, if it is missing. So
the lock file is named the same way before the first scan as after. A symlink the lock file is named the same way before the first scan as after. A symlink
@@ -1382,7 +1395,7 @@ All of the following, run in this directory, must pass:
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
printf 'y' > "$d/tiny3" # 1-byte non-duplicate printf 'y' > "$d/tiny3" # 1-byte non-duplicate
: > "$d/empty1"; : > "$d/empty2" # empty duplicates : > "$d/empty1"; : > "$d/empty2" # empty files
# duplicate trees: t1 and t2 are identical; t3 differs by one filename # duplicate trees: t1 and t2 are identical; t3 differs by one filename
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub" mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
head -c 3000 /dev/urandom > "$d/t1/f1" head -c 3000 /dev/urandom > "$d/t1/f1"
@@ -1407,8 +1420,8 @@ All of the following, run in this directory, must pass:
is the path that sorts first. is the path that sorts first.
- `t1/f1`, `t2/f1` and `t3/f1` form one group. - `t1/f1`, `t2/f1` and `t3/f1` form one group.
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group. - `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
- `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`. - `tiny1` and `tiny2` form a pair.
- `unique.bin` and `tiny3` appear nowhere. - `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
- The groups come in order of size, largest first. - The groups come in order of size, largest first.
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`. `trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
+9
View File
@@ -29,6 +29,15 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- Without `SFDUPES_DATABASE`, a user other than root keeps the database in
`sfdupes/db.sqlite` under the XDG state directory, `~/.local/state` by
default. Root keeps `/var/lib/sfdupes/db.sqlite`. `sfdupes --help` says where
the database is (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/171)
- `report` and `trees` leave out empty files. `report` writes no group of size
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
counts toward a tree's digest (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/178)
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the - The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
command, and it calls `Main` in the package `internal/sfdupes`. `main.go` command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
became `internal/sfdupes/cli.go`. Every other file kept its name. The install became `internal/sfdupes/cli.go`. Every other file kept its name. The install
+4 -2
View File
@@ -5,8 +5,10 @@
// KiB, and of the content. The content hash covers the whole file under // KiB, and of the content. The content hash covers the whole file under
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large // 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
// file is never read in full. scan maintains a persistent SQLite // file is never read in full. scan maintains a persistent SQLite
// database of file signatures (SFDUPES_DATABASE, default // database of file signatures that the other subcommands read. The
// /var/lib/sfdupes/db.sqlite) that the other subcommands read. // database is SFDUPES_DATABASE, or by default /var/lib/sfdupes/db.sqlite
// for root and sfdupes/db.sqlite in the XDG state directory for any
// other user.
// //
// Usage: // Usage:
// //
+1
View File
@@ -3,6 +3,7 @@ module sneak.berlin/go/sfdupes
go 1.25.7 go 1.25.7
require ( require (
github.com/adrg/xdg v0.5.3
github.com/schollz/progressbar/v3 v3.19.1 github.com/schollz/progressbar/v3 v3.19.1
github.com/spf13/cobra v1.10.2 github.com/spf13/cobra v1.10.2
golang.org/x/sys v0.46.0 golang.org/x/sys v0.46.0
+2
View File
@@ -1,3 +1,5 @@
github.com/adrg/xdg v0.5.3 h1:xRnxJXne7+oWDatRhR1JLnvuccuIeCoBu2rtuLqQB78=
github.com/adrg/xdg v0.5.3/go.mod h1:nlTsY+NNiCBGCK2tpm09vRqfVzrc2fLmXGpBLF0zlTQ=
github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM= github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM=
github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY= github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g= github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
+11 -1
View File
@@ -116,6 +116,13 @@ func run(args []string, stdout, stderr io.Writer) int {
} }
} }
// databaseHelp says in sfdupes --help where the database is.
const databaseHelp = `The database is the file SFDUPES_DATABASE names.
When that is unset or empty, root uses /var/lib/sfdupes/db.sqlite. Any
other user uses sfdupes/db.sqlite in the XDG state directory:
$XDG_STATE_HOME, or ~/.local/state when that is unset
(~/Library/Application Support on macOS).`
// newRootCommand builds the command tree. Everything on stdout is // newRootCommand builds the command tree. Everything on stdout is
// machine-readable data, the version line included; all human-facing // machine-readable data, the version line included; all human-facing
// output (help, usage, errors) goes to stderr. // output (help, usage, errors) goes to stderr.
@@ -134,9 +141,12 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
return nil return nil
}) })
short := "Find candidate duplicate files by size and head/tail/content SHA-256"
root := &cobra.Command{ root := &cobra.Command{
Use: "sfdupes", Use: "sfdupes",
Short: "Find candidate duplicate files by size and head/tail/content SHA-256", Short: short,
Long: short + ".\n\n" + databaseHelp,
Args: cobra.NoArgs, Args: cobra.NoArgs,
RunE: func(cmd *cobra.Command, args []string) error { RunE: func(cmd *cobra.Command, args []string) error {
if showVersion { if showVersion {
+42 -8
View File
@@ -15,6 +15,7 @@ import (
"strings" "strings"
"testing" "testing"
"github.com/adrg/xdg"
"golang.org/x/sys/unix" "golang.org/x/sys/unix"
) )
@@ -553,13 +554,14 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
// line. // line.
t.Setenv(databaseEnv, "") t.Setenv(databaseEnv, "")
// Whether scan calls the database new depends on that machine too. dbPath := defaultDatabasePath(os.Geteuid())
scanWant := "scan: database " + defaultDatabasePath + " (default)\n"
_, err := os.Stat(defaultDatabasePath) // Whether scan calls the database new depends on that machine too.
scanWant := "scan: database " + dbPath + " (default)\n"
_, err := os.Stat(dbPath)
if errors.Is(err, fs.ErrNotExist) { if errors.Is(err, fs.ErrNotExist) {
scanWant = "scan: new database " + defaultDatabasePath + scanWant = "scan: new database " + dbPath + " (default)\n"
" (default)\n"
} }
missing := filepath.Join(t.TempDir(), "nope") missing := filepath.Join(t.TempDir(), "nope")
@@ -578,21 +580,21 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
run: func() error { run: func() error {
return runReport(cancelledContext(t), io.Discard) return runReport(cancelledContext(t), io.Discard)
}, },
want: "report: database " + defaultDatabasePath + " (default)\n" + want: "report: database " + dbPath + " (default)\n" +
reportOpenLine + "\n" + reportOpenLine + "\n", reportOpenLine + "\n" + reportOpenLine + "\n",
}, },
cmdTrees: { cmdTrees: {
run: func() error { run: func() error {
return runTrees(cancelledContext(t), io.Discard) return runTrees(cancelledContext(t), io.Discard)
}, },
want: "trees: database " + defaultDatabasePath + " (default)\n" + want: "trees: database " + dbPath + " (default)\n" +
treesOpenLine + "\n" + treesOpenLine + "\n", treesOpenLine + "\n" + treesOpenLine + "\n",
}, },
cmdInfo: { cmdInfo: {
run: func() error { run: func() error {
return runInfo(cancelledContext(t), io.Discard) return runInfo(cancelledContext(t), io.Discard)
}, },
want: "info: database " + defaultDatabasePath + " (default)\n" + want: "info: database " + dbPath + " (default)\n" +
infoOpenLine + "\n" + infoOpenLine + "\n", infoOpenLine + "\n" + infoOpenLine + "\n",
}, },
} }
@@ -711,6 +713,20 @@ func TestRunHelp(t *testing.T) {
t.Errorf("run(%v) stdout = %q, want nothing (data only)", t.Errorf("run(%v) stdout = %q, want nothing (data only)",
args, got) args, got)
} }
if args[0] == cmdScan {
continue
}
// README §Location: sfdupes --help says where the database is.
for _, want := range []string{
databaseEnv, rootDatabasePath, "XDG_STATE_HOME", "~/.local/state",
} {
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) stderr = %q, want it to name %q",
args, stderr.String(), want)
}
}
} }
} }
@@ -1496,6 +1512,24 @@ func TestRunDatabasePathUsedAsGiven(t *testing.T) {
}) })
} }
func TestRunNonRootUsesStateDirectory(t *testing.T) {
// README §Location: without SFDUPES_DATABASE, a user other than
// root keeps the database in the XDG state directory. scan creates
// the missing directories, and report and trees read the same file.
if os.Geteuid() == 0 {
t.Skip("root uses " + rootDatabasePath)
}
t.Cleanup(xdg.Reload)
state := t.TempDir()
t.Setenv("XDG_STATE_HOME", state)
t.Setenv(databaseEnv, "")
xdg.Reload()
assertRunsUseDatabase(t, filepath.Join(state, "sfdupes"), "db.sqlite")
}
// holdScanLock takes the lock on the database at path, as a running // holdScanLock takes the lock on the database at path, as a running
// scan does, and holds it until the test ends. It fails the test when // scan does, and holds it until the test ends. It fails the test when
// the lock is already held. // the lock is already held.
+20 -5
View File
@@ -15,15 +15,16 @@ import (
"strings" "strings"
"time" "time"
"github.com/adrg/xdg"
"golang.org/x/sys/unix" "golang.org/x/sys/unix"
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo // The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
// disabled. // disabled.
_ "modernc.org/sqlite" _ "modernc.org/sqlite"
) )
// defaultDatabasePath is where the persistent scan database lives when // rootDatabasePath is the database root uses when SFDUPES_DATABASE is
// SFDUPES_DATABASE is not set. // not set.
const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite" const rootDatabasePath = "/var/lib/sfdupes/db.sqlite"
// databaseEnv is the environment variable that overrides the database // databaseEnv is the environment variable that overrides the database
// path. // path.
@@ -144,13 +145,27 @@ var errScanRunning = errors.New("another scan is running")
// 32 bytes of a SHA-256. // 32 bytes of a SHA-256.
var errHashSize = errors.New("hash is not 32 bytes") var errHashSize = errors.New("hash is not 32 bytes")
// defaultDatabasePath is the database used when SFDUPES_DATABASE is not
// set: rootDatabasePath for root (euid 0), and db.sqlite in the
// directory sfdupes under the XDG state directory for any other user.
// xdg reads XDG_STATE_HOME and HOME once, when the program starts. The
// path is joined as text, not cleaned, for the reason absolutePath
// gives.
func defaultDatabasePath(euid int) string {
if euid == 0 {
return rootDatabasePath
}
return strings.TrimSuffix(xdg.StateHome, "/") + "/sfdupes/db.sqlite"
}
// databasePath resolves the database location: SFDUPES_DATABASE when // databasePath resolves the database location: SFDUPES_DATABASE when
// set and non-empty, the compiled-in default otherwise. It returns the // set and non-empty, defaultDatabasePath otherwise. It returns the
// path, made absolute by absolutePath, and where the path came from, // path, made absolute by absolutePath, and where the path came from,
// "default" or "from SFDUPES_DATABASE", for the line each subcommand // "default" or "from SFDUPES_DATABASE", for the line each subcommand
// prints first on stderr. // prints first on stderr.
func databasePath() (string, string, error) { func databasePath() (string, string, error) {
path, source := defaultDatabasePath, "default" path, source := defaultDatabasePath(os.Geteuid()), "default"
if p := os.Getenv(databaseEnv); p != "" { if p := os.Getenv(databaseEnv); p != "" {
path, source = p, "from "+databaseEnv path, source = p, "from "+databaseEnv
} }
+39 -1
View File
@@ -9,10 +9,13 @@ import (
"io/fs" "io/fs"
"os" "os"
"path/filepath" "path/filepath"
"runtime"
"slices" "slices"
"strings" "strings"
"testing" "testing"
"time" "time"
"github.com/adrg/xdg"
) )
// testDBPath returns a database path inside a fresh temp dir, with // testDBPath returns a database path inside a fresh temp dir, with
@@ -60,7 +63,7 @@ func TestDatabasePath(t *testing.T) {
cases := []struct { cases := []struct {
env, path, source string env, path, source string
}{ }{
{"", defaultDatabasePath, "default"}, {"", defaultDatabasePath(os.Geteuid()), "default"},
{"/custom/place.sqlite", "/custom/place.sqlite", fromEnv}, {"/custom/place.sqlite", "/custom/place.sqlite", fromEnv},
{"/custom/link/../place.sqlite", "/custom/link/../place.sqlite", {"/custom/link/../place.sqlite", "/custom/link/../place.sqlite",
fromEnv}, fromEnv},
@@ -89,6 +92,41 @@ func TestDatabasePath(t *testing.T) {
} }
} }
func TestDefaultDatabasePath(t *testing.T) {
// README §Location: root's database is /var/lib/sfdupes/db.sqlite.
// Any other user's is db.sqlite in sfdupes under XDG_STATE_HOME, or
// under ~/.local/state when XDG_STATE_HOME is unset or not absolute.
if runtime.GOOS != goosLinux {
t.Skip("macOS has another default state directory")
}
t.Cleanup(xdg.Reload)
t.Setenv("HOME", "/home/someone")
cases := []struct {
euid int
stateHome string
want string
}{
{0, "", rootDatabasePath},
{0, "/elsewhere/state", rootDatabasePath},
{1000, "", "/home/someone/.local/state/sfdupes/db.sqlite"},
{1000, "relative/state", "/home/someone/.local/state/sfdupes/db.sqlite"},
{1000, "/elsewhere/state", "/elsewhere/state/sfdupes/db.sqlite"},
{1000, "/elsewhere/state/", "/elsewhere/state/sfdupes/db.sqlite"},
}
for _, tc := range cases {
t.Setenv("XDG_STATE_HOME", tc.stateHome)
xdg.Reload()
if got := defaultDatabasePath(tc.euid); got != tc.want {
t.Errorf("defaultDatabasePath(%d) with XDG_STATE_HOME=%q = %q, want %q",
tc.euid, tc.stateHome, got, tc.want)
}
}
}
func TestOpenScanDatabaseCreates(t *testing.T) { func TestOpenScanDatabaseCreates(t *testing.T) {
t.Parallel() t.Parallel()
+4 -3
View File
@@ -195,10 +195,11 @@ func (w *reportWriter) add(r scanRec) error {
} }
// endGroup ends the paths read with w.sig: two or more are a duplicate // endGroup ends the paths read with w.sig: two or more are a duplicate
// group, held with its paths sorted until writeGroups writes it. // group, held with its paths sorted until writeGroups writes it. Empty
// files form no group, since removing one frees nothing.
func (w *reportWriter) endGroup() { func (w *reportWriter) endGroup() {
if len(w.paths) < minGroupSize { if len(w.paths) < minGroupSize || w.sig.size == 0 {
// A path with no duplicate is dropped, and its slice reused. // The paths are dropped, and their slice reused.
w.paths = w.paths[:0] w.paths = w.paths[:0]
return return
+21
View File
@@ -434,6 +434,27 @@ func TestDupeGroups(t *testing.T) {
} }
} }
// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
// duplicate group, while 1-byte files still do.
func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
t.Parallel()
e, x := sha256.Sum256(nil), testHash("x")
recs := []scanRec{
{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
}
groups := dupeGroupsOf(t, recs)
if len(groups) != 1 || groups[0].size != 1 ||
!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
}
}
func TestDupeGroupsContentSeparates(t *testing.T) { func TestDupeGroupsContentSeparates(t *testing.T) {
t.Parallel() t.Parallel()
+6 -5
View File
@@ -1185,11 +1185,12 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
t.Helper() t.Helper()
groups := dupeGroups(t, db) groups := dupeGroups(t, db)
if len(groups) != 5 { if len(groups) != 4 {
t.Fatalf("len(groups) = %d, want 5", len(groups)) t.Fatalf("len(groups) = %d, want 4", len(groups))
} }
wantSizes := []int64{3000, 2000, 100, 1, 0} // The empty files empty1 and empty2 form no group.
wantSizes := []int64{3000, 2000, 100, 1}
for i, g := range groups { for i, g := range groups {
if g.size != wantSizes[i] { if g.size != wantSizes[i] {
t.Errorf("groups[%d].size = %d, want %d", t.Errorf("groups[%d].size = %d, want %d",
@@ -2367,8 +2368,8 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
report := runStdout(t, cmdReport) report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") || if strings.Contains(report, "db.sqlite") ||
strings.Contains(report, hardLink) || strings.Contains(report, hardLink) ||
!strings.Contains(report, filepath.Join(tree, "empty2")) { !strings.Contains(report, filepath.Join(tree, "b")) {
t.Errorf("report = %q, want the empty files and none of "+ t.Errorf("report = %q, want a and b and none of "+
"the database's files", report) "the database's files", report)
} }
}) })
+6 -2
View File
@@ -301,7 +301,9 @@ func fileEntry(name string, r scanRec) string {
// collectTreeGroups groups directories by digest, counting each one on // collectTreeGroups groups directories by digest, counting each one on
// prog as it is grouped, and returns every maximal group with two or // prog as it is grouped, and returns every maximal group with two or
// more members, each group's members sorted by path, groups ordered by // more members, each group's members sorted by path, groups ordered by
// tree size descending then by first path ascending. // tree size descending then by first path ascending. A group whose
// trees hold 0 bytes is left out, since removing one frees nothing. The
// groups it implies hold 0 bytes too, so leaving it out hides nothing.
func collectTreeGroups(allDirs []*treeNode, super *treeNode, func collectTreeGroups(allDirs []*treeNode, super *treeNode,
prog *progress, prog *progress,
) [][]*treeNode { ) [][]*treeNode {
@@ -315,7 +317,9 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
var dupes [][]*treeNode var dupes [][]*treeNode
for _, g := range groups { for _, g := range groups {
if len(g) < minGroupSize || suppressed(g, super) { // The members share a digest, so they share a total size.
if len(g) < minGroupSize || g[0].totalSize == 0 ||
suppressed(g, super) {
continue continue
} }
+30
View File
@@ -2,6 +2,7 @@ package sfdupes //nolint:testpackage // checks the unexported treeNode results
import ( import (
"bytes" "bytes"
"crypto/sha256"
"database/sql" "database/sql"
"os" "os"
"slices" "slices"
@@ -435,6 +436,35 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
} }
} }
// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
// holding only empty files form no group, and that an empty file still
// counts toward the digest of a tree that holds other files.
func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
t.Parallel()
h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
sha256.Sum256(nil)
recs := []scanRec{
{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
// Without the empty file, /p/x3 is no copy of /p/x1.
{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
}
super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/x1", "/p/x2"}}
if !slices.EqualFunc(got, want, slices.Equal) {
t.Fatalf("groups = %v, want %v", got, want)
}
}
func TestCollectTreeGroupsDifferingParents(t *testing.T) { func TestCollectTreeGroupsDifferingParents(t *testing.T) {
t.Parallel() t.Parallel()