Compare commits
2
Commits
main
...
6ded88234a
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6ded88234a | ||
|
|
6c529fcc6d |
No files matched your search
@@ -29,7 +29,6 @@ first look at a new machine with one command.
|
|||||||
|
|
||||||
```sh
|
```sh
|
||||||
make build
|
make build
|
||||||
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
|
||||||
./sfdupes scan /srv
|
./sfdupes scan /srv
|
||||||
./sfdupes report > dupes.tsv
|
./sfdupes report > dupes.tsv
|
||||||
./sfdupes trees > dupetrees.tsv
|
./sfdupes trees > dupetrees.tsv
|
||||||
@@ -45,8 +44,9 @@ that stands for a piece of hardware, such as a disk or a terminal. `report`
|
|||||||
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
||||||
`info` prints the database's path, size and record count.
|
`info` prints the database's path, size and record count.
|
||||||
|
|
||||||
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
Root keeps the database in `/var/lib/sfdupes/db.sqlite`. On Linux, any other
|
||||||
another file. The first `scan` creates it.
|
user keeps it in `~/.local/state/sfdupes/db.sqlite`. `SFDUPES_DATABASE` names
|
||||||
|
another file (see "Location"). The first `scan` creates the database.
|
||||||
|
|
||||||
`report`, `trees` and `info` write their data to stdout. Progress, warnings and
|
`report`, `trees` and `info` write their data to stdout. Progress, warnings and
|
||||||
errors go to stderr. stdout and stderr are a program's two output streams. The
|
errors go to stderr. stdout and stderr are a program's two output streams. The
|
||||||
@@ -146,8 +146,9 @@ walk"). As the file `/etc/cron.d/sfdupes`:
|
|||||||
- `report`, `trees` and `info` need only read access to the database. The umask
|
- `report`, `trees` and `info` need only read access to the database. The umask
|
||||||
is the setting that removes permissions from the files a process creates.
|
is the setting that removes permissions from the files a process creates.
|
||||||
Under the usual umask of `022`, the first scan creates the database readable
|
Under the usual umask of `022`, the first scan creates the database readable
|
||||||
by every user. An unprivileged user can then run those three commands against
|
by every user. Another user can then run those three commands against root's
|
||||||
root's database.
|
database, with `SFDUPES_DATABASE=/var/lib/sfdupes/db.sqlite` before the
|
||||||
|
command.
|
||||||
|
|
||||||
### Reading the reports
|
### Reading the reports
|
||||||
|
|
||||||
@@ -283,9 +284,9 @@ The reference below is also the program's specification.
|
|||||||
### Settings
|
### Settings
|
||||||
|
|
||||||
- `SFDUPES_DATABASE` names the database file. When it is unset or empty, the
|
- `SFDUPES_DATABASE` names the database file. When it is unset or empty, the
|
||||||
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
|
database is at its default path (see "Location"). There is no command-line
|
||||||
it. The path names the file exactly, whatever characters it holds. A relative
|
flag for it. The path names the file exactly, whatever characters it holds. A
|
||||||
path is relative to the working directory.
|
relative path is relative to the working directory.
|
||||||
- `--workers N` (`scan`) sets how many directories `scan` reads at once while it
|
- `--workers N` (`scan`) sets how many directories `scan` reads at once while it
|
||||||
lists the trees. It also sets how many files `scan` reads at once while it
|
lists the trees. It also sets how many files `scan` reads at once while it
|
||||||
hashes them. The default is the number of CPUs. `N` must be at least 1. A
|
hashes them. The default is the number of CPUs. `N` must be at least 1. A
|
||||||
@@ -665,6 +666,7 @@ Processing:
|
|||||||
- Records are grouped by signature, the four values
|
- Records are grouped by signature, the four values
|
||||||
`(size, head, tail, content)`.
|
`(size, head, tail, content)`.
|
||||||
- Every group with two or more paths is a duplicate group.
|
- Every group with two or more paths is a duplicate group.
|
||||||
|
- Empty files form no group, because removing one frees nothing.
|
||||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||||
every other path is a `dupe`.
|
every other path is a `dupe`.
|
||||||
- Groups are ordered by size, largest first, then by `first` path. The output is
|
- Groups are ordered by size, largest first, then by `first` path. The output is
|
||||||
@@ -713,7 +715,7 @@ The summary line on stderr gives:
|
|||||||
- the reclaimable bytes, the sum of `size` over all rows.
|
- the reclaimable bytes, the sum of `size` over all rows.
|
||||||
|
|
||||||
```
|
```
|
||||||
report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable
|
report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
||||||
```
|
```
|
||||||
|
|
||||||
### `trees` mode
|
### `trees` mode
|
||||||
@@ -741,6 +743,9 @@ Processing:
|
|||||||
- `trees` builds the directory tree and computes every directory's digest. It
|
- `trees` builds the directory tree and computes every directory's digest. It
|
||||||
then groups the directories by digest. Every group with two or more
|
then groups the directories by digest. Every group with two or more
|
||||||
directories is a duplicate-tree group.
|
directories is a duplicate-tree group.
|
||||||
|
- `trees` leaves out a group whose trees hold 0 bytes, because removing one
|
||||||
|
frees nothing. An empty file still counts toward the digest of a tree that
|
||||||
|
holds other files.
|
||||||
- `trees` leaves out a group that a group of its members' parents already
|
- `trees` leaves out a group that a group of its members' parents already
|
||||||
implies. A group is left out when its members' parents are distinct
|
implies. A group is left out when its members' parents are distinct
|
||||||
directories that all have the same digest. The parents' group, or an
|
directories that all have the same digest. The parents' group, or an
|
||||||
@@ -876,7 +881,15 @@ All five subcommands work on one SQLite database file.
|
|||||||
#### Location
|
#### Location
|
||||||
|
|
||||||
The database is the file named by `SFDUPES_DATABASE`, when that is set and not
|
The database is the file named by `SFDUPES_DATABASE`, when that is set and not
|
||||||
empty, and otherwise `/var/lib/sfdupes/db.sqlite`.
|
empty. Otherwise it is at its default path, which depends on the user:
|
||||||
|
|
||||||
|
- For root, it is `/var/lib/sfdupes/db.sqlite`.
|
||||||
|
- For any other user, it is `sfdupes/db.sqlite` in the XDG state directory. That
|
||||||
|
directory is `$XDG_STATE_HOME` when that is set to an absolute path. Otherwise
|
||||||
|
it is `~/.local/state` on Linux and `~/Library/Application Support` on macOS.
|
||||||
|
|
||||||
|
All five subcommands use the same default, so a user's `report` reads the
|
||||||
|
database that user's `scan` wrote.
|
||||||
|
|
||||||
Each subcommand prints one line on stderr naming the database it uses, before
|
Each subcommand prints one line on stderr naming the database it uses, before
|
||||||
anything else and whether or not stderr is a terminal. The line gives the path
|
anything else and whether or not stderr is a terminal. The line gives the path
|
||||||
@@ -916,9 +929,9 @@ Only one `scan` at a time can hold the lock.
|
|||||||
- For its whole run, `scan` holds a lock on a lock file beside the database. The
|
- For its whole run, `scan` holds a lock on a lock file beside the database. The
|
||||||
lock is taken with the `flock(2)` system call, a request a program makes to
|
lock is taken with the `flock(2)` system call, a request a program makes to
|
||||||
the kernel, and only one process can hold it. The lock file's name is the
|
the kernel, and only one process can hold it. The lock file's name is the
|
||||||
database's path, with symlinks resolved, plus `.lock`. By default that is
|
database's path, with symlinks resolved, plus `.lock`. For root, by default,
|
||||||
`/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same
|
that is `/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the
|
||||||
lock file.
|
same lock file.
|
||||||
- `scan` takes the lock before it walks the filesystem or opens the database.
|
- `scan` takes the lock before it walks the filesystem or opens the database.
|
||||||
Just before that, it creates the database file, empty, if it is missing. So
|
Just before that, it creates the database file, empty, if it is missing. So
|
||||||
the lock file is named the same way before the first scan as after. A symlink
|
the lock file is named the same way before the first scan as after. A symlink
|
||||||
@@ -1382,7 +1395,7 @@ All of the following, run in this directory, must pass:
|
|||||||
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
||||||
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
||||||
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
||||||
: > "$d/empty1"; : > "$d/empty2" # empty duplicates
|
: > "$d/empty1"; : > "$d/empty2" # empty files
|
||||||
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
||||||
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
||||||
head -c 3000 /dev/urandom > "$d/t1/f1"
|
head -c 3000 /dev/urandom > "$d/t1/f1"
|
||||||
@@ -1407,8 +1420,8 @@ All of the following, run in this directory, must pass:
|
|||||||
is the path that sorts first.
|
is the path that sorts first.
|
||||||
- `t1/f1`, `t2/f1` and `t3/f1` form one group.
|
- `t1/f1`, `t2/f1` and `t3/f1` form one group.
|
||||||
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
|
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
|
||||||
- `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`.
|
- `tiny1` and `tiny2` form a pair.
|
||||||
- `unique.bin` and `tiny3` appear nowhere.
|
- `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
|
||||||
- The groups come in order of size, largest first.
|
- The groups come in order of size, largest first.
|
||||||
|
|
||||||
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
|
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
|
||||||
|
|||||||
@@ -29,6 +29,15 @@
|
|||||||
Newest first. An entry ends with its date and its issue or branch, where those
|
Newest first. An entry ends with its date and its issue or branch, where those
|
||||||
are known.
|
are known.
|
||||||
|
|
||||||
|
- Without `SFDUPES_DATABASE`, a user other than root keeps the database in
|
||||||
|
`sfdupes/db.sqlite` under the XDG state directory, `~/.local/state` by
|
||||||
|
default. Root keeps `/var/lib/sfdupes/db.sqlite`. `sfdupes --help` says where
|
||||||
|
the database is (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/171)
|
||||||
|
|
||||||
|
- `report` and `trees` leave out empty files. `report` writes no group of size
|
||||||
|
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
|
||||||
|
counts toward a tree's digest (2026-11-11,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/178)
|
||||||
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
|
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
|
||||||
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
|
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
|
||||||
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
|
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
|
||||||
|
|||||||
+4
-2
@@ -5,8 +5,10 @@
|
|||||||
// KiB, and of the content. The content hash covers the whole file under
|
// KiB, and of the content. The content hash covers the whole file under
|
||||||
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
||||||
// file is never read in full. scan maintains a persistent SQLite
|
// file is never read in full. scan maintains a persistent SQLite
|
||||||
// database of file signatures (SFDUPES_DATABASE, default
|
// database of file signatures that the other subcommands read. The
|
||||||
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
|
// database is SFDUPES_DATABASE, or by default /var/lib/sfdupes/db.sqlite
|
||||||
|
// for root and sfdupes/db.sqlite in the XDG state directory for any
|
||||||
|
// other user.
|
||||||
//
|
//
|
||||||
// Usage:
|
// Usage:
|
||||||
//
|
//
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ module sneak.berlin/go/sfdupes
|
|||||||
go 1.25.7
|
go 1.25.7
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
github.com/adrg/xdg v0.5.3
|
||||||
github.com/schollz/progressbar/v3 v3.19.1
|
github.com/schollz/progressbar/v3 v3.19.1
|
||||||
github.com/spf13/cobra v1.10.2
|
github.com/spf13/cobra v1.10.2
|
||||||
golang.org/x/sys v0.46.0
|
golang.org/x/sys v0.46.0
|
||||||
|
|||||||
@@ -1,3 +1,5 @@
|
|||||||
|
github.com/adrg/xdg v0.5.3 h1:xRnxJXne7+oWDatRhR1JLnvuccuIeCoBu2rtuLqQB78=
|
||||||
|
github.com/adrg/xdg v0.5.3/go.mod h1:nlTsY+NNiCBGCK2tpm09vRqfVzrc2fLmXGpBLF0zlTQ=
|
||||||
github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM=
|
github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM=
|
||||||
github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY=
|
github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY=
|
||||||
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
||||||
|
|||||||
+11
-1
@@ -116,6 +116,13 @@ func run(args []string, stdout, stderr io.Writer) int {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// databaseHelp says in sfdupes --help where the database is.
|
||||||
|
const databaseHelp = `The database is the file SFDUPES_DATABASE names.
|
||||||
|
When that is unset or empty, root uses /var/lib/sfdupes/db.sqlite. Any
|
||||||
|
other user uses sfdupes/db.sqlite in the XDG state directory:
|
||||||
|
$XDG_STATE_HOME, or ~/.local/state when that is unset
|
||||||
|
(~/Library/Application Support on macOS).`
|
||||||
|
|
||||||
// newRootCommand builds the command tree. Everything on stdout is
|
// newRootCommand builds the command tree. Everything on stdout is
|
||||||
// machine-readable data, the version line included; all human-facing
|
// machine-readable data, the version line included; all human-facing
|
||||||
// output (help, usage, errors) goes to stderr.
|
// output (help, usage, errors) goes to stderr.
|
||||||
@@ -134,9 +141,12 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
|||||||
return nil
|
return nil
|
||||||
})
|
})
|
||||||
|
|
||||||
|
short := "Find candidate duplicate files by size and head/tail/content SHA-256"
|
||||||
|
|
||||||
root := &cobra.Command{
|
root := &cobra.Command{
|
||||||
Use: "sfdupes",
|
Use: "sfdupes",
|
||||||
Short: "Find candidate duplicate files by size and head/tail/content SHA-256",
|
Short: short,
|
||||||
|
Long: short + ".\n\n" + databaseHelp,
|
||||||
Args: cobra.NoArgs,
|
Args: cobra.NoArgs,
|
||||||
RunE: func(cmd *cobra.Command, args []string) error {
|
RunE: func(cmd *cobra.Command, args []string) error {
|
||||||
if showVersion {
|
if showVersion {
|
||||||
|
|||||||
@@ -15,6 +15,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/adrg/xdg"
|
||||||
"golang.org/x/sys/unix"
|
"golang.org/x/sys/unix"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -553,13 +554,14 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
|||||||
// line.
|
// line.
|
||||||
t.Setenv(databaseEnv, "")
|
t.Setenv(databaseEnv, "")
|
||||||
|
|
||||||
// Whether scan calls the database new depends on that machine too.
|
dbPath := defaultDatabasePath(os.Geteuid())
|
||||||
scanWant := "scan: database " + defaultDatabasePath + " (default)\n"
|
|
||||||
|
|
||||||
_, err := os.Stat(defaultDatabasePath)
|
// Whether scan calls the database new depends on that machine too.
|
||||||
|
scanWant := "scan: database " + dbPath + " (default)\n"
|
||||||
|
|
||||||
|
_, err := os.Stat(dbPath)
|
||||||
if errors.Is(err, fs.ErrNotExist) {
|
if errors.Is(err, fs.ErrNotExist) {
|
||||||
scanWant = "scan: new database " + defaultDatabasePath +
|
scanWant = "scan: new database " + dbPath + " (default)\n"
|
||||||
" (default)\n"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
missing := filepath.Join(t.TempDir(), "nope")
|
missing := filepath.Join(t.TempDir(), "nope")
|
||||||
@@ -578,21 +580,21 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
|||||||
run: func() error {
|
run: func() error {
|
||||||
return runReport(cancelledContext(t), io.Discard)
|
return runReport(cancelledContext(t), io.Discard)
|
||||||
},
|
},
|
||||||
want: "report: database " + defaultDatabasePath + " (default)\n" +
|
want: "report: database " + dbPath + " (default)\n" +
|
||||||
reportOpenLine + "\n" + reportOpenLine + "\n",
|
reportOpenLine + "\n" + reportOpenLine + "\n",
|
||||||
},
|
},
|
||||||
cmdTrees: {
|
cmdTrees: {
|
||||||
run: func() error {
|
run: func() error {
|
||||||
return runTrees(cancelledContext(t), io.Discard)
|
return runTrees(cancelledContext(t), io.Discard)
|
||||||
},
|
},
|
||||||
want: "trees: database " + defaultDatabasePath + " (default)\n" +
|
want: "trees: database " + dbPath + " (default)\n" +
|
||||||
treesOpenLine + "\n" + treesOpenLine + "\n",
|
treesOpenLine + "\n" + treesOpenLine + "\n",
|
||||||
},
|
},
|
||||||
cmdInfo: {
|
cmdInfo: {
|
||||||
run: func() error {
|
run: func() error {
|
||||||
return runInfo(cancelledContext(t), io.Discard)
|
return runInfo(cancelledContext(t), io.Discard)
|
||||||
},
|
},
|
||||||
want: "info: database " + defaultDatabasePath + " (default)\n" +
|
want: "info: database " + dbPath + " (default)\n" +
|
||||||
infoOpenLine + "\n" + infoOpenLine + "\n",
|
infoOpenLine + "\n" + infoOpenLine + "\n",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -711,6 +713,20 @@ func TestRunHelp(t *testing.T) {
|
|||||||
t.Errorf("run(%v) stdout = %q, want nothing (data only)",
|
t.Errorf("run(%v) stdout = %q, want nothing (data only)",
|
||||||
args, got)
|
args, got)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if args[0] == cmdScan {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// README §Location: sfdupes --help says where the database is.
|
||||||
|
for _, want := range []string{
|
||||||
|
databaseEnv, rootDatabasePath, "XDG_STATE_HOME", "~/.local/state",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(stderr.String(), want) {
|
||||||
|
t.Errorf("run(%v) stderr = %q, want it to name %q",
|
||||||
|
args, stderr.String(), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1496,6 +1512,24 @@ func TestRunDatabasePathUsedAsGiven(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunNonRootUsesStateDirectory(t *testing.T) {
|
||||||
|
// README §Location: without SFDUPES_DATABASE, a user other than
|
||||||
|
// root keeps the database in the XDG state directory. scan creates
|
||||||
|
// the missing directories, and report and trees read the same file.
|
||||||
|
if os.Geteuid() == 0 {
|
||||||
|
t.Skip("root uses " + rootDatabasePath)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Cleanup(xdg.Reload)
|
||||||
|
|
||||||
|
state := t.TempDir()
|
||||||
|
t.Setenv("XDG_STATE_HOME", state)
|
||||||
|
t.Setenv(databaseEnv, "")
|
||||||
|
xdg.Reload()
|
||||||
|
|
||||||
|
assertRunsUseDatabase(t, filepath.Join(state, "sfdupes"), "db.sqlite")
|
||||||
|
}
|
||||||
|
|
||||||
// holdScanLock takes the lock on the database at path, as a running
|
// holdScanLock takes the lock on the database at path, as a running
|
||||||
// scan does, and holds it until the test ends. It fails the test when
|
// scan does, and holds it until the test ends. It fails the test when
|
||||||
// the lock is already held.
|
// the lock is already held.
|
||||||
|
|||||||
+20
-5
@@ -15,15 +15,16 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/adrg/xdg"
|
||||||
"golang.org/x/sys/unix"
|
"golang.org/x/sys/unix"
|
||||||
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
|
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
|
||||||
// disabled.
|
// disabled.
|
||||||
_ "modernc.org/sqlite"
|
_ "modernc.org/sqlite"
|
||||||
)
|
)
|
||||||
|
|
||||||
// defaultDatabasePath is where the persistent scan database lives when
|
// rootDatabasePath is the database root uses when SFDUPES_DATABASE is
|
||||||
// SFDUPES_DATABASE is not set.
|
// not set.
|
||||||
const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
|
const rootDatabasePath = "/var/lib/sfdupes/db.sqlite"
|
||||||
|
|
||||||
// databaseEnv is the environment variable that overrides the database
|
// databaseEnv is the environment variable that overrides the database
|
||||||
// path.
|
// path.
|
||||||
@@ -144,13 +145,27 @@ var errScanRunning = errors.New("another scan is running")
|
|||||||
// 32 bytes of a SHA-256.
|
// 32 bytes of a SHA-256.
|
||||||
var errHashSize = errors.New("hash is not 32 bytes")
|
var errHashSize = errors.New("hash is not 32 bytes")
|
||||||
|
|
||||||
|
// defaultDatabasePath is the database used when SFDUPES_DATABASE is not
|
||||||
|
// set: rootDatabasePath for root (euid 0), and db.sqlite in the
|
||||||
|
// directory sfdupes under the XDG state directory for any other user.
|
||||||
|
// xdg reads XDG_STATE_HOME and HOME once, when the program starts. The
|
||||||
|
// path is joined as text, not cleaned, for the reason absolutePath
|
||||||
|
// gives.
|
||||||
|
func defaultDatabasePath(euid int) string {
|
||||||
|
if euid == 0 {
|
||||||
|
return rootDatabasePath
|
||||||
|
}
|
||||||
|
|
||||||
|
return strings.TrimSuffix(xdg.StateHome, "/") + "/sfdupes/db.sqlite"
|
||||||
|
}
|
||||||
|
|
||||||
// databasePath resolves the database location: SFDUPES_DATABASE when
|
// databasePath resolves the database location: SFDUPES_DATABASE when
|
||||||
// set and non-empty, the compiled-in default otherwise. It returns the
|
// set and non-empty, defaultDatabasePath otherwise. It returns the
|
||||||
// path, made absolute by absolutePath, and where the path came from,
|
// path, made absolute by absolutePath, and where the path came from,
|
||||||
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
||||||
// prints first on stderr.
|
// prints first on stderr.
|
||||||
func databasePath() (string, string, error) {
|
func databasePath() (string, string, error) {
|
||||||
path, source := defaultDatabasePath, "default"
|
path, source := defaultDatabasePath(os.Geteuid()), "default"
|
||||||
if p := os.Getenv(databaseEnv); p != "" {
|
if p := os.Getenv(databaseEnv); p != "" {
|
||||||
path, source = p, "from "+databaseEnv
|
path, source = p, "from "+databaseEnv
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,10 +9,13 @@ import (
|
|||||||
"io/fs"
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/adrg/xdg"
|
||||||
)
|
)
|
||||||
|
|
||||||
// testDBPath returns a database path inside a fresh temp dir, with
|
// testDBPath returns a database path inside a fresh temp dir, with
|
||||||
@@ -60,7 +63,7 @@ func TestDatabasePath(t *testing.T) {
|
|||||||
cases := []struct {
|
cases := []struct {
|
||||||
env, path, source string
|
env, path, source string
|
||||||
}{
|
}{
|
||||||
{"", defaultDatabasePath, "default"},
|
{"", defaultDatabasePath(os.Geteuid()), "default"},
|
||||||
{"/custom/place.sqlite", "/custom/place.sqlite", fromEnv},
|
{"/custom/place.sqlite", "/custom/place.sqlite", fromEnv},
|
||||||
{"/custom/link/../place.sqlite", "/custom/link/../place.sqlite",
|
{"/custom/link/../place.sqlite", "/custom/link/../place.sqlite",
|
||||||
fromEnv},
|
fromEnv},
|
||||||
@@ -89,6 +92,41 @@ func TestDatabasePath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestDefaultDatabasePath(t *testing.T) {
|
||||||
|
// README §Location: root's database is /var/lib/sfdupes/db.sqlite.
|
||||||
|
// Any other user's is db.sqlite in sfdupes under XDG_STATE_HOME, or
|
||||||
|
// under ~/.local/state when XDG_STATE_HOME is unset or not absolute.
|
||||||
|
if runtime.GOOS != goosLinux {
|
||||||
|
t.Skip("macOS has another default state directory")
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Cleanup(xdg.Reload)
|
||||||
|
t.Setenv("HOME", "/home/someone")
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
euid int
|
||||||
|
stateHome string
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{0, "", rootDatabasePath},
|
||||||
|
{0, "/elsewhere/state", rootDatabasePath},
|
||||||
|
{1000, "", "/home/someone/.local/state/sfdupes/db.sqlite"},
|
||||||
|
{1000, "relative/state", "/home/someone/.local/state/sfdupes/db.sqlite"},
|
||||||
|
{1000, "/elsewhere/state", "/elsewhere/state/sfdupes/db.sqlite"},
|
||||||
|
{1000, "/elsewhere/state/", "/elsewhere/state/sfdupes/db.sqlite"},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Setenv("XDG_STATE_HOME", tc.stateHome)
|
||||||
|
xdg.Reload()
|
||||||
|
|
||||||
|
if got := defaultDatabasePath(tc.euid); got != tc.want {
|
||||||
|
t.Errorf("defaultDatabasePath(%d) with XDG_STATE_HOME=%q = %q, want %q",
|
||||||
|
tc.euid, tc.stateHome, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestOpenScanDatabaseCreates(t *testing.T) {
|
func TestOpenScanDatabaseCreates(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -195,10 +195,11 @@ func (w *reportWriter) add(r scanRec) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// endGroup ends the paths read with w.sig: two or more are a duplicate
|
// endGroup ends the paths read with w.sig: two or more are a duplicate
|
||||||
// group, held with its paths sorted until writeGroups writes it.
|
// group, held with its paths sorted until writeGroups writes it. Empty
|
||||||
|
// files form no group, since removing one frees nothing.
|
||||||
func (w *reportWriter) endGroup() {
|
func (w *reportWriter) endGroup() {
|
||||||
if len(w.paths) < minGroupSize {
|
if len(w.paths) < minGroupSize || w.sig.size == 0 {
|
||||||
// A path with no duplicate is dropped, and its slice reused.
|
// The paths are dropped, and their slice reused.
|
||||||
w.paths = w.paths[:0]
|
w.paths = w.paths[:0]
|
||||||
|
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -434,6 +434,27 @@ func TestDupeGroups(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
|
||||||
|
// duplicate group, while 1-byte files still do.
|
||||||
|
func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
e, x := sha256.Sum256(nil), testHash("x")
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
|
||||||
|
{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
|
||||||
|
{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
|
||||||
|
{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
|
||||||
|
{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
|
||||||
|
}
|
||||||
|
|
||||||
|
groups := dupeGroupsOf(t, recs)
|
||||||
|
if len(groups) != 1 || groups[0].size != 1 ||
|
||||||
|
!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
|
||||||
|
t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestDupeGroupsContentSeparates(t *testing.T) {
|
func TestDupeGroupsContentSeparates(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -1185,11 +1185,12 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
groups := dupeGroups(t, db)
|
groups := dupeGroups(t, db)
|
||||||
if len(groups) != 5 {
|
if len(groups) != 4 {
|
||||||
t.Fatalf("len(groups) = %d, want 5", len(groups))
|
t.Fatalf("len(groups) = %d, want 4", len(groups))
|
||||||
}
|
}
|
||||||
|
|
||||||
wantSizes := []int64{3000, 2000, 100, 1, 0}
|
// The empty files empty1 and empty2 form no group.
|
||||||
|
wantSizes := []int64{3000, 2000, 100, 1}
|
||||||
for i, g := range groups {
|
for i, g := range groups {
|
||||||
if g.size != wantSizes[i] {
|
if g.size != wantSizes[i] {
|
||||||
t.Errorf("groups[%d].size = %d, want %d",
|
t.Errorf("groups[%d].size = %d, want %d",
|
||||||
@@ -2367,8 +2368,8 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
|||||||
report := runStdout(t, cmdReport)
|
report := runStdout(t, cmdReport)
|
||||||
if strings.Contains(report, "db.sqlite") ||
|
if strings.Contains(report, "db.sqlite") ||
|
||||||
strings.Contains(report, hardLink) ||
|
strings.Contains(report, hardLink) ||
|
||||||
!strings.Contains(report, filepath.Join(tree, "empty2")) {
|
!strings.Contains(report, filepath.Join(tree, "b")) {
|
||||||
t.Errorf("report = %q, want the empty files and none of "+
|
t.Errorf("report = %q, want a and b and none of "+
|
||||||
"the database's files", report)
|
"the database's files", report)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -301,7 +301,9 @@ func fileEntry(name string, r scanRec) string {
|
|||||||
// collectTreeGroups groups directories by digest, counting each one on
|
// collectTreeGroups groups directories by digest, counting each one on
|
||||||
// prog as it is grouped, and returns every maximal group with two or
|
// prog as it is grouped, and returns every maximal group with two or
|
||||||
// more members, each group's members sorted by path, groups ordered by
|
// more members, each group's members sorted by path, groups ordered by
|
||||||
// tree size descending then by first path ascending.
|
// tree size descending then by first path ascending. A group whose
|
||||||
|
// trees hold 0 bytes is left out, since removing one frees nothing. The
|
||||||
|
// groups it implies hold 0 bytes too, so leaving it out hides nothing.
|
||||||
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||||
prog *progress,
|
prog *progress,
|
||||||
) [][]*treeNode {
|
) [][]*treeNode {
|
||||||
@@ -315,7 +317,9 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
|||||||
var dupes [][]*treeNode
|
var dupes [][]*treeNode
|
||||||
|
|
||||||
for _, g := range groups {
|
for _, g := range groups {
|
||||||
if len(g) < minGroupSize || suppressed(g, super) {
|
// The members share a digest, so they share a total size.
|
||||||
|
if len(g) < minGroupSize || g[0].totalSize == 0 ||
|
||||||
|
suppressed(g, super) {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package sfdupes //nolint:testpackage // checks the unexported treeNode results
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
|
"crypto/sha256"
|
||||||
"database/sql"
|
"database/sql"
|
||||||
"os"
|
"os"
|
||||||
"slices"
|
"slices"
|
||||||
@@ -435,6 +436,35 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
|
||||||
|
// holding only empty files form no group, and that an empty file still
|
||||||
|
// counts toward the digest of a tree that holds other files.
|
||||||
|
func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
|
||||||
|
sha256.Sum256(nil)
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
|
||||||
|
{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
|
||||||
|
{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
|
||||||
|
{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
|
||||||
|
{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
|
||||||
|
{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
|
||||||
|
// Without the empty file, /p/x3 is no copy of /p/x1.
|
||||||
|
{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
|
||||||
|
}
|
||||||
|
|
||||||
|
super, dirs := treeOf(t, recs)
|
||||||
|
|
||||||
|
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||||
|
|
||||||
|
want := [][]string{{"/p/x1", "/p/x2"}}
|
||||||
|
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||||
|
t.Fatalf("groups = %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user