Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cb469a2210 |
No files matched your search
@@ -99,15 +99,13 @@ Each `-v` flag below makes a bind mount, which shows a host directory at a path
|
|||||||
inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between
|
inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between
|
||||||
runs. Mount the tree to scan at the same path inside the container as on the
|
runs. Mount the tree to scan at the same path inside the container as on the
|
||||||
host. Read-only access is enough. The database then records the host's paths,
|
host. Read-only access is enough. The database then records the host's paths,
|
||||||
and so do the reports. `report` and `trees` need that mount too. Without it,
|
and so do the reports.
|
||||||
every recorded path counts as gone, and each report holds only its header line.
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
make docker
|
make docker
|
||||||
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
||||||
sfdupes scan /srv
|
sfdupes scan /srv
|
||||||
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
|
||||||
sfdupes report > dupes.tsv
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Daily scan from `cron`
|
### Daily scan from `cron`
|
||||||
@@ -213,9 +211,9 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
|
|||||||
of reads, at the price of a comparison that is not exact.
|
of reads, at the price of a comparison that is not exact.
|
||||||
|
|
||||||
The hashes live in a database that is kept between runs. A rescan reads only new
|
The hashes live in a database that is kept between runs. A rescan reads only new
|
||||||
files, and files whose size or modification time (mtime) differs from their
|
files, and files whose size changed or whose modification time (mtime) is newer
|
||||||
record. All analysis then works from the database. The reports only check that
|
than recorded. All analysis then works from the database. The reports only check
|
||||||
each copy they name still exists.
|
that each copy they name still exists.
|
||||||
|
|
||||||
The goal is whole duplicated trees, not only single files. A download extracted
|
The goal is whole duplicated trees, not only single files. A download extracted
|
||||||
twice, or a project copied once too often, is such a tree. It can be removed as
|
twice, or a project copied once too often, is such a tree. It can be removed as
|
||||||
@@ -417,16 +415,17 @@ operands:
|
|||||||
|
|
||||||
- A file not yet in the database is hashed and inserted.
|
- A file not yet in the database is hashed and inserted.
|
||||||
- A file already in the database is not read again when its size and mtime match
|
- A file already in the database is not read again when its size and mtime match
|
||||||
its record. This is what makes a daily rescan cheap.
|
its record. The mtime matches when it is not newer than the recorded one. This
|
||||||
|
is what makes a daily rescan cheap.
|
||||||
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
|
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
|
||||||
updates `disk_size` alone. It still does not read the file, and counts it as
|
updates `disk_size` alone. It still does not read the file, and counts it as
|
||||||
unchanged. ZFS, for example, counts a new file's blocks up to seconds after
|
unchanged. ZFS, for example, counts a new file's blocks up to seconds after
|
||||||
the write. A block is the unit in which a filesystem allocates disk space. A
|
the write. A block is the unit in which a filesystem allocates disk space. A
|
||||||
file scanned within those seconds gets its real `disk_size` at the next scan.
|
file scanned within those seconds gets its real `disk_size` at the next scan.
|
||||||
- `scan` hashes a file again when its size or its mtime differs from its record.
|
- `scan` hashes a file again when its size differs or its mtime is newer. The
|
||||||
An older mtime counts as much as a newer one. A file put back by `cp -p`,
|
mtime is compared to the nanosecond, as finely as the filesystem records it. A
|
||||||
`rsync -a` or a restore from backup often has an older mtime. The mtime is
|
same-size rewrite counts as a change whenever the filesystem gives it a later
|
||||||
compared to the nanosecond, as finely as the filesystem records it.
|
mtime, even within the same second.
|
||||||
- The scan deletes each record under a scanned operand whose file it could not
|
- The scan deletes each record under a scanned operand whose file it could not
|
||||||
verify. That covers deleted files, and paths the scan could not stat or read
|
verify. That covers deleted files, and paths the scan could not stat or read
|
||||||
this run. To stat a file is to read its details, such as its size, without
|
this run. To stat a file is to read its details, such as its size, without
|
||||||
@@ -666,7 +665,7 @@ each path in a group still exists, and looks at no other path. The check is
|
|||||||
`lstat`, the system call that looks up a path without following a symlink:
|
`lstat`, the system call that looks up a path without following a symlink:
|
||||||
|
|
||||||
- A path is gone when `lstat` finds that it, or a directory above it, does not
|
- A path is gone when `lstat` finds that it, or a directory above it, does not
|
||||||
exist.
|
exist. A path on a filesystem that is not mounted is gone too.
|
||||||
- Any other error, such as a permission error, counts as present. So a user who
|
- Any other error, such as a permission error, counts as present. So a user who
|
||||||
may not look inside a directory still sees its copies.
|
may not look inside a directory still sees its copies.
|
||||||
|
|
||||||
@@ -679,7 +678,6 @@ Processing:
|
|||||||
`(size, head, tail, content)`.
|
`(size, head, tail, content)`.
|
||||||
- In a group with two or more paths, a path that is gone is left out. A group
|
- In a group with two or more paths, a path that is gone is left out. A group
|
||||||
that still has two or more paths is a duplicate group.
|
that still has two or more paths is a duplicate group.
|
||||||
- Empty files form no group, because removing one frees nothing.
|
|
||||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||||
every other path is a `dupe`.
|
every other path is a `dupe`.
|
||||||
- Groups are ordered by size, largest first, then by `first` path. The output is
|
- Groups are ordered by size, largest first, then by `first` path. The output is
|
||||||
@@ -728,7 +726,7 @@ The summary line on stderr gives:
|
|||||||
- the reclaimable bytes, the sum of `size` over all rows.
|
- the reclaimable bytes, the sum of `size` over all rows.
|
||||||
|
|
||||||
```
|
```
|
||||||
report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable
|
||||||
```
|
```
|
||||||
|
|
||||||
### `trees` mode
|
### `trees` mode
|
||||||
@@ -739,6 +737,7 @@ when they hold the same relative paths, and each path holds a file with the same
|
|||||||
signature in both. It rebuilds the directory tree from the paths in the records,
|
signature in both. It rebuilds the directory tree from the paths in the records,
|
||||||
split on `/`. Like `report`, it never opens a scanned file. It checks only that
|
split on `/`. Like `report`, it never opens a scanned file. It checks only that
|
||||||
the directory at the top of each copy still exists, as `report` checks a path.
|
the directory at the top of each copy still exists, as `report` checks a path.
|
||||||
|
It looks at no file or directory inside a copy.
|
||||||
|
|
||||||
Definitions:
|
Definitions:
|
||||||
|
|
||||||
@@ -757,16 +756,13 @@ Processing:
|
|||||||
- `trees` builds the directory tree and computes every directory's digest. It
|
- `trees` builds the directory tree and computes every directory's digest. It
|
||||||
then groups the directories by digest. Every group with two or more
|
then groups the directories by digest. Every group with two or more
|
||||||
directories is a duplicate-tree group.
|
directories is a duplicate-tree group.
|
||||||
- `trees` leaves out a group whose trees hold 0 bytes, because removing one
|
|
||||||
frees nothing. An empty file still counts toward the digest of a tree that
|
|
||||||
holds other files.
|
|
||||||
- `trees` leaves out a group that a group of its members' parents already
|
- `trees` leaves out a group that a group of its members' parents already
|
||||||
implies. A group is left out when its members' parents are distinct
|
implies. A group is left out when its members' parents are distinct
|
||||||
directories that all have the same digest. The parents' group, or an
|
directories that all have the same digest. The parents' group, or an
|
||||||
ancestor's, already implies it. A group is always reported when it holds
|
ancestor's, already implies it. A group is always reported when it holds
|
||||||
sibling directories, or when its members' parents have different digests.
|
sibling directories, or when its members' parents have different digests.
|
||||||
- A directory that is gone is dropped from its group. A group left with fewer
|
- In each group that is not left out, a directory that is gone is left out. A
|
||||||
than two directories is not reported.
|
group that still has two or more directories is reported.
|
||||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||||
every other path is a `dupe`.
|
every other path is a `dupe`.
|
||||||
- Groups are ordered by total tree size, largest first, then by `first` path.
|
- Groups are ordered by total tree size, largest first, then by `first` path.
|
||||||
@@ -1406,7 +1402,7 @@ All of the following, run in this directory, must pass:
|
|||||||
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
||||||
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
||||||
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
||||||
: > "$d/empty1"; : > "$d/empty2" # empty files
|
: > "$d/empty1"; : > "$d/empty2" # empty duplicates
|
||||||
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
||||||
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
||||||
head -c 3000 /dev/urandom > "$d/t1/f1"
|
head -c 3000 /dev/urandom > "$d/t1/f1"
|
||||||
@@ -1431,8 +1427,8 @@ All of the following, run in this directory, must pass:
|
|||||||
is the path that sorts first.
|
is the path that sorts first.
|
||||||
- `t1/f1`, `t2/f1` and `t3/f1` form one group.
|
- `t1/f1`, `t2/f1` and `t3/f1` form one group.
|
||||||
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
|
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
|
||||||
- `tiny1` and `tiny2` form a pair.
|
- `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`.
|
||||||
- `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
|
- `unique.bin` and `tiny3` appear nowhere.
|
||||||
- The groups come in order of size, largest first.
|
- The groups come in order of size, largest first.
|
||||||
|
|
||||||
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
|
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
|
||||||
|
|||||||
@@ -34,13 +34,7 @@ are known.
|
|||||||
top of a tree. A group left with one copy is not reported. A missing `scan`
|
top of a tree. A group left with one copy is not reported. A missing `scan`
|
||||||
operand is still a fatal error (2026-11-11,
|
operand is still a fatal error (2026-11-11,
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/173)
|
https://git.eeqj.de/sneak/sfdupes/issues/173)
|
||||||
- The usage line of `scan` no longer ends in `[flags]`. It is the line
|
|
||||||
`README.md` shows under "Subcommands" (2026-11-11,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/169)
|
|
||||||
- `report` and `trees` leave out empty files. `report` writes no group of size
|
|
||||||
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
|
|
||||||
counts toward a tree's digest (2026-11-11,
|
|
||||||
https://git.eeqj.de/sneak/sfdupes/issues/178)
|
|
||||||
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
|
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
|
||||||
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
|
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
|
||||||
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
|
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
|
||||||
|
|||||||
@@ -215,13 +215,10 @@ func newScanCommand() *cobra.Command {
|
|||||||
short := "Walk trees and synchronize the scan database"
|
short := "Walk trees and synchronize the scan database"
|
||||||
|
|
||||||
scanCmd := &cobra.Command{
|
scanCmd := &cobra.Command{
|
||||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
||||||
// The Use line already names every flag, so cobra must not
|
Short: short,
|
||||||
// append its generic "[flags]" to it.
|
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
||||||
DisableFlagsInUseLine: true,
|
Args: cobra.MinimumNArgs(1),
|
||||||
Short: short,
|
|
||||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
|
||||||
Args: cobra.MinimumNArgs(1),
|
|
||||||
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
||||||
return checkScanWorkers(cmd, scanWorkers)
|
return checkScanWorkers(cmd, scanWorkers)
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -714,34 +714,6 @@ func TestRunHelp(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunScanUsageLine(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
// README §Subcommands shows this usage line for scan, and the help
|
|
||||||
// and the usage error both print it.
|
|
||||||
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...\n"
|
|
||||||
|
|
||||||
cases := map[string][]string{
|
|
||||||
"help": {cmdScan, "-h"},
|
|
||||||
"missing paths": {cmdScan},
|
|
||||||
}
|
|
||||||
|
|
||||||
for name, args := range cases {
|
|
||||||
t.Run(name, func(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
var stdout, stderr bytes.Buffer
|
|
||||||
|
|
||||||
run(args, &stdout, &stderr)
|
|
||||||
|
|
||||||
if !strings.Contains(stderr.String(), want) {
|
|
||||||
t.Errorf("run(%v) stderr = %q, want the usage line %q",
|
|
||||||
args, stderr.String(), want)
|
|
||||||
}
|
|
||||||
})
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestRunVersion(t *testing.T) {
|
func TestRunVersion(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -205,18 +205,17 @@ func (w *reportWriter) add(r scanRec) error {
|
|||||||
|
|
||||||
// endGroup ends the paths read with w.sig: two or more that are still
|
// endGroup ends the paths read with w.sig: two or more that are still
|
||||||
// on disk are a duplicate group, held with its paths sorted until
|
// on disk are a duplicate group, held with its paths sorted until
|
||||||
// writeGroups writes it. Empty files form no group, since removing one
|
// writeGroups writes it. Only the paths of a possible group are
|
||||||
// frees nothing. Only the paths of a possible group are checked, never
|
// checked, never a path with no duplicate.
|
||||||
// a path with no duplicate or an empty file.
|
|
||||||
func (w *reportWriter) endGroup() {
|
func (w *reportWriter) endGroup() {
|
||||||
if len(w.paths) >= minGroupSize && w.sig.size > 0 {
|
if len(w.paths) >= minGroupSize {
|
||||||
w.paths = slices.DeleteFunc(w.paths, func(p string) bool {
|
w.paths = slices.DeleteFunc(w.paths, func(p string) bool {
|
||||||
return !w.exists(p)
|
return !w.exists(p)
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
if len(w.paths) < minGroupSize || w.sig.size == 0 {
|
if len(w.paths) < minGroupSize {
|
||||||
// The paths are dropped, and their slice reused.
|
// A path with no duplicate is dropped, and its slice reused.
|
||||||
w.paths = w.paths[:0]
|
w.paths = w.paths[:0]
|
||||||
|
|
||||||
return
|
return
|
||||||
|
|||||||
@@ -462,140 +462,6 @@ func TestDupeGroups(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
|
|
||||||
// duplicate group, while 1-byte files still do.
|
|
||||||
func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
e, x := sha256.Sum256(nil), testHash("x")
|
|
||||||
recs := []scanRec{
|
|
||||||
{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
|
|
||||||
{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
|
|
||||||
{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
|
|
||||||
{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
|
|
||||||
{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
|
|
||||||
}
|
|
||||||
|
|
||||||
groups := dupeGroupsOf(t, recs)
|
|
||||||
if len(groups) != 1 || groups[0].size != 1 ||
|
|
||||||
!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
|
|
||||||
t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestReadReportChecksOnlyPossibleDuplicates checks that report asks
|
|
||||||
// whether a path is on disk only when its signature has two or more
|
|
||||||
// records, and never for an empty file.
|
|
||||||
func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
a, u, e := testHash("a"), testHash("u"), sha256.Sum256(nil)
|
|
||||||
recs := []scanRec{
|
|
||||||
{size: 9, head: a, tail: a, content: a, path: "/copy/1"},
|
|
||||||
{size: 9, head: a, tail: a, content: a, path: "/copy/2"},
|
|
||||||
{size: 9, head: a, tail: a, content: a, path: "/copy/3"},
|
|
||||||
{size: 9, head: u, tail: u, content: u, path: "/lonely"},
|
|
||||||
{size: 0, head: e, tail: e, content: e, path: "/empty1"},
|
|
||||||
{size: 0, head: e, tail: e, content: e, path: "/empty2"},
|
|
||||||
}
|
|
||||||
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
err := applyChanges(t.Context(), db, recs, nil, nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
var checked []string
|
|
||||||
|
|
||||||
exists := func(path string) bool {
|
|
||||||
checked = append(checked, path)
|
|
||||||
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
_, err = readReport(t.Context(), db, "", io.Discard, exists, nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.Sort(checked)
|
|
||||||
|
|
||||||
want := []string{"/copy/1", "/copy/2", "/copy/3"}
|
|
||||||
if !slices.Equal(checked, want) {
|
|
||||||
t.Errorf("paths checked on disk = %q, want %q", checked, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestReadReportDropsGroupLeftWithOnePath checks that a group of two
|
|
||||||
// paths, one of them gone, is no duplicate group. If kept, it would
|
|
||||||
// write no row, but the summary line would count it.
|
|
||||||
func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
a := testHash("a")
|
|
||||||
recs := []scanRec{
|
|
||||||
{size: 9, head: a, tail: a, content: a, path: "/a/kept"},
|
|
||||||
{size: 9, head: a, tail: a, content: a, path: "/a/gone"},
|
|
||||||
}
|
|
||||||
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
err := applyChanges(t.Context(), db, recs, nil, nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
exists := func(path string) bool { return path != "/a/gone" }
|
|
||||||
|
|
||||||
report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if report.dupeGroups != 0 {
|
|
||||||
t.Errorf("duplicate groups = %d, want 0", report.dupeGroups)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestReadReportKeepsGroupLeftWithTwoPaths checks that a group of three
|
|
||||||
// paths, one of them gone, is a duplicate group of the other two.
|
|
||||||
func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
a := testHash("a")
|
|
||||||
recs := []scanRec{
|
|
||||||
{size: 9, head: a, tail: a, content: a, path: "/b/kept1"},
|
|
||||||
{size: 9, head: a, tail: a, content: a, path: "/b/gone"},
|
|
||||||
{size: 9, head: a, tail: a, content: a, path: "/b/kept2"},
|
|
||||||
}
|
|
||||||
|
|
||||||
db := openTestDB(t)
|
|
||||||
|
|
||||||
err := applyChanges(t.Context(), db, recs, nil, nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
exists := func(path string) bool { return path != "/b/gone" }
|
|
||||||
|
|
||||||
var out bytes.Buffer
|
|
||||||
|
|
||||||
report, err := readReport(t.Context(), db, "", &out, exists, nil)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
if report.dupeGroups != 1 {
|
|
||||||
t.Errorf("duplicate groups = %d, want 1", report.dupeGroups)
|
|
||||||
}
|
|
||||||
|
|
||||||
want := "first\tdupe\tsize\n/b/kept1\t/b/kept2\t9\n"
|
|
||||||
if got := out.String(); got != want {
|
|
||||||
t.Errorf("stdout = %q, want %q", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestDupeGroupsContentSeparates(t *testing.T) {
|
func TestDupeGroupsContentSeparates(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -545,7 +545,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
|||||||
prog.increment()
|
prog.increment()
|
||||||
|
|
||||||
old, ok := s.existing[ev.rec.path]
|
old, ok := s.existing[ev.rec.path]
|
||||||
if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) {
|
if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) {
|
||||||
s.toHash = append(s.toHash, ev.rec)
|
s.toHash = append(s.toHash, ev.rec)
|
||||||
|
|
||||||
continue
|
continue
|
||||||
@@ -780,6 +780,16 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
|||||||
return applyChanges(ctx, s.db, nil, deletes, prog)
|
return applyChanges(ctx, s.db, nil, deletes, prog)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// mtimeAfter reports whether mtime a is later than mtime b.
|
||||||
|
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
|
||||||
|
func mtimeAfter(a, b time.Time) bool {
|
||||||
|
if a.Unix() != b.Unix() {
|
||||||
|
return a.Unix() > b.Unix()
|
||||||
|
}
|
||||||
|
|
||||||
|
return a.Nanosecond() > b.Nanosecond()
|
||||||
|
}
|
||||||
|
|
||||||
// underAnyRoot reports whether path is any of the roots or lies under
|
// underAnyRoot reports whether path is any of the roots or lies under
|
||||||
// one of them.
|
// one of them.
|
||||||
func underAnyRoot(path string, roots []string) bool {
|
func underAnyRoot(path string, roots []string) bool {
|
||||||
|
|||||||
@@ -1185,12 +1185,11 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
groups := dupeGroups(t, db)
|
groups := dupeGroups(t, db)
|
||||||
if len(groups) != 4 {
|
if len(groups) != 5 {
|
||||||
t.Fatalf("len(groups) = %d, want 4", len(groups))
|
t.Fatalf("len(groups) = %d, want 5", len(groups))
|
||||||
}
|
}
|
||||||
|
|
||||||
// The empty files empty1 and empty2 form no group.
|
wantSizes := []int64{3000, 2000, 100, 1, 0}
|
||||||
wantSizes := []int64{3000, 2000, 100, 1}
|
|
||||||
for i, g := range groups {
|
for i, g := range groups {
|
||||||
if g.size != wantSizes[i] {
|
if g.size != wantSizes[i] {
|
||||||
t.Errorf("groups[%d].size = %d, want %d",
|
t.Errorf("groups[%d].size = %d, want %d",
|
||||||
@@ -1384,47 +1383,6 @@ func TestSyncScanMtimeBump(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestSyncScanMtimeMovedBack replaces a file with other content of the
|
|
||||||
// same size and an older mtime, as cp -p or a restore from backup does.
|
|
||||||
// The next scan must hash the file again.
|
|
||||||
func TestSyncScanMtimeMovedBack(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
dir := t.TempDir()
|
|
||||||
db := openTestDB(t)
|
|
||||||
a := writeFile(t, dir, "a.bin", pattern(1, 500))
|
|
||||||
|
|
||||||
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
|
||||||
|
|
||||||
err := os.Chtimes(a, recorded, recorded)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
syncTree(t, db, dir)
|
|
||||||
|
|
||||||
replaced := pattern(3, 500)
|
|
||||||
writeFile(t, dir, "a.bin", replaced)
|
|
||||||
|
|
||||||
older := recorded.Add(-time.Hour)
|
|
||||||
|
|
||||||
err = os.Chtimes(a, older, older)
|
|
||||||
if err != nil {
|
|
||||||
t.Fatal(err)
|
|
||||||
}
|
|
||||||
|
|
||||||
st := syncTree(t, db, dir)
|
|
||||||
if st != (scanStats{walked: 1, updated: 1}) {
|
|
||||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
|
||||||
}
|
|
||||||
|
|
||||||
assertWholeFileHashed(t, db, a, replaced)
|
|
||||||
|
|
||||||
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
|
|
||||||
t.Fatalf("mtime = %v, want %v", r.mtime, older)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// assertWholeFileHashed fails unless the record for path holds the
|
// assertWholeFileHashed fails unless the record for path holds the
|
||||||
// whole-file hash of data as its head, tail, and content.
|
// whole-file hash of data as its head, tail, and content.
|
||||||
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
|
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
|
||||||
@@ -2331,7 +2289,8 @@ func TestRunReportsLeaveOutRemovedCopies(t *testing.T) {
|
|||||||
p("t1/f1") + "\t" + p("t2/f1") + "\t3000\n" +
|
p("t1/f1") + "\t" + p("t2/f1") + "\t3000\n" +
|
||||||
p("t1/f1") + "\t" + p("t3/f1") + "\t3000\n" +
|
p("t1/f1") + "\t" + p("t3/f1") + "\t3000\n" +
|
||||||
p("a/one.bin") + "\t" + p("b/copy.bin") + "\t2000\n" +
|
p("a/one.bin") + "\t" + p("b/copy.bin") + "\t2000\n" +
|
||||||
p("t1/sub/f2") + "\t" + p("t3/sub/f2renamed") + "\t100\n"
|
p("t1/sub/f2") + "\t" + p("t3/sub/f2renamed") + "\t100\n" +
|
||||||
|
p("empty1") + "\t" + p("empty2") + "\t0\n"
|
||||||
if got := runStdout(t, cmdReport); got != want {
|
if got := runStdout(t, cmdReport); got != want {
|
||||||
t.Errorf("report stdout = %q, want %q", got, want)
|
t.Errorf("report stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
@@ -2472,8 +2431,8 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
|||||||
report := runStdout(t, cmdReport)
|
report := runStdout(t, cmdReport)
|
||||||
if strings.Contains(report, "db.sqlite") ||
|
if strings.Contains(report, "db.sqlite") ||
|
||||||
strings.Contains(report, hardLink) ||
|
strings.Contains(report, hardLink) ||
|
||||||
!strings.Contains(report, filepath.Join(tree, "b")) {
|
!strings.Contains(report, filepath.Join(tree, "empty2")) {
|
||||||
t.Errorf("report = %q, want a and b and none of "+
|
t.Errorf("report = %q, want the empty files and none of "+
|
||||||
"the database's files", report)
|
"the database's files", report)
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -303,10 +303,8 @@ func fileEntry(name string, r scanRec) string {
|
|||||||
// prog as it is grouped, and returns every maximal group with two or
|
// prog as it is grouped, and returns every maximal group with two or
|
||||||
// more members for which exists returns true, each group's members
|
// more members for which exists returns true, each group's members
|
||||||
// sorted by path, groups ordered by tree size descending then by first
|
// sorted by path, groups ordered by tree size descending then by first
|
||||||
// path ascending. A group whose trees hold 0 bytes is left out, since
|
// path ascending. exists is called only on the members of maximal
|
||||||
// removing one frees nothing. The groups it implies hold 0 bytes too,
|
// groups.
|
||||||
// so leaving it out hides nothing. exists is called only on the members
|
|
||||||
// of maximal groups.
|
|
||||||
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||||
exists func(path string) bool, prog *progress,
|
exists func(path string) bool, prog *progress,
|
||||||
) [][]*treeNode {
|
) [][]*treeNode {
|
||||||
@@ -320,9 +318,7 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
|||||||
var dupes [][]*treeNode
|
var dupes [][]*treeNode
|
||||||
|
|
||||||
for _, g := range groups {
|
for _, g := range groups {
|
||||||
// The members share a digest, so they share a total size.
|
if len(g) < minGroupSize || suppressed(g, super) {
|
||||||
if len(g) < minGroupSize || g[0].totalSize == 0 ||
|
|
||||||
suppressed(g, super) {
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2,7 +2,6 @@ package sfdupes //nolint:testpackage // checks the unexported treeNode results
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"bytes"
|
"bytes"
|
||||||
"crypto/sha256"
|
|
||||||
"database/sql"
|
"database/sql"
|
||||||
"os"
|
"os"
|
||||||
"slices"
|
"slices"
|
||||||
@@ -448,35 +447,6 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
|
|
||||||
// holding only empty files form no group, and that an empty file still
|
|
||||||
// counts toward the digest of a tree that holds other files.
|
|
||||||
func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
|
|
||||||
sha256.Sum256(nil)
|
|
||||||
recs := []scanRec{
|
|
||||||
{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
|
|
||||||
{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
|
|
||||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
|
|
||||||
{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
|
|
||||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
|
|
||||||
{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
|
|
||||||
// Without the empty file, /p/x3 is no copy of /p/x1.
|
|
||||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
|
|
||||||
}
|
|
||||||
|
|
||||||
super, dirs := treeOf(t, recs)
|
|
||||||
|
|
||||||
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
|
|
||||||
|
|
||||||
want := [][]string{{"/p/x1", "/p/x2"}}
|
|
||||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
|
||||||
t.Fatalf("groups = %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -499,91 +469,3 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
|||||||
t.Fatalf("groups = %v, want %v", got, want)
|
t.Fatalf("groups = %v, want %v", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestCollectTreeGroupsChecksOnlyReportedCopies checks that trees asks
|
|
||||||
// whether a directory is on disk only for the members of the groups it
|
|
||||||
// returns. It never asks about /a/proj/sub, which lies inside a copy
|
|
||||||
// and whose group the group of /a/proj implies.
|
|
||||||
func TestCollectTreeGroupsChecksOnlyReportedCopies(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
f, g, u := testHash("f"), testHash("g"), testHash("u")
|
|
||||||
recs := []scanRec{
|
|
||||||
{size: 10, head: f, tail: f, content: f, path: "/a/proj/f"},
|
|
||||||
{size: 20, head: g, tail: g, content: g, path: "/a/proj/sub/g"},
|
|
||||||
{size: 10, head: f, tail: f, content: f, path: "/b/proj/f"},
|
|
||||||
{size: 20, head: g, tail: g, content: g, path: "/b/proj/sub/g"},
|
|
||||||
// /a lacks this file, so /a and /b are no copies of each other.
|
|
||||||
{size: 30, head: u, tail: u, content: u, path: "/b/lonely"},
|
|
||||||
}
|
|
||||||
|
|
||||||
super, dirs := treeOf(t, recs)
|
|
||||||
|
|
||||||
var checked []string
|
|
||||||
|
|
||||||
exists := func(path string) bool {
|
|
||||||
checked = append(checked, path)
|
|
||||||
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
|
|
||||||
|
|
||||||
want := [][]string{{"/a/proj", "/b/proj"}}
|
|
||||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
|
||||||
t.Fatalf("groups = %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
|
|
||||||
slices.Sort(checked)
|
|
||||||
|
|
||||||
if !slices.Equal(checked, want[0]) {
|
|
||||||
t.Errorf("directories checked on disk = %q, want %q",
|
|
||||||
checked, want[0])
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCollectTreeGroupsDropsGroupLeftWithOneCopy checks that a group of
|
|
||||||
// two copies, one of them gone, is not returned. If kept, it would
|
|
||||||
// write no row, but the summary line would count it.
|
|
||||||
func TestCollectTreeGroupsDropsGroupLeftWithOneCopy(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
f := testHash("f")
|
|
||||||
recs := []scanRec{
|
|
||||||
{size: 10, head: f, tail: f, content: f, path: "/c/kept/f"},
|
|
||||||
{size: 10, head: f, tail: f, content: f, path: "/c/gone/f"},
|
|
||||||
}
|
|
||||||
|
|
||||||
super, dirs := treeOf(t, recs)
|
|
||||||
|
|
||||||
exists := func(path string) bool { return path != "/c/gone" }
|
|
||||||
|
|
||||||
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
|
|
||||||
if len(got) != 0 {
|
|
||||||
t.Errorf("groups = %v, want none", got)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies checks that a group
|
|
||||||
// of three copies, one of them gone, is returned with the other two.
|
|
||||||
func TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies(t *testing.T) {
|
|
||||||
t.Parallel()
|
|
||||||
|
|
||||||
f := testHash("f")
|
|
||||||
recs := []scanRec{
|
|
||||||
{size: 10, head: f, tail: f, content: f, path: "/d/kept1/f"},
|
|
||||||
{size: 10, head: f, tail: f, content: f, path: "/d/gone/f"},
|
|
||||||
{size: 10, head: f, tail: f, content: f, path: "/d/kept2/f"},
|
|
||||||
}
|
|
||||||
|
|
||||||
super, dirs := treeOf(t, recs)
|
|
||||||
|
|
||||||
exists := func(path string) bool { return path != "/d/gone" }
|
|
||||||
|
|
||||||
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
|
|
||||||
|
|
||||||
want := [][]string{{"/d/kept1", "/d/kept2"}}
|
|
||||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
|
||||||
t.Errorf("groups = %v, want %v", got, want)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Reference in new issue
Block a user