Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1477c57024 | ||
|
|
a27d651532 | ||
|
|
a1ec1e0ca4 |
No files matched your search
@@ -21,9 +21,10 @@ file takes too long. It has five subcommands. `scan` walks the filesystem and
|
|||||||
records the files it finds in an SQLite database. SQLite keeps a whole database
|
records the files it finds in an SQLite database. SQLite keeps a whole database
|
||||||
in one file. The database is kept between runs. A rescan reads only new and
|
in one file. The database is kept between runs. A rescan reads only new and
|
||||||
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
||||||
system's job scheduler. `report`, `trees` and `info` read the database alone and
|
system's job scheduler. `report`, `trees` and `info` work from the database and
|
||||||
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
|
can run at any time. `report` and `trees` leave out a copy that was deleted
|
||||||
first look at a new machine with one command.
|
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first
|
||||||
|
look at a new machine with one command.
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
@@ -98,13 +99,15 @@ Each `-v` flag below makes a bind mount, which shows a host directory at a path
|
|||||||
inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between
|
inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between
|
||||||
runs. Mount the tree to scan at the same path inside the container as on the
|
runs. Mount the tree to scan at the same path inside the container as on the
|
||||||
host. Read-only access is enough. The database then records the host's paths,
|
host. Read-only access is enough. The database then records the host's paths,
|
||||||
and so do the reports.
|
and so do the reports. `report` and `trees` need that mount too. Without it,
|
||||||
|
every recorded path counts as gone, and each report holds only its header line.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
make docker
|
make docker
|
||||||
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
||||||
sfdupes scan /srv
|
sfdupes scan /srv
|
||||||
docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
|
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
||||||
|
sfdupes report > dupes.tsv
|
||||||
```
|
```
|
||||||
|
|
||||||
### Daily scan from `cron`
|
### Daily scan from `cron`
|
||||||
@@ -159,8 +162,10 @@ to keep is your choice.
|
|||||||
|
|
||||||
A row is a candidate, not proof:
|
A row is a candidate, not proof:
|
||||||
|
|
||||||
- The reports show each file as it was when a scan last recorded it. A file may
|
- The reports show each file as it was when a scan last recorded it. They check
|
||||||
have changed or gone since.
|
only that each copy still exists: the file, or the directory at the top of a
|
||||||
|
tree. A file may have changed since, and a file inside a tree may be gone. Run
|
||||||
|
`scan` again to bring the reports up to date.
|
||||||
- A file of 100 MiB or more is compared only on samples of its content (see
|
- A file of 100 MiB or more is compared only on samples of its content (see
|
||||||
"Duplicate detection").
|
"Duplicate detection").
|
||||||
- Two paths that are hard links, two names for one file, are listed as
|
- Two paths that are hard links, two names for one file, are listed as
|
||||||
@@ -208,8 +213,9 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
|
|||||||
of reads, at the price of a comparison that is not exact.
|
of reads, at the price of a comparison that is not exact.
|
||||||
|
|
||||||
The hashes live in a database that is kept between runs. A rescan reads only new
|
The hashes live in a database that is kept between runs. A rescan reads only new
|
||||||
files, and files whose size changed or whose modification time (mtime) is newer
|
files, and files whose size or modification time (mtime) differs from their
|
||||||
than recorded. All analysis then works from the database alone.
|
record. All analysis then works from the database. The reports only check that
|
||||||
|
each copy they name still exists.
|
||||||
|
|
||||||
The goal is whole duplicated trees, not only single files. A download extracted
|
The goal is whole duplicated trees, not only single files. A download extracted
|
||||||
twice, or a project copied once too often, is such a tree. It can be removed as
|
twice, or a project copied once too often, is such a tree. It can be removed as
|
||||||
@@ -235,9 +241,10 @@ In order:
|
|||||||
the files of the directories it is reading.
|
the files of the directories it is reading.
|
||||||
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
|
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
|
||||||
between runs, and an unchanged file is never read again. `report`, `trees`
|
between runs, and an unchanged file is never read again. `report`, `trees`
|
||||||
and `info` work from the database alone and never touch the scanned
|
and `info` work from the database and never read a scanned file. `report` and
|
||||||
filesystem. `scan` is built to run from `cron`. The reports run at any time,
|
`trees` check only that each copy they name still exists, and leave out a
|
||||||
against what the database holds.
|
copy that is gone. `scan` is built to run from `cron`. The reports run at any
|
||||||
|
time, against what the database holds.
|
||||||
4. **Keep the streams apart.** Everything on stdout is machine-readable data.
|
4. **Keep the streams apart.** Everything on stdout is machine-readable data.
|
||||||
Everything for a human reader, such as progress and warnings, goes to stderr.
|
Everything for a human reader, such as progress and warnings, goes to stderr.
|
||||||
|
|
||||||
@@ -410,17 +417,16 @@ operands:
|
|||||||
|
|
||||||
- A file not yet in the database is hashed and inserted.
|
- A file not yet in the database is hashed and inserted.
|
||||||
- A file already in the database is not read again when its size and mtime match
|
- A file already in the database is not read again when its size and mtime match
|
||||||
its record. The mtime matches when it is not newer than the recorded one. This
|
its record. This is what makes a daily rescan cheap.
|
||||||
is what makes a daily rescan cheap.
|
|
||||||
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
|
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
|
||||||
updates `disk_size` alone. It still does not read the file, and counts it as
|
updates `disk_size` alone. It still does not read the file, and counts it as
|
||||||
unchanged. ZFS, for example, counts a new file's blocks up to seconds after
|
unchanged. ZFS, for example, counts a new file's blocks up to seconds after
|
||||||
the write. A block is the unit in which a filesystem allocates disk space. A
|
the write. A block is the unit in which a filesystem allocates disk space. A
|
||||||
file scanned within those seconds gets its real `disk_size` at the next scan.
|
file scanned within those seconds gets its real `disk_size` at the next scan.
|
||||||
- `scan` hashes a file again when its size differs or its mtime is newer. The
|
- `scan` hashes a file again when its size or its mtime differs from its record.
|
||||||
mtime is compared to the nanosecond, as finely as the filesystem records it. A
|
An older mtime counts as much as a newer one. A file put back by `cp -p`,
|
||||||
same-size rewrite counts as a change whenever the filesystem gives it a later
|
`rsync -a` or a restore from backup often has an older mtime. The mtime is
|
||||||
mtime, even within the same second.
|
compared to the nanosecond, as finely as the filesystem records it.
|
||||||
- The scan deletes each record under a scanned operand whose file it could not
|
- The scan deletes each record under a scanned operand whose file it could not
|
||||||
verify. That covers deleted files, and paths the scan could not stat or read
|
verify. That covers deleted files, and paths the scan could not stat or read
|
||||||
this run. To stat a file is to read its details, such as its size, without
|
this run. To stat a file is to read its details, such as its size, without
|
||||||
@@ -655,21 +661,29 @@ and `scan` leaves it ignored.
|
|||||||
|
|
||||||
`report` reads every record from the database. It takes no positional arguments.
|
`report` reads every record from the database. It takes no positional arguments.
|
||||||
A positional argument is anything on the command line that is not a flag, such
|
A positional argument is anything on the command line that is not a flag, such
|
||||||
as the paths `scan` takes. `report` never touches the scanned filesystem. It
|
as the paths `scan` takes. `report` never opens a scanned file. It checks that
|
||||||
neither stats nor opens any path in the records. It reads the database and
|
each path in a group still exists, and looks at no other path. The check is
|
||||||
writes to stdout and stderr, and nothing else. Its output is the same whether or
|
`lstat`, the system call that looks up a path without following a symlink:
|
||||||
not the scanned filesystem is still mounted.
|
|
||||||
|
- A path is gone when `lstat` finds that it, or a directory above it, does not
|
||||||
|
exist.
|
||||||
|
- Any other error, such as a permission error, counts as present. So a user who
|
||||||
|
may not look inside a directory still sees its copies.
|
||||||
|
|
||||||
|
The check does not notice a file that changed after the scan. Run `scan` again
|
||||||
|
to bring the report up to date.
|
||||||
|
|
||||||
Processing:
|
Processing:
|
||||||
|
|
||||||
- Records are grouped by signature, the four values
|
- Records are grouped by signature, the four values
|
||||||
`(size, head, tail, content)`.
|
`(size, head, tail, content)`.
|
||||||
- Every group with two or more paths is a duplicate group.
|
- In a group with two or more paths, a path that is gone is left out. A group
|
||||||
|
that still has two or more paths is a duplicate group.
|
||||||
- Empty files form no group, because removing one frees nothing.
|
- Empty files form no group, because removing one frees nothing.
|
||||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||||
every other path is a `dupe`.
|
every other path is a `dupe`.
|
||||||
- Groups are ordered by size, largest first, then by `first` path. The output is
|
- Groups are ordered by size, largest first, then by `first` path. The output is
|
||||||
the same for the same database contents.
|
the same for the same database contents and the same paths on disk.
|
||||||
|
|
||||||
`report` has two phases, each with its own progress display on stderr (see
|
`report` has two phases, each with its own progress display on stderr (see
|
||||||
"Progress"). `open` opens the database. `read` gets the number of records from
|
"Progress"). `open` opens the database. `read` gets the number of records from
|
||||||
@@ -722,8 +736,9 @@ report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
|||||||
`trees` reads the same database as `report` and takes no positional arguments.
|
`trees` reads the same database as `report` and takes no positional arguments.
|
||||||
It reports whole duplicate directory trees. Two directories are duplicate trees
|
It reports whole duplicate directory trees. Two directories are duplicate trees
|
||||||
when they hold the same relative paths, and each path holds a file with the same
|
when they hold the same relative paths, and each path holds a file with the same
|
||||||
signature in both. Like `report`, it never touches the scanned filesystem. It
|
signature in both. It rebuilds the directory tree from the paths in the records,
|
||||||
rebuilds the directory tree from the paths in the records, split on `/`.
|
split on `/`. Like `report`, it never opens a scanned file. It checks only that
|
||||||
|
the directory at the top of each copy still exists, as `report` checks a path.
|
||||||
|
|
||||||
Definitions:
|
Definitions:
|
||||||
|
|
||||||
@@ -750,10 +765,13 @@ Processing:
|
|||||||
directories that all have the same digest. The parents' group, or an
|
directories that all have the same digest. The parents' group, or an
|
||||||
ancestor's, already implies it. A group is always reported when it holds
|
ancestor's, already implies it. A group is always reported when it holds
|
||||||
sibling directories, or when its members' parents have different digests.
|
sibling directories, or when its members' parents have different digests.
|
||||||
|
- A directory that is gone is dropped from its group. A group left with fewer
|
||||||
|
than two directories is not reported.
|
||||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||||
every other path is a `dupe`.
|
every other path is a `dupe`.
|
||||||
- Groups are ordered by total tree size, largest first, then by `first` path.
|
- Groups are ordered by total tree size, largest first, then by `first` path.
|
||||||
The output is the same for the same database contents.
|
The output is the same for the same database contents and the same paths on
|
||||||
|
disk.
|
||||||
|
|
||||||
Two limitations are accepted:
|
Two limitations are accepted:
|
||||||
|
|
||||||
@@ -767,8 +785,9 @@ Two limitations are accepted:
|
|||||||
`trees` has four phases, each with its own progress display on stderr (see
|
`trees` has four phases, each with its own progress display on stderr (see
|
||||||
"Progress"). `open` opens the database. `read` gets the number of records from
|
"Progress"). `open` opens the database. `read` gets the number of records from
|
||||||
the `totals` table, then reads them in path order. It builds the directory tree
|
the `totals` table, then reads them in path order. It builds the directory tree
|
||||||
and computes the digests as it goes. `group` groups the directories by digest
|
and computes the digests as it goes. `group` groups the directories by digest,
|
||||||
and picks the groups to report. `write` writes them to stdout.
|
picks the groups to report and checks that their directories still exist.
|
||||||
|
`write` writes the groups to stdout.
|
||||||
|
|
||||||
#### Trees output format
|
#### Trees output format
|
||||||
|
|
||||||
@@ -826,8 +845,8 @@ The report goes to stdout, exactly as `trees` prints it. On stderr comes what
|
|||||||
dwim: run "sfdupes trees" to print this report again
|
dwim: run "sfdupes trees" to print this report again
|
||||||
```
|
```
|
||||||
|
|
||||||
The database keeps the scan, so `sfdupes trees` prints the same report later,
|
The database keeps the scan, so `sfdupes trees` prints the report again later,
|
||||||
even after the terminal is gone.
|
even after the terminal is gone. It then leaves out a copy deleted in between.
|
||||||
|
|
||||||
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
|
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
|
||||||
`dwim` then prints no report and exits 1. A fatal scan error also ends it before
|
`dwim` then prints no report and exits 1. A fatal scan error also ends it before
|
||||||
@@ -1119,9 +1138,10 @@ in, and a phase that may wait in two places names both, as `disk/db`:
|
|||||||
- `hash` says `disk/db`, because it reads files and writes their records to the
|
- `hash` says `disk/db`, because it reads files and writes their records to the
|
||||||
database between them.
|
database between them.
|
||||||
- `load`, `update` and the `read` phase of `trees` say `db`.
|
- `load`, `update` and the `read` phase of `trees` say `db`.
|
||||||
- The `read` phase of `report` says `db/out`, because it writes the report to
|
- The `read` phase of `report` says `db/disk/out`. It checks that the paths of
|
||||||
stdout while it reads.
|
each group still exist, and writes the report to stdout, while it reads.
|
||||||
- `group` of `trees` says `cpu`, for work in memory, and its `write` says `out`.
|
- `group` of `trees` says `cpu/disk`, for work in memory and the check that each
|
||||||
|
copy still exists. Its `write` says `out`.
|
||||||
- A phase still waiting for its total says what it waits for, and nothing more.
|
- A phase still waiting for its total says what it waits for, and nothing more.
|
||||||
|
|
||||||
On a terminal:
|
On a terminal:
|
||||||
@@ -1361,8 +1381,8 @@ commands of their own:
|
|||||||
not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
|
not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
|
||||||
cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
|
cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
|
||||||
exception, because the race detector needs cgo (see `script/test`).
|
exception, because the race detector needs cgo (see `script/test`).
|
||||||
- `report` and `trees` give the same output for the same database contents,
|
- `report` and `trees` give the same output for the same database contents and
|
||||||
whatever order the records were inserted in.
|
the same paths on disk, whatever order the records were inserted in.
|
||||||
|
|
||||||
### Definition of done
|
### Definition of done
|
||||||
|
|
||||||
@@ -1427,7 +1447,7 @@ All of the following, run in this directory, must pass:
|
|||||||
|
|
||||||
The test suite automates this scenario (see
|
The test suite automates this scenario (see
|
||||||
`internal/sfdupes/scan_test.go`). It also checks that `report` and `trees`
|
`internal/sfdupes/scan_test.go`). It also checks that `report` and `trees`
|
||||||
never touch the scanned filesystem.
|
leave out a copy removed after the scan.
|
||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
|
|||||||
@@ -29,6 +29,14 @@
|
|||||||
Newest first. An entry ends with its date and its issue or branch, where those
|
Newest first. An entry ends with its date and its issue or branch, where those
|
||||||
are known.
|
are known.
|
||||||
|
|
||||||
|
- `report` and `trees` leave out a copy that was deleted after the scan. Each
|
||||||
|
checks only the copy itself with `lstat`: the file, or the directory at the
|
||||||
|
top of a tree. A group left with one copy is not reported. A missing `scan`
|
||||||
|
operand is still a fatal error (2026-11-11,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/173)
|
||||||
|
- The usage line of `scan` no longer ends in `[flags]`. It is the line
|
||||||
|
`README.md` shows under "Subcommands" (2026-11-11,
|
||||||
|
https://git.eeqj.de/sneak/sfdupes/issues/169)
|
||||||
- `report` and `trees` leave out empty files. `report` writes no group of size
|
- `report` and `trees` leave out empty files. `report` writes no group of size
|
||||||
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
|
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
|
||||||
counts toward a tree's digest (2026-11-11,
|
counts toward a tree's digest (2026-11-11,
|
||||||
|
|||||||
@@ -215,10 +215,13 @@ func newScanCommand() *cobra.Command {
|
|||||||
short := "Walk trees and synchronize the scan database"
|
short := "Walk trees and synchronize the scan database"
|
||||||
|
|
||||||
scanCmd := &cobra.Command{
|
scanCmd := &cobra.Command{
|
||||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
||||||
Short: short,
|
// The Use line already names every flag, so cobra must not
|
||||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
// append its generic "[flags]" to it.
|
||||||
Args: cobra.MinimumNArgs(1),
|
DisableFlagsInUseLine: true,
|
||||||
|
Short: short,
|
||||||
|
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
||||||
|
Args: cobra.MinimumNArgs(1),
|
||||||
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
||||||
return checkScanWorkers(cmd, scanWorkers)
|
return checkScanWorkers(cmd, scanWorkers)
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -714,6 +714,34 @@ func TestRunHelp(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunScanUsageLine(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
// README §Subcommands shows this usage line for scan, and the help
|
||||||
|
// and the usage error both print it.
|
||||||
|
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...\n"
|
||||||
|
|
||||||
|
cases := map[string][]string{
|
||||||
|
"help": {cmdScan, "-h"},
|
||||||
|
"missing paths": {cmdScan},
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, args := range cases {
|
||||||
|
t.Run(name, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
|
run(args, &stdout, &stderr)
|
||||||
|
|
||||||
|
if !strings.Contains(stderr.String(), want) {
|
||||||
|
t.Errorf("run(%v) stderr = %q, want the usage line %q",
|
||||||
|
args, stderr.String(), want)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunVersion(t *testing.T) {
|
func TestRunVersion(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -642,9 +642,9 @@ func TestTerminalStalledLines(t *testing.T) {
|
|||||||
{label: scanPhase("hash"), stalledOn: hashStalledOn},
|
{label: scanPhase("hash"), stalledOn: hashStalledOn},
|
||||||
{label: scanPhase("update"), stalledOn: "db"},
|
{label: scanPhase("update"), stalledOn: "db"},
|
||||||
{label: treesPhase("read"), stalledOn: "db"},
|
{label: treesPhase("read"), stalledOn: "db"},
|
||||||
{label: treesPhase("group"), stalledOn: "cpu"},
|
{label: treesPhase("group"), stalledOn: "cpu/disk"},
|
||||||
{label: treesPhase("write"), stalledOn: "out"},
|
{label: treesPhase("write"), stalledOn: "out"},
|
||||||
{label: reportPhase("read"), stalledOn: "db/out"},
|
{label: reportPhase("read"), stalledOn: "db/disk/out"},
|
||||||
}
|
}
|
||||||
|
|
||||||
for _, params := range phases {
|
for _, params := range phases {
|
||||||
|
|||||||
+43
-16
@@ -4,11 +4,14 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"crypto/sha256"
|
"crypto/sha256"
|
||||||
"database/sql"
|
"database/sql"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
|
"syscall"
|
||||||
"time"
|
"time"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -41,10 +44,10 @@ func reportPhase(name string) string {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// runReport implements the report subcommand: it prints the file-level
|
// runReport implements the report subcommand: it prints the file-level
|
||||||
// duplicates report as TSV on stdout. It never touches the scanned
|
// duplicates report as TSV on stdout. Of the scanned filesystem it
|
||||||
// filesystem; its only I/O is the database, stdout, and stderr. It
|
// only checks, with pathExists, that each path in a duplicate group is
|
||||||
// names the database on stderr before it opens it. Any database
|
// still there. It names the database on stderr before it opens it. Any
|
||||||
// problem, including a missing database, is fatal.
|
// database problem, including a missing database, is fatal.
|
||||||
//
|
//
|
||||||
// The work runs in two phases, each under a progress display of its
|
// The work runs in two phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database) and read (read the number of
|
// own on stderr: open (open the database) and read (read the number of
|
||||||
@@ -77,13 +80,14 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
|||||||
|
|
||||||
defer func() { _ = db.Close() }()
|
defer func() { _ = db.Close() }()
|
||||||
|
|
||||||
// The read phase writes the report to stdout while it reads the
|
// The read phase checks the paths of each duplicate group on disk
|
||||||
// records, so it may wait on either.
|
// and writes the report to stdout while it reads the records, so it
|
||||||
|
// may wait on any of the three.
|
||||||
prog = newProgress(progressParams{
|
prog = newProgress(progressParams{
|
||||||
label: reportPhase("read"), unit: "records", total: -1,
|
label: reportPhase("read"), unit: "records", total: -1,
|
||||||
waitingFor: countingRecords, stalledOn: "db/out",
|
waitingFor: countingRecords, stalledOn: "db/disk/out",
|
||||||
})
|
})
|
||||||
report, err := readReport(ctx, db, dbPath, stdout, prog)
|
report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog)
|
||||||
|
|
||||||
prog.finish()
|
prog.finish()
|
||||||
|
|
||||||
@@ -104,14 +108,15 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
|||||||
// dbPath, as its total, then reads every record in signature order,
|
// dbPath, as its total, then reads every record in signature order,
|
||||||
// counting each on prog, and writes the report to stdout: the header
|
// counting each on prog, and writes the report to stdout: the header
|
||||||
// line, then the rows of each size's duplicate groups once that size is
|
// line, then the rows of each size's duplicate groups once that size is
|
||||||
// read. stdout may be the terminal prog is drawn on, so the lines go
|
// read. A path for which exists returns false is left out of its group.
|
||||||
// through prog's writer. It returns what it wrote, with the totals for
|
// stdout may be the terminal prog is drawn on, so the lines go through
|
||||||
// the summary line.
|
// prog's writer. It returns what it wrote, with the totals for the
|
||||||
|
// summary line.
|
||||||
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||||
stdout io.Writer, prog *progress,
|
stdout io.Writer, exists func(path string) bool, prog *progress,
|
||||||
) (*reportWriter, error) {
|
) (*reportWriter, error) {
|
||||||
out := prog.writer(stdout)
|
out := prog.writer(stdout)
|
||||||
report := &reportWriter{out: out}
|
report := &reportWriter{out: out, exists: exists}
|
||||||
|
|
||||||
_, err := fmt.Fprintln(out, "first\tdupe\tsize")
|
_, err := fmt.Fprintln(out, "first\tdupe\tsize")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -158,6 +163,10 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
|||||||
type reportWriter struct {
|
type reportWriter struct {
|
||||||
out io.Writer
|
out io.Writer
|
||||||
|
|
||||||
|
// exists reports whether a path is still on disk. A path that is
|
||||||
|
// gone is left out of its group.
|
||||||
|
exists func(path string) bool
|
||||||
|
|
||||||
sig scanRec // the signature being read; its path is not used
|
sig scanRec // the signature being read; its path is not used
|
||||||
paths []string // the paths read so far with sig
|
paths []string // the paths read so far with sig
|
||||||
groups [][]string // the duplicate groups of sig.size, each sorted
|
groups [][]string // the duplicate groups of sig.size, each sorted
|
||||||
@@ -194,10 +203,18 @@ func (w *reportWriter) add(r scanRec) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
// endGroup ends the paths read with w.sig: two or more are a duplicate
|
// endGroup ends the paths read with w.sig: two or more that are still
|
||||||
// group, held with its paths sorted until writeGroups writes it. Empty
|
// on disk are a duplicate group, held with its paths sorted until
|
||||||
// files form no group, since removing one frees nothing.
|
// writeGroups writes it. Empty files form no group, since removing one
|
||||||
|
// frees nothing. Only the paths of a possible group are checked, never
|
||||||
|
// a path with no duplicate or an empty file.
|
||||||
func (w *reportWriter) endGroup() {
|
func (w *reportWriter) endGroup() {
|
||||||
|
if len(w.paths) >= minGroupSize && w.sig.size > 0 {
|
||||||
|
w.paths = slices.DeleteFunc(w.paths, func(p string) bool {
|
||||||
|
return !w.exists(p)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
if len(w.paths) < minGroupSize || w.sig.size == 0 {
|
if len(w.paths) < minGroupSize || w.sig.size == 0 {
|
||||||
// The paths are dropped, and their slice reused.
|
// The paths are dropped, and their slice reused.
|
||||||
w.paths = w.paths[:0]
|
w.paths = w.paths[:0]
|
||||||
@@ -246,6 +263,16 @@ func (w *reportWriter) finish() error {
|
|||||||
return w.writeGroups()
|
return w.writeGroups()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// pathExists reports whether path is still on disk, for report and
|
||||||
|
// trees. It is false only when lstat finds that the path, or a
|
||||||
|
// directory above it, is gone. Any other error counts as present, so
|
||||||
|
// a user who may not look inside a directory still sees its copies.
|
||||||
|
func pathExists(path string) bool {
|
||||||
|
_, err := os.Lstat(path)
|
||||||
|
|
||||||
|
return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
|
||||||
|
}
|
||||||
|
|
||||||
// escapePath returns a path as it is written in a report column (README
|
// escapePath returns a path as it is written in a report column (README
|
||||||
// "Report output format"): a backslash, tab, newline or carriage return
|
// "Report output format"): a backslash, tab, newline or carriage return
|
||||||
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
||||||
|
|||||||
@@ -72,15 +72,37 @@ type dupeGroup struct {
|
|||||||
paths []string
|
paths []string
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// everyPathExists stands in for pathExists where a test's records name
|
||||||
|
// made-up paths that are not on disk.
|
||||||
|
func everyPathExists(string) bool { return true }
|
||||||
|
|
||||||
|
// recsOnDisk returns a new directory and recs with each path moved
|
||||||
|
// under it, after creating an empty file at each moved path: report and
|
||||||
|
// trees then find every path still on disk.
|
||||||
|
func recsOnDisk(t *testing.T, recs []scanRec) (string, []scanRec) {
|
||||||
|
t.Helper()
|
||||||
|
|
||||||
|
dir := realTempDir(t)
|
||||||
|
moved := make([]scanRec, len(recs))
|
||||||
|
|
||||||
|
for i, r := range recs {
|
||||||
|
r.path = writeFile(t, dir, r.path, nil)
|
||||||
|
moved[i] = r
|
||||||
|
}
|
||||||
|
|
||||||
|
return dir, moved
|
||||||
|
}
|
||||||
|
|
||||||
// dupeGroups returns the duplicate groups report reads from db, in
|
// dupeGroups returns the duplicate groups report reads from db, in
|
||||||
// report order, from the rows it writes: a group's rows come together,
|
// report order, from the rows it writes: a group's rows come together,
|
||||||
// each naming the group's first path.
|
// each naming the group's first path. Every path counts as still on
|
||||||
|
// disk.
|
||||||
func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
|
func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
|
|
||||||
var out bytes.Buffer
|
var out bytes.Buffer
|
||||||
|
|
||||||
_, err := readReport(t.Context(), db, "", &out, nil)
|
_, err := readReport(t.Context(), db, "", &out, everyPathExists, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -127,7 +149,8 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunReportEscapesPaths(t *testing.T) {
|
func TestRunReportEscapesPaths(t *testing.T) {
|
||||||
t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
|
dir, recs := recsOnDisk(t, awkwardPairRecs())
|
||||||
|
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||||
|
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
@@ -138,7 +161,7 @@ func TestRunReportEscapesPaths(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
want := "first\tdupe\tsize\n" +
|
want := "first\tdupe\tsize\n" +
|
||||||
`/d/\tone\ntwo\rthree\\four/f` + "\t/d/A/f\t5\n"
|
dir + `/d/\tone\ntwo\rthree\\four/f` + "\t" + dir + "/d/A/f\t5\n"
|
||||||
if got := stdout.String(); got != want {
|
if got := stdout.String(); got != want {
|
||||||
t.Errorf("stdout = %q, want %q", got, want)
|
t.Errorf("stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
@@ -172,7 +195,8 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
|
|||||||
// Counts the records read, and draws nothing.
|
// Counts the records read, and draws nothing.
|
||||||
prog := &progress{now: time.Now}
|
prog := &progress{now: time.Now}
|
||||||
|
|
||||||
_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
|
_, err = readReport(t.Context(), db, "", failingWriter{},
|
||||||
|
everyPathExists, prog)
|
||||||
if !errors.Is(err, errWriteFailed) ||
|
if !errors.Is(err, errWriteFailed) ||
|
||||||
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
||||||
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
|
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
|
||||||
@@ -203,7 +227,8 @@ const smokeReport = "first\tdupe\tsize\n" +
|
|||||||
// it ends; read counts the 6 records before it reads them. The summary
|
// it ends; read counts the 6 records before it reads them. The summary
|
||||||
// comes last, and stdout holds the report alone.
|
// comes last, and stdout holds the report alone.
|
||||||
func TestRunReportProgressNamesEachPhase(t *testing.T) {
|
func TestRunReportProgressNamesEachPhase(t *testing.T) {
|
||||||
path := seedDatabase(t, smokeTreeRecs())
|
dir, recs := recsOnDisk(t, smokeTreeRecs())
|
||||||
|
path := seedDatabase(t, recs)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
stderr := captureStderr(t)
|
stderr := captureStderr(t)
|
||||||
@@ -216,8 +241,9 @@ func TestRunReportProgressNamesEachPhase(t *testing.T) {
|
|||||||
code, exitOK, stderr())
|
code, exitOK, stderr())
|
||||||
}
|
}
|
||||||
|
|
||||||
if got := stdout.String(); got != smokeReport {
|
report := strings.ReplaceAll(smokeReport, "/d/", dir+"/d/")
|
||||||
t.Errorf("stdout = %q, want %q", got, smokeReport)
|
if got := stdout.String(); got != report {
|
||||||
|
t.Errorf("stdout = %q, want %q", got, report)
|
||||||
}
|
}
|
||||||
|
|
||||||
// How each line starts; the rate at the end of a phase varies.
|
// How each line starts; the rate at the end of a phase varies.
|
||||||
@@ -266,7 +292,7 @@ func TestReadReportOnTerminal(t *testing.T) {
|
|||||||
waitingFor: countingRecords,
|
waitingFor: countingRecords,
|
||||||
}, terminalWidth, time.Hour, time.Now)
|
}, terminalWidth, time.Hour, time.Now)
|
||||||
|
|
||||||
_, err = readReport(t.Context(), db, "", os.Stderr, p)
|
_, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p)
|
||||||
|
|
||||||
p.finish()
|
p.finish()
|
||||||
|
|
||||||
@@ -301,6 +327,8 @@ func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
|
|||||||
scanRec{size: 50, head: a, tail: a, content: a, path: "/x/1"},
|
scanRec{size: 50, head: a, tail: a, content: a, path: "/x/1"},
|
||||||
)
|
)
|
||||||
|
|
||||||
|
_, recs = recsOnDisk(t, recs)
|
||||||
|
|
||||||
reversed := slices.Clone(recs)
|
reversed := slices.Clone(recs)
|
||||||
slices.Reverse(reversed)
|
slices.Reverse(reversed)
|
||||||
|
|
||||||
@@ -455,6 +483,119 @@ func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestReadReportChecksOnlyPossibleDuplicates checks that report asks
|
||||||
|
// whether a path is on disk only when its signature has two or more
|
||||||
|
// records, and never for an empty file.
|
||||||
|
func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
a, u, e := testHash("a"), testHash("u"), sha256.Sum256(nil)
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 9, head: a, tail: a, content: a, path: "/copy/1"},
|
||||||
|
{size: 9, head: a, tail: a, content: a, path: "/copy/2"},
|
||||||
|
{size: 9, head: a, tail: a, content: a, path: "/copy/3"},
|
||||||
|
{size: 9, head: u, tail: u, content: u, path: "/lonely"},
|
||||||
|
{size: 0, head: e, tail: e, content: e, path: "/empty1"},
|
||||||
|
{size: 0, head: e, tail: e, content: e, path: "/empty2"},
|
||||||
|
}
|
||||||
|
|
||||||
|
db := openTestDB(t)
|
||||||
|
|
||||||
|
err := applyChanges(t.Context(), db, recs, nil, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
var checked []string
|
||||||
|
|
||||||
|
exists := func(path string) bool {
|
||||||
|
checked = append(checked, path)
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = readReport(t.Context(), db, "", io.Discard, exists, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
slices.Sort(checked)
|
||||||
|
|
||||||
|
want := []string{"/copy/1", "/copy/2", "/copy/3"}
|
||||||
|
if !slices.Equal(checked, want) {
|
||||||
|
t.Errorf("paths checked on disk = %q, want %q", checked, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReadReportDropsGroupLeftWithOnePath checks that a group of two
|
||||||
|
// paths, one of them gone, is no duplicate group. If kept, it would
|
||||||
|
// write no row, but the summary line would count it.
|
||||||
|
func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
a := testHash("a")
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 9, head: a, tail: a, content: a, path: "/a/kept"},
|
||||||
|
{size: 9, head: a, tail: a, content: a, path: "/a/gone"},
|
||||||
|
}
|
||||||
|
|
||||||
|
db := openTestDB(t)
|
||||||
|
|
||||||
|
err := applyChanges(t.Context(), db, recs, nil, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
exists := func(path string) bool { return path != "/a/gone" }
|
||||||
|
|
||||||
|
report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if report.dupeGroups != 0 {
|
||||||
|
t.Errorf("duplicate groups = %d, want 0", report.dupeGroups)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReadReportKeepsGroupLeftWithTwoPaths checks that a group of three
|
||||||
|
// paths, one of them gone, is a duplicate group of the other two.
|
||||||
|
func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
a := testHash("a")
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 9, head: a, tail: a, content: a, path: "/b/kept1"},
|
||||||
|
{size: 9, head: a, tail: a, content: a, path: "/b/gone"},
|
||||||
|
{size: 9, head: a, tail: a, content: a, path: "/b/kept2"},
|
||||||
|
}
|
||||||
|
|
||||||
|
db := openTestDB(t)
|
||||||
|
|
||||||
|
err := applyChanges(t.Context(), db, recs, nil, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
exists := func(path string) bool { return path != "/b/gone" }
|
||||||
|
|
||||||
|
var out bytes.Buffer
|
||||||
|
|
||||||
|
report, err := readReport(t.Context(), db, "", &out, exists, nil)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if report.dupeGroups != 1 {
|
||||||
|
t.Errorf("duplicate groups = %d, want 1", report.dupeGroups)
|
||||||
|
}
|
||||||
|
|
||||||
|
want := "first\tdupe\tsize\n/b/kept1\t/b/kept2\t9\n"
|
||||||
|
if got := out.String(); got != want {
|
||||||
|
t.Errorf("stdout = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestDupeGroupsContentSeparates(t *testing.T) {
|
func TestDupeGroupsContentSeparates(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -545,7 +545,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
|||||||
prog.increment()
|
prog.increment()
|
||||||
|
|
||||||
old, ok := s.existing[ev.rec.path]
|
old, ok := s.existing[ev.rec.path]
|
||||||
if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) {
|
if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) {
|
||||||
s.toHash = append(s.toHash, ev.rec)
|
s.toHash = append(s.toHash, ev.rec)
|
||||||
|
|
||||||
continue
|
continue
|
||||||
@@ -780,16 +780,6 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
|||||||
return applyChanges(ctx, s.db, nil, deletes, prog)
|
return applyChanges(ctx, s.db, nil, deletes, prog)
|
||||||
}
|
}
|
||||||
|
|
||||||
// mtimeAfter reports whether mtime a is later than mtime b.
|
|
||||||
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
|
|
||||||
func mtimeAfter(a, b time.Time) bool {
|
|
||||||
if a.Unix() != b.Unix() {
|
|
||||||
return a.Unix() > b.Unix()
|
|
||||||
}
|
|
||||||
|
|
||||||
return a.Nanosecond() > b.Nanosecond()
|
|
||||||
}
|
|
||||||
|
|
||||||
// underAnyRoot reports whether path is any of the roots or lies under
|
// underAnyRoot reports whether path is any of the roots or lies under
|
||||||
// one of them.
|
// one of them.
|
||||||
func underAnyRoot(path string, roots []string) bool {
|
func underAnyRoot(path string, roots []string) bool {
|
||||||
|
|||||||
+114
-10
@@ -1215,7 +1215,7 @@ func assertSmokeTreeGroups(t *testing.T, dir string, db *sql.DB) {
|
|||||||
|
|
||||||
super, dirs := dbTree(t, db)
|
super, dirs := dbTree(t, db)
|
||||||
|
|
||||||
tg := collectTreeGroups(dirs, super, nil)
|
tg := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||||
if len(tg) != 1 {
|
if len(tg) != 1 {
|
||||||
t.Fatalf("len(tree groups) = %d, want 1", len(tg))
|
t.Fatalf("len(tree groups) = %d, want 1", len(tg))
|
||||||
}
|
}
|
||||||
@@ -1384,6 +1384,47 @@ func TestSyncScanMtimeBump(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestSyncScanMtimeMovedBack replaces a file with other content of the
|
||||||
|
// same size and an older mtime, as cp -p or a restore from backup does.
|
||||||
|
// The next scan must hash the file again.
|
||||||
|
func TestSyncScanMtimeMovedBack(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
db := openTestDB(t)
|
||||||
|
a := writeFile(t, dir, "a.bin", pattern(1, 500))
|
||||||
|
|
||||||
|
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||||
|
|
||||||
|
err := os.Chtimes(a, recorded, recorded)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
syncTree(t, db, dir)
|
||||||
|
|
||||||
|
replaced := pattern(3, 500)
|
||||||
|
writeFile(t, dir, "a.bin", replaced)
|
||||||
|
|
||||||
|
older := recorded.Add(-time.Hour)
|
||||||
|
|
||||||
|
err = os.Chtimes(a, older, older)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
st := syncTree(t, db, dir)
|
||||||
|
if st != (scanStats{walked: 1, updated: 1}) {
|
||||||
|
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
||||||
|
}
|
||||||
|
|
||||||
|
assertWholeFileHashed(t, db, a, replaced)
|
||||||
|
|
||||||
|
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
|
||||||
|
t.Fatalf("mtime = %v, want %v", r.mtime, older)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// assertWholeFileHashed fails unless the record for path holds the
|
// assertWholeFileHashed fails unless the record for path holds the
|
||||||
// whole-file hash of data as its head, tail, and content.
|
// whole-file hash of data as its head, tail, and content.
|
||||||
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
|
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
|
||||||
@@ -2266,23 +2307,86 @@ func TestPruneRoots(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReportsNeverTouchFilesystem(t *testing.T) {
|
// TestRunReportsLeaveOutRemovedCopies scans the smoke tree, then
|
||||||
t.Parallel()
|
// removes copies the reports name. report leaves out each removed file,
|
||||||
|
// and a group left with one path. trees checks only the root of each
|
||||||
|
// duplicate tree: with a file under t2 removed it still names t2, and
|
||||||
|
// with t2 removed it names nothing.
|
||||||
|
func TestRunReportsLeaveOutRemovedCopies(t *testing.T) {
|
||||||
|
t.Setenv(databaseEnv, testDBPath(t))
|
||||||
|
|
||||||
dir := buildSmokeTree(t)
|
dir := buildSmokeTree(t)
|
||||||
db := openTestDB(t)
|
p := func(name string) string { return filepath.Join(dir, name) }
|
||||||
|
|
||||||
syncTree(t, db, dir)
|
scanOK(t, dir)
|
||||||
|
|
||||||
// Remove the scanned tree entirely; the analysis must be
|
for _, name := range []string{"b/copy2.bin", "tiny2", "t2/sub/f2"} {
|
||||||
// unaffected because it reads the database alone.
|
err := os.Remove(p(name))
|
||||||
err := os.RemoveAll(dir)
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
want := "first\tdupe\tsize\n" +
|
||||||
|
p("t1/f1") + "\t" + p("t2/f1") + "\t3000\n" +
|
||||||
|
p("t1/f1") + "\t" + p("t3/f1") + "\t3000\n" +
|
||||||
|
p("a/one.bin") + "\t" + p("b/copy.bin") + "\t2000\n" +
|
||||||
|
p("t1/sub/f2") + "\t" + p("t3/sub/f2renamed") + "\t100\n"
|
||||||
|
if got := runStdout(t, cmdReport); got != want {
|
||||||
|
t.Errorf("report stdout = %q, want %q", got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
// disk_size, the last column, depends on the filesystem.
|
||||||
|
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
|
||||||
|
|
||||||
|
want = header + p("t1") + "\t" + p("t2") + "\t2\t3100\t"
|
||||||
|
if got := runStdout(t, cmdTrees); !strings.HasPrefix(got, want) ||
|
||||||
|
strings.Count(got, "\n") != 2 {
|
||||||
|
t.Errorf("trees stdout = %q, want one row starting %q", got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
err := os.RemoveAll(p("t2"))
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
assertSmokeDupeGroups(t, dir, db)
|
if got := runStdout(t, cmdTrees); got != header {
|
||||||
assertSmokeTreeGroups(t, dir, db)
|
t.Errorf("trees stdout after removing t2 = %q, want %q", got, header)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPathExists(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
dir := t.TempDir()
|
||||||
|
file := writeFile(t, dir, "file", nil)
|
||||||
|
locked := writeFile(t, dir, "locked/file", nil)
|
||||||
|
|
||||||
|
// Without search permission on its directory, a file cannot be
|
||||||
|
// looked at by anyone but root.
|
||||||
|
err := os.Chmod(filepath.Dir(locked), 0)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Runs before t.TempDir's own cleanup, which must delete the files.
|
||||||
|
t.Cleanup(func() {
|
||||||
|
//nolint:gosec // removing the directory needs its search bit back
|
||||||
|
_ = os.Chmod(filepath.Dir(locked), 0o700)
|
||||||
|
})
|
||||||
|
|
||||||
|
cases := map[string]bool{
|
||||||
|
dir: true,
|
||||||
|
file: true,
|
||||||
|
locked: true,
|
||||||
|
filepath.Join(dir, "gone"): false,
|
||||||
|
filepath.Join(file, "under-a-file"): false,
|
||||||
|
}
|
||||||
|
for path, want := range cases {
|
||||||
|
if got := pathExists(path); got != want {
|
||||||
|
t.Errorf("pathExists(%q) = %v, want %v", path, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestUnderRoot(t *testing.T) {
|
func TestUnderRoot(t *testing.T) {
|
||||||
|
|||||||
+26
-13
@@ -38,10 +38,11 @@ func treesPhase(name string) string {
|
|||||||
// runTrees implements the trees subcommand: it reads every record from
|
// runTrees implements the trees subcommand: it reads every record from
|
||||||
// the database in path order, reconstructs the directory hierarchy from
|
// the database in path order, reconstructs the directory hierarchy from
|
||||||
// the record paths, computes a Merkle-style digest per directory, and
|
// the record paths, computes a Merkle-style digest per directory, and
|
||||||
// prints maximal duplicate-tree groups as TSV on stdout. It never
|
// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned
|
||||||
// touches the scanned filesystem; its only I/O is the database, stdout,
|
// filesystem it only checks, with pathExists, that the root of each
|
||||||
// and stderr. It names the database on stderr before it opens it. Any
|
// duplicate tree is still there, never the files below it. It names the
|
||||||
// database problem, including a missing database, is fatal.
|
// database on stderr before it opens it. Any database problem,
|
||||||
|
// including a missing database, is fatal.
|
||||||
//
|
//
|
||||||
// The work runs in four phases, each under a progress display of its
|
// The work runs in four phases, each under a progress display of its
|
||||||
// own on stderr: open (open the database), read (read the number of
|
// own on stderr: open (open the database), read (read the number of
|
||||||
@@ -86,13 +87,13 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
|||||||
return fmt.Errorf("database %s: %w", dbPath, err)
|
return fmt.Errorf("database %s: %w", dbPath, err)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Grouping works in memory, so it waits on nothing but the
|
// Grouping works in memory, then checks the root of each duplicate
|
||||||
// processor.
|
// tree on disk, so it waits on the processor or the disk.
|
||||||
prog = newProgress(progressParams{
|
prog = newProgress(progressParams{
|
||||||
label: treesPhase("group"), unit: "directories",
|
label: treesPhase("group"), unit: "directories",
|
||||||
total: int64(len(allDirs)), stalledOn: "cpu",
|
total: int64(len(allDirs)), stalledOn: "cpu/disk",
|
||||||
})
|
})
|
||||||
dupes := collectTreeGroups(allDirs, super, prog)
|
dupes := collectTreeGroups(allDirs, super, pathExists, prog)
|
||||||
|
|
||||||
prog.finish()
|
prog.finish()
|
||||||
|
|
||||||
@@ -300,12 +301,14 @@ func fileEntry(name string, r scanRec) string {
|
|||||||
|
|
||||||
// collectTreeGroups groups directories by digest, counting each one on
|
// collectTreeGroups groups directories by digest, counting each one on
|
||||||
// prog as it is grouped, and returns every maximal group with two or
|
// prog as it is grouped, and returns every maximal group with two or
|
||||||
// more members, each group's members sorted by path, groups ordered by
|
// more members for which exists returns true, each group's members
|
||||||
// tree size descending then by first path ascending. A group whose
|
// sorted by path, groups ordered by tree size descending then by first
|
||||||
// trees hold 0 bytes is left out, since removing one frees nothing. The
|
// path ascending. A group whose trees hold 0 bytes is left out, since
|
||||||
// groups it implies hold 0 bytes too, so leaving it out hides nothing.
|
// removing one frees nothing. The groups it implies hold 0 bytes too,
|
||||||
|
// so leaving it out hides nothing. exists is called only on the members
|
||||||
|
// of maximal groups.
|
||||||
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||||
prog *progress,
|
exists func(path string) bool, prog *progress,
|
||||||
) [][]*treeNode {
|
) [][]*treeNode {
|
||||||
groups := make(map[[sha256.Size]byte][]*treeNode)
|
groups := make(map[[sha256.Size]byte][]*treeNode)
|
||||||
for _, d := range allDirs {
|
for _, d := range allDirs {
|
||||||
@@ -323,6 +326,16 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
|||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A member that is gone leaves the group. A suppressed group is
|
||||||
|
// still implied: its members still on disk have parents still
|
||||||
|
// on disk, and those parents' group is checked the same way.
|
||||||
|
g = slices.DeleteFunc(g, func(n *treeNode) bool {
|
||||||
|
return !exists(n.path)
|
||||||
|
})
|
||||||
|
if len(g) < minGroupSize {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
slices.SortFunc(g, func(a, b *treeNode) int {
|
slices.SortFunc(g, func(a, b *treeNode) int {
|
||||||
return strings.Compare(a.path, b.path)
|
return strings.Compare(a.path, b.path)
|
||||||
})
|
})
|
||||||
|
|||||||
+116
-16
@@ -6,6 +6,7 @@ import (
|
|||||||
"database/sql"
|
"database/sql"
|
||||||
"os"
|
"os"
|
||||||
"slices"
|
"slices"
|
||||||
|
"strconv"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -179,7 +180,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
|
|||||||
t.Fatalf("directory paths = %q, want %q", got, want)
|
t.Fatalf("directory paths = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
|
|
||||||
groups := collectTreeGroups(dirs, super, nil)
|
groups := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||||
|
|
||||||
gotGroups := groupPaths(groups)
|
gotGroups := groupPaths(groups)
|
||||||
wantGroups := [][]string{{"/a", "/c"}}
|
wantGroups := [][]string{{"/a", "/c"}}
|
||||||
@@ -195,7 +196,8 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func TestRunTreesEscapesPaths(t *testing.T) {
|
func TestRunTreesEscapesPaths(t *testing.T) {
|
||||||
t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
|
dir, recs := recsOnDisk(t, awkwardPairRecs())
|
||||||
|
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||||
|
|
||||||
var stdout, stderr bytes.Buffer
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
@@ -206,7 +208,7 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||||
`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\t4096\n"
|
dir + `/d/\tone\ntwo\rthree\\four` + "\t" + dir + "/d/A\t1\t5\t4096\n"
|
||||||
if got := stdout.String(); got != want {
|
if got := stdout.String(); got != want {
|
||||||
t.Errorf("stdout = %q, want %q", got, want)
|
t.Errorf("stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
@@ -224,9 +226,11 @@ func TestRunTreesDiskSize(t *testing.T) {
|
|||||||
{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
|
{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
|
||||||
}
|
}
|
||||||
|
|
||||||
|
dir, recs := recsOnDisk(t, recs)
|
||||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||||
|
|
||||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n/a\t/b\t2\t30\t1536\n"
|
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||||
|
dir + "/a\t" + dir + "/b\t2\t30\t1536\n"
|
||||||
if got := runStdout(t, cmdTrees); got != want {
|
if got := runStdout(t, cmdTrees); got != want {
|
||||||
t.Errorf("stdout = %q, want %q", got, want)
|
t.Errorf("stdout = %q, want %q", got, want)
|
||||||
}
|
}
|
||||||
@@ -237,10 +241,11 @@ func TestRunTreesDiskSize(t *testing.T) {
|
|||||||
// there. After the line naming the database, each phase prints a line
|
// there. After the line naming the database, each phase prints a line
|
||||||
// naming it with its place among the four as it starts and another as
|
// naming it with its place among the four as it starts and another as
|
||||||
// it ends. read counts the 6 records before it reads them, group counts
|
// it ends. read counts the 6 records before it reads them, group counts
|
||||||
// the 8 directories, and write the 1 group. The summary comes last, and
|
// the directories, and write the 1 group. The summary comes last, and
|
||||||
// stdout holds the report alone.
|
// stdout holds the report alone.
|
||||||
func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
||||||
path := seedDatabase(t, smokeTreeRecs())
|
dir, recs := recsOnDisk(t, smokeTreeRecs())
|
||||||
|
path := seedDatabase(t, recs)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
stderr := captureStderr(t)
|
stderr := captureStderr(t)
|
||||||
@@ -254,11 +259,15 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||||
"/d/t1\t/d/t2\t2\t3100\t8192\n"
|
dir + "/d/t1\t" + dir + "/d/t2\t2\t3100\t8192\n"
|
||||||
if got := stdout.String(); got != report {
|
if got := stdout.String(); got != report {
|
||||||
t.Errorf("stdout = %q, want %q", got, report)
|
t.Errorf("stdout = %q, want %q", got, report)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The smoke-test records name 8 directories, / included. Moved under
|
||||||
|
// dir, they name one more for each slash in dir.
|
||||||
|
dirs := strconv.Itoa(8 + strings.Count(dir, "/"))
|
||||||
|
|
||||||
// How each line starts; the rate at the end of a phase varies.
|
// How each line starts; the rate at the end of a phase varies.
|
||||||
want := []string{
|
want := []string{
|
||||||
"trees: database " + path + " (from " + databaseEnv + ")",
|
"trees: database " + path + " (from " + databaseEnv + ")",
|
||||||
@@ -266,8 +275,9 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
|||||||
treesOpenLine,
|
treesOpenLine,
|
||||||
"[2/4] read: counting the records, elapsed 0s, remain unknown",
|
"[2/4] read: counting the records, elapsed 0s, remain unknown",
|
||||||
"[2/4] read: [6/6] 100%, ",
|
"[2/4] read: [6/6] 100%, ",
|
||||||
"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, remain unknown",
|
"[3/4] group: [0/" + dirs + "] 0%, 0 directories/s, elapsed 0s, " +
|
||||||
"[3/4] group: [8/8] 100%, ",
|
"remain unknown",
|
||||||
|
"[3/4] group: [" + dirs + "/" + dirs + "] 100%, ",
|
||||||
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown",
|
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown",
|
||||||
"[4/4] write: [1/1] 100%, ",
|
"[4/4] write: [1/1] 100%, ",
|
||||||
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
|
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
|
||||||
@@ -296,7 +306,7 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
|||||||
func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
||||||
stderr := captureStderr(t)
|
stderr := captureStderr(t)
|
||||||
super, dirs := treeOf(t, smokeTreeRecs())
|
super, dirs := treeOf(t, smokeTreeRecs())
|
||||||
dupes := collectTreeGroups(dirs, super, nil)
|
dupes := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||||
|
|
||||||
// No timed draws: the display is drawn as it starts and as it ends.
|
// No timed draws: the display is drawn as it starts and as it ends.
|
||||||
p := startProgress(progressParams{
|
p := startProgress(progressParams{
|
||||||
@@ -377,7 +387,7 @@ func TestCollectTreeGroupsMaximal(t *testing.T) {
|
|||||||
|
|
||||||
super, dirs := treeOf(t, smokeTreeRecs())
|
super, dirs := treeOf(t, smokeTreeRecs())
|
||||||
|
|
||||||
groups := collectTreeGroups(dirs, super, nil)
|
groups := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||||
|
|
||||||
want := [][]string{{"/d/t1", "/d/t2"}}
|
want := [][]string{{"/d/t1", "/d/t2"}}
|
||||||
if got := groupPaths(groups); !slices.EqualFunc(got, want,
|
if got := groupPaths(groups); !slices.EqualFunc(got, want,
|
||||||
@@ -401,14 +411,16 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
|
|||||||
|
|
||||||
super, dirs := treeOf(t, recs)
|
super, dirs := treeOf(t, recs)
|
||||||
|
|
||||||
forward := groupPaths(collectTreeGroups(dirs, super, nil))
|
forward := groupPaths(collectTreeGroups(dirs, super, everyPathExists,
|
||||||
|
nil))
|
||||||
|
|
||||||
reversed := slices.Clone(recs)
|
reversed := slices.Clone(recs)
|
||||||
slices.Reverse(reversed)
|
slices.Reverse(reversed)
|
||||||
|
|
||||||
superR, dirsR := treeOf(t, reversed)
|
superR, dirsR := treeOf(t, reversed)
|
||||||
|
|
||||||
backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
|
backward := groupPaths(collectTreeGroups(dirsR, superR, everyPathExists,
|
||||||
|
nil))
|
||||||
if !slices.EqualFunc(forward, backward, slices.Equal) {
|
if !slices.EqualFunc(forward, backward, slices.Equal) {
|
||||||
t.Fatalf("output depends on record order: %v vs %v",
|
t.Fatalf("output depends on record order: %v vs %v",
|
||||||
forward, backward)
|
forward, backward)
|
||||||
@@ -428,7 +440,7 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
|
|||||||
|
|
||||||
super, dirs := treeOf(t, recs)
|
super, dirs := treeOf(t, recs)
|
||||||
|
|
||||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
|
||||||
|
|
||||||
want := [][]string{{"/p/x1", "/p/x2"}}
|
want := [][]string{{"/p/x1", "/p/x2"}}
|
||||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||||
@@ -457,7 +469,7 @@ func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
|
|||||||
|
|
||||||
super, dirs := treeOf(t, recs)
|
super, dirs := treeOf(t, recs)
|
||||||
|
|
||||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
|
||||||
|
|
||||||
want := [][]string{{"/p/x1", "/p/x2"}}
|
want := [][]string{{"/p/x1", "/p/x2"}}
|
||||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||||
@@ -480,10 +492,98 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
|||||||
|
|
||||||
super, dirs := treeOf(t, recs)
|
super, dirs := treeOf(t, recs)
|
||||||
|
|
||||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
|
||||||
|
|
||||||
want := [][]string{{"/p/a/x", "/q/b/x"}}
|
want := [][]string{{"/p/a/x", "/q/b/x"}}
|
||||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||||
t.Fatalf("groups = %v, want %v", got, want)
|
t.Fatalf("groups = %v, want %v", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestCollectTreeGroupsChecksOnlyReportedCopies checks that trees asks
|
||||||
|
// whether a directory is on disk only for the members of the groups it
|
||||||
|
// returns. It never asks about /a/proj/sub, which lies inside a copy
|
||||||
|
// and whose group the group of /a/proj implies.
|
||||||
|
func TestCollectTreeGroupsChecksOnlyReportedCopies(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f, g, u := testHash("f"), testHash("g"), testHash("u")
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/a/proj/f"},
|
||||||
|
{size: 20, head: g, tail: g, content: g, path: "/a/proj/sub/g"},
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/b/proj/f"},
|
||||||
|
{size: 20, head: g, tail: g, content: g, path: "/b/proj/sub/g"},
|
||||||
|
// /a lacks this file, so /a and /b are no copies of each other.
|
||||||
|
{size: 30, head: u, tail: u, content: u, path: "/b/lonely"},
|
||||||
|
}
|
||||||
|
|
||||||
|
super, dirs := treeOf(t, recs)
|
||||||
|
|
||||||
|
var checked []string
|
||||||
|
|
||||||
|
exists := func(path string) bool {
|
||||||
|
checked = append(checked, path)
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
|
||||||
|
|
||||||
|
want := [][]string{{"/a/proj", "/b/proj"}}
|
||||||
|
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||||
|
t.Fatalf("groups = %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
|
||||||
|
slices.Sort(checked)
|
||||||
|
|
||||||
|
if !slices.Equal(checked, want[0]) {
|
||||||
|
t.Errorf("directories checked on disk = %q, want %q",
|
||||||
|
checked, want[0])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCollectTreeGroupsDropsGroupLeftWithOneCopy checks that a group of
|
||||||
|
// two copies, one of them gone, is not returned. If kept, it would
|
||||||
|
// write no row, but the summary line would count it.
|
||||||
|
func TestCollectTreeGroupsDropsGroupLeftWithOneCopy(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f := testHash("f")
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/c/kept/f"},
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/c/gone/f"},
|
||||||
|
}
|
||||||
|
|
||||||
|
super, dirs := treeOf(t, recs)
|
||||||
|
|
||||||
|
exists := func(path string) bool { return path != "/c/gone" }
|
||||||
|
|
||||||
|
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
|
||||||
|
if len(got) != 0 {
|
||||||
|
t.Errorf("groups = %v, want none", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies checks that a group
|
||||||
|
// of three copies, one of them gone, is returned with the other two.
|
||||||
|
func TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f := testHash("f")
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/d/kept1/f"},
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/d/gone/f"},
|
||||||
|
{size: 10, head: f, tail: f, content: f, path: "/d/kept2/f"},
|
||||||
|
}
|
||||||
|
|
||||||
|
super, dirs := treeOf(t, recs)
|
||||||
|
|
||||||
|
exists := func(path string) bool { return path != "/d/gone" }
|
||||||
|
|
||||||
|
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
|
||||||
|
|
||||||
|
want := [][]string{{"/d/kept1", "/d/kept2"}}
|
||||||
|
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||||
|
t.Errorf("groups = %v, want %v", got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in new issue
Block a user