Compare commits
19
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
1477c57024 | ||
|
|
a27d651532 | ||
|
|
a1ec1e0ca4 | ||
|
|
6c529fcc6d | ||
|
|
202f083cc0 | ||
|
|
1017332d68 | ||
|
|
5c7284c351 | ||
|
|
7cb10ed303 | ||
|
|
da75b713aa | ||
|
|
24b0fc486d | ||
|
|
f52bbe5d95 | ||
|
|
30dae35c51 | ||
|
|
4ecf8dc484 | ||
|
|
2322f8bb02 | ||
|
|
4ed2e370b1 | ||
|
|
98193b2896 | ||
|
|
eebf579f11 | ||
|
|
8a4616039b | ||
|
|
c2804d088b |
No files matched your search
@@ -4,7 +4,7 @@ export CGO_ENABLED = 0
|
||||
|
||||
BINARY := sfdupes
|
||||
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
|
||||
LDFLAGS := -X main.Version=$(VERSION)
|
||||
LDFLAGS := -X sneak.berlin/go/sfdupes/internal/sfdupes.Version=$(VERSION)
|
||||
|
||||
.PHONY: sfdupes build bootstrap setup test lint fmt fmt-check check docker hooks clean
|
||||
|
||||
@@ -14,7 +14,7 @@ LDFLAGS := -X main.Version=$(VERSION)
|
||||
# Default target: build the binary. Phony so go build (which has its
|
||||
# own build cache) always decides what to recompile.
|
||||
sfdupes:
|
||||
go build -ldflags "$(LDFLAGS)" -o $(BINARY)
|
||||
go build -ldflags "$(LDFLAGS)" -o $(BINARY) ./cmd/sfdupes
|
||||
|
||||
build: sfdupes
|
||||
|
||||
|
||||
@@ -17,12 +17,14 @@ gigabyte. A large file is never read in full. A match between two large files is
|
||||
a strong candidate, not proof.
|
||||
|
||||
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every
|
||||
file takes too long. It has four subcommands. `scan` walks the filesystem and
|
||||
file takes too long. It has five subcommands. `scan` walks the filesystem and
|
||||
records the files it finds in an SQLite database. SQLite keeps a whole database
|
||||
in one file. The database is kept between runs. A rescan reads only new and
|
||||
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
||||
system's job scheduler. `report`, `trees` and `info` read the database alone and
|
||||
can run at any time.
|
||||
system's job scheduler. `report`, `trees` and `info` work from the database and
|
||||
can run at any time. `report` and `trees` leave out a copy that was deleted
|
||||
after the scan. `dwim` scans `/` and then prints the `trees` report, for a first
|
||||
look at a new machine with one command.
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -38,9 +40,11 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
||||
`scan` walks the trees named by its operands, the paths given after `scan` on
|
||||
the command line. The walk is the listing of every directory under them. `scan`
|
||||
keeps one database record per regular file. A regular file is an ordinary file,
|
||||
as opposed to a directory, a symlink or a device. `report` prints the file-level
|
||||
duplicates. `trees` prints the duplicate directory trees. `info` prints the
|
||||
database's path, size and record count.
|
||||
as opposed to a directory, a symlink or a device node. A symlink is a file that
|
||||
holds the path of another file and stands in for it. A device node is a file
|
||||
that stands for a piece of hardware, such as a disk or a terminal. `report`
|
||||
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
||||
`info` prints the database's path, size and record count.
|
||||
|
||||
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
||||
another file. The first `scan` creates it.
|
||||
@@ -54,24 +58,25 @@ shell shows both on the terminal, and `>` redirects stdout alone.
|
||||
With Go installed, this builds and installs the newest tagged release:
|
||||
|
||||
```sh
|
||||
go install sneak.berlin/go/sfdupes@latest
|
||||
go install sneak.berlin/go/sfdupes/cmd/sfdupes@latest
|
||||
```
|
||||
|
||||
`@main` installs the tip of the `main` branch instead:
|
||||
|
||||
```sh
|
||||
go install sneak.berlin/go/sfdupes@main
|
||||
go install sneak.berlin/go/sfdupes/cmd/sfdupes@main
|
||||
```
|
||||
|
||||
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
|
||||
|
||||
`sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from
|
||||
`go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
|
||||
a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
|
||||
string for an untagged commit. That string ends in the commit's hash. A binary
|
||||
built from a clone carries the output of `git describe --tags --always --dirty`
|
||||
in that clone, such as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no
|
||||
version is known. To see everything Go recorded in an installed binary:
|
||||
a release tag. For `@main` it is the commit's tag when it has one. Otherwise it
|
||||
is a pseudo-version, Go's version string for an untagged commit. That string
|
||||
ends in the commit's hash. A binary built from a clone carries the output of
|
||||
`git describe --tags --always --dirty` in that clone, such as `v0.0.3` on a
|
||||
tagged commit. `VERSION` is `dev` when no version is known. To see everything Go
|
||||
recorded in an installed binary:
|
||||
|
||||
```sh
|
||||
go version -m "$(command -v sfdupes)"
|
||||
@@ -94,20 +99,23 @@ Each `-v` flag below makes a bind mount, which shows a host directory at a path
|
||||
inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between
|
||||
runs. Mount the tree to scan at the same path inside the container as on the
|
||||
host. Read-only access is enough. The database then records the host's paths,
|
||||
and so do the reports.
|
||||
and so do the reports. `report` and `trees` need that mount too. Without it,
|
||||
every recorded path counts as gone, and each report holds only its header line.
|
||||
|
||||
```sh
|
||||
make docker
|
||||
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
||||
sfdupes scan /srv
|
||||
docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
|
||||
docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
||||
sfdupes report > dupes.tsv
|
||||
```
|
||||
|
||||
### Daily scan from `cron`
|
||||
|
||||
Run `scan` as root, so that it can read every file. A path that it cannot read
|
||||
is skipped with a warning and loses its database record. As the file
|
||||
`/etc/cron.d/sfdupes`:
|
||||
Run `scan` as root, so that it can read every file. `scan` warns about a
|
||||
directory it cannot read, and about a new or changed file it cannot read. It
|
||||
skips the path and deletes the records at or below it (see "Rules for the
|
||||
walk"). As the file `/etc/cron.d/sfdupes`:
|
||||
|
||||
```
|
||||
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
||||
@@ -123,8 +131,9 @@ is skipped with a warning and loses its database record. As the file
|
||||
- a warning for each path it skips;
|
||||
- a line for each system or temporary directory the walk leaves out (see
|
||||
"Rules for the walk");
|
||||
- a line for each path left out because its filesystem is one the kernel
|
||||
makes up or keeps in memory, such as `tmpfs`;
|
||||
- a line for each path left out because its filesystem is one the kernel,
|
||||
the core of the operating system, makes up or keeps in memory, such as
|
||||
`tmpfs`;
|
||||
- the summary line;
|
||||
- a warning when a report still has the database open as the scan ends (see
|
||||
"Concurrent reads").
|
||||
@@ -153,8 +162,10 @@ to keep is your choice.
|
||||
|
||||
A row is a candidate, not proof:
|
||||
|
||||
- The reports show the files as they were at the last scan. A file may have
|
||||
changed or gone since.
|
||||
- The reports show each file as it was when a scan last recorded it. They check
|
||||
only that each copy still exists: the file, or the directory at the top of a
|
||||
tree. A file may have changed since, and a file inside a tree may be gone. Run
|
||||
`scan` again to bring the reports up to date.
|
||||
- A file of 100 MiB or more is compared only on samples of its content (see
|
||||
"Duplicate detection").
|
||||
- Two paths that are hard links, two names for one file, are listed as
|
||||
@@ -201,9 +212,10 @@ two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
|
||||
makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
|
||||
of reads, at the price of a comparison that is not exact.
|
||||
|
||||
The hashes live in a database that is kept between runs. A rescan reads only the
|
||||
files that are new or whose size or modification time (mtime) changed. All
|
||||
analysis then works from the database alone.
|
||||
The hashes live in a database that is kept between runs. A rescan reads only new
|
||||
files, and files whose size or modification time (mtime) differs from their
|
||||
record. All analysis then works from the database. The reports only check that
|
||||
each copy they name still exists.
|
||||
|
||||
The goal is whole duplicated trees, not only single files. A download extracted
|
||||
twice, or a project copied once too often, is such a tree. It can be removed as
|
||||
@@ -224,15 +236,15 @@ In order:
|
||||
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
|
||||
in full. The target is a filesystem of about ten million files and 150 TB, on
|
||||
disks that may be slow or busy. During a scan, one small record per file is
|
||||
held in memory. It holds the path and the two values the walk compares, the
|
||||
size and the mtime. The hashes stay in the database. `report` holds the
|
||||
duplicate paths of one file size at a time. `trees` holds one entry per
|
||||
directory, plus the files of the directories it is reading.
|
||||
held in memory. The hashes stay in the database. `report` holds the duplicate
|
||||
paths of one file size at a time. `trees` holds one entry per directory, plus
|
||||
the files of the directories it is reading.
|
||||
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
|
||||
between runs, and an unchanged file is never read again. `report`, `trees`
|
||||
and `info` work from the database alone and never touch the scanned
|
||||
filesystem. `scan` is built to run from `cron`. The reports run at any time,
|
||||
against what the database holds.
|
||||
and `info` work from the database and never read a scanned file. `report` and
|
||||
`trees` check only that each copy they name still exists, and leave out a
|
||||
copy that is gone. `scan` is built to run from `cron`. The reports run at any
|
||||
time, against what the database holds.
|
||||
4. **Keep the streams apart.** Everything on stdout is machine-readable data.
|
||||
Everything for a human reader, such as progress and warnings, goes to stderr.
|
||||
|
||||
@@ -243,12 +255,15 @@ In order:
|
||||
- `report` prints the file-level duplicate report from the database.
|
||||
- `trees` prints the duplicate-tree report from the database.
|
||||
- `info` prints the database's details.
|
||||
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
|
||||
report.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes info
|
||||
sfdupes dwim
|
||||
sfdupes --version
|
||||
sfdupes [command] --help
|
||||
```
|
||||
@@ -261,11 +276,14 @@ The reference below is also the program's specification.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No byte-for-byte compare, and no deleting or linking of duplicates. The
|
||||
reports are advisory. Acting on them is your job.
|
||||
- No byte-for-byte compare.
|
||||
- No deleting of duplicates.
|
||||
- No replacing a copy with a hard link to another copy. The reports are
|
||||
advisory. Acting on them is your job.
|
||||
- The SQLite database is the only stored data. There are no export or import
|
||||
formats.
|
||||
- No daemon or filesystem watcher. Scheduling rescans is `cron`'s job.
|
||||
- `sfdupes` does not run in the background or watch the filesystem for changes.
|
||||
Scheduling rescans is `cron`'s job.
|
||||
|
||||
## Reference
|
||||
|
||||
@@ -280,9 +298,9 @@ The reference below is also the program's specification.
|
||||
hashes them. The default is the number of CPUs. `N` must be at least 1. A
|
||||
smaller value is a usage error, reported in one line on stderr with exit 2
|
||||
before anything is scanned.
|
||||
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand on that
|
||||
operand's filesystem, as the same flag does for `du` and `rsync`. By default
|
||||
the walk also enters directories on other filesystems.
|
||||
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
|
||||
directories on other filesystems, as the same flag does for `du` and `rsync`.
|
||||
By default the walk also enters directories on other filesystems.
|
||||
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
|
||||
Those are the system and temporary directories the walk otherwise leaves out,
|
||||
such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||
@@ -341,16 +359,26 @@ default. An operand may be a directory or a regular file.
|
||||
|
||||
- `scan` with no operand is a usage error. The usage text goes to stderr, and
|
||||
the exit code is 2.
|
||||
- An operand that does not exist is a fatal error. `scan` then stops before it
|
||||
walks anything, and exits 1.
|
||||
- Each operand is made absolute before the walk. A `.` or `..` in it is resolved
|
||||
as text, and symlinks are not followed. Each record is stored under that
|
||||
absolute path, so results do not depend on the working directory.
|
||||
- An operand that does not exist is a fatal error, and so is a symlink whose
|
||||
target does not exist. `scan` then stops before it walks anything, and exits
|
||||
with code 1.
|
||||
- Each operand is resolved to its real path before the walk, as `realpath`
|
||||
prints it. A relative operand is joined to the working directory as `pwd -P`
|
||||
prints it. Every symlink in the path is then followed. A `..` after a symlink
|
||||
counts from the directory the symlink leads to. From then on the operand is
|
||||
that real path. Its files are recorded under it, and warnings name it.
|
||||
- An operand that is a symlink is scanned at its target's real path. After a
|
||||
`cd` through a symlink, `scan .` records its files under the real path of the
|
||||
working directory.
|
||||
- All operands belong to one scan, and one walk covers them all at once (see
|
||||
"Concurrency").
|
||||
- An operand that repeats another is dropped before the walk, without a line.
|
||||
- An operand that repeats another is dropped before the walk. Two operands
|
||||
repeat each other when their real paths are the same.
|
||||
- An operand under another operand is dropped the same way, unless one of the
|
||||
cases below applies. No path is then walked twice.
|
||||
cases below applies. No path is then walked twice. Whether one operand is
|
||||
under another is judged on their real paths. With `/srv/link` a symlink to a
|
||||
directory outside `/srv`, `scan /srv /srv/link/sub` walks `sub` on its own,
|
||||
because the real path of `sub` is not under `/srv`.
|
||||
- An operand under another is kept, and walked on its own, when the inner
|
||||
operand or a directory between the two is one of these:
|
||||
- a default exclude (see "Rules for the walk");
|
||||
@@ -359,28 +387,27 @@ default. An operand may be a directory or a regular file.
|
||||
It is also kept when a directory between the two is on a filesystem that is
|
||||
not persistent. Such a filesystem is one the kernel makes up or keeps in
|
||||
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
||||
such a directory, so the inner operand is walked on its own. For an inner
|
||||
operand that is a regular file, the check starts at its directory.
|
||||
such a directory, so the inner operand is walked on its own.
|
||||
|
||||
- No other case keeps an operand under another. An operand reached through a
|
||||
symlinked directory below the outer operand is dropped too, without a line.
|
||||
The outer operand's walk does not follow the symlink, so no file under the
|
||||
inner operand is scanned. The records that an earlier scan left under it are
|
||||
deleted.
|
||||
- An operand that is neither a regular file nor a directory is not scanned.
|
||||
`scan` prints a warning naming the path and what it is. It counts the operand
|
||||
as skipped and drops it. An operand beneath a dropped one is scanned all the
|
||||
same. The scan keeps the records beneath a dropped operand, unless another
|
||||
operand covers them.
|
||||
- No other case keeps an operand under another.
|
||||
- An operand that is neither a regular file nor a directory is not scanned. Such
|
||||
an operand is, for example, a symlink to a device node. `scan` prints a
|
||||
warning naming the real path and what it is. It counts the operand as skipped
|
||||
and drops it. The scan keeps the records beneath a dropped operand, unless
|
||||
another operand covers them.
|
||||
- An operand at or inside a directory named `.zfs` is never scanned, whether
|
||||
given alone or under another operand. The check looks at the path as given and
|
||||
at the path with its symlinks resolved. `scan` prints a warning naming the
|
||||
path, counts the operand as skipped and drops it. The records beneath it are
|
||||
kept, unless it lies under another operand. Then they are deleted, because
|
||||
that operand's walk never enters `.zfs` (see "Rules for the walk").
|
||||
given alone or under another operand. The check looks at the real path, so a
|
||||
symlink into a `.zfs` directory is not scanned either. `scan` prints a warning
|
||||
naming the real path, counts the operand as skipped and drops it. The records
|
||||
beneath it are kept, unless it lies under another operand. Then they are
|
||||
deleted, because that operand's walk never enters `.zfs` (see "Rules for the
|
||||
walk").
|
||||
- An operand on a filesystem that is not persistent is dropped too. `scan`
|
||||
prints a line naming it and its filesystem type, and does not count it as
|
||||
skipped.
|
||||
- `scan` prints these warnings, and the line for an operand that is not on a
|
||||
persistent filesystem, even when the operand repeats another or lies under
|
||||
another.
|
||||
- A scan whose every operand is dropped walks nothing and exits 0.
|
||||
|
||||
#### What a scan does to the database
|
||||
@@ -390,17 +417,22 @@ operands:
|
||||
|
||||
- A file not yet in the database is hashed and inserted.
|
||||
- A file already in the database is not read again when its size and mtime match
|
||||
its record. The mtime matches when it is not newer than the recorded one. This
|
||||
is what makes a daily rescan cheap.
|
||||
- A file whose size differs, or whose mtime is newer, is hashed again. The mtime
|
||||
is compared to the nanosecond, as finely as the filesystem records it. A
|
||||
same-size rewrite counts as a change whenever the filesystem gives it a later
|
||||
mtime, even within the same second.
|
||||
its record. This is what makes a daily rescan cheap.
|
||||
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
|
||||
updates `disk_size` alone. It still does not read the file, and counts it as
|
||||
unchanged. ZFS, for example, counts a new file's blocks up to seconds after
|
||||
the write. A block is the unit in which a filesystem allocates disk space. A
|
||||
file scanned within those seconds gets its real `disk_size` at the next scan.
|
||||
- `scan` hashes a file again when its size or its mtime differs from its record.
|
||||
An older mtime counts as much as a newer one. A file put back by `cp -p`,
|
||||
`rsync -a` or a restore from backup often has an older mtime. The mtime is
|
||||
compared to the nanosecond, as finely as the filesystem records it.
|
||||
- The scan deletes each record under a scanned operand whose file it could not
|
||||
verify. That covers deleted files, and paths the scan could not stat or read
|
||||
this run. To stat a file is to read its details, such as its size, without
|
||||
opening it. So the database holds only what the latest scan over each path
|
||||
could verify. A later successful scan adds such files again.
|
||||
opening it. So after a complete scan, the database holds only what that scan
|
||||
could verify under its operands. A later successful scan adds such files
|
||||
again.
|
||||
- Records outside the scanned operands are left alone, so separate trees can be
|
||||
scanned on different schedules into one database.
|
||||
- The scan keeps every record at or below a path that the walk leaves out for
|
||||
@@ -411,9 +443,8 @@ operands:
|
||||
|
||||
Such records count as outside the scanned operands. So with `-x`, a
|
||||
filesystem mounted under an operand can be scanned on its own schedule into
|
||||
the same database. An operand may lie at or inside a default exclude, or,
|
||||
with `-x`, at or inside a directory on another filesystem. Such an operand
|
||||
is walked on its own. Its walk verifies the records below it, so they are
|
||||
the same database. An operand may itself lie at or inside such a path, as
|
||||
"Operands" describes. Its walk verifies the records below it, so they are
|
||||
deleted when their files are gone.
|
||||
|
||||
#### Phases
|
||||
@@ -422,14 +453,17 @@ operands:
|
||||
phases.
|
||||
|
||||
1. `load` reads the existing records. Each record under the operands is kept in
|
||||
memory by path, with its size and mtime, for the walk to compare against.
|
||||
memory by path, with its size, its size on disk and its mtime, for the walk
|
||||
to compare against.
|
||||
2. `walk` lists the trees under all operands. Several workers share one queue of
|
||||
directories to read. A worker is one thread of the program, and "Concurrency"
|
||||
says how many there are. Each worker reads one directory at a time. It hands
|
||||
the subdirectories it finds back to the queue, and it stats each regular file
|
||||
as it finds it. The walk compares each file with its record. An unchanged
|
||||
file needs nothing more. Every new or changed file is passed to the hash
|
||||
phase.
|
||||
directories to read. A worker is one thread of the program. A thread is one
|
||||
of several parts of a program that run at the same time and share its memory.
|
||||
"Concurrency" says how many workers there are. Each worker reads one
|
||||
directory at a time. It hands the subdirectories it finds back to the queue,
|
||||
and it stats each regular file as it finds it. The walk compares each file
|
||||
with its record. An unchanged file is not read again. When its size on disk
|
||||
has changed, the update phase writes its new `disk_size`. Every new or
|
||||
changed file is passed to the hash phase.
|
||||
3. `hash` reads every new or changed file and computes its three hashes. Files
|
||||
are read in inode order. An inode is the filesystem's own record of a file.
|
||||
Files with nearby inode numbers usually lie near each other on disk. Reading
|
||||
@@ -438,8 +472,9 @@ phases.
|
||||
tree made of hard links therefore costs one read per file. Completed records
|
||||
are written to the database 10,000 at a time while hashing runs. So a scan
|
||||
interrupted after hours keeps everything hashed so far.
|
||||
4. `update` writes the hashed records not yet written, then deletes the records
|
||||
the scan did not verify.
|
||||
4. `update` writes the hashed records not yet written, and the new `disk_size`
|
||||
of each unchanged file whose size on disk changed. Then it deletes the
|
||||
records the scan did not verify.
|
||||
|
||||
Reading a tree one directory at a time waits on the disk at every step and takes
|
||||
hours at millions of files. Reading many directories at once is what makes the
|
||||
@@ -448,7 +483,7 @@ walk practical on large or busy storage pools.
|
||||
#### Rules for the walk
|
||||
|
||||
- Only regular files are recorded. Every other kind of file, such as a symlink
|
||||
or a device node, is passed over. Symlinks are never followed.
|
||||
or a device node, is passed over. The walk follows no symlink it finds.
|
||||
- The walk never enters a directory named `.zfs`. Such a directory holds the
|
||||
filesystem's ZFS snapshots, which are its saved earlier states. Walking it
|
||||
would list every file once per snapshot. `scan` never starts a walk at or
|
||||
@@ -462,18 +497,18 @@ walk practical on large or busy storage pools.
|
||||
The walk knows them by device ID and inode number. The device ID is the
|
||||
number by which the kernel tells filesystems apart. The inode number tells
|
||||
the files of one filesystem apart. So the walk passes over them whatever
|
||||
path leads to them. The walk follows no symlink it finds. It does open each
|
||||
operand at the path given, even when that path runs through a symlinked
|
||||
directory. Such an operand can lead to the database files, and the walk
|
||||
passes over them there as well. The walk prints no line for the database
|
||||
files and does not count them as skipped. A record that an earlier scan left
|
||||
for one of them is deleted.
|
||||
path leads to them. The walk follows no symlink it finds. It opens each
|
||||
operand at its real path, not the path given. An operand given through a
|
||||
symlink can lead to the database files, and the walk passes over them there
|
||||
as well. The walk prints no line for the database files and does not count
|
||||
them as skipped. A record that an earlier scan left for one of them is
|
||||
deleted.
|
||||
|
||||
- By default the walk enters directories on other filesystems than its
|
||||
operand's. With `-x`, it never enters a directory on a different filesystem
|
||||
than its operand. `scan` prints no line for that directory and does not count
|
||||
it as skipped. The records below it are kept, as described under "What a scan
|
||||
does to the database".
|
||||
than its operand. `scan` prints no line for a directory that only `-x` leaves
|
||||
out, and does not count it as skipped. The records below it are kept, as
|
||||
described under "What a scan does to the database".
|
||||
- The walk never records anything on a filesystem that is not persistent. A
|
||||
filesystem is not persistent when the kernel makes it up or keeps it in
|
||||
memory. These are the types, as `sfdupes scan --help` lists them on Linux and
|
||||
@@ -509,11 +544,10 @@ walk practical on large or busy storage pools.
|
||||
The path is not counted as skipped. The records below it are kept, as
|
||||
described under "What a scan does to the database".
|
||||
|
||||
- Below an operand, the walk does not enter a default exclude. The default
|
||||
excludes hold the kernel's view of the running system and temporary files, so
|
||||
that `sfdupes scan /` walks only the files worth comparing. They depend on the
|
||||
system, and `sfdupes scan --help` lists those of the system it runs on. On
|
||||
Linux they are:
|
||||
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
||||
default excludes hold temporary files and the kernel's view of the running
|
||||
system. They depend on the system, and `sfdupes scan --help` lists those of
|
||||
the system it runs on. On Linux they are:
|
||||
|
||||
```
|
||||
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
||||
@@ -542,17 +576,22 @@ walk practical on large or busy storage pools.
|
||||
scan: skipped /var/tmp (default exclude)
|
||||
```
|
||||
|
||||
The directory is not counted as skipped. A default exclude on a filesystem
|
||||
that is not persistent, such as `/proc`, gets the line for its filesystem
|
||||
instead. One that is itself an operand gets no line, because it is walked as
|
||||
that operand. The scan keeps the records below a default exclude that the
|
||||
The directory is not counted as skipped. `scan` prints the filesystem line
|
||||
instead for a default exclude that is not on a persistent filesystem, such
|
||||
as `/proc`. One that is itself an operand gets no line, because it is walked
|
||||
as that operand. The scan keeps the records below a default exclude that the
|
||||
walk did not enter, as described under "What a scan does to the database".
|
||||
|
||||
- A path that cannot be read, for lack of permission or any other error, gets a
|
||||
one-line warning on stderr and is skipped. The scan continues, and the summary
|
||||
line says how many paths were skipped. A skipped path that has a record from
|
||||
an earlier scan loses that record, and so does every path under an unreadable
|
||||
directory. The database mirrors what the latest scan could verify.
|
||||
- `scan` skips each of these, with a one-line warning on stderr:
|
||||
- a directory it cannot read;
|
||||
- a path it cannot stat;
|
||||
- a new or changed file it cannot read.
|
||||
|
||||
An unchanged file is not opened, so it keeps its record even when it has
|
||||
become unreadable. The scan continues, and the summary line says how many
|
||||
paths were skipped. A skipped path that has a record from an earlier scan
|
||||
loses that record, and so does every path under an unreadable directory.
|
||||
|
||||
- A file or directory that disappears while the scan runs is not an error. The
|
||||
scan prints no line for it and does not count it as skipped. A path gone when
|
||||
the walk reaches it gets no record, and so does a file gone when the hash
|
||||
@@ -575,8 +614,8 @@ from a thread of its own and never blocks the workers.
|
||||
|
||||
- one line naming the database (see "Database");
|
||||
- for each dropped operand, in operand order, the warning or the line described
|
||||
under "Operands". An operand dropped as a repeat of another, or as lying under
|
||||
another, gets neither;
|
||||
under "Operands". `scan` prints neither for an operand it drops only because
|
||||
it repeats another or lies under another;
|
||||
- the progress display of each phase (see "Progress"). `scan` prints each
|
||||
warning for a skipped path as it happens. It prints a line the same way for
|
||||
each path left out as a default exclude or as not on a persistent filesystem;
|
||||
@@ -605,12 +644,12 @@ signal `kill` sends by default:
|
||||
- Nothing else is written or deleted. Records are deleted only after a complete
|
||||
walk. So the records under paths that an interrupted walk never reached are
|
||||
kept.
|
||||
- The database is closed and the lock released, as on any other exit.
|
||||
- The database is closed and the lock released.
|
||||
- The line `scan: interrupted after N files` goes to stderr. `N` is the number
|
||||
of files the walk reached. The exit code is 1.
|
||||
|
||||
The next scan reads none of the files already recorded and brings the database
|
||||
up to date as usual.
|
||||
The next scan reads none of the unchanged files already recorded, and brings the
|
||||
database up to date as usual.
|
||||
|
||||
After the first signal `scan` stops catching signals. A second one ends it at
|
||||
once, as any uncaught signal does. The records not yet written are lost, and the
|
||||
@@ -622,30 +661,39 @@ and `scan` leaves it ignored.
|
||||
|
||||
`report` reads every record from the database. It takes no positional arguments.
|
||||
A positional argument is anything on the command line that is not a flag, such
|
||||
as the paths `scan` takes. `report` never touches the scanned filesystem. It
|
||||
neither stats nor opens any path in the records. It reads the database and
|
||||
writes to stdout and stderr, and nothing else. Its output is the same whether or
|
||||
not the scanned filesystem is still mounted.
|
||||
as the paths `scan` takes. `report` never opens a scanned file. It checks that
|
||||
each path in a group still exists, and looks at no other path. The check is
|
||||
`lstat`, the system call that looks up a path without following a symlink:
|
||||
|
||||
- A path is gone when `lstat` finds that it, or a directory above it, does not
|
||||
exist.
|
||||
- Any other error, such as a permission error, counts as present. So a user who
|
||||
may not look inside a directory still sees its copies.
|
||||
|
||||
The check does not notice a file that changed after the scan. Run `scan` again
|
||||
to bring the report up to date.
|
||||
|
||||
Processing:
|
||||
|
||||
- Records are grouped by signature, the four values
|
||||
`(size, head, tail, content)`.
|
||||
- Every group with two or more paths is a duplicate group.
|
||||
- In a group with two or more paths, a path that is gone is left out. A group
|
||||
that still has two or more paths is a duplicate group.
|
||||
- Empty files form no group, because removing one frees nothing.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
every other path is a `dupe`.
|
||||
- Groups are ordered by size, largest first, then by `first` path. The output is
|
||||
the same for the same database contents.
|
||||
the same for the same database contents and the same paths on disk.
|
||||
|
||||
`report` has two phases, each with its own progress display on stderr (see
|
||||
"Progress"). `open` opens the database. `read` counts the records, then reads
|
||||
them in signature order, largest size first. The database keeps an index named
|
||||
`files_signature`, a sorted list of the records by signature (see "Schema").
|
||||
Reading it backwards gives that order, so SQLite sorts nothing and returns the
|
||||
first record at once. Records with one signature come together, and so do the
|
||||
records of one size. So `report` holds only the duplicate groups of the size it
|
||||
is reading. Once it reaches a smaller size, it writes them, ordered by `first`
|
||||
path.
|
||||
"Progress"). `open` opens the database. `read` gets the number of records from
|
||||
the `totals` table (see "Schema"), then reads them in signature order, largest
|
||||
size first. The database keeps an index named `files_signature`, a sorted list
|
||||
of the records by signature (see "Schema"). Reading it backwards gives that
|
||||
order, so SQLite sorts nothing and returns the first record at once. Records
|
||||
with one signature come together, and so do the records of one size. So `report`
|
||||
holds only the duplicate groups of the size it is reading. Once it reaches a
|
||||
smaller size, it writes them, ordered by `first` path.
|
||||
|
||||
#### Report output format
|
||||
|
||||
@@ -680,7 +728,7 @@ The summary line on stderr gives:
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
|
||||
```
|
||||
report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable
|
||||
report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
||||
```
|
||||
|
||||
### `trees` mode
|
||||
@@ -688,8 +736,9 @@ report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable
|
||||
`trees` reads the same database as `report` and takes no positional arguments.
|
||||
It reports whole duplicate directory trees. Two directories are duplicate trees
|
||||
when they hold the same relative paths, and each path holds a file with the same
|
||||
signature in both. Like `report`, it never touches the scanned filesystem. It
|
||||
rebuilds the directory tree from the paths in the records, split on `/`.
|
||||
signature in both. It rebuilds the directory tree from the paths in the records,
|
||||
split on `/`. Like `report`, it never opens a scanned file. It checks only that
|
||||
the directory at the top of each copy still exists, as `report` checks a path.
|
||||
|
||||
Definitions:
|
||||
|
||||
@@ -708,15 +757,21 @@ Processing:
|
||||
- `trees` builds the directory tree and computes every directory's digest. It
|
||||
then groups the directories by digest. Every group with two or more
|
||||
directories is a duplicate-tree group.
|
||||
- Only the outermost duplicate trees are reported. A group is left out when its
|
||||
members' parents are distinct directories that all have the same digest. The
|
||||
parents' group, or an ancestor's, already implies it. A group holding sibling
|
||||
directories, or members whose parents have different digests, is always
|
||||
reported.
|
||||
- `trees` leaves out a group whose trees hold 0 bytes, because removing one
|
||||
frees nothing. An empty file still counts toward the digest of a tree that
|
||||
holds other files.
|
||||
- `trees` leaves out a group that a group of its members' parents already
|
||||
implies. A group is left out when its members' parents are distinct
|
||||
directories that all have the same digest. The parents' group, or an
|
||||
ancestor's, already implies it. A group is always reported when it holds
|
||||
sibling directories, or when its members' parents have different digests.
|
||||
- A directory that is gone is dropped from its group. A group left with fewer
|
||||
than two directories is not reported.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
every other path is a `dupe`.
|
||||
- Groups are ordered by total tree size, largest first, then by `first` path.
|
||||
The output is the same for the same database contents.
|
||||
The output is the same for the same database contents and the same paths on
|
||||
disk.
|
||||
|
||||
Two limitations are accepted:
|
||||
|
||||
@@ -724,26 +779,40 @@ Two limitations are accepted:
|
||||
They share storage, so removing one frees nothing. The scan uses the inode to
|
||||
avoid reading a file twice, and does not store it.
|
||||
- Only the regular files in the database define a tree. Empty directories are
|
||||
invisible. A file skipped during the scan in one copy but not in the other
|
||||
makes two otherwise identical trees differ.
|
||||
invisible. When the scan skips a file in one copy but not in the other, the
|
||||
two trees differ.
|
||||
|
||||
`trees` has four phases, each with its own progress display on stderr (see
|
||||
"Progress"). `open` opens the database. `read` counts the records, then reads
|
||||
them in path order. It builds the directory tree and computes the digests as it
|
||||
goes. `group` groups the directories by digest and picks the groups to report.
|
||||
`write` writes them to stdout.
|
||||
"Progress"). `open` opens the database. `read` gets the number of records from
|
||||
the `totals` table, then reads them in path order. It builds the directory tree
|
||||
and computes the digests as it goes. `group` groups the directories by digest,
|
||||
picks the groups to report and checks that their directories still exist.
|
||||
`write` writes the groups to stdout.
|
||||
|
||||
#### Trees output format
|
||||
|
||||
The report is TSV on stdout, with a header line and then one row per duplicate
|
||||
tree. A group of N trees gives N-1 rows. `files` is the number of regular files
|
||||
in one copy of the tree, and `size` their total size in bytes:
|
||||
tree. A group of N trees gives N-1 rows. After the `first` and `dupe` paths come
|
||||
three numbers:
|
||||
|
||||
- `files`, the number of regular files in one copy of the tree;
|
||||
- `size`, their total size in bytes;
|
||||
- `disk_size`, the bytes the `dupe` copy takes on disk, its files' `disk_size`
|
||||
added up.
|
||||
|
||||
```
|
||||
first dupe files size
|
||||
/srv/a/project /srv/backup/project 3417 104857600
|
||||
first dupe files size disk_size
|
||||
/srv/a/project /srv/backup/project 3417 104857600 61865984
|
||||
```
|
||||
|
||||
A file's `disk_size` is the space its blocks take on disk. It can be more than
|
||||
the file's size, because space is allocated in whole blocks. It is less than the
|
||||
file's size on a compressed or sparse file. A sparse file has gaps that take no
|
||||
space on disk. So two copies of one tree can differ in `disk_size`. When a file
|
||||
in one copy is a hard link to one in the other, both copies count its blocks in
|
||||
full. Removing it from one copy then frees nothing, as the limitation under
|
||||
"`trees` mode" says.
|
||||
|
||||
The paths are escaped as described under "Report output format". The root
|
||||
directory's path is `/`.
|
||||
|
||||
@@ -758,6 +827,33 @@ The summary line on stderr gives:
|
||||
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
|
||||
```
|
||||
|
||||
### `dwim` mode
|
||||
|
||||
`dwim`, short for "do what I mean", scans a new machine and reports its
|
||||
duplicate trees in one command. It takes no positional arguments and no flags of
|
||||
its own. Like `scan`, it should run as root, so that it can read every file.
|
||||
|
||||
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
|
||||
excludes and crosses into other filesystems. `--workers` keeps its default. The
|
||||
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
|
||||
what `sfdupes trees` runs. The largest trees come first.
|
||||
|
||||
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
|
||||
`scan` prints, then what `trees` prints, then one last line:
|
||||
|
||||
```
|
||||
dwim: run "sfdupes trees" to print this report again
|
||||
```
|
||||
|
||||
The database keeps the scan, so `sfdupes trees` prints the report again later,
|
||||
even after the terminal is gone. It then leaves out a copy deleted in between.
|
||||
|
||||
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
|
||||
`dwim` then prints no report and exits 1. A fatal scan error also ends it before
|
||||
the report, with the same message and exit code as `scan`. Warnings for skipped
|
||||
files do not stop it. `dwim` stops catching signals when the scan ends, so a
|
||||
signal during the report ends it at once, as it ends `trees`.
|
||||
|
||||
### `info` mode
|
||||
|
||||
`info` takes no positional arguments and prints the database's details on
|
||||
@@ -790,15 +886,15 @@ size of recorded files 150000000000000 (136.4 TiB)
|
||||
of all recorded files, in bytes, then rounded to a unit such as TiB.
|
||||
|
||||
`info` has two phases, each with its own progress display on stderr (see
|
||||
"Progress"). `open` opens the database. `count` reads the schema version and
|
||||
counts the records in one transaction. A transaction is a group of reads or
|
||||
writes that SQLite carries out as one step, against the database as it stands at
|
||||
one moment. So both describe the database at one moment, even while a scan is
|
||||
writing. `info` writes its lines after the `count` display has ended.
|
||||
"Progress"). `open` opens the database. `count` reads the schema version and the
|
||||
row of `totals` (see "Schema") in one transaction. A transaction is a group of
|
||||
reads or writes that SQLite carries out as one step, against the database as it
|
||||
stands at one moment. So both describe the database at one moment, even while a
|
||||
scan is writing. `info` writes its lines after the `count` display has ended.
|
||||
|
||||
### Database
|
||||
|
||||
All four subcommands work on one SQLite database file.
|
||||
All five subcommands work on one SQLite database file.
|
||||
|
||||
#### Location
|
||||
|
||||
@@ -814,6 +910,8 @@ scan: new database /var/lib/sfdupes/db.sqlite (default)
|
||||
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
|
||||
```
|
||||
|
||||
`dwim` prints the `scan` line first, and the `trees` line after the scan.
|
||||
|
||||
A relative path is printed with the working directory in front of it. The
|
||||
working directory is printed as `pwd -P` prints it, with its symlinks resolved.
|
||||
The rest of the path is printed as set, so a `..` in it stays. The line names
|
||||
@@ -836,20 +934,21 @@ the error for a missing database comes after it.
|
||||
|
||||
#### The scan lock
|
||||
|
||||
Only one `scan` runs against a database at a time.
|
||||
Only one `scan` at a time can hold the lock.
|
||||
|
||||
- For its whole run, `scan` holds a lock on a lock file beside the database. The
|
||||
lock is taken with the `flock(2)` system call, and only one process can hold
|
||||
it. The lock file's name is the database's path, with symlinks resolved, plus
|
||||
`.lock`. By default that is `/var/lib/sfdupes/db.sqlite.lock`. Every path to
|
||||
one database therefore names the same lock file.
|
||||
lock is taken with the `flock(2)` system call, a request a program makes to
|
||||
the kernel, and only one process can hold it. The lock file's name is the
|
||||
database's path, with symlinks resolved, plus `.lock`. By default that is
|
||||
`/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same
|
||||
lock file.
|
||||
- `scan` takes the lock before it walks the filesystem or opens the database.
|
||||
Just before that, it creates the database file, empty, if it is missing. So
|
||||
the lock file is named the same way before the first scan as after. A symlink
|
||||
at the database path is followed, and its target is created.
|
||||
- A second `scan` against the same database does not wait. It fails at once with
|
||||
an error naming the lock file, and exits 1. It has walked nothing and opened
|
||||
nothing. The running scan carries on.
|
||||
- A second `scan` that finds the lock held does not wait. It fails at once with
|
||||
an error naming the lock file, and exits 1. It has walked nothing and has not
|
||||
opened the database. The running scan carries on.
|
||||
- The lock file is created on first use and never deleted. Only its owner may
|
||||
open it. A leftover file blocks nothing. The lock ends with the process that
|
||||
holds it, however that process ends. Deleting the file while a scan runs would
|
||||
@@ -862,8 +961,9 @@ Only one `scan` runs against a database at a time.
|
||||
SQLite appends changes to a `-wal` file beside the database, and keeps a
|
||||
lookup table for that file in a `-shm` file. Readers do not block the writer,
|
||||
so running a report during a scan is safe.
|
||||
- Every connection has a busy timeout of 10 seconds. That is how long it waits
|
||||
for a database another connection has locked before it gives up.
|
||||
- Every subcommand opens the database with a busy timeout of 10 seconds. That is
|
||||
how long it waits for a database another process has locked before it gives
|
||||
up.
|
||||
- The filesystem is authoritative, and the database follows it. Hashed records
|
||||
are written 10,000 at a time while the scan runs, so a report may see a scan's
|
||||
changes partly applied. A scan that dies partway leaves a valid database
|
||||
@@ -888,31 +988,36 @@ Only one `scan` runs against a database at a time.
|
||||
- Then, before the walk as well, it writes:
|
||||
- the `-wal` and `-shm` files, which stay beside the database while the scan
|
||||
runs;
|
||||
- in a new database, the table, the index and the schema version described
|
||||
under "Schema".
|
||||
- in a new database, the tables, the index, the triggers and the schema
|
||||
version described under "Schema".
|
||||
- During the `load` and `walk` phases, no records are written. A first scan's
|
||||
database holds no records until the hash phase, however long its walk runs.
|
||||
- During the `hash` phase, records are written 10,000 at a time.
|
||||
- During the `update` phase, the rest of the hashed records are written, and the
|
||||
deletions are made.
|
||||
- During the `update` phase, the rest of the hashed records are written, and so
|
||||
is the new `disk_size` of each unchanged file whose size on disk changed. Then
|
||||
the deletions are made.
|
||||
- As `scan` closes the database, the `-wal` and `-shm` files go away, unless a
|
||||
report still has the database open.
|
||||
|
||||
#### Schema
|
||||
|
||||
The schema is the database's layout, its table and its index.
|
||||
`PRAGMA user_version` holds the schema version, currently 1. A database with any
|
||||
other version is a fatal error. `scan` creates the table and the index and sets
|
||||
the version in one transaction (see "`info` mode"). So the three land together
|
||||
or not at all. A first scan stopped while doing so leaves an empty database,
|
||||
which the next scan sets up. A database at version 0 that already has a `files`
|
||||
table was not made by `sfdupes`. Every subcommand refuses it, with an error
|
||||
telling you to remove the file and rescan.
|
||||
The schema is the database's layout: its tables, its index and its triggers. A
|
||||
trigger is a statement SQLite runs by itself when a table changes.
|
||||
`PRAGMA user_version` holds the schema version, currently 1. An empty database
|
||||
is at version 0, and `scan` sets it up. Any other version is a fatal error.
|
||||
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
|
||||
error too. `scan` creates the tables, the index and the triggers and sets the
|
||||
version in one transaction (see "`info` mode"). So they all land together or not
|
||||
at all. A first scan stopped while doing so leaves an empty database, which the
|
||||
next scan sets up. A database at version 0 that already has a `files` table was
|
||||
not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
|
||||
remove the file and rescan.
|
||||
|
||||
```sql
|
||||
CREATE TABLE files (
|
||||
path BLOB PRIMARY KEY, -- absolute path, raw bytes
|
||||
size INTEGER NOT NULL, -- bytes
|
||||
disk_size INTEGER NOT NULL, -- bytes allocated on disk (st_blocks × 512)
|
||||
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime
|
||||
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second
|
||||
head BLOB NOT NULL, -- raw 32-byte SHA-256 of the first 64 KiB
|
||||
@@ -920,10 +1025,28 @@ CREATE TABLE files (
|
||||
content BLOB NOT NULL -- raw 32-byte SHA-256; whole file under 100 MiB, else samples
|
||||
) WITHOUT ROWID;
|
||||
CREATE INDEX files_signature ON files (size, head, tail, content);
|
||||
CREATE TABLE totals (
|
||||
records INTEGER NOT NULL, -- number of records in files
|
||||
size INTEGER NOT NULL -- sizes of all records in files added up
|
||||
);
|
||||
INSERT INTO totals (records, size) VALUES (0, 0);
|
||||
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
|
||||
UPDATE totals SET records = records + 1, size = size + NEW.size;
|
||||
END;
|
||||
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
|
||||
UPDATE totals SET records = records - 1, size = size - OLD.size;
|
||||
END;
|
||||
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
|
||||
UPDATE totals SET size = size + NEW.size - OLD.size;
|
||||
END;
|
||||
```
|
||||
|
||||
- `path` is a BLOB, a column of raw bytes, because Unix paths are bytes and not
|
||||
necessarily UTF-8 text.
|
||||
- `disk_size` is the space the file's blocks take on disk. `lstat` is the system
|
||||
call that stats a file without following a symlink. Its `st_blocks` field
|
||||
counts that space in units of 512 bytes. `disk_size` is less than `size` on a
|
||||
compressed or sparse file (see "Trees output format").
|
||||
- `mtime` holds the whole seconds of the file's mtime, rounded down, and
|
||||
`mtime_nsec` the nanoseconds past that second. Split this way, they hold any
|
||||
mtime a filesystem can record, even one before 1678 or after 2262. One 64-bit
|
||||
@@ -934,18 +1057,29 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
|
||||
not 32 bytes.
|
||||
- `files_signature` lets `report` read the records in signature order without
|
||||
sorting the table.
|
||||
- `totals` holds one row: the number of records in `files` and the sum of their
|
||||
sizes. Three triggers on `files` keep that row current. Each runs in the
|
||||
transaction of the change to `files` that fires it, so the row and the records
|
||||
always agree. `info` reads both numbers from this row. The `load` phase of
|
||||
`scan` and the `read` phase of `report` and `trees` read the number of records
|
||||
from it as their total. Counting the records in `files` would read every
|
||||
record. That takes minutes on a database of tens of millions. The triggers
|
||||
make each insert, each delete and each change of a record's `size` slower, so
|
||||
a first scan of many small files takes about two fifths longer. On large
|
||||
files, reading them takes most of the time, so the difference is small.
|
||||
|
||||
### Progress
|
||||
|
||||
Every subcommand always shows that it is working. Each phase gets a display of
|
||||
its own on stderr from the moment it starts. The display names the phase and
|
||||
shows its progress. The parts of its line are listed below.
|
||||
Each phase of every subcommand gets a display of its own on stderr from the
|
||||
moment it starts. The display names the phase and shows its progress. The parts
|
||||
of its line are listed below.
|
||||
|
||||
The phases, and what each display counts:
|
||||
|
||||
- `scan` has four phases:
|
||||
- `[1/4] load` counts the records first, then reads them against that count.
|
||||
While it counts, the display says `counting the records`.
|
||||
- `[1/4] load` gets the number of records from the `totals` table first,
|
||||
then reads them against that number. Until it has the number, the display
|
||||
says `counting the records`.
|
||||
- `[2/4] walk` has no exact total. Its estimate is the number of records
|
||||
under the operands, shown with a `~`. A scan with no records under its
|
||||
operands, such as a first scan, has no estimate. The time remaining is
|
||||
@@ -957,8 +1091,9 @@ The phases, and what each display counts:
|
||||
- `report` has two phases:
|
||||
- `[1/2] open` has no total. The display says `opening the database`, also
|
||||
while it waits on a busy one.
|
||||
- `[2/2] read` counts the records first, then reads them against that count.
|
||||
It writes rows to stdout as it goes.
|
||||
- `[2/2] read` gets the number of records from the `totals` table first,
|
||||
then reads them against that number. Until it has the number, the display
|
||||
says `counting the records`. It writes rows to stdout as it goes.
|
||||
- `trees` has four phases:
|
||||
- `[1/4] open` and `[2/4] read` are as for `report`, except that `read`
|
||||
writes nothing.
|
||||
@@ -966,9 +1101,8 @@ The phases, and what each display counts:
|
||||
- `[4/4] write` has the duplicate-tree groups as its total.
|
||||
- `info` has two phases:
|
||||
- `[1/2] open` is as for `report`.
|
||||
- `[2/2] count` has no total either, because one request to the database
|
||||
counts everything. The display says `counting the records` until that
|
||||
request returns.
|
||||
- `[2/2] count` has no total either, because it reads one row. The display
|
||||
says `counting the records` until it has the row.
|
||||
|
||||
Each display is one line holding:
|
||||
|
||||
@@ -1000,13 +1134,14 @@ phase waits, in place of the rate. That happens while one large file is read, or
|
||||
while a disk or the database is slow. Each phase names the places it may wait
|
||||
in, and a phase that may wait in two places names both, as `disk/db`:
|
||||
|
||||
- `walk` says `disk`, for files.
|
||||
- `walk` says `disk`, because it reads directories and stats files.
|
||||
- `hash` says `disk/db`, because it reads files and writes their records to the
|
||||
database between them.
|
||||
- `load`, `update` and the `read` phase of `trees` say `db`.
|
||||
- The `read` phase of `report` says `db/out`, because it writes the report to
|
||||
stdout while it reads.
|
||||
- `group` of `trees` says `cpu`, for work in memory, and its `write` says `out`.
|
||||
- The `read` phase of `report` says `db/disk/out`. It checks that the paths of
|
||||
each group still exist, and writes the report to stdout, while it reads.
|
||||
- `group` of `trees` says `cpu/disk`, for work in memory and the check that each
|
||||
copy still exists. Its `write` says `out`.
|
||||
- A phase still waiting for its total says what it waits for, and nothing more.
|
||||
|
||||
On a terminal:
|
||||
@@ -1028,11 +1163,10 @@ On a terminal:
|
||||
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
||||
```
|
||||
|
||||
- No line reaches the terminal's last column, however narrow the terminal is. A
|
||||
line too wide for the terminal first leaves out its rate, the words of a
|
||||
stalled phase, or what the phase is waiting for. It leaves out its bar next.
|
||||
It is cut short at the edge only when the rest does not fit on its own. The
|
||||
rest is:
|
||||
- No line reaches the terminal's last column. A line too wide for the terminal
|
||||
first leaves out its rate, the words of a stalled phase, or what the phase is
|
||||
waiting for. It leaves out its bar next. It is cut short at the edge only when
|
||||
the rest does not fit on its own. The rest is:
|
||||
- the phase and its place;
|
||||
- the progress, when the phase shows one;
|
||||
- the elapsed time;
|
||||
@@ -1056,22 +1190,24 @@ Each phase prints its line:
|
||||
- every 5 seconds while it runs, whether or not an item has completed;
|
||||
- once more as it ends.
|
||||
|
||||
A phase that stops short of its total, as an interrupted one does, keeps showing
|
||||
the count it stopped at.
|
||||
When a phase stops short of its total, as an interrupted one does, it keeps
|
||||
showing the count it stopped at.
|
||||
|
||||
### Error handling and exit codes
|
||||
|
||||
- `0`: success, even when files were skipped with warnings.
|
||||
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal
|
||||
error is one of these:
|
||||
- an operand that does not exist, or whose details `scan` cannot read;
|
||||
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
|
||||
stopped by either during its scan. A fatal error is one of these:
|
||||
- an operand that does not exist, or whose real path `scan` cannot find;
|
||||
- a list of mounted filesystems that `scan` cannot read;
|
||||
- another `scan` already running against the same database;
|
||||
- a database that cannot be created or opened, or whose reads or writes
|
||||
fail, its directory and its lock file included;
|
||||
- another `scan` holding the lock (see "The scan lock");
|
||||
- a database, its directory or its lock file that cannot be created or
|
||||
opened;
|
||||
- a read or write of the database that fails;
|
||||
- a relative database path when the working directory cannot be read;
|
||||
- a database with another schema version, or one at version 0 that already
|
||||
has a `files` table;
|
||||
- a database at a schema version other than 0 or 1;
|
||||
- a database at version 0 read by `report`, `trees` or `info`;
|
||||
- a database at version 0 that already has a `files` table (see "Schema");
|
||||
- a record with a hash that is not 32 bytes, read by `report` or `trees`;
|
||||
- a missing database for `report`, `trees` or `info`;
|
||||
- a failed write to stdout.
|
||||
@@ -1082,7 +1218,7 @@ the count it stopped at.
|
||||
`--workers abc`;
|
||||
- `scan` with no operand;
|
||||
- `--workers` below 1;
|
||||
- `report`, `trees` or `info` with a positional argument.
|
||||
- `report`, `trees`, `info` or `dwim` with a positional argument.
|
||||
|
||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||
stdout, such as a full disk, is reported the same way. Two cases never reach
|
||||
@@ -1120,8 +1256,8 @@ the repository root itself, so it can be run from any directory.
|
||||
- An installed `node` is used when its major version is 22. A `yarn` already
|
||||
on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
|
||||
activated through `corepack`, the `yarn` installer that ships with `node`.
|
||||
When there is no `corepack` either, `npm`, the package manager that ships
|
||||
with `node`, installs it.
|
||||
When there is no `corepack` either, the script installs `yarn` with `npm`,
|
||||
the package manager that ships with `node`.
|
||||
- Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
|
||||
version manager, and activates `yarn` 1.22.22 under that `node` through
|
||||
`corepack`. When `nvm` is missing, the script first downloads it and
|
||||
@@ -1151,9 +1287,9 @@ the repository root itself, so it can be run from any directory.
|
||||
- `script/lint` runs the linter, `golangci-lint`, which checks the Go sources
|
||||
for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
|
||||
writes no image. The stage starts from the `golangci/golangci-lint` image,
|
||||
pinned by its digest, and runs three build steps:
|
||||
- the `gofmt` check, which fails on a Go source that `gofmt`, Go's own
|
||||
formatter, would change;
|
||||
pinned by its digest, and runs these checks as build steps:
|
||||
- the `gofmt` check, which fails when `gofmt`, Go's own formatter, would
|
||||
change a Go source;
|
||||
- `golangci-lint config verify`, which checks the linter's settings file
|
||||
`.golangci.yml`;
|
||||
- `golangci-lint run`, which lints the Go sources.
|
||||
@@ -1187,8 +1323,8 @@ the repository root itself, so it can be run from any directory.
|
||||
every check passed. The tests and the linter run twice, once in `script/check`
|
||||
and again as stages of the image build.
|
||||
- `script/precommit` runs from the `git` pre-commit hook, a script that `git`
|
||||
runs before each commit. It fails the commit when `go mod tidy` would change
|
||||
anything. Then it runs `script/check`.
|
||||
runs before each commit. It runs `go mod tidy`, and fails the commit when that
|
||||
changes `go.mod` or `go.sum`. Then it runs `script/check`.
|
||||
- `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
|
||||
`script/precommit`.
|
||||
|
||||
@@ -1245,8 +1381,8 @@ commands of their own:
|
||||
not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
|
||||
cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
|
||||
exception, because the race detector needs cgo (see `script/test`).
|
||||
- `report` and `trees` give the same output for the same database contents,
|
||||
whatever order the records were inserted in.
|
||||
- `report` and `trees` give the same output for the same database contents and
|
||||
the same paths on disk, whatever order the records were inserted in.
|
||||
|
||||
### Definition of done
|
||||
|
||||
@@ -1260,6 +1396,7 @@ All of the following, run in this directory, must pass:
|
||||
data.
|
||||
|
||||
```sh
|
||||
make build
|
||||
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
|
||||
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
|
||||
mkdir -p "$d/a" "$d/b"
|
||||
@@ -1269,7 +1406,7 @@ All of the following, run in this directory, must pass:
|
||||
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
||||
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
||||
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
||||
: > "$d/empty1"; : > "$d/empty2" # empty duplicates
|
||||
: > "$d/empty1"; : > "$d/empty2" # empty files
|
||||
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
||||
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
||||
head -c 3000 /dev/urandom > "$d/t1/f1"
|
||||
@@ -1294,21 +1431,23 @@ All of the following, run in this directory, must pass:
|
||||
is the path that sorts first.
|
||||
- `t1/f1`, `t2/f1` and `t3/f1` form one group.
|
||||
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
|
||||
- `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`.
|
||||
- `unique.bin` and `tiny3` appear nowhere.
|
||||
- `tiny1` and `tiny2` form a pair.
|
||||
- `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
|
||||
- The groups come in order of size, largest first.
|
||||
|
||||
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
|
||||
The row shows 2 files and 3100 bytes. The two `sub` directories are left
|
||||
out, because the `t1` and `t2` group implies them. `t3` appears nowhere,
|
||||
since one of its file names differs.
|
||||
The row shows 2 files, 3100 bytes, and a `disk_size` that depends on the
|
||||
filesystem. The two `sub` directories are left out, because the `t1` and
|
||||
`t2` group implies them. `t3` appears nowhere, since one of its file names
|
||||
differs.
|
||||
|
||||
After the rescan, the second `report` shows `one.bin` out of its group,
|
||||
because its content changed. `copy.bin` and `copy2.bin` remain as one pair.
|
||||
`unique.bin` is gone from the database.
|
||||
|
||||
The test suite automates this scenario (see `scan_test.go`). It also checks
|
||||
that `report` and `trees` never touch the scanned filesystem.
|
||||
The test suite automates this scenario (see
|
||||
`internal/sfdupes/scan_test.go`). It also checks that `report` and `trees`
|
||||
leave out a copy removed after the scan.
|
||||
|
||||
## TODO
|
||||
|
||||
|
||||
@@ -0,0 +1,29 @@
|
||||
// Command sfdupes quickly identifies candidate duplicate files across
|
||||
// very large filesystems without reading every byte of every file.
|
||||
// Files are considered duplicates when their sizes are equal and they
|
||||
// agree on three SHA-256 hashes: of the first 64 KiB, of the last 64
|
||||
// KiB, and of the content. The content hash covers the whole file under
|
||||
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
||||
// file is never read in full. scan maintains a persistent SQLite
|
||||
// database of file signatures (SFDUPES_DATABASE, default
|
||||
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
// sfdupes report > dupes.tsv
|
||||
// sfdupes trees > dupetrees.tsv
|
||||
// sfdupes info
|
||||
// sfdupes dwim
|
||||
// sfdupes --version
|
||||
//
|
||||
// See README.md for the complete specification.
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"sneak.berlin/go/sfdupes/internal/sfdupes"
|
||||
)
|
||||
|
||||
func main() { os.Exit(sfdupes.Main()) }
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // cancels the unexported runScan and hashWorker
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -1,23 +1,6 @@
|
||||
// Command sfdupes quickly identifies candidate duplicate files across
|
||||
// very large filesystems without reading every byte of every file.
|
||||
// Files are considered duplicates when their sizes are equal and they
|
||||
// agree on three SHA-256 hashes: of the first 64 KiB, of the last 64
|
||||
// KiB, and of the content. The content hash covers the whole file under
|
||||
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
||||
// file is never read in full. scan maintains a persistent SQLite
|
||||
// database of file signatures (SFDUPES_DATABASE, default
|
||||
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
// sfdupes report > dupes.tsv
|
||||
// sfdupes trees > dupetrees.tsv
|
||||
// sfdupes info
|
||||
// sfdupes --version
|
||||
//
|
||||
// See README.md for the complete specification.
|
||||
package main
|
||||
// Package sfdupes implements the sfdupes command and the database its
|
||||
// subcommands share.
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -45,6 +28,7 @@ const (
|
||||
cmdReport = "report"
|
||||
cmdTrees = "trees"
|
||||
cmdInfo = "info"
|
||||
cmdDwim = "dwim"
|
||||
)
|
||||
|
||||
// errNoSubcommand is returned by the root command when it is invoked
|
||||
@@ -84,20 +68,22 @@ func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
|
||||
return info.Main.Version
|
||||
}
|
||||
|
||||
func main() {
|
||||
// Main runs the command line in os.Args and returns the process exit
|
||||
// code.
|
||||
func Main() int {
|
||||
// Once the reader of a stdout pipe has gone, as in "sfdupes report |
|
||||
// head", the Go runtime ends the process with SIGPIPE on the next
|
||||
// write instead of returning an error (README "Error handling").
|
||||
// Registering for SIGPIPE with os/signal would change that.
|
||||
os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))
|
||||
return run(os.Args[1:], os.Stdout, os.Stderr)
|
||||
}
|
||||
|
||||
// run executes args against the command tree and returns the process
|
||||
// exit code. It is the program's single exit point: the subcommands
|
||||
// return their errors instead of exiting, so every deferred cleanup —
|
||||
// above all closing the database, which checkpoints the SQLite WAL —
|
||||
// runs before the process ends. The report, trees and info subcommands
|
||||
// write their data to stdout.
|
||||
// runs before the process ends. The report, trees, info and dwim
|
||||
// subcommands write their data to stdout.
|
||||
func run(args []string, stdout, stderr io.Writer) int {
|
||||
// A nil slice makes cobra fall back to os.Args, which would let a
|
||||
// test binary's own flags reach the command tree.
|
||||
@@ -201,7 +187,16 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
}),
|
||||
}
|
||||
|
||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
|
||||
dwimCmd := &cobra.Command{
|
||||
Use: cmdDwim,
|
||||
Short: "Scan / with the default excludes, then print the duplicate-tree report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runDwim(ctx, "/", stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
|
||||
|
||||
return root
|
||||
}
|
||||
@@ -220,10 +215,13 @@ func newScanCommand() *cobra.Command {
|
||||
short := "Walk trees and synchronize the scan database"
|
||||
|
||||
scanCmd := &cobra.Command{
|
||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
||||
Short: short,
|
||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
|
||||
// The Use line already names every flag, so cobra must not
|
||||
// append its generic "[flags]" to it.
|
||||
DisableFlagsInUseLine: true,
|
||||
Short: short,
|
||||
Long: short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
|
||||
Args: cobra.MinimumNArgs(1),
|
||||
PreRunE: func(cmd *cobra.Command, _ []string) error {
|
||||
return checkScanWorkers(cmd, scanWorkers)
|
||||
},
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // drives the unexported run
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -14,6 +14,8 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// usageMarker is the first line of cobra's usage text, which a usage
|
||||
@@ -180,9 +182,9 @@ func capture(t *testing.T, std **os.File) func() string {
|
||||
}
|
||||
|
||||
// brokenDatabase writes a database that opens cleanly and passes the
|
||||
// schema-version check but has no files table, so the first query
|
||||
// fails with the database already open: a fatal error on a path that
|
||||
// owns an open database. It closes the database the way scan does.
|
||||
// schema-version check but has no tables, so the first query fails
|
||||
// with the database already open: a fatal error on a path that owns an
|
||||
// open database. It closes the database the way scan does.
|
||||
func brokenDatabase(t *testing.T) string {
|
||||
t.Helper()
|
||||
|
||||
@@ -265,8 +267,9 @@ func TestRunFatalAfterOpenClosesDatabase(t *testing.T) {
|
||||
assertFatalOutput(t, stderr.String(), stdout())
|
||||
|
||||
// Proof that the failure happened after the open: only a
|
||||
// query against the opened database can report this.
|
||||
if !strings.Contains(stderr.String(), "no such table: files") {
|
||||
// query against the opened database can report this. Each
|
||||
// subcommand's first query reads totals.
|
||||
if !strings.Contains(stderr.String(), "no such table: totals") {
|
||||
t.Errorf("stderr = %q, want the failure to come from a "+
|
||||
"query on the open database", stderr.String())
|
||||
}
|
||||
@@ -619,6 +622,7 @@ func TestRunUsageErrors(t *testing.T) {
|
||||
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
||||
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
||||
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
||||
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
|
||||
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
||||
"unknown subcommand": {[]string{"nope"}, "unknown command"},
|
||||
}
|
||||
@@ -710,6 +714,34 @@ func TestRunHelp(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunScanUsageLine(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// README §Subcommands shows this usage line for scan, and the help
|
||||
// and the usage error both print it.
|
||||
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...\n"
|
||||
|
||||
cases := map[string][]string{
|
||||
"help": {cmdScan, "-h"},
|
||||
"missing paths": {cmdScan},
|
||||
}
|
||||
|
||||
for name, args := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
run(args, &stdout, &stderr)
|
||||
|
||||
if !strings.Contains(stderr.String(), want) {
|
||||
t.Errorf("run(%v) stderr = %q, want the usage line %q",
|
||||
args, stderr.String(), want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunVersion(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -796,7 +828,7 @@ func TestRunVersionWriteFailureIsFatal(t *testing.T) {
|
||||
func scanFixture(t *testing.T) []string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
|
||||
dupes := []string{
|
||||
writeFile(t, dir, "one/a.bin", pattern(1, 300)),
|
||||
@@ -817,6 +849,23 @@ func scanFixture(t *testing.T) []string {
|
||||
return dupes
|
||||
}
|
||||
|
||||
// storedDiskSize returns the disk_size of the record for path in the
|
||||
// database at dbPath. A test of a trees run compares with this rather
|
||||
// than with a new lstat, which on ZFS can differ from the scan's (see
|
||||
// diskSizeOf).
|
||||
func storedDiskSize(t *testing.T, dbPath, path string) int64 {
|
||||
t.Helper()
|
||||
|
||||
db, err := openReportDatabase(t.Context(), dbPath)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
return recordByPath(t, dbRecords(t, db), path).diskSize
|
||||
}
|
||||
|
||||
// scanOK runs scan over operands, fails the test unless it exits 0 with
|
||||
// nothing on stdout, and returns everything it printed to stderr.
|
||||
func scanOK(t *testing.T, operands ...string) string {
|
||||
@@ -849,7 +898,7 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
writeFile(t, dir, "a.bin", pattern(1, 300))
|
||||
|
||||
// Same size as a.bin, so the scan reads it, and the read fails.
|
||||
@@ -877,58 +926,278 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
||||
assertNoSidecars(t, path)
|
||||
}
|
||||
|
||||
func TestRunScanSkipsSymlinkOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
// realTempDir returns a new temporary directory by its real path, so
|
||||
// tests hold where the temporary directory is reached through a
|
||||
// symlink.
|
||||
func realTempDir(t *testing.T) string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||
|
||||
link := filepath.Join(dir, "link")
|
||||
|
||||
err := os.Symlink(filepath.Join(dir, "target"), link)
|
||||
dir, err := filepath.EvalSymlinks(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Scanning a directory through the symlink stores a record beneath
|
||||
// the symlink's own path for a file beneath its target.
|
||||
scanOK(t, filepath.Join(link, "sub"))
|
||||
|
||||
assertOperandSkipped(t, path, link, "symlink",
|
||||
filepath.Join(link, "sub", "f"))
|
||||
return dir
|
||||
}
|
||||
|
||||
// symlink creates link pointing at target.
|
||||
func symlink(t *testing.T, target, link string) {
|
||||
t.Helper()
|
||||
|
||||
err := os.Symlink(target, link)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanFollowsSymlinkOperand scans link, a symlink to target.
|
||||
// The operand is scanned at its real path, and its file is recorded
|
||||
// there.
|
||||
func TestRunScanFollowsSymlinkOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||
|
||||
link := filepath.Join(dir, "link")
|
||||
symlink(t, filepath.Join(dir, "target"), link)
|
||||
|
||||
stderr := scanOK(t, link)
|
||||
|
||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 0 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanWalksOperandUnderSymlinkOperand scans link and link/sub,
|
||||
// with link a symlink to target. Both resolve under target, so
|
||||
// target/sub is walked once, as part of target.
|
||||
func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||
|
||||
link := filepath.Join(dir, "link")
|
||||
symlink(t, filepath.Join(dir, "target"), link)
|
||||
|
||||
err := os.Symlink(filepath.Join(dir, "target"), link)
|
||||
stderr := scanOK(t, link, filepath.Join(link, "sub"))
|
||||
|
||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 0 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanWalksSymlinkedOperandBelowOperand is the case of
|
||||
// https://git.eeqj.de/sneak/sfdupes/issues/144: scan dir dir/link/sub,
|
||||
// with dir/link a symlink to a directory outside dir. The walk of dir
|
||||
// does not follow link, so dir/link/sub is walked on its own at its
|
||||
// real path, and the record an earlier scan stored there survives.
|
||||
func TestRunScanWalksSymlinkedOperandBelowOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
a := writeFile(t, dir, "a", pattern(1, 10))
|
||||
|
||||
outside := realTempDir(t)
|
||||
f := writeFile(t, outside, "sub/f", pattern(2, 10))
|
||||
|
||||
symlink(t, outside, filepath.Join(dir, "link"))
|
||||
|
||||
scanOK(t, filepath.Join(outside, "sub"))
|
||||
|
||||
stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub"))
|
||||
|
||||
summary := "scan: 2 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"1 unchanged), 0 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
want := []string{a, f}
|
||||
slices.Sort(want)
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, want) {
|
||||
t.Errorf("records = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanDropsSymlinkOperandInsideOperand scans dir and link, with
|
||||
// link, outside dir, a symlink to dir/real. link resolves inside dir,
|
||||
// so it is dropped as covered by dir, and its file is recorded once.
|
||||
func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "real/f", pattern(1, 10))
|
||||
|
||||
link := filepath.Join(realTempDir(t), "link")
|
||||
symlink(t, filepath.Join(dir, "real"), link)
|
||||
|
||||
stderr := scanOK(t, dir, link)
|
||||
|
||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 0 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanDanglingSymlinkOperandIsFatal scans a valid directory and
|
||||
// a symlink whose target does not exist. The scan fails with exit 1 and
|
||||
// an error naming the symlink as given, before it walks anything, so no
|
||||
// database is created.
|
||||
func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
writeFile(t, dir, "f", pattern(1, 10))
|
||||
|
||||
dangling := filepath.Join(realTempDir(t), "dangling")
|
||||
symlink(t, filepath.Join(dir, "missing"), dangling)
|
||||
|
||||
stdout := captureStdout(t)
|
||||
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := run([]string{cmdScan, dir, dangling}, os.Stdout, &stderr)
|
||||
if code != exitFatal {
|
||||
t.Errorf("run(scan %s %s) = %d, want %d", dir, dangling, code,
|
||||
exitFatal)
|
||||
}
|
||||
|
||||
assertFatalOutput(t, stderr.String(), stdout())
|
||||
|
||||
want := "sfdupes: resolve " + dangling + ": "
|
||||
if !strings.Contains(stderr.String(), want) {
|
||||
t.Errorf("stderr = %q, want %q", stderr.String(), want)
|
||||
}
|
||||
|
||||
_, err := os.Stat(path)
|
||||
if !errors.Is(err, fs.ErrNotExist) {
|
||||
t.Errorf("database: stat error %v, want it missing", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath runs scan .
|
||||
// after a cd through home/proj, a symlink to data/proj. The file is
|
||||
// recorded under data/proj, its real path.
|
||||
func TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
|
||||
|
||||
err := os.Mkdir(filepath.Join(dir, "home"), 0o750)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// link is dropped as a symlink, but link/sub must still be scanned,
|
||||
// not dropped as lying under link.
|
||||
scanOK(t, link, filepath.Join(link, "sub"))
|
||||
symlink(t, filepath.Join(dir, "data", "proj"),
|
||||
filepath.Join(dir, "home", "proj"))
|
||||
|
||||
db, err := openDB(path, reportParams)
|
||||
// t.Chdir also sets PWD to the path through the symlink, as a
|
||||
// shell does after cd.
|
||||
t.Chdir(filepath.Join(dir, "home", "proj"))
|
||||
|
||||
scanOK(t, ".")
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanFollowsSymlinkBeforeDotDot scans a path with a ".." after
|
||||
// home/proj, a symlink to data/proj: home/proj/.. from dir, and .. after
|
||||
// a cd through home/proj. As with realpath, the ".." counts from
|
||||
// data/proj, so data is scanned and home is not.
|
||||
func TestRunScanFollowsSymlinkBeforeDotDot(t *testing.T) {
|
||||
for _, tc := range []struct{ wd, operand string }{
|
||||
{"", "home/proj/.."},
|
||||
{"home/proj", ".."},
|
||||
} {
|
||||
t.Run(tc.operand, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
|
||||
g := writeFile(t, dir, "data/g", pattern(2, 10))
|
||||
writeFile(t, dir, "home/h", pattern(3, 10))
|
||||
symlink(t, filepath.Join(dir, "data", "proj"),
|
||||
filepath.Join(dir, "home", "proj"))
|
||||
|
||||
// t.Chdir also sets PWD to the path given, as a shell does
|
||||
// after cd.
|
||||
t.Chdir(filepath.Join(dir, tc.wd))
|
||||
|
||||
scanOK(t, tc.operand)
|
||||
|
||||
want := []string{f, g}
|
||||
slices.Sort(want)
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, want) {
|
||||
t.Errorf("records = %q, want %q", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanSkipsSymlinkToFIFOOperand scans link, a symlink to a FIFO.
|
||||
// The scan warns about the FIFO by its real path, counts it as skipped,
|
||||
// and exits 0.
|
||||
func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) {
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
dir := realTempDir(t)
|
||||
fifo := filepath.Join(dir, "fifo")
|
||||
|
||||
err := unix.Mkfifo(fifo, 0o600)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
link := filepath.Join(dir, "link")
|
||||
symlink(t, fifo, link)
|
||||
|
||||
recordByPath(t, dbRecords(t, db), filepath.Join(link, "sub", "f"))
|
||||
stderr := scanOK(t, link)
|
||||
|
||||
warning := "walk " + fifo + ": skipping FIFO operand\n"
|
||||
if !strings.Contains(stderr, warning) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||
}
|
||||
|
||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 1 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunScanSkipsZFSOperand(t *testing.T) {
|
||||
zfs := filepath.Join(t.TempDir(), ".zfs")
|
||||
zfs := filepath.Join(realTempDir(t), ".zfs")
|
||||
f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
|
||||
|
||||
// No scan records a file inside a .zfs directory, so the record
|
||||
@@ -979,7 +1248,7 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
|
||||
writeFile(t, hourly, "f", pattern(1, 10))
|
||||
f := writeFile(t, dir, "f", pattern(2, 10))
|
||||
@@ -1023,23 +1292,19 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
|
||||
|
||||
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
|
||||
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
|
||||
// by the path given, counted as skipped, and not walked.
|
||||
// by its real path, counted as skipped, and not walked.
|
||||
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
|
||||
|
||||
snaps := filepath.Join(dir, "snaps")
|
||||
symlink(t, filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
|
||||
|
||||
err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
daily := filepath.Join(snaps, "daily")
|
||||
stderr := scanOK(t, daily)
|
||||
stderr := scanOK(t, filepath.Join(snaps, "daily"))
|
||||
|
||||
daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily")
|
||||
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
|
||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 1 skipped\n"
|
||||
@@ -1051,24 +1316,19 @@ func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
|
||||
// a symlink to a directory outside any .zfs directory. The operand is
|
||||
// scanned as usual, and its file is recorded by the path given.
|
||||
// a symlink to pool/data. The operand is scanned as usual, and its file
|
||||
// is recorded by its real path.
|
||||
func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
|
||||
|
||||
data := filepath.Join(dir, "data")
|
||||
symlink(t, filepath.Join(dir, "pool", "data"), data)
|
||||
|
||||
err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
daily := filepath.Join(data, "daily")
|
||||
stderr := scanOK(t, daily)
|
||||
stderr := scanOK(t, filepath.Join(data, "daily"))
|
||||
|
||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 0 skipped\n"
|
||||
@@ -1076,14 +1336,9 @@ func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
db, err := openDB(path, reportParams)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
|
||||
}
|
||||
|
||||
func TestRunReportSucceeds(t *testing.T) {
|
||||
@@ -1140,8 +1395,9 @@ func TestRunTreesSucceeds(t *testing.T) {
|
||||
|
||||
// The two directories holding the duplicate pair are duplicate
|
||||
// trees of each other.
|
||||
want := "first\tdupe\tfiles\tsize\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
|
||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
||||
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n"
|
||||
if got := stdout.String(); got != want {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
@@ -1175,9 +1431,10 @@ func TestRunReportsNeedOnlyReadAccess(t *testing.T) {
|
||||
cases := map[string]string{
|
||||
cmdReport: "first\tdupe\tsize\n" +
|
||||
dupes[0] + "\t" + dupes[1] + "\t300\n",
|
||||
cmdTrees: "first\tdupe\tfiles\tsize\n" +
|
||||
cmdTrees: "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) +
|
||||
"\t1\t300\n",
|
||||
"\t1\t300\t" +
|
||||
strconv.FormatInt(storedDiskSize(t, path, dupes[1]), 10) + "\n",
|
||||
}
|
||||
|
||||
for name, want := range cases {
|
||||
@@ -1212,8 +1469,11 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
|
||||
t.Errorf("report stdout = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
want = "first\tdupe\tfiles\tsize\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\n"
|
||||
diskSize := storedDiskSize(t, filepath.Join(dir, name), dupes[1])
|
||||
|
||||
want = "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
filepath.Dir(dupes[0]) + "\t" + filepath.Dir(dupes[1]) + "\t1\t300\t" +
|
||||
strconv.FormatInt(diskSize, 10) + "\n"
|
||||
if got := runStdout(t, cmdTrees); got != want {
|
||||
t.Errorf("trees stdout = %q, want %q", got, want)
|
||||
}
|
||||
@@ -1279,9 +1539,8 @@ func holdScanLock(t *testing.T, path string) {
|
||||
}
|
||||
|
||||
func TestRunSecondScanFails(t *testing.T) {
|
||||
// README §Database: while one scan holds the lock, a second scan
|
||||
// fails at once, naming the lock file, without opening the
|
||||
// database.
|
||||
// README §Database: a second scan that finds the lock held fails
|
||||
// at once, naming the lock file, without opening the database.
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
@@ -1340,7 +1599,7 @@ func assertSecondScanFails(t *testing.T, path string) {
|
||||
}
|
||||
|
||||
func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
|
||||
// README §Database: every path to one database names the same lock
|
||||
// README §Database: a path through a symlink names the same lock
|
||||
// file, so while a scan holds it through one path, a scan through
|
||||
// another fails, naming the lock file beside the database's real
|
||||
// path.
|
||||
+179
-68
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -47,14 +47,15 @@ const dbFilePerm = 0o644
|
||||
const lockFilePerm = 0o600
|
||||
|
||||
// createTableSQL is the schema applied to a fresh database. Paths are
|
||||
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8. mtime
|
||||
// holds whole Unix seconds and mtime_nsec the nanoseconds within that
|
||||
// second. head, tail and content each hold the raw 32 bytes of a
|
||||
// SHA-256.
|
||||
// BLOBs because Unix paths are raw bytes, not guaranteed UTF-8.
|
||||
// disk_size holds the bytes the file takes on disk. mtime holds whole
|
||||
// Unix seconds and mtime_nsec the nanoseconds within that second. head,
|
||||
// tail and content each hold the raw 32 bytes of a SHA-256.
|
||||
const createTableSQL = `
|
||||
CREATE TABLE files (
|
||||
path BLOB PRIMARY KEY,
|
||||
size INTEGER NOT NULL,
|
||||
disk_size INTEGER NOT NULL,
|
||||
mtime INTEGER NOT NULL,
|
||||
mtime_nsec INTEGER NOT NULL,
|
||||
head BLOB NOT NULL,
|
||||
@@ -69,14 +70,59 @@ const createIndexSQL = `
|
||||
CREATE INDEX files_signature ON files (size, head, tail, content)
|
||||
`
|
||||
|
||||
// createTotalsSQL creates the totals table, which holds one row: the
|
||||
// number of records and the sizes of all records added up. Counting
|
||||
// them over files would read every record. The three triggers below
|
||||
// keep the row current, in the transaction of each change to files.
|
||||
const createTotalsSQL = `
|
||||
CREATE TABLE totals (
|
||||
records INTEGER NOT NULL,
|
||||
size INTEGER NOT NULL
|
||||
)
|
||||
`
|
||||
|
||||
// insertTotalsSQL adds the one row of totals, for an empty files table.
|
||||
const insertTotalsSQL = `
|
||||
INSERT INTO totals (records, size) VALUES (0, 0)
|
||||
`
|
||||
|
||||
// totalsAfterInsertSQL counts each inserted record into totals.
|
||||
const totalsAfterInsertSQL = `
|
||||
CREATE TRIGGER totals_after_insert AFTER INSERT ON files BEGIN
|
||||
UPDATE totals SET records = records + 1, size = size + NEW.size;
|
||||
END
|
||||
`
|
||||
|
||||
// totalsAfterDeleteSQL takes each deleted record out of totals.
|
||||
const totalsAfterDeleteSQL = `
|
||||
CREATE TRIGGER totals_after_delete AFTER DELETE ON files BEGIN
|
||||
UPDATE totals SET records = records - 1, size = size - OLD.size;
|
||||
END
|
||||
`
|
||||
|
||||
// totalsAfterUpdateSQL replaces a record's old size in totals with its
|
||||
// new one. scan's upsert of an existing record is such an update.
|
||||
const totalsAfterUpdateSQL = `
|
||||
CREATE TRIGGER totals_after_update AFTER UPDATE OF size ON files BEGIN
|
||||
UPDATE totals SET size = size + NEW.size - OLD.size;
|
||||
END
|
||||
`
|
||||
|
||||
// recordCountSQL reads the number of records from totals.
|
||||
const recordCountSQL = `
|
||||
SELECT records FROM totals
|
||||
`
|
||||
|
||||
// upsertSQL inserts one file record, replacing any existing record for
|
||||
// the same path.
|
||||
const upsertSQL = `
|
||||
INSERT INTO files (path, size, mtime, mtime_nsec, head, tail, content)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?)
|
||||
INSERT INTO files (
|
||||
path, size, disk_size, mtime, mtime_nsec, head, tail, content
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT (path) DO UPDATE SET
|
||||
size = excluded.size, mtime = excluded.mtime,
|
||||
mtime_nsec = excluded.mtime_nsec,
|
||||
size = excluded.size, disk_size = excluded.disk_size,
|
||||
mtime = excluded.mtime, mtime_nsec = excluded.mtime_nsec,
|
||||
head = excluded.head, tail = excluded.tail,
|
||||
content = excluded.content
|
||||
`
|
||||
@@ -100,33 +146,44 @@ var errHashSize = errors.New("hash is not 32 bytes")
|
||||
|
||||
// databasePath resolves the database location: SFDUPES_DATABASE when
|
||||
// set and non-empty, the compiled-in default otherwise. It returns the
|
||||
// path, with the working directory, its symlinks resolved, put in front
|
||||
// of it when it is relative, and where the path came from, "default" or
|
||||
// "from SFDUPES_DATABASE", for the line each subcommand prints first on
|
||||
// stderr.
|
||||
// path, made absolute by absolutePath, and where the path came from,
|
||||
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
||||
// prints first on stderr.
|
||||
func databasePath() (string, string, error) {
|
||||
path, source := defaultDatabasePath, "default"
|
||||
if p := os.Getenv(databaseEnv); p != "" {
|
||||
path, source = p, "from "+databaseEnv
|
||||
}
|
||||
|
||||
if filepath.IsAbs(path) {
|
||||
return path, source, nil
|
||||
}
|
||||
|
||||
// The working directory as the kernel reports it, with no symlink
|
||||
// in it, as README "Database" says. os.Getwd may return PWD
|
||||
// instead, which can name it through a symlink.
|
||||
wd, err := unix.Getwd()
|
||||
abs, err := absolutePath(path)
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
||||
}
|
||||
|
||||
// Not filepath.Abs or filepath.Join: they clean the path, which
|
||||
// drops each ".." together with the directory name before it. When
|
||||
// that directory is a symlink, the cleaned path names another file.
|
||||
return abs, source, nil
|
||||
}
|
||||
|
||||
// absolutePath returns path unchanged when it is absolute, and
|
||||
// otherwise the working directory, a "/" and path. The working
|
||||
// directory is the one the kernel reports, with no symlink in it;
|
||||
// os.Getwd may return PWD instead, which can name it through a symlink.
|
||||
// The result is not cleaned, as filepath.Abs and filepath.Join would
|
||||
// clean it. Cleaning drops each ".." together with the directory name
|
||||
// before it. When that name is a symlink, the cleaned path names
|
||||
// another file than the one the system reaches by following the symlink
|
||||
// first.
|
||||
func absolutePath(path string) (string, error) {
|
||||
if filepath.IsAbs(path) {
|
||||
return path, nil
|
||||
}
|
||||
|
||||
wd, err := unix.Getwd()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// The working directory ends in "/" only when it is the root.
|
||||
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
|
||||
return strings.TrimSuffix(wd, "/") + "/" + path, nil
|
||||
}
|
||||
|
||||
// createDatabaseDirectory creates the directory the database at path
|
||||
@@ -150,9 +207,14 @@ func createDatabaseDirectory(path string) error {
|
||||
// scanParams are the connection parameters for scan: read-write, with
|
||||
// WAL journaling and a busy timeout, so a report can run while a cron
|
||||
// scan is in progress. closeScanDatabase leaves WAL mode again.
|
||||
//
|
||||
// Each write to files fires a trigger, so SQLite keeps a statement
|
||||
// journal for it. temp_store(MEMORY) keeps that journal in memory,
|
||||
// which costs far less than a temporary file.
|
||||
const scanParams = "_pragma=busy_timeout(10000)" +
|
||||
"&_pragma=journal_mode(WAL)" +
|
||||
"&_pragma=synchronous(NORMAL)"
|
||||
"&_pragma=synchronous(NORMAL)" +
|
||||
"&_pragma=temp_store(MEMORY)"
|
||||
|
||||
// reportParams are the connection parameters for report, trees and
|
||||
// info: read-only, with the same busy timeout. They set no journal
|
||||
@@ -426,14 +488,14 @@ func createSchema(ctx context.Context, db *sql.DB) error {
|
||||
|
||||
defer func() { _ = tx.Rollback() }()
|
||||
|
||||
_, err = tx.ExecContext(ctx, createTableSQL)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create schema: %w", err)
|
||||
}
|
||||
|
||||
_, err = tx.ExecContext(ctx, createIndexSQL)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create schema: %w", err)
|
||||
for _, stmt := range []string{
|
||||
createTableSQL, createIndexSQL, createTotalsSQL, insertTotalsSQL,
|
||||
totalsAfterInsertSQL, totalsAfterDeleteSQL, totalsAfterUpdateSQL,
|
||||
} {
|
||||
_, err = tx.ExecContext(ctx, stmt)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create schema: %w", err)
|
||||
}
|
||||
}
|
||||
|
||||
_, err = tx.ExecContext(ctx,
|
||||
@@ -480,7 +542,7 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
||||
|
||||
var n int64
|
||||
|
||||
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
||||
err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
|
||||
if err != nil {
|
||||
return fmt.Errorf("count records: %w", err)
|
||||
}
|
||||
@@ -488,8 +550,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
||||
total(n)
|
||||
|
||||
rows, err := tx.QueryContext(ctx,
|
||||
"SELECT path, size, mtime, mtime_nsec, head, tail, content "+
|
||||
"FROM files ORDER BY path")
|
||||
"SELECT path, size, disk_size, mtime, mtime_nsec, head, tail, "+
|
||||
"content FROM files ORDER BY path")
|
||||
if err != nil {
|
||||
return fmt.Errorf("read records: %w", err)
|
||||
}
|
||||
@@ -503,7 +565,8 @@ func loadFileRows(ctx context.Context, db *sql.DB, total func(n int64),
|
||||
r scanRec
|
||||
)
|
||||
|
||||
err = rows.Scan(&path, &r.size, &sec, &nsec, &head, &tail, &content)
|
||||
err = rows.Scan(&path, &r.size, &r.diskSize, &sec, &nsec,
|
||||
&head, &tail, &content)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read record: %w", err)
|
||||
}
|
||||
@@ -538,10 +601,11 @@ ORDER BY size DESC, head DESC, tail DESC, content DESC
|
||||
`
|
||||
|
||||
// loadSignatureRows gives total the number of records, then streams
|
||||
// every record's signature and path to fn, without its mtime, in the
|
||||
// order of signatureRowsSQL. The count and the records are read in one
|
||||
// transaction, so they agree while a scan is committing. An error from
|
||||
// fn stops the reading and is returned as it is.
|
||||
// every record's signature and path to fn, without its mtime or its
|
||||
// disk_size, in the order of signatureRowsSQL. The count and the
|
||||
// records are read in one transaction, so they agree while a scan is
|
||||
// committing. An error from fn stops the reading and is returned as it
|
||||
// is.
|
||||
func loadSignatureRows(ctx context.Context, db *sql.DB,
|
||||
total func(n int64), fn func(r scanRec) error,
|
||||
) error {
|
||||
@@ -556,7 +620,7 @@ func loadSignatureRows(ctx context.Context, db *sql.DB,
|
||||
|
||||
var n int64
|
||||
|
||||
err = tx.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
||||
err = tx.QueryRowContext(ctx, recordCountSQL).Scan(&n)
|
||||
if err != nil {
|
||||
return fmt.Errorf("count records: %w", err)
|
||||
}
|
||||
@@ -622,7 +686,7 @@ func (r *scanRec) setHashes(head, tail, content []byte) error {
|
||||
func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
|
||||
var n int64
|
||||
|
||||
err := db.QueryRowContext(ctx, "SELECT COUNT(*) FROM files").Scan(&n)
|
||||
err := db.QueryRowContext(ctx, recordCountSQL).Scan(&n)
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("count records: %w", err)
|
||||
}
|
||||
@@ -630,12 +694,10 @@ func countRecords(ctx context.Context, db *sql.DB) (int64, error) {
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// databaseCountsSQL counts the records and adds up their sizes, in one
|
||||
// pass over the table. A SUM over no rows is NULL, so on an empty table
|
||||
// the SUM falls back to 0.
|
||||
// databaseCountsSQL reads the number of records and the sizes of all
|
||||
// records added up from totals.
|
||||
const databaseCountsSQL = `
|
||||
SELECT COUNT(*), COALESCE(SUM(size), 0)
|
||||
FROM files
|
||||
SELECT records, size FROM totals
|
||||
`
|
||||
|
||||
// databaseCounts is what info reads from the database: its schema
|
||||
@@ -678,15 +740,15 @@ func readDatabaseCounts(ctx context.Context,
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// loadFileMeta streams every record's path, size and mtime to fn. Scan
|
||||
// change detection needs no hash values, and skipping the hash columns
|
||||
// keeps the scan's in-memory index small on multi-million-file
|
||||
// databases.
|
||||
// loadFileMeta streams every record's path, size, disk_size and mtime
|
||||
// to fn. Scan change detection needs no hash values, and skipping the
|
||||
// hash columns keeps the scan's in-memory index small on
|
||||
// multi-million-file databases.
|
||||
func loadFileMeta(ctx context.Context, db *sql.DB,
|
||||
fn func(path string, size int64, mtime time.Time),
|
||||
fn func(path string, m fileMeta),
|
||||
) error {
|
||||
rows, err := db.QueryContext(ctx,
|
||||
"SELECT path, size, mtime, mtime_nsec FROM files")
|
||||
"SELECT path, size, disk_size, mtime, mtime_nsec FROM files")
|
||||
if err != nil {
|
||||
return fmt.Errorf("read records: %w", err)
|
||||
}
|
||||
@@ -695,16 +757,19 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
|
||||
|
||||
for rows.Next() {
|
||||
var (
|
||||
path []byte
|
||||
size, sec, nsec int64
|
||||
path []byte
|
||||
m fileMeta
|
||||
sec, nsec int64
|
||||
)
|
||||
|
||||
err = rows.Scan(&path, &size, &sec, &nsec)
|
||||
err = rows.Scan(&path, &m.size, &m.diskSize, &sec, &nsec)
|
||||
if err != nil {
|
||||
return fmt.Errorf("read record: %w", err)
|
||||
}
|
||||
|
||||
fn(string(path), size, time.Unix(sec, nsec))
|
||||
m.mtime = time.Unix(sec, nsec)
|
||||
|
||||
fn(string(path), m)
|
||||
}
|
||||
|
||||
err = rows.Err()
|
||||
@@ -722,21 +787,21 @@ func loadFileMeta(ctx context.Context, db *sql.DB,
|
||||
// WAL small and let concurrent reports observe progress.
|
||||
const updateBatchSize = 10000
|
||||
|
||||
// applyChanges writes one scan's database changes — upserts for new and
|
||||
// changed files, deletes for vanished ones — in batched transactions.
|
||||
// Progress is rendered on prog (one increment per change).
|
||||
// applyChanges writes upserts for new and changed files and deletes for
|
||||
// vanished ones, in batched transactions. Progress is rendered on prog
|
||||
// (one increment per change).
|
||||
func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||
deletes []string, prog *progress,
|
||||
) error {
|
||||
for batch := range slices.Chunk(upserts, updateBatchSize) {
|
||||
err := applyBatch(ctx, db, batch, nil, prog)
|
||||
err := applyBatch(ctx, db, batch, nil, nil, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
for batch := range slices.Chunk(deletes, updateBatchSize) {
|
||||
err := applyBatch(ctx, db, nil, batch, prog)
|
||||
err := applyBatch(ctx, db, nil, nil, batch, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -745,10 +810,26 @@ func applyChanges(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyBatch commits one batch of upserts and deletes in a single
|
||||
// transaction.
|
||||
// applyDiskSizes writes the disk_size of each of recs into its record,
|
||||
// in batched transactions, and leaves the rest of the record alone.
|
||||
// Progress is rendered on prog (one increment per record).
|
||||
func applyDiskSizes(ctx context.Context, db *sql.DB, recs []fileRec,
|
||||
prog *progress,
|
||||
) error {
|
||||
for batch := range slices.Chunk(recs, updateBatchSize) {
|
||||
err := applyBatch(ctx, db, nil, batch, nil, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// applyBatch commits one batch of upserts, disk_size updates and
|
||||
// deletes in a single transaction.
|
||||
func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||
deletes []string, prog *progress,
|
||||
diskSizes []fileRec, deletes []string, prog *progress,
|
||||
) error {
|
||||
tx, err := db.BeginTx(ctx, nil)
|
||||
if err != nil {
|
||||
@@ -762,6 +843,11 @@ func applyBatch(ctx context.Context, db *sql.DB, upserts []scanRec,
|
||||
return err
|
||||
}
|
||||
|
||||
err = execDiskSizes(ctx, tx, diskSizes, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = execDeletes(ctx, tx, deletes, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -787,7 +873,7 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
|
||||
defer func() { _ = st.Close() }()
|
||||
|
||||
for _, r := range upserts {
|
||||
_, err = st.ExecContext(ctx, []byte(r.path), r.size,
|
||||
_, err = st.ExecContext(ctx, []byte(r.path), r.size, r.diskSize,
|
||||
r.mtime.Unix(), r.mtime.Nanosecond(),
|
||||
r.head[:], r.tail[:], r.content[:])
|
||||
if err != nil {
|
||||
@@ -800,6 +886,31 @@ func execUpserts(ctx context.Context, tx *sql.Tx, upserts []scanRec,
|
||||
return nil
|
||||
}
|
||||
|
||||
// execDiskSizes sets the disk_size of the record of each of recs. It
|
||||
// does not touch size, so the totals triggers do not fire.
|
||||
func execDiskSizes(ctx context.Context, tx *sql.Tx, recs []fileRec,
|
||||
prog *progress,
|
||||
) error {
|
||||
st, err := tx.PrepareContext(ctx,
|
||||
"UPDATE files SET disk_size = ? WHERE path = ?")
|
||||
if err != nil {
|
||||
return fmt.Errorf("prepare disk_size update: %w", err)
|
||||
}
|
||||
|
||||
defer func() { _ = st.Close() }()
|
||||
|
||||
for _, r := range recs {
|
||||
_, err = st.ExecContext(ctx, r.diskSize, []byte(r.path))
|
||||
if err != nil {
|
||||
return fmt.Errorf("update disk_size of %s: %w", r.path, err)
|
||||
}
|
||||
|
||||
prog.increment()
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
// execDeletes removes the records for paths no longer present.
|
||||
func execDeletes(ctx context.Context, tx *sql.Tx, deletes []string,
|
||||
prog *progress,
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // calls the unexported database openers
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -535,17 +535,12 @@ func TestLoadRowsRefuseWrongSizeHash(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
|
||||
// from the files_signature index alone, with nothing to sort: the
|
||||
// first record comes at once, so report's read phase counts each
|
||||
// record as it comes rather than waiting on SQLite for all of them.
|
||||
func TestSignatureRowsNeedNoSort(t *testing.T) {
|
||||
t.Parallel()
|
||||
// queryPlan returns the detail of each line of SQLite's EXPLAIN QUERY
|
||||
// PLAN for query.
|
||||
func queryPlan(t *testing.T, db *sql.DB, query string) []string {
|
||||
t.Helper()
|
||||
|
||||
db := openTestDB(t)
|
||||
|
||||
rows, err := db.QueryContext(t.Context(),
|
||||
"EXPLAIN QUERY PLAN "+signatureRowsSQL)
|
||||
rows, err := db.QueryContext(t.Context(), "EXPLAIN QUERY PLAN "+query)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -573,12 +568,164 @@ func TestSignatureRowsNeedNoSort(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
return plan
|
||||
}
|
||||
|
||||
// TestSignatureRowsNeedNoSort checks that SQLite reads signatureRowsSQL
|
||||
// from the files_signature index alone, with nothing to sort: the
|
||||
// first record comes at once, so report's read phase counts each
|
||||
// record as it comes rather than waiting on SQLite for all of them.
|
||||
func TestSignatureRowsNeedNoSort(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := openTestDB(t)
|
||||
|
||||
plan := queryPlan(t, db, signatureRowsSQL)
|
||||
|
||||
want := []string{"SCAN files USING COVERING INDEX files_signature"}
|
||||
if !slices.Equal(plan, want) {
|
||||
t.Errorf("query plan = %q, want %q", plan, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCountsReadOnlyTotals checks that SQLite answers the queries for
|
||||
// the number of records and their total size from the one row of
|
||||
// totals. A plan that reads files or its index reads every record.
|
||||
func TestCountsReadOnlyTotals(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
db := openTestDB(t)
|
||||
|
||||
for _, query := range []string{recordCountSQL, databaseCountsSQL} {
|
||||
plan := queryPlan(t, db, query)
|
||||
|
||||
want := []string{"SCAN totals"}
|
||||
if !slices.Equal(plan, want) {
|
||||
t.Errorf("query plan of %q = %q, want %q", query, plan, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestCountsComeFromTotals sets the row of totals to numbers that files
|
||||
// does not hold, then checks that scan's load phase, the read phases of
|
||||
// trees and report, and info each report the numbers from totals.
|
||||
func TestCountsComeFromTotals(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// files is empty, so a count over files gives 0 records of 0 bytes.
|
||||
const records, size = 7, 9000
|
||||
|
||||
db := openTestDB(t)
|
||||
|
||||
_, err := db.ExecContext(t.Context(),
|
||||
"UPDATE totals SET records = ?, size = ?", records, size)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
scanTotal, err := countRecords(t.Context(), db)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var treesTotal, reportTotal int64
|
||||
|
||||
err = loadFileRows(t.Context(), db,
|
||||
func(n int64) { treesTotal = n }, func(scanRec) {})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
err = loadSignatureRows(t.Context(), db,
|
||||
func(n int64) { reportTotal = n },
|
||||
func(scanRec) error { return nil })
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
info, err := readDatabaseCounts(t.Context(), db)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
for name, got := range map[string]int64{
|
||||
"scan's load": scanTotal,
|
||||
"trees' read": treesTotal,
|
||||
"report's read": reportTotal,
|
||||
"info": info.records,
|
||||
} {
|
||||
if got != records {
|
||||
t.Errorf("%s: %d records, want %d from totals",
|
||||
name, got, records)
|
||||
}
|
||||
}
|
||||
|
||||
if info.size != size {
|
||||
t.Errorf("info: size %d, want %d from totals", info.size, size)
|
||||
}
|
||||
}
|
||||
|
||||
// assertTotalsMatchFiles fails unless the row of totals holds the
|
||||
// number of records and the sizes of all records added up, as counted
|
||||
// over files.
|
||||
func assertTotalsMatchFiles(t *testing.T, db *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
got, err := readDatabaseCounts(t.Context(), db)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var records, size int64
|
||||
|
||||
err = db.QueryRowContext(t.Context(),
|
||||
"SELECT COUNT(*), COALESCE(SUM(size), 0) FROM files").
|
||||
Scan(&records, &size)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if got.records != records || got.size != size {
|
||||
t.Errorf("totals = %d records of %d bytes, files holds %d of %d",
|
||||
got.records, got.size, records, size)
|
||||
}
|
||||
}
|
||||
|
||||
// TestTotalsMatchFiles scans the smoke tree, rescans it after two files
|
||||
// change size, and rescans it after two files are removed. After each
|
||||
// scan the row of totals must agree with files.
|
||||
func TestTotalsMatchFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := buildSmokeTree(t)
|
||||
db := openTestDB(t)
|
||||
|
||||
syncTree(t, db, dir)
|
||||
assertTotalsMatchFiles(t, db)
|
||||
|
||||
writeFile(t, dir, "a/one.bin", pattern(10, 2500))
|
||||
writeFile(t, dir, "t1/f1", pattern(30, 1000))
|
||||
|
||||
if st := syncTree(t, db, dir); st.updated != 2 {
|
||||
t.Fatalf("rescan stats = %+v, want 2 updated", st)
|
||||
}
|
||||
|
||||
assertTotalsMatchFiles(t, db)
|
||||
|
||||
for _, name := range []string{"a/unique.bin", "t2/sub/f2"} {
|
||||
err := os.Remove(filepath.Join(dir, name))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
if st := syncTree(t, db, dir); st.removed != 2 {
|
||||
t.Fatalf("rescan stats = %+v, want 2 removed", st)
|
||||
}
|
||||
|
||||
assertTotalsMatchFiles(t, db)
|
||||
}
|
||||
|
||||
func TestApplyChangesRoundTrip(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"runtime"
|
||||
)
|
||||
|
||||
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
|
||||
// root" runs with no flags, then what "sfdupes trees" runs, and ends
|
||||
// with a line on stderr saying how to print the report again. The dwim
|
||||
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
|
||||
// does, but only during the scan. An error from the scan, an interrupt
|
||||
// included, ends it before the report. During the report, a signal
|
||||
// ends it at once, as it ends trees.
|
||||
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
||||
scanCtx, stop := interruptContext(ctx)
|
||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
|
||||
|
||||
stop()
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = runTrees(ctx, stdout)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Fprintln(os.Stderr,
|
||||
`dwim: run "sfdupes trees" to print this report again`)
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package sfdupes //nolint:testpackage // calls the unexported runDwim
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
|
||||
// dwim scans its root, then prints the trees report, the larger of
|
||||
// two duplicate-tree pairs first, and ends by saying how to print
|
||||
// the report again.
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
root := realTempDir(t)
|
||||
|
||||
small := []string{
|
||||
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
|
||||
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
|
||||
}
|
||||
large := []string{
|
||||
filepath.Dir(writeFile(t, root, "b1/f.bin", pattern(2, 5000))),
|
||||
filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000))),
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
err := runDwim(t.Context(), root, &stdout)
|
||||
if err != nil {
|
||||
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
||||
}
|
||||
|
||||
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
|
||||
if len(lines) != 3 {
|
||||
t.Fatalf("stdout = %q, want a header and two groups",
|
||||
stdout.String())
|
||||
}
|
||||
|
||||
wantFirst := large[0] + "\t" + large[1] + "\t1\t5000\t"
|
||||
if !strings.HasPrefix(lines[1], wantFirst) {
|
||||
t.Errorf("first group = %q, want it to start %q", lines[1], wantFirst)
|
||||
}
|
||||
|
||||
wantSecond := small[0] + "\t" + small[1] + "\t1\t300\t"
|
||||
if !strings.HasPrefix(lines[2], wantSecond) {
|
||||
t.Errorf("second group = %q, want it to start %q",
|
||||
lines[2], wantSecond)
|
||||
}
|
||||
|
||||
last := "\ndwim: run \"sfdupes trees\" to print this report again\n"
|
||||
if got := stderr(); !strings.HasSuffix(got, last) {
|
||||
t.Errorf("stderr = %q, want it to end %q", got, last)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
||||
// A fatal scan error ends dwim before the report: a root that does
|
||||
// not exist fails the scan as it fails scan itself.
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
missing := filepath.Join(realTempDir(t), "missing")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
err := runDwim(t.Context(), missing, &stdout)
|
||||
if err == nil {
|
||||
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != "" {
|
||||
t.Errorf("stdout = %q, want no report", got)
|
||||
}
|
||||
|
||||
if got := stderr(); strings.Contains(got, "trees:") ||
|
||||
strings.Contains(got, "dwim:") {
|
||||
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
// This file holds two rules for the walk (README "Rules for the walk"):
|
||||
// the default excludes, the paths a scan's walk does not enter below an
|
||||
@@ -1,6 +1,6 @@
|
||||
//go:build darwin
|
||||
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"slices"
|
||||
@@ -1,6 +1,6 @@
|
||||
//go:build !darwin
|
||||
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // checks the unexported defaultExcludes
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -286,7 +286,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
|
||||
t.Skip(dir + " is not inside a default exclude")
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -26,11 +26,11 @@ func infoPhase(name string) string {
|
||||
// it. Any database problem, including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in two phases, each under a progress display of its
|
||||
// own on stderr: open (open the database) and count (count the records,
|
||||
// with one query). Each display ends before the next phase starts, and
|
||||
// before an error is returned to be printed. The details are written
|
||||
// once the last display has ended, so no display is drawn while they
|
||||
// are written.
|
||||
// own on stderr: open (open the database) and count (read the number of
|
||||
// records and their total size from the totals table). Each display
|
||||
// ends before the next phase starts, and before an error is returned
|
||||
// to be printed. The details are written once the last display has
|
||||
// ended, so no display is drawn while they are written.
|
||||
func runInfo(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
@@ -56,7 +56,7 @@ func runInfo(ctx context.Context, stdout io.Writer) error {
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
// One query counts everything, so the phase never has a total.
|
||||
// One row holds both counts, so the phase never has a total.
|
||||
prog = newProgress(progressParams{
|
||||
label: infoPhase("count"), total: -1,
|
||||
waitingFor: countingRecords,
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // seeds the database with unexported scanRec
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -60,7 +60,7 @@ const fallbackWidth = 80
|
||||
const spinnerMargin = 5
|
||||
|
||||
// countingRecords is what the display of a phase that reads the records
|
||||
// says while it counts them, before it has its total.
|
||||
// says until it has the number of records, its total.
|
||||
const countingRecords = "counting the records"
|
||||
|
||||
// openPhase is the name of the phase in which report, trees and info
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // drives the unexported startProgress
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -642,9 +642,9 @@ func TestTerminalStalledLines(t *testing.T) {
|
||||
{label: scanPhase("hash"), stalledOn: hashStalledOn},
|
||||
{label: scanPhase("update"), stalledOn: "db"},
|
||||
{label: treesPhase("read"), stalledOn: "db"},
|
||||
{label: treesPhase("group"), stalledOn: "cpu"},
|
||||
{label: treesPhase("group"), stalledOn: "cpu/disk"},
|
||||
{label: treesPhase("write"), stalledOn: "out"},
|
||||
{label: reportPhase("read"), stalledOn: "db/out"},
|
||||
{label: reportPhase("read"), stalledOn: "db/disk/out"},
|
||||
}
|
||||
|
||||
for _, params := range phases {
|
||||
@@ -1,14 +1,17 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/fs"
|
||||
"os"
|
||||
"slices"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -21,14 +24,16 @@ const minGroupSize = 2
|
||||
|
||||
// scanRec is one file record from the database. The signature (size,
|
||||
// head, tail, content) is the duplicate key; mtime is informational
|
||||
// only and used by scan for change detection.
|
||||
// only and used by scan for change detection. diskSize is the bytes the
|
||||
// file takes on disk, which trees adds up.
|
||||
type scanRec struct {
|
||||
size int64
|
||||
mtime time.Time
|
||||
head [sha256.Size]byte
|
||||
tail [sha256.Size]byte
|
||||
content [sha256.Size]byte
|
||||
path string
|
||||
size int64
|
||||
diskSize int64
|
||||
mtime time.Time
|
||||
head [sha256.Size]byte
|
||||
tail [sha256.Size]byte
|
||||
content [sha256.Size]byte
|
||||
path string
|
||||
}
|
||||
|
||||
// reportPhase names one phase of report with its place among all of
|
||||
@@ -39,17 +44,17 @@ func reportPhase(name string) string {
|
||||
}
|
||||
|
||||
// runReport implements the report subcommand: it prints the file-level
|
||||
// duplicates report as TSV on stdout. It never touches the scanned
|
||||
// filesystem; its only I/O is the database, stdout, and stderr. It
|
||||
// names the database on stderr before it opens it. Any database
|
||||
// problem, including a missing database, is fatal.
|
||||
// duplicates report as TSV on stdout. Of the scanned filesystem it
|
||||
// only checks, with pathExists, that each path in a duplicate group is
|
||||
// still there. It names the database on stderr before it opens it. Any
|
||||
// database problem, including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in two phases, each under a progress display of its
|
||||
// own on stderr: open (open the database) and read (count the records,
|
||||
// then read them in signature order, writing each size's duplicate
|
||||
// groups to stdout once that size is read; see readReport). Each
|
||||
// display ends before the next phase starts, and before an error is
|
||||
// returned to be printed.
|
||||
// own on stderr: open (open the database) and read (read the number of
|
||||
// records from the totals table, then read the records in signature
|
||||
// order, writing each size's duplicate groups to stdout once that size
|
||||
// is read; see readReport). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
@@ -75,13 +80,14 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
|
||||
defer func() { _ = db.Close() }()
|
||||
|
||||
// The read phase writes the report to stdout while it reads the
|
||||
// records, so it may wait on either.
|
||||
// The read phase checks the paths of each duplicate group on disk
|
||||
// and writes the report to stdout while it reads the records, so it
|
||||
// may wait on any of the three.
|
||||
prog = newProgress(progressParams{
|
||||
label: reportPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db/out",
|
||||
waitingFor: countingRecords, stalledOn: "db/disk/out",
|
||||
})
|
||||
report, err := readReport(ctx, db, dbPath, stdout, prog)
|
||||
report, err := readReport(ctx, db, dbPath, stdout, pathExists, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -102,14 +108,15 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
// dbPath, as its total, then reads every record in signature order,
|
||||
// counting each on prog, and writes the report to stdout: the header
|
||||
// line, then the rows of each size's duplicate groups once that size is
|
||||
// read. stdout may be the terminal prog is drawn on, so the lines go
|
||||
// through prog's writer. It returns what it wrote, with the totals for
|
||||
// the summary line.
|
||||
// read. A path for which exists returns false is left out of its group.
|
||||
// stdout may be the terminal prog is drawn on, so the lines go through
|
||||
// prog's writer. It returns what it wrote, with the totals for the
|
||||
// summary line.
|
||||
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
stdout io.Writer, prog *progress,
|
||||
stdout io.Writer, exists func(path string) bool, prog *progress,
|
||||
) (*reportWriter, error) {
|
||||
out := prog.writer(stdout)
|
||||
report := &reportWriter{out: out}
|
||||
report := &reportWriter{out: out, exists: exists}
|
||||
|
||||
_, err := fmt.Fprintln(out, "first\tdupe\tsize")
|
||||
if err != nil {
|
||||
@@ -156,6 +163,10 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
type reportWriter struct {
|
||||
out io.Writer
|
||||
|
||||
// exists reports whether a path is still on disk. A path that is
|
||||
// gone is left out of its group.
|
||||
exists func(path string) bool
|
||||
|
||||
sig scanRec // the signature being read; its path is not used
|
||||
paths []string // the paths read so far with sig
|
||||
groups [][]string // the duplicate groups of sig.size, each sorted
|
||||
@@ -192,11 +203,20 @@ func (w *reportWriter) add(r scanRec) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// endGroup ends the paths read with w.sig: two or more are a duplicate
|
||||
// group, held with its paths sorted until writeGroups writes it.
|
||||
// endGroup ends the paths read with w.sig: two or more that are still
|
||||
// on disk are a duplicate group, held with its paths sorted until
|
||||
// writeGroups writes it. Empty files form no group, since removing one
|
||||
// frees nothing. Only the paths of a possible group are checked, never
|
||||
// a path with no duplicate or an empty file.
|
||||
func (w *reportWriter) endGroup() {
|
||||
if len(w.paths) < minGroupSize {
|
||||
// A path with no duplicate is dropped, and its slice reused.
|
||||
if len(w.paths) >= minGroupSize && w.sig.size > 0 {
|
||||
w.paths = slices.DeleteFunc(w.paths, func(p string) bool {
|
||||
return !w.exists(p)
|
||||
})
|
||||
}
|
||||
|
||||
if len(w.paths) < minGroupSize || w.sig.size == 0 {
|
||||
// The paths are dropped, and their slice reused.
|
||||
w.paths = w.paths[:0]
|
||||
|
||||
return
|
||||
@@ -243,6 +263,16 @@ func (w *reportWriter) finish() error {
|
||||
return w.writeGroups()
|
||||
}
|
||||
|
||||
// pathExists reports whether path is still on disk, for report and
|
||||
// trees. It is false only when lstat finds that the path, or a
|
||||
// directory above it, is gone. Any other error counts as present, so
|
||||
// a user who may not look inside a directory still sees its copies.
|
||||
func pathExists(path string) bool {
|
||||
_, err := os.Lstat(path)
|
||||
|
||||
return !errors.Is(err, fs.ErrNotExist) && !errors.Is(err, syscall.ENOTDIR)
|
||||
}
|
||||
|
||||
// escapePath returns a path as it is written in a report column (README
|
||||
// "Report output format"): a backslash, tab, newline or carriage return
|
||||
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // checks the unexported dupeGroup results
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -38,8 +38,8 @@ func awkwardPairRecs() []scanRec {
|
||||
h, tl, c := testHash("h"), testHash("t"), testHash("c")
|
||||
|
||||
return []scanRec{
|
||||
{size: 5, head: h, tail: tl, content: c, path: "/d/A/f"},
|
||||
{size: 5, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
|
||||
{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: "/d/A/f"},
|
||||
{size: 5, diskSize: 4096, head: h, tail: tl, content: c, path: awkwardDir + "/f"},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -72,15 +72,37 @@ type dupeGroup struct {
|
||||
paths []string
|
||||
}
|
||||
|
||||
// everyPathExists stands in for pathExists where a test's records name
|
||||
// made-up paths that are not on disk.
|
||||
func everyPathExists(string) bool { return true }
|
||||
|
||||
// recsOnDisk returns a new directory and recs with each path moved
|
||||
// under it, after creating an empty file at each moved path: report and
|
||||
// trees then find every path still on disk.
|
||||
func recsOnDisk(t *testing.T, recs []scanRec) (string, []scanRec) {
|
||||
t.Helper()
|
||||
|
||||
dir := realTempDir(t)
|
||||
moved := make([]scanRec, len(recs))
|
||||
|
||||
for i, r := range recs {
|
||||
r.path = writeFile(t, dir, r.path, nil)
|
||||
moved[i] = r
|
||||
}
|
||||
|
||||
return dir, moved
|
||||
}
|
||||
|
||||
// dupeGroups returns the duplicate groups report reads from db, in
|
||||
// report order, from the rows it writes: a group's rows come together,
|
||||
// each naming the group's first path.
|
||||
// each naming the group's first path. Every path counts as still on
|
||||
// disk.
|
||||
func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
|
||||
t.Helper()
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
_, err := readReport(t.Context(), db, "", &out, nil)
|
||||
_, err := readReport(t.Context(), db, "", &out, everyPathExists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -127,7 +149,8 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
|
||||
}
|
||||
|
||||
func TestRunReportEscapesPaths(t *testing.T) {
|
||||
t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
|
||||
dir, recs := recsOnDisk(t, awkwardPairRecs())
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
@@ -138,7 +161,7 @@ func TestRunReportEscapesPaths(t *testing.T) {
|
||||
}
|
||||
|
||||
want := "first\tdupe\tsize\n" +
|
||||
`/d/\tone\ntwo\rthree\\four/f` + "\t/d/A/f\t5\n"
|
||||
dir + `/d/\tone\ntwo\rthree\\four/f` + "\t" + dir + "/d/A/f\t5\n"
|
||||
if got := stdout.String(); got != want {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
@@ -172,7 +195,8 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
|
||||
// Counts the records read, and draws nothing.
|
||||
prog := &progress{now: time.Now}
|
||||
|
||||
_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
|
||||
_, err = readReport(t.Context(), db, "", failingWriter{},
|
||||
everyPathExists, prog)
|
||||
if !errors.Is(err, errWriteFailed) ||
|
||||
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
||||
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
|
||||
@@ -203,7 +227,8 @@ const smokeReport = "first\tdupe\tsize\n" +
|
||||
// it ends; read counts the 6 records before it reads them. The summary
|
||||
// comes last, and stdout holds the report alone.
|
||||
func TestRunReportProgressNamesEachPhase(t *testing.T) {
|
||||
path := seedDatabase(t, smokeTreeRecs())
|
||||
dir, recs := recsOnDisk(t, smokeTreeRecs())
|
||||
path := seedDatabase(t, recs)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
stderr := captureStderr(t)
|
||||
@@ -216,8 +241,9 @@ func TestRunReportProgressNamesEachPhase(t *testing.T) {
|
||||
code, exitOK, stderr())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != smokeReport {
|
||||
t.Errorf("stdout = %q, want %q", got, smokeReport)
|
||||
report := strings.ReplaceAll(smokeReport, "/d/", dir+"/d/")
|
||||
if got := stdout.String(); got != report {
|
||||
t.Errorf("stdout = %q, want %q", got, report)
|
||||
}
|
||||
|
||||
// How each line starts; the rate at the end of a phase varies.
|
||||
@@ -266,7 +292,7 @@ func TestReadReportOnTerminal(t *testing.T) {
|
||||
waitingFor: countingRecords,
|
||||
}, terminalWidth, time.Hour, time.Now)
|
||||
|
||||
_, err = readReport(t.Context(), db, "", os.Stderr, p)
|
||||
_, err = readReport(t.Context(), db, "", os.Stderr, everyPathExists, p)
|
||||
|
||||
p.finish()
|
||||
|
||||
@@ -301,6 +327,8 @@ func TestRunReportsIgnoreInsertionOrder(t *testing.T) {
|
||||
scanRec{size: 50, head: a, tail: a, content: a, path: "/x/1"},
|
||||
)
|
||||
|
||||
_, recs = recsOnDisk(t, recs)
|
||||
|
||||
reversed := slices.Clone(recs)
|
||||
slices.Reverse(reversed)
|
||||
|
||||
@@ -434,6 +462,140 @@ func TestDupeGroups(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
|
||||
// duplicate group, while 1-byte files still do.
|
||||
func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e, x := sha256.Sum256(nil), testHash("x")
|
||||
recs := []scanRec{
|
||||
{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
|
||||
{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
|
||||
{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
|
||||
}
|
||||
|
||||
groups := dupeGroupsOf(t, recs)
|
||||
if len(groups) != 1 || groups[0].size != 1 ||
|
||||
!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
|
||||
t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadReportChecksOnlyPossibleDuplicates checks that report asks
|
||||
// whether a path is on disk only when its signature has two or more
|
||||
// records, and never for an empty file.
|
||||
func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
a, u, e := testHash("a"), testHash("u"), sha256.Sum256(nil)
|
||||
recs := []scanRec{
|
||||
{size: 9, head: a, tail: a, content: a, path: "/copy/1"},
|
||||
{size: 9, head: a, tail: a, content: a, path: "/copy/2"},
|
||||
{size: 9, head: a, tail: a, content: a, path: "/copy/3"},
|
||||
{size: 9, head: u, tail: u, content: u, path: "/lonely"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/empty1"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/empty2"},
|
||||
}
|
||||
|
||||
db := openTestDB(t)
|
||||
|
||||
err := applyChanges(t.Context(), db, recs, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
var checked []string
|
||||
|
||||
exists := func(path string) bool {
|
||||
checked = append(checked, path)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
_, err = readReport(t.Context(), db, "", io.Discard, exists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
slices.Sort(checked)
|
||||
|
||||
want := []string{"/copy/1", "/copy/2", "/copy/3"}
|
||||
if !slices.Equal(checked, want) {
|
||||
t.Errorf("paths checked on disk = %q, want %q", checked, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadReportDropsGroupLeftWithOnePath checks that a group of two
|
||||
// paths, one of them gone, is no duplicate group. If kept, it would
|
||||
// write no row, but the summary line would count it.
|
||||
func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
a := testHash("a")
|
||||
recs := []scanRec{
|
||||
{size: 9, head: a, tail: a, content: a, path: "/a/kept"},
|
||||
{size: 9, head: a, tail: a, content: a, path: "/a/gone"},
|
||||
}
|
||||
|
||||
db := openTestDB(t)
|
||||
|
||||
err := applyChanges(t.Context(), db, recs, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
exists := func(path string) bool { return path != "/a/gone" }
|
||||
|
||||
report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if report.dupeGroups != 0 {
|
||||
t.Errorf("duplicate groups = %d, want 0", report.dupeGroups)
|
||||
}
|
||||
}
|
||||
|
||||
// TestReadReportKeepsGroupLeftWithTwoPaths checks that a group of three
|
||||
// paths, one of them gone, is a duplicate group of the other two.
|
||||
func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
a := testHash("a")
|
||||
recs := []scanRec{
|
||||
{size: 9, head: a, tail: a, content: a, path: "/b/kept1"},
|
||||
{size: 9, head: a, tail: a, content: a, path: "/b/gone"},
|
||||
{size: 9, head: a, tail: a, content: a, path: "/b/kept2"},
|
||||
}
|
||||
|
||||
db := openTestDB(t)
|
||||
|
||||
err := applyChanges(t.Context(), db, recs, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
exists := func(path string) bool { return path != "/b/gone" }
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
report, err := readReport(t.Context(), db, "", &out, exists, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
if report.dupeGroups != 1 {
|
||||
t.Errorf("duplicate groups = %d, want 1", report.dupeGroups)
|
||||
}
|
||||
|
||||
want := "first\tdupe\tsize\n/b/kept1\t/b/kept2\t9\n"
|
||||
if got := out.String(); got != want {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDupeGroupsContentSeparates(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"cmp"
|
||||
@@ -52,23 +52,26 @@ const workQueueDepth = 1024
|
||||
// has already printed its line, so run prints nothing more.
|
||||
var errInterrupted = errors.New("scan interrupted")
|
||||
|
||||
// fileRec carries one statted file between the scan phases. dev and
|
||||
// ino identify the underlying inode so hard-linked paths can share
|
||||
// one read; both are zero when the platform exposes no inode.
|
||||
// fileRec carries one statted file between the scan phases. diskSize is
|
||||
// the bytes the file takes on disk. dev and ino identify the underlying
|
||||
// inode so hard-linked paths can share one read; both are zero when the
|
||||
// platform exposes no inode.
|
||||
type fileRec struct {
|
||||
path string
|
||||
size int64
|
||||
mtime time.Time
|
||||
dev uint64
|
||||
ino uint64
|
||||
path string
|
||||
size int64
|
||||
diskSize int64
|
||||
mtime time.Time
|
||||
dev uint64
|
||||
ino uint64
|
||||
}
|
||||
|
||||
// fileMeta is the in-memory index entry for one existing database
|
||||
// record: just enough for change detection. Hashes stay on disk; at
|
||||
// record: what the walk compares the file with. Hashes stay on disk; at
|
||||
// tens of millions of records they would dominate the scan's memory.
|
||||
type fileMeta struct {
|
||||
size int64
|
||||
mtime time.Time
|
||||
size int64
|
||||
diskSize int64
|
||||
mtime time.Time
|
||||
}
|
||||
|
||||
// scanUnit is what every scan phase counts, as its progress display
|
||||
@@ -206,29 +209,30 @@ func interrupted(walked int) error {
|
||||
return errInterrupted
|
||||
}
|
||||
|
||||
// resolveRoots converts each PATH operand to an absolute, lexically
|
||||
// cleaned path (symlinks are not resolved) and verifies that it
|
||||
// exists. Database records are keyed by absolute path, so scan results
|
||||
// must not depend on the working directory.
|
||||
// resolveRoots converts each PATH operand to its real path: absolute,
|
||||
// with every symlink in it followed. Database records are keyed by
|
||||
// that path, so scan results depend neither on the working directory
|
||||
// nor on the symlinks an operand was given through. An operand that
|
||||
// cannot be resolved, because it or a symlink's target does not exist
|
||||
// or cannot be read, is a fatal error before any scanning.
|
||||
func resolveRoots(roots []string) ([]string, error) {
|
||||
abs := make([]string, 0, len(roots))
|
||||
resolved := make([]string, 0, len(roots))
|
||||
|
||||
for _, root := range roots {
|
||||
a, err := filepath.Abs(root)
|
||||
a, err := absolutePath(root)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
}
|
||||
|
||||
// A nonexistent operand is a fatal error before any scanning.
|
||||
_, err = os.Lstat(a)
|
||||
r, err := filepath.EvalSymlinks(a)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
}
|
||||
|
||||
abs = append(abs, a)
|
||||
resolved = append(resolved, r)
|
||||
}
|
||||
|
||||
return abs, nil
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
// pruneRoots drops operands already covered by another operand:
|
||||
@@ -339,11 +343,12 @@ type scanStats struct {
|
||||
// whatever remains after the walk and hash phases is deleted by the
|
||||
// update phase.
|
||||
type scanState struct {
|
||||
db *sql.DB
|
||||
existing map[string]fileMeta
|
||||
toHash []fileRec // new and changed files: must be read
|
||||
batch []scanRec
|
||||
st scanStats
|
||||
db *sql.DB
|
||||
existing map[string]fileMeta
|
||||
toHash []fileRec // new and changed files: must be read
|
||||
diskSizes []fileRec // unchanged files whose disk_size changed
|
||||
batch []scanRec
|
||||
st scanStats
|
||||
}
|
||||
|
||||
// syncScan synchronizes the database with the filesystem under roots;
|
||||
@@ -413,14 +418,16 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
||||
|
||||
// walkableRoots returns the operands the walk can start from: regular
|
||||
// files, and directories not named .zfs, none of them inside a
|
||||
// directory named .zfs. Every other operand is warned about, counted as
|
||||
// skipped, and dropped. A dropped operand is no longer a root, so the
|
||||
// records stored beneath it are treated as if it had not been given:
|
||||
// those under no other root count as outside the roots and are not
|
||||
// deleted as unverified. An operand on a filesystem in rules that is not
|
||||
// persistent is dropped the same way, with a line saying so that is not
|
||||
// a warning, and is not counted as skipped. An operand that fails lstat
|
||||
// here is kept, and the walk warns about it unless it no longer exists.
|
||||
// directory named .zfs. The operands are real paths from resolveRoots,
|
||||
// so these checks apply to what a symlink operand points at. Every
|
||||
// other operand is warned about, counted as skipped, and dropped. A
|
||||
// dropped operand is no longer a root, so the records stored beneath it
|
||||
// are treated as if it had not been given: those under no other root
|
||||
// count as outside the roots and are not deleted as unverified. An
|
||||
// operand on a filesystem in rules that is not persistent is dropped
|
||||
// the same way, with a line saying so that is not a warning, and is not
|
||||
// counted as skipped. An operand that fails lstat here is kept, and the
|
||||
// walk warns about it unless it no longer exists.
|
||||
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
||||
kept := make([]string, 0, len(roots))
|
||||
|
||||
@@ -455,8 +462,8 @@ func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
||||
// included, so its display counts against the number of records.
|
||||
func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||
// Indexing tens of millions of records takes real time; without a
|
||||
// display the scan looks hung before the walk begins. Counting them
|
||||
// first gives the display its total.
|
||||
// display the scan looks hung before the walk begins. The number of
|
||||
// records, read first, gives the display its total.
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("load"), unit: scanUnit, total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db",
|
||||
@@ -472,24 +479,24 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
|
||||
|
||||
s.existing = make(map[string]fileMeta)
|
||||
|
||||
return loadFileMeta(ctx, s.db,
|
||||
func(path string, size int64, mtime time.Time) {
|
||||
prog.increment()
|
||||
return loadFileMeta(ctx, s.db, func(path string, m fileMeta) {
|
||||
prog.increment()
|
||||
|
||||
if underAnyRoot(path, roots) {
|
||||
s.existing[path] = fileMeta{size: size, mtime: mtime}
|
||||
}
|
||||
})
|
||||
if underAnyRoot(path, roots) {
|
||||
s.existing[path] = m
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// walkPhase drains the walk and queues every new or changed file for
|
||||
// the hash phase. An unchanged file needs nothing more, and its record
|
||||
// counts as verified. A record at or below a path the walk left out,
|
||||
// because -x stopped at it, it is a default exclude, or it is not on a
|
||||
// persistent filesystem, counts as outside the roots: it is not
|
||||
// deleted. That does not hold below a root at or inside that path,
|
||||
// since that root's own walk verifies the records below it; nor is a
|
||||
// line printed for such a root.
|
||||
// the hash phase. An unchanged file is not read, and its record counts
|
||||
// as verified. When its disk_size differs from the record's, it is
|
||||
// queued for the update phase, which writes the new disk_size. A record
|
||||
// at or below a path the walk left out, because -x stopped at it, it is
|
||||
// a default exclude, or it is not on a persistent filesystem, counts as
|
||||
// outside the roots: it is not deleted. That does not hold below a root
|
||||
// at or inside that path, since that root's own walk verifies the
|
||||
// records below it; nor is a line printed for such a root.
|
||||
func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
// The records the last scan left under the roots estimate how many
|
||||
// files the walk will find. A first scan has none, and no estimate.
|
||||
@@ -538,7 +545,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
prog.increment()
|
||||
|
||||
old, ok := s.existing[ev.rec.path]
|
||||
if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) {
|
||||
if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) {
|
||||
s.toHash = append(s.toHash, ev.rec)
|
||||
|
||||
continue
|
||||
@@ -547,6 +554,13 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
delete(s.existing, ev.rec.path)
|
||||
|
||||
s.st.unchanged++
|
||||
|
||||
// The space a file takes on disk can change while its size and
|
||||
// mtime stay the same. ZFS, for one, counts a new file's blocks
|
||||
// up to seconds after the write.
|
||||
if old.diskSize != ev.rec.diskSize {
|
||||
s.diskSizes = append(s.diskSizes, ev.rec)
|
||||
}
|
||||
}
|
||||
|
||||
prog.finish()
|
||||
@@ -695,12 +709,13 @@ func (s *scanState) recordRun(ctx context.Context, r hashResult) error {
|
||||
s.resolve(rec.path)
|
||||
|
||||
s.batch = append(s.batch, scanRec{
|
||||
size: rec.size,
|
||||
mtime: rec.mtime,
|
||||
head: r.head,
|
||||
tail: r.tail,
|
||||
content: r.content,
|
||||
path: rec.path,
|
||||
size: rec.size,
|
||||
diskSize: rec.diskSize,
|
||||
mtime: rec.mtime,
|
||||
head: r.head,
|
||||
tail: r.tail,
|
||||
content: r.content,
|
||||
path: rec.path,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -716,7 +731,7 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
err := applyBatch(ctx, s.db, s.batch, nil, nil)
|
||||
err := applyBatch(ctx, s.db, s.batch, nil, nil, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -727,9 +742,10 @@ func (s *scanState) commitFullBatch(ctx context.Context) error {
|
||||
}
|
||||
|
||||
// updatePhase writes the scan's tail under one progress display: the
|
||||
// final partial batch of hashed records, and deletions for every record
|
||||
// the scan did not verify (vanished files, plus paths that failed to
|
||||
// stat or hash).
|
||||
// final partial batch of hashed records, the new disk_size of every
|
||||
// unchanged file whose disk_size changed, and deletions for every
|
||||
// record the scan did not verify (vanished files, plus paths that
|
||||
// failed to stat or hash).
|
||||
func (s *scanState) updatePhase(ctx context.Context) error {
|
||||
deletes := make([]string, 0, len(s.existing))
|
||||
for path := range s.existing {
|
||||
@@ -741,7 +757,7 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
||||
|
||||
s.st.removed = len(deletes)
|
||||
|
||||
total := len(s.batch) + len(deletes)
|
||||
total := len(s.batch) + len(s.diskSizes) + len(deletes)
|
||||
prog := newProgress(progressParams{
|
||||
label: scanPhase("update"), unit: scanUnit, total: int64(total),
|
||||
stalledOn: "db",
|
||||
@@ -756,17 +772,12 @@ func (s *scanState) updatePhase(ctx context.Context) error {
|
||||
|
||||
s.batch = nil
|
||||
|
||||
return applyChanges(ctx, s.db, nil, deletes, prog)
|
||||
}
|
||||
|
||||
// mtimeAfter reports whether mtime a is later than mtime b.
|
||||
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
|
||||
func mtimeAfter(a, b time.Time) bool {
|
||||
if a.Unix() != b.Unix() {
|
||||
return a.Unix() > b.Unix()
|
||||
err = applyDiskSizes(ctx, s.db, s.diskSizes, prog)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return a.Nanosecond() > b.Nanosecond()
|
||||
return applyChanges(ctx, s.db, nil, deletes, prog)
|
||||
}
|
||||
|
||||
// underAnyRoot reports whether path is any of the roots or lies under
|
||||
@@ -876,19 +887,13 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
|
||||
// operandWarning returns the one-line warning for an operand the walk
|
||||
// does not start from, naming the path and what it is, or "" for one it
|
||||
// does: a regular file, or a directory not named .zfs, that is not
|
||||
// inside a directory named .zfs, neither as given nor with its symlinks
|
||||
// resolved. Symlinks are never followed, including as operands.
|
||||
// inside a directory named .zfs. resolveRoots has already followed the
|
||||
// operand's symlinks, so a symlink here is an operand replaced by one
|
||||
// after it was resolved, and is not followed.
|
||||
func operandWarning(root string, fi fs.FileInfo) string {
|
||||
// The walk never enters a .zfs directory, so it starts from nothing
|
||||
// inside one either, not even through a symlink. An operand whose
|
||||
// symlinks cannot be resolved is checked as given.
|
||||
resolved, err := filepath.EvalSymlinks(root)
|
||||
if err != nil {
|
||||
resolved = root
|
||||
}
|
||||
|
||||
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
|
||||
slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
|
||||
// inside one either.
|
||||
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") {
|
||||
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
|
||||
root)
|
||||
}
|
||||
@@ -961,11 +966,12 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
|
||||
|
||||
dev, ino := inodeOfInfo(fi)
|
||||
rec := fileRec{
|
||||
path: root,
|
||||
size: fi.Size(),
|
||||
mtime: fi.ModTime(),
|
||||
dev: dev,
|
||||
ino: ino,
|
||||
path: root,
|
||||
size: fi.Size(),
|
||||
diskSize: diskSizeOfInfo(fi),
|
||||
mtime: fi.ModTime(),
|
||||
dev: dev,
|
||||
ino: ino,
|
||||
}
|
||||
|
||||
if !isDatabaseFile(rec, rules) {
|
||||
@@ -1145,11 +1151,12 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
||||
|
||||
dev, ino := inodeOfInfo(info)
|
||||
rec := fileRec{
|
||||
path: p,
|
||||
size: info.Size(),
|
||||
mtime: info.ModTime(),
|
||||
dev: dev,
|
||||
ino: ino,
|
||||
path: p,
|
||||
size: info.Size(),
|
||||
diskSize: diskSizeOfInfo(info),
|
||||
mtime: info.ModTime(),
|
||||
dev: dev,
|
||||
ino: ino,
|
||||
}
|
||||
|
||||
if !isDatabaseFile(rec, rules) {
|
||||
@@ -1241,6 +1248,23 @@ func inodeOfInfo(fi fs.FileInfo) (uint64, uint64) {
|
||||
return statDev(st), st.Ino
|
||||
}
|
||||
|
||||
// statBlockSize is the unit of st_blocks in an lstat result on Linux and
|
||||
// macOS, whatever the filesystem's own block size.
|
||||
const statBlockSize = 512
|
||||
|
||||
// diskSizeOfInfo returns the bytes a file takes on disk: its allocated
|
||||
// blocks, st_blocks × 512. On a compressed or sparse file it is less
|
||||
// than the file's size. It is 0 when the platform exposes no block
|
||||
// count.
|
||||
func diskSizeOfInfo(fi fs.FileInfo) int64 {
|
||||
st, ok := fi.Sys().(*syscall.Stat_t)
|
||||
if !ok {
|
||||
return 0
|
||||
}
|
||||
|
||||
return st.Blocks * statBlockSize
|
||||
}
|
||||
|
||||
// hashResult carries the hashes computed for one inode run (or the
|
||||
// error that prevented computing them) from the pool's workers to the
|
||||
// hash phase. Its run leaves out the paths found removed.
|
||||
@@ -1,6 +1,6 @@
|
||||
//go:build darwin
|
||||
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import "syscall"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
//go:build !darwin
|
||||
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import "syscall"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // drives the unexported syncTree
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -42,6 +42,30 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
|
||||
return p
|
||||
}
|
||||
|
||||
// diskSizeOf returns the bytes the file at path takes on disk, from its
|
||||
// own lstat: st_blocks counts 512-byte units. ZFS counts a write's
|
||||
// blocks only when it commits the write, up to seconds later, and one
|
||||
// commit can take the file through several growing sizes. So this value
|
||||
// can change at any moment. Neither fsync nor unix.Sync hastens that,
|
||||
// but a unix.Sync call often takes long enough that the blocks are
|
||||
// counted when it returns. A test that needs the value to change waits
|
||||
// for the change. A test that compares it with a scan's disk_size reads
|
||||
// it before and after the scan. The rewrite test rescans until the two
|
||||
// reads match, then compares exactly. The other two tests fail only on
|
||||
// a disk_size outside the two reads.
|
||||
func diskSizeOf(t *testing.T, path string) int64 {
|
||||
t.Helper()
|
||||
|
||||
var st syscall.Stat_t
|
||||
|
||||
err := syscall.Lstat(path, &st)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
return st.Blocks * 512
|
||||
}
|
||||
|
||||
// pattern returns n bytes of deterministic content seeded by tag.
|
||||
func pattern(tag byte, n int) []byte {
|
||||
data := make([]byte, n)
|
||||
@@ -1070,7 +1094,7 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
|
||||
func buildSmokeTree(t *testing.T) string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
one := pattern(10, 2000)
|
||||
f1 := pattern(30, 3000)
|
||||
f2 := pattern(40, 100)
|
||||
@@ -1161,11 +1185,12 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
groups := dupeGroups(t, db)
|
||||
if len(groups) != 5 {
|
||||
t.Fatalf("len(groups) = %d, want 5", len(groups))
|
||||
if len(groups) != 4 {
|
||||
t.Fatalf("len(groups) = %d, want 4", len(groups))
|
||||
}
|
||||
|
||||
wantSizes := []int64{3000, 2000, 100, 1, 0}
|
||||
// The empty files empty1 and empty2 form no group.
|
||||
wantSizes := []int64{3000, 2000, 100, 1}
|
||||
for i, g := range groups {
|
||||
if g.size != wantSizes[i] {
|
||||
t.Errorf("groups[%d].size = %d, want %d",
|
||||
@@ -1190,7 +1215,7 @@ func assertSmokeTreeGroups(t *testing.T, dir string, db *sql.DB) {
|
||||
|
||||
super, dirs := dbTree(t, db)
|
||||
|
||||
tg := collectTreeGroups(dirs, super, nil)
|
||||
tg := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||
if len(tg) != 1 {
|
||||
t.Fatalf("len(tree groups) = %d, want 1", len(tg))
|
||||
}
|
||||
@@ -1229,9 +1254,9 @@ func TestScanPipeline(t *testing.T) {
|
||||
// TestScanProgressNamesEachPhase checks the plain progress lines of a
|
||||
// first scan, and of a rescan of one operand holding two of the
|
||||
// database's records. Each phase's first line names it with its place
|
||||
// among the four: load counts the records before reading them, and the
|
||||
// walk estimates its total from the records under its operand, or says
|
||||
// its time remaining is unknown when there are none.
|
||||
// among the four: load gets the number of records before reading them,
|
||||
// and the walk estimates its total from the records under its operand,
|
||||
// or says its time remaining is unknown when there are none.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestScanProgressNamesEachPhase(t *testing.T) {
|
||||
@@ -1359,6 +1384,47 @@ func TestSyncScanMtimeBump(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestSyncScanMtimeMovedBack replaces a file with other content of the
|
||||
// same size and an older mtime, as cp -p or a restore from backup does.
|
||||
// The next scan must hash the file again.
|
||||
func TestSyncScanMtimeMovedBack(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t)
|
||||
a := writeFile(t, dir, "a.bin", pattern(1, 500))
|
||||
|
||||
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
|
||||
|
||||
err := os.Chtimes(a, recorded, recorded)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
syncTree(t, db, dir)
|
||||
|
||||
replaced := pattern(3, 500)
|
||||
writeFile(t, dir, "a.bin", replaced)
|
||||
|
||||
older := recorded.Add(-time.Hour)
|
||||
|
||||
err = os.Chtimes(a, older, older)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
st := syncTree(t, db, dir)
|
||||
if st != (scanStats{walked: 1, updated: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
||||
}
|
||||
|
||||
assertWholeFileHashed(t, db, a, replaced)
|
||||
|
||||
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
|
||||
t.Fatalf("mtime = %v, want %v", r.mtime, older)
|
||||
}
|
||||
}
|
||||
|
||||
// assertWholeFileHashed fails unless the record for path holds the
|
||||
// whole-file hash of data as its head, tail, and content.
|
||||
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
|
||||
@@ -1417,6 +1483,159 @@ func TestScanStoresRawHashes(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestScanStoresDiskSize scans a sparse file, which takes less space on
|
||||
// disk than its length, once through its directory and once as the
|
||||
// operand itself. Each scan stores the disk_size of the file's lstat.
|
||||
func TestScanStoresDiskSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
const length = 1024 * 1024
|
||||
|
||||
p := sparseFile(t, t.TempDir(), "sparse", length)
|
||||
pokeAt(t, p, 0, []byte("x"))
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
|
||||
size := diskSizeOf(t, p)
|
||||
if size == 0 || size >= length {
|
||||
t.Fatalf("the file takes %d bytes on disk, want more than none "+
|
||||
"and fewer than its length %d", size, length)
|
||||
}
|
||||
|
||||
for _, operand := range []string{filepath.Dir(p), p} {
|
||||
db := openTestDB(t)
|
||||
|
||||
// ZFS may count the blocks during the scan; see diskSizeOf.
|
||||
before := diskSizeOf(t, p)
|
||||
|
||||
syncTree(t, db, operand)
|
||||
|
||||
after := diskSizeOf(t, p)
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got < before || got > after {
|
||||
t.Errorf("scan of %s: disk_size = %d, want %d to %d",
|
||||
operand, got, before, after)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSyncScanRewriteUpdatesDiskSize writes into the hole of a scanned
|
||||
// sparse file and moves its mtime on. The rescan hashes the file again
|
||||
// and rewrites its record with the new disk_size.
|
||||
func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t)
|
||||
p := sparseFile(t, dir, "sparse", 1024*1024)
|
||||
pokeAt(t, p, 0, []byte("x"))
|
||||
|
||||
syncTree(t, db, dir)
|
||||
|
||||
recorded := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
|
||||
pokeAt(t, p, 512*1024, pattern(7, 64*1024))
|
||||
|
||||
// The write may leave the mtime the scan recorded; a later one makes
|
||||
// the rescan read the file again.
|
||||
later := time.Now().Add(time.Hour)
|
||||
|
||||
err := os.Chtimes(p, later, later)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Wait for ZFS to count the write; see diskSizeOf.
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for diskSizeOf(t, p) == recorded {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("the file still takes %d bytes on disk 30 seconds "+
|
||||
"after the write into its hole", recorded)
|
||||
}
|
||||
|
||||
unix.Sync()
|
||||
}
|
||||
|
||||
// ZFS may still be counting, so the size may change during a rescan.
|
||||
// Only a rescan with the same size on both sides of it can be compared
|
||||
// exactly. A rescan that keeps the old size still fails, because the
|
||||
// wait made before differ from it.
|
||||
for {
|
||||
before := diskSizeOf(t, p)
|
||||
|
||||
st := syncTree(t, db, dir)
|
||||
|
||||
after := diskSizeOf(t, p)
|
||||
|
||||
if st != (scanStats{walked: 1, updated: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
||||
}
|
||||
|
||||
if before == after {
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got != after {
|
||||
t.Errorf("disk_size = %d, want %d", got, after)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("the size on disk was still changing 30 seconds after "+
|
||||
"the write into its hole: %d, then %d", before, after)
|
||||
}
|
||||
|
||||
// A later mtime makes the next rescan read the file again.
|
||||
later = later.Add(time.Second)
|
||||
|
||||
err = os.Chtimes(p, later, later)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestSyncScanUnchangedUpdatesDiskSize gives a file's record a
|
||||
// disk_size the file does not have, as when ZFS had not yet counted the
|
||||
// file's blocks at the last scan. The rescan writes the file's own
|
||||
// disk_size into the record without reading the file again.
|
||||
func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
db := openTestDB(t)
|
||||
p := writeFile(t, dir, "a.bin", pattern(1, 500))
|
||||
|
||||
syncTree(t, db, dir)
|
||||
|
||||
// lstat counts 512-byte units, so no file takes 1 byte on disk.
|
||||
_, err := db.ExecContext(t.Context(),
|
||||
"UPDATE files SET disk_size = 1 WHERE path = ?", []byte(p))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
|
||||
// ZFS may count the blocks during the rescan; see diskSizeOf.
|
||||
before := diskSizeOf(t, p)
|
||||
|
||||
// A file read again would count as updated.
|
||||
st := syncTree(t, db, dir)
|
||||
|
||||
after := diskSizeOf(t, p)
|
||||
|
||||
if st != (scanStats{walked: 1, unchanged: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
|
||||
}
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got < before || got > after {
|
||||
t.Errorf("disk_size = %d, want %d to %d", got, before, after)
|
||||
}
|
||||
}
|
||||
|
||||
// TestSyncScanSameSecondRewrite rewrites a file in place at the same
|
||||
// size with an mtime later in the same second as the recorded one: the
|
||||
// next scan must notice the change and re-hash the file. The operand is
|
||||
@@ -2088,23 +2307,86 @@ func TestPruneRoots(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestReportsNeverTouchFilesystem(t *testing.T) {
|
||||
t.Parallel()
|
||||
// TestRunReportsLeaveOutRemovedCopies scans the smoke tree, then
|
||||
// removes copies the reports name. report leaves out each removed file,
|
||||
// and a group left with one path. trees checks only the root of each
|
||||
// duplicate tree: with a file under t2 removed it still names t2, and
|
||||
// with t2 removed it names nothing.
|
||||
func TestRunReportsLeaveOutRemovedCopies(t *testing.T) {
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
dir := buildSmokeTree(t)
|
||||
db := openTestDB(t)
|
||||
p := func(name string) string { return filepath.Join(dir, name) }
|
||||
|
||||
syncTree(t, db, dir)
|
||||
scanOK(t, dir)
|
||||
|
||||
// Remove the scanned tree entirely; the analysis must be
|
||||
// unaffected because it reads the database alone.
|
||||
err := os.RemoveAll(dir)
|
||||
for _, name := range []string{"b/copy2.bin", "tiny2", "t2/sub/f2"} {
|
||||
err := os.Remove(p(name))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
want := "first\tdupe\tsize\n" +
|
||||
p("t1/f1") + "\t" + p("t2/f1") + "\t3000\n" +
|
||||
p("t1/f1") + "\t" + p("t3/f1") + "\t3000\n" +
|
||||
p("a/one.bin") + "\t" + p("b/copy.bin") + "\t2000\n" +
|
||||
p("t1/sub/f2") + "\t" + p("t3/sub/f2renamed") + "\t100\n"
|
||||
if got := runStdout(t, cmdReport); got != want {
|
||||
t.Errorf("report stdout = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
// disk_size, the last column, depends on the filesystem.
|
||||
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
|
||||
|
||||
want = header + p("t1") + "\t" + p("t2") + "\t2\t3100\t"
|
||||
if got := runStdout(t, cmdTrees); !strings.HasPrefix(got, want) ||
|
||||
strings.Count(got, "\n") != 2 {
|
||||
t.Errorf("trees stdout = %q, want one row starting %q", got, want)
|
||||
}
|
||||
|
||||
err := os.RemoveAll(p("t2"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
assertSmokeDupeGroups(t, dir, db)
|
||||
assertSmokeTreeGroups(t, dir, db)
|
||||
if got := runStdout(t, cmdTrees); got != header {
|
||||
t.Errorf("trees stdout after removing t2 = %q, want %q", got, header)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPathExists(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
dir := t.TempDir()
|
||||
file := writeFile(t, dir, "file", nil)
|
||||
locked := writeFile(t, dir, "locked/file", nil)
|
||||
|
||||
// Without search permission on its directory, a file cannot be
|
||||
// looked at by anyone but root.
|
||||
err := os.Chmod(filepath.Dir(locked), 0)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Runs before t.TempDir's own cleanup, which must delete the files.
|
||||
t.Cleanup(func() {
|
||||
//nolint:gosec // removing the directory needs its search bit back
|
||||
_ = os.Chmod(filepath.Dir(locked), 0o700)
|
||||
})
|
||||
|
||||
cases := map[string]bool{
|
||||
dir: true,
|
||||
file: true,
|
||||
locked: true,
|
||||
filepath.Join(dir, "gone"): false,
|
||||
filepath.Join(file, "under-a-file"): false,
|
||||
}
|
||||
for path, want := range cases {
|
||||
if got := pathExists(path); got != want {
|
||||
t.Errorf("pathExists(%q) = %v, want %v", path, got, want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnderRoot(t *testing.T) {
|
||||
@@ -2133,33 +2415,24 @@ func TestUnderRoot(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
|
||||
// database, reached directly and through a symlinked directory. The
|
||||
// walk passes over the database file, its -wal and -shm files and its
|
||||
// .lock file with no line and no count, and the record an earlier scan
|
||||
// left for the .lock file is deleted. A rescan of the unchanged tree
|
||||
// updates nothing, and report lists none of the database's files.
|
||||
// database, and in one case also a hard link to the database file. The
|
||||
// walk knows the database's files by device ID and inode number, so it
|
||||
// passes over the database file, its -wal and -shm files, its .lock
|
||||
// file and the hard link with no line and no count. The record an
|
||||
// earlier scan left for the .lock file is deleted. A rescan of the
|
||||
// unchanged tree updates nothing, and report lists none of the
|
||||
// database's files.
|
||||
func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||
for _, viaSymlink := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
tree := filepath.Join(base, "real", "tree")
|
||||
operand := tree
|
||||
for _, viaHardLink := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("via hard link %v", viaHardLink), func(t *testing.T) {
|
||||
tree := realTempDir(t)
|
||||
hardLink := filepath.Join(tree, "hardlink")
|
||||
|
||||
writeFile(t, tree, "a", pattern(1, 100))
|
||||
writeFile(t, tree, "b", pattern(1, 100))
|
||||
writeFile(t, tree, "empty1", nil)
|
||||
writeFile(t, tree, "empty2", nil)
|
||||
|
||||
if viaSymlink {
|
||||
err := os.Symlink(filepath.Join(base, "real"),
|
||||
filepath.Join(base, "link"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
operand = filepath.Join(base, "link", "tree")
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tree, "db.sqlite")
|
||||
t.Setenv(databaseEnv, dbPath)
|
||||
|
||||
@@ -2169,7 +2442,7 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||
}
|
||||
|
||||
err = applyChanges(t.Context(), db, []scanRec{
|
||||
{path: filepath.Join(operand, "db.sqlite.lock")},
|
||||
{path: filepath.Join(tree, "db.sqlite.lock")},
|
||||
}, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -2177,17 +2450,30 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||
|
||||
closeScanDatabase(t.Context(), db, dbPath)
|
||||
|
||||
assertScanSummary(t,
|
||||
if viaHardLink {
|
||||
err = os.Link(dbPath, hardLink)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
stderr := assertScanSummary(t,
|
||||
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
|
||||
operand)
|
||||
assertScanSummary(t,
|
||||
tree)
|
||||
stderr += assertScanSummary(t,
|
||||
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
|
||||
operand)
|
||||
tree)
|
||||
|
||||
if strings.Contains(stderr, hardLink) {
|
||||
t.Errorf("scan stderr = %q, want no line naming %s",
|
||||
stderr, hardLink)
|
||||
}
|
||||
|
||||
report := runStdout(t, cmdReport)
|
||||
if strings.Contains(report, "db.sqlite") ||
|
||||
!strings.Contains(report, filepath.Join(operand, "empty2")) {
|
||||
t.Errorf("report = %q, want the empty files and none of "+
|
||||
strings.Contains(report, hardLink) ||
|
||||
!strings.Contains(report, filepath.Join(tree, "b")) {
|
||||
t.Errorf("report = %q, want a and b and none of "+
|
||||
"the database's files", report)
|
||||
}
|
||||
})
|
||||
@@ -2211,8 +2497,9 @@ func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
|
||||
|
||||
// assertScanSummary scans operands and fails the test unless the
|
||||
// summary line gives counts with nothing skipped, and the line naming
|
||||
// the database is the only one that names one of its files.
|
||||
func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
||||
// the database is the only one that names one of its files. It returns
|
||||
// everything the scan printed to stderr.
|
||||
func assertScanSummary(t *testing.T, counts string, operands ...string) string {
|
||||
t.Helper()
|
||||
|
||||
stderr := scanOK(t, operands...)
|
||||
@@ -2226,4 +2513,6 @@ func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
||||
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
|
||||
"database to name one of its files", operands, stderr)
|
||||
}
|
||||
|
||||
return stderr
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -22,6 +22,10 @@ type treeNode struct {
|
||||
digest [sha256.Size]byte
|
||||
fileCount int64
|
||||
totalSize int64
|
||||
// diskSize is the bytes the directory's files take on disk. Two
|
||||
// copies of one tree have the same totalSize, but their diskSize
|
||||
// can differ.
|
||||
diskSize int64
|
||||
}
|
||||
|
||||
// treesPhase names one phase of trees with its place among all of them,
|
||||
@@ -34,17 +38,18 @@ func treesPhase(name string) string {
|
||||
// runTrees implements the trees subcommand: it reads every record from
|
||||
// the database in path order, reconstructs the directory hierarchy from
|
||||
// the record paths, computes a Merkle-style digest per directory, and
|
||||
// prints maximal duplicate-tree groups as TSV on stdout. It never
|
||||
// touches the scanned filesystem; its only I/O is the database, stdout,
|
||||
// and stderr. It names the database on stderr before it opens it. Any
|
||||
// database problem, including a missing database, is fatal.
|
||||
// prints maximal duplicate-tree groups as TSV on stdout. Of the scanned
|
||||
// filesystem it only checks, with pathExists, that the root of each
|
||||
// duplicate tree is still there, never the files below it. It names the
|
||||
// database on stderr before it opens it. Any database problem,
|
||||
// including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in four phases, each under a progress display of its
|
||||
// own on stderr: open (open the database), read (count the records,
|
||||
// then read them into the hierarchy), group (group the directories by
|
||||
// digest) and write (write the groups to stdout). Each display ends
|
||||
// before the next phase starts, and before an error is returned to be
|
||||
// printed.
|
||||
// own on stderr: open (open the database), read (read the number of
|
||||
// records from the totals table, then read the records into the
|
||||
// hierarchy), group (group the directories by digest) and write (write
|
||||
// the groups to stdout). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
@@ -82,13 +87,13 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
return fmt.Errorf("database %s: %w", dbPath, err)
|
||||
}
|
||||
|
||||
// Grouping works in memory, so it waits on nothing but the
|
||||
// processor.
|
||||
// Grouping works in memory, then checks the root of each duplicate
|
||||
// tree on disk, so it waits on the processor or the disk.
|
||||
prog = newProgress(progressParams{
|
||||
label: treesPhase("group"), unit: "directories",
|
||||
total: int64(len(allDirs)), stalledOn: "cpu",
|
||||
total: int64(len(allDirs)), stalledOn: "cpu/disk",
|
||||
})
|
||||
dupes := collectTreeGroups(allDirs, super, prog)
|
||||
dupes := collectTreeGroups(allDirs, super, pathExists, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -139,15 +144,16 @@ func readTree(ctx context.Context, db *sql.DB,
|
||||
|
||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||
// a row for every member of each group but its first, counting each
|
||||
// group on prog once its rows are written. stdout may be the terminal
|
||||
// prog is drawn on, so the lines go through prog's writer. It returns
|
||||
// the number of rows (dupe trees) and the bytes they could reclaim.
|
||||
// group on prog once its rows are written. The disk_size column is the
|
||||
// dupe's own. stdout may be the terminal prog is drawn on, so the lines
|
||||
// go through prog's writer. It returns the number of rows (dupe trees)
|
||||
// and the bytes they could reclaim.
|
||||
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||
prog *progress,
|
||||
) (int, int64, error) {
|
||||
out := prog.writer(stdout)
|
||||
|
||||
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize")
|
||||
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size")
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
@@ -159,9 +165,9 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||
for _, g := range dupes {
|
||||
first := g[0]
|
||||
for _, n := range g[1:] {
|
||||
_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\n",
|
||||
_, err = fmt.Fprintf(out, "%s\t%s\t%d\t%d\t%d\n",
|
||||
escapePath(first.path), escapePath(n.path),
|
||||
first.fileCount, first.totalSize)
|
||||
first.fileCount, first.totalSize, n.diskSize)
|
||||
if err != nil {
|
||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
||||
}
|
||||
@@ -232,6 +238,7 @@ func (b *treeBuilder) add(r scanRec) {
|
||||
dir.entries = append(dir.entries, fileEntry(name, r))
|
||||
dir.fileCount++
|
||||
dir.totalSize += r.size
|
||||
dir.diskSize += r.diskSize
|
||||
}
|
||||
|
||||
// openDir starts the directory called name inside the innermost open
|
||||
@@ -270,6 +277,7 @@ func (b *treeBuilder) closeTo(n int) {
|
||||
"d\x00"+name+"\x00"+string(dir.digest[:]))
|
||||
dir.parent.fileCount += dir.fileCount
|
||||
dir.parent.totalSize += dir.totalSize
|
||||
dir.parent.diskSize += dir.diskSize
|
||||
b.dirs = append(b.dirs, dir)
|
||||
}
|
||||
}
|
||||
@@ -293,10 +301,14 @@ func fileEntry(name string, r scanRec) string {
|
||||
|
||||
// collectTreeGroups groups directories by digest, counting each one on
|
||||
// prog as it is grouped, and returns every maximal group with two or
|
||||
// more members, each group's members sorted by path, groups ordered by
|
||||
// tree size descending then by first path ascending.
|
||||
// more members for which exists returns true, each group's members
|
||||
// sorted by path, groups ordered by tree size descending then by first
|
||||
// path ascending. A group whose trees hold 0 bytes is left out, since
|
||||
// removing one frees nothing. The groups it implies hold 0 bytes too,
|
||||
// so leaving it out hides nothing. exists is called only on the members
|
||||
// of maximal groups.
|
||||
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||
prog *progress,
|
||||
exists func(path string) bool, prog *progress,
|
||||
) [][]*treeNode {
|
||||
groups := make(map[[sha256.Size]byte][]*treeNode)
|
||||
for _, d := range allDirs {
|
||||
@@ -308,7 +320,19 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||
var dupes [][]*treeNode
|
||||
|
||||
for _, g := range groups {
|
||||
if len(g) < minGroupSize || suppressed(g, super) {
|
||||
// The members share a digest, so they share a total size.
|
||||
if len(g) < minGroupSize || g[0].totalSize == 0 ||
|
||||
suppressed(g, super) {
|
||||
continue
|
||||
}
|
||||
|
||||
// A member that is gone leaves the group. A suppressed group is
|
||||
// still implied: its members still on disk have parents still
|
||||
// on disk, and those parents' group is checked the same way.
|
||||
g = slices.DeleteFunc(g, func(n *treeNode) bool {
|
||||
return !exists(n.path)
|
||||
})
|
||||
if len(g) < minGroupSize {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // checks the unexported treeNode results
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"os"
|
||||
"slices"
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -17,19 +19,24 @@ const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
|
||||
|
||||
// smokeTreeRecs mirrors the tree of the test under README "Definition
|
||||
// of done": /d/t1 and /d/t2 are identical, /d/t3 differs from them only
|
||||
// by one filename.
|
||||
// by one filename. Each file takes one 4 KiB block on disk.
|
||||
func smokeTreeRecs() []scanRec {
|
||||
f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
|
||||
f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")
|
||||
|
||||
return []scanRec{
|
||||
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t1/f1"},
|
||||
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t1/sub/f2"},
|
||||
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t2/f1"},
|
||||
{size: 100, head: f2Head, tail: f2Tail, content: f2Content, path: "/d/t2/sub/f2"},
|
||||
{size: 3000, head: f1Head, tail: f1Tail, content: f1Content, path: "/d/t3/f1"},
|
||||
{size: 100, head: f2Head, tail: f2Tail, content: f2Content,
|
||||
path: "/d/t3/sub/f2renamed"},
|
||||
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
||||
content: f1Content, path: "/d/t1/f1"},
|
||||
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
||||
content: f2Content, path: "/d/t1/sub/f2"},
|
||||
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
||||
content: f1Content, path: "/d/t2/f1"},
|
||||
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
||||
content: f2Content, path: "/d/t2/sub/f2"},
|
||||
{size: 3000, diskSize: 4096, head: f1Head, tail: f1Tail,
|
||||
content: f1Content, path: "/d/t3/f1"},
|
||||
{size: 100, diskSize: 4096, head: f2Head, tail: f2Tail,
|
||||
content: f2Content, path: "/d/t3/sub/f2renamed"},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -173,7 +180,7 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
|
||||
t.Fatalf("directory paths = %q, want %q", got, want)
|
||||
}
|
||||
|
||||
groups := collectTreeGroups(dirs, super, nil)
|
||||
groups := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||
|
||||
gotGroups := groupPaths(groups)
|
||||
wantGroups := [][]string{{"/a", "/c"}}
|
||||
@@ -189,7 +196,8 @@ func TestTreeNamesSortingBeforeSlash(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRunTreesEscapesPaths(t *testing.T) {
|
||||
t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
|
||||
dir, recs := recsOnDisk(t, awkwardPairRecs())
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
@@ -199,22 +207,45 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
||||
code, exitOK, stderr.String())
|
||||
}
|
||||
|
||||
want := "first\tdupe\tfiles\tsize\n" +
|
||||
`/d/\tone\ntwo\rthree\\four` + "\t/d/A\t1\t5\n"
|
||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
dir + `/d/\tone\ntwo\rthree\\four` + "\t" + dir + "/d/A\t1\t5\t4096\n"
|
||||
if got := stdout.String(); got != want {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
||||
// take different space on disk. The row's disk_size is the sum over the
|
||||
// dupe copy's files, those in its subdirectory included.
|
||||
func TestRunTreesDiskSize(t *testing.T) {
|
||||
f, g := testHash("f"), testHash("g")
|
||||
recs := []scanRec{
|
||||
{size: 10, diskSize: 4096, head: f, tail: f, content: f, path: "/a/f"},
|
||||
{size: 20, diskSize: 4096, head: g, tail: g, content: g, path: "/a/sub/g"},
|
||||
{size: 10, diskSize: 512, head: f, tail: f, content: f, path: "/b/f"},
|
||||
{size: 20, diskSize: 1024, head: g, tail: g, content: g, path: "/b/sub/g"},
|
||||
}
|
||||
|
||||
dir, recs := recsOnDisk(t, recs)
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
want := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
dir + "/a\t" + dir + "/b\t2\t30\t1536\n"
|
||||
if got := runStdout(t, cmdTrees); got != want {
|
||||
t.Errorf("stdout = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesProgressNamesEachPhase runs trees on the smoke-test
|
||||
// records with stderr not a terminal, and checks every line it prints
|
||||
// there. After the line naming the database, each phase prints a line
|
||||
// naming it with its place among the four as it starts and another as
|
||||
// it ends. read counts the 6 records before it reads them, group counts
|
||||
// the 8 directories, and write the 1 group. The summary comes last, and
|
||||
// the directories, and write the 1 group. The summary comes last, and
|
||||
// stdout holds the report alone.
|
||||
func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
||||
path := seedDatabase(t, smokeTreeRecs())
|
||||
dir, recs := recsOnDisk(t, smokeTreeRecs())
|
||||
path := seedDatabase(t, recs)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
stderr := captureStderr(t)
|
||||
@@ -227,11 +258,16 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
||||
code, exitOK, stderr())
|
||||
}
|
||||
|
||||
report := "first\tdupe\tfiles\tsize\n/d/t1\t/d/t2\t2\t3100\n"
|
||||
report := "first\tdupe\tfiles\tsize\tdisk_size\n" +
|
||||
dir + "/d/t1\t" + dir + "/d/t2\t2\t3100\t8192\n"
|
||||
if got := stdout.String(); got != report {
|
||||
t.Errorf("stdout = %q, want %q", got, report)
|
||||
}
|
||||
|
||||
// The smoke-test records name 8 directories, / included. Moved under
|
||||
// dir, they name one more for each slash in dir.
|
||||
dirs := strconv.Itoa(8 + strings.Count(dir, "/"))
|
||||
|
||||
// How each line starts; the rate at the end of a phase varies.
|
||||
want := []string{
|
||||
"trees: database " + path + " (from " + databaseEnv + ")",
|
||||
@@ -239,8 +275,9 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
||||
treesOpenLine,
|
||||
"[2/4] read: counting the records, elapsed 0s, remain unknown",
|
||||
"[2/4] read: [6/6] 100%, ",
|
||||
"[3/4] group: [0/8] 0%, 0 directories/s, elapsed 0s, remain unknown",
|
||||
"[3/4] group: [8/8] 100%, ",
|
||||
"[3/4] group: [0/" + dirs + "] 0%, 0 directories/s, elapsed 0s, " +
|
||||
"remain unknown",
|
||||
"[3/4] group: [" + dirs + "/" + dirs + "] 100%, ",
|
||||
"[4/4] write: [0/1] 0%, 0 groups/s, elapsed 0s, remain unknown",
|
||||
"[4/4] write: [1/1] 100%, ",
|
||||
"trees: 6 records read, 1 duplicate tree groups, 1 dupe trees, ",
|
||||
@@ -269,7 +306,7 @@ func TestRunTreesProgressNamesEachPhase(t *testing.T) {
|
||||
func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
||||
stderr := captureStderr(t)
|
||||
super, dirs := treeOf(t, smokeTreeRecs())
|
||||
dupes := collectTreeGroups(dirs, super, nil)
|
||||
dupes := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||
|
||||
// No timed draws: the display is drawn as it starts and as it ends.
|
||||
p := startProgress(progressParams{
|
||||
@@ -292,7 +329,9 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
||||
shown = append(shown, line[strings.LastIndex(line, "\r")+1:])
|
||||
}
|
||||
|
||||
want := []string{"first\tdupe\tfiles\tsize", "/d/t1\t/d/t2\t2\t3100"}
|
||||
want := []string{
|
||||
"first\tdupe\tfiles\tsize\tdisk_size", "/d/t1\t/d/t2\t2\t3100\t8192",
|
||||
}
|
||||
if len(shown) < len(want) || !slices.Equal(shown[:len(want)], want) {
|
||||
t.Errorf("terminal shows lines %q, want %q first", shown, want)
|
||||
}
|
||||
@@ -348,7 +387,7 @@ func TestCollectTreeGroupsMaximal(t *testing.T) {
|
||||
|
||||
super, dirs := treeOf(t, smokeTreeRecs())
|
||||
|
||||
groups := collectTreeGroups(dirs, super, nil)
|
||||
groups := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||
|
||||
want := [][]string{{"/d/t1", "/d/t2"}}
|
||||
if got := groupPaths(groups); !slices.EqualFunc(got, want,
|
||||
@@ -372,14 +411,16 @@ func TestCollectTreeGroupsDeterministic(t *testing.T) {
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
forward := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||
forward := groupPaths(collectTreeGroups(dirs, super, everyPathExists,
|
||||
nil))
|
||||
|
||||
reversed := slices.Clone(recs)
|
||||
slices.Reverse(reversed)
|
||||
|
||||
superR, dirsR := treeOf(t, reversed)
|
||||
|
||||
backward := groupPaths(collectTreeGroups(dirsR, superR, nil))
|
||||
backward := groupPaths(collectTreeGroups(dirsR, superR, everyPathExists,
|
||||
nil))
|
||||
if !slices.EqualFunc(forward, backward, slices.Equal) {
|
||||
t.Fatalf("output depends on record order: %v vs %v",
|
||||
forward, backward)
|
||||
@@ -399,7 +440,36 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
|
||||
|
||||
want := [][]string{{"/p/x1", "/p/x2"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
t.Fatalf("groups = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
|
||||
// holding only empty files form no group, and that an empty file still
|
||||
// counts toward the digest of a tree that holds other files.
|
||||
func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
|
||||
sha256.Sum256(nil)
|
||||
recs := []scanRec{
|
||||
{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
|
||||
// Without the empty file, /p/x3 is no copy of /p/x1.
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
|
||||
}
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
|
||||
|
||||
want := [][]string{{"/p/x1", "/p/x2"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
@@ -422,10 +492,98 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||
got := groupPaths(collectTreeGroups(dirs, super, everyPathExists, nil))
|
||||
|
||||
want := [][]string{{"/p/a/x", "/q/b/x"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
t.Fatalf("groups = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectTreeGroupsChecksOnlyReportedCopies checks that trees asks
|
||||
// whether a directory is on disk only for the members of the groups it
|
||||
// returns. It never asks about /a/proj/sub, which lies inside a copy
|
||||
// and whose group the group of /a/proj implies.
|
||||
func TestCollectTreeGroupsChecksOnlyReportedCopies(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f, g, u := testHash("f"), testHash("g"), testHash("u")
|
||||
recs := []scanRec{
|
||||
{size: 10, head: f, tail: f, content: f, path: "/a/proj/f"},
|
||||
{size: 20, head: g, tail: g, content: g, path: "/a/proj/sub/g"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/b/proj/f"},
|
||||
{size: 20, head: g, tail: g, content: g, path: "/b/proj/sub/g"},
|
||||
// /a lacks this file, so /a and /b are no copies of each other.
|
||||
{size: 30, head: u, tail: u, content: u, path: "/b/lonely"},
|
||||
}
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
var checked []string
|
||||
|
||||
exists := func(path string) bool {
|
||||
checked = append(checked, path)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
|
||||
|
||||
want := [][]string{{"/a/proj", "/b/proj"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
t.Fatalf("groups = %v, want %v", got, want)
|
||||
}
|
||||
|
||||
slices.Sort(checked)
|
||||
|
||||
if !slices.Equal(checked, want[0]) {
|
||||
t.Errorf("directories checked on disk = %q, want %q",
|
||||
checked, want[0])
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectTreeGroupsDropsGroupLeftWithOneCopy checks that a group of
|
||||
// two copies, one of them gone, is not returned. If kept, it would
|
||||
// write no row, but the summary line would count it.
|
||||
func TestCollectTreeGroupsDropsGroupLeftWithOneCopy(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := testHash("f")
|
||||
recs := []scanRec{
|
||||
{size: 10, head: f, tail: f, content: f, path: "/c/kept/f"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/c/gone/f"},
|
||||
}
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
exists := func(path string) bool { return path != "/c/gone" }
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
|
||||
if len(got) != 0 {
|
||||
t.Errorf("groups = %v, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies checks that a group
|
||||
// of three copies, one of them gone, is returned with the other two.
|
||||
func TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
f := testHash("f")
|
||||
recs := []scanRec{
|
||||
{size: 10, head: f, tail: f, content: f, path: "/d/kept1/f"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/d/gone/f"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/d/kept2/f"},
|
||||
}
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
exists := func(path string) bool { return path != "/d/gone" }
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
|
||||
|
||||
want := [][]string{{"/d/kept1", "/d/kept2"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
t.Errorf("groups = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user