Compare commits
13
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
97a02da837 | ||
|
|
6c529fcc6d | ||
|
|
202f083cc0 | ||
|
|
1017332d68 | ||
|
|
5c7284c351 | ||
|
|
7cb10ed303 | ||
|
|
da75b713aa | ||
|
|
24b0fc486d | ||
|
|
f52bbe5d95 | ||
|
|
30dae35c51 | ||
|
|
eebf579f11 | ||
|
|
8a4616039b | ||
|
|
c2804d088b |
No files matched your search
@@ -4,7 +4,7 @@ export CGO_ENABLED = 0
|
||||
|
||||
BINARY := sfdupes
|
||||
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
|
||||
LDFLAGS := -X main.Version=$(VERSION)
|
||||
LDFLAGS := -X sneak.berlin/go/sfdupes/internal/sfdupes.Version=$(VERSION)
|
||||
|
||||
.PHONY: sfdupes build bootstrap setup test lint fmt fmt-check check docker hooks clean
|
||||
|
||||
@@ -14,7 +14,7 @@ LDFLAGS := -X main.Version=$(VERSION)
|
||||
# Default target: build the binary. Phony so go build (which has its
|
||||
# own build cache) always decides what to recompile.
|
||||
sfdupes:
|
||||
go build -ldflags "$(LDFLAGS)" -o $(BINARY)
|
||||
go build -ldflags "$(LDFLAGS)" -o $(BINARY) ./cmd/sfdupes
|
||||
|
||||
build: sfdupes
|
||||
|
||||
|
||||
@@ -17,12 +17,13 @@ gigabyte. A large file is never read in full. A match between two large files is
|
||||
a strong candidate, not proof.
|
||||
|
||||
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every
|
||||
file takes too long. It has four subcommands. `scan` walks the filesystem and
|
||||
file takes too long. It has five subcommands. `scan` walks the filesystem and
|
||||
records the files it finds in an SQLite database. SQLite keeps a whole database
|
||||
in one file. The database is kept between runs. A rescan reads only new and
|
||||
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
||||
system's job scheduler. `report`, `trees` and `info` read the database alone and
|
||||
can run at any time.
|
||||
can run at any time. `dwim` scans `/` and then prints the `trees` report, for a
|
||||
first look at a new machine with one command.
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -42,7 +43,9 @@ as opposed to a directory, a symlink or a device node. A symlink is a file that
|
||||
holds the path of another file and stands in for it. A device node is a file
|
||||
that stands for a piece of hardware, such as a disk or a terminal. `report`
|
||||
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
||||
`info` prints the database's path, size and record count.
|
||||
Both cover the whole database, or only the paths given after them, as in
|
||||
`sfdupes report /srv/photos`. `info` prints the database's path, size and record
|
||||
count.
|
||||
|
||||
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
||||
another file. The first `scan` creates it.
|
||||
@@ -56,25 +59,25 @@ shell shows both on the terminal, and `>` redirects stdout alone.
|
||||
With Go installed, this builds and installs the newest tagged release:
|
||||
|
||||
```sh
|
||||
go install sneak.berlin/go/sfdupes@latest
|
||||
go install sneak.berlin/go/sfdupes/cmd/sfdupes@latest
|
||||
```
|
||||
|
||||
`@main` installs the tip of the `main` branch instead:
|
||||
|
||||
```sh
|
||||
go install sneak.berlin/go/sfdupes@main
|
||||
go install sneak.berlin/go/sfdupes/cmd/sfdupes@main
|
||||
```
|
||||
|
||||
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
|
||||
|
||||
`sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from
|
||||
`go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
|
||||
a release tag such as `v0.0.3`. For `@main` it is the commit's tag when it has
|
||||
one. Otherwise it is a pseudo-version, Go's version string for an untagged
|
||||
commit. That string ends in the commit's hash. A binary built from a clone
|
||||
carries the output of `git describe --tags --always --dirty` in that clone, such
|
||||
as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no version is known. To
|
||||
see everything Go recorded in an installed binary:
|
||||
a release tag. For `@main` it is the commit's tag when it has one. Otherwise it
|
||||
is a pseudo-version, Go's version string for an untagged commit. That string
|
||||
ends in the commit's hash. A binary built from a clone carries the output of
|
||||
`git describe --tags --always --dirty` in that clone, such as `v0.0.3` on a
|
||||
tagged commit. `VERSION` is `dev` when no version is known. To see everything Go
|
||||
recorded in an installed binary:
|
||||
|
||||
```sh
|
||||
go version -m "$(command -v sfdupes)"
|
||||
@@ -247,12 +250,15 @@ In order:
|
||||
- `report` prints the file-level duplicate report from the database.
|
||||
- `trees` prints the duplicate-tree report from the database.
|
||||
- `info` prints the database's details.
|
||||
- `dwim` scans `/` with the default excludes, then prints the duplicate-tree
|
||||
report.
|
||||
|
||||
```
|
||||
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
sfdupes report > dupes.tsv
|
||||
sfdupes trees > dupetrees.tsv
|
||||
sfdupes report [PATH...] > dupes.tsv
|
||||
sfdupes trees [PATH...] > dupetrees.tsv
|
||||
sfdupes info
|
||||
sfdupes dwim
|
||||
sfdupes --version
|
||||
sfdupes [command] --help
|
||||
```
|
||||
@@ -348,17 +354,26 @@ default. An operand may be a directory or a regular file.
|
||||
|
||||
- `scan` with no operand is a usage error. The usage text goes to stderr, and
|
||||
the exit code is 2.
|
||||
- An operand that does not exist is a fatal error. `scan` then stops before it
|
||||
walks anything, and exits 1.
|
||||
- Each operand is made absolute before the walk. A relative operand is joined to
|
||||
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
|
||||
`..` in it is resolved as text, and symlinks are not followed. Each record is
|
||||
stored under that absolute path.
|
||||
- An operand that does not exist is a fatal error, and so is a symlink whose
|
||||
target does not exist. `scan` then stops before it walks anything, and exits
|
||||
with code 1.
|
||||
- Each operand is resolved to its real path before the walk, as `realpath`
|
||||
prints it. A relative operand is joined to the working directory as `pwd -P`
|
||||
prints it. Every symlink in the path is then followed. A `..` after a symlink
|
||||
counts from the directory the symlink leads to. From then on the operand is
|
||||
that real path. Its files are recorded under it, and warnings name it.
|
||||
- An operand that is a symlink is scanned at its target's real path. After a
|
||||
`cd` through a symlink, `scan .` records its files under the real path of the
|
||||
working directory.
|
||||
- All operands belong to one scan, and one walk covers them all at once (see
|
||||
"Concurrency").
|
||||
- An operand that repeats another is dropped before the walk.
|
||||
- An operand that repeats another is dropped before the walk. Two operands
|
||||
repeat each other when their real paths are the same.
|
||||
- An operand under another operand is dropped the same way, unless one of the
|
||||
cases below applies. No path is then walked twice.
|
||||
cases below applies. No path is then walked twice. Whether one operand is
|
||||
under another is judged on their real paths. With `/srv/link` a symlink to a
|
||||
directory outside `/srv`, `scan /srv /srv/link/sub` walks `sub` on its own,
|
||||
because the real path of `sub` is not under `/srv`.
|
||||
- An operand under another is kept, and walked on its own, when the inner
|
||||
operand or a directory between the two is one of these:
|
||||
- a default exclude (see "Rules for the walk");
|
||||
@@ -369,25 +384,19 @@ default. An operand may be a directory or a regular file.
|
||||
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
||||
such a directory, so the inner operand is walked on its own.
|
||||
|
||||
- No other case keeps an operand under another. An operand reached through a
|
||||
symlinked directory below the outer operand is dropped too. The outer
|
||||
operand's walk does not follow the symlink, so no file under the inner operand
|
||||
is scanned. The records that an earlier scan left under it are deleted. The
|
||||
`-x` case above also covers an operand reached that way.
|
||||
- An operand that is neither a regular file nor a directory is not scanned.
|
||||
`scan` prints a warning naming the path and what it is. It counts the operand
|
||||
as skipped and drops it. An operand beneath a symlink operand is scanned all
|
||||
the same. The scan keeps the records beneath a dropped operand, unless another
|
||||
operand covers them. After a `cd` through a symlink, `scan .` names the
|
||||
symlink and is skipped as a symlink operand. `scan sub` in that directory
|
||||
records its files under the symlink's path. `report` can then list each as a
|
||||
duplicate of the same file recorded under its real path.
|
||||
- No other case keeps an operand under another.
|
||||
- An operand that is neither a regular file nor a directory is not scanned. Such
|
||||
an operand is, for example, a symlink to a device node. `scan` prints a
|
||||
warning naming the real path and what it is. It counts the operand as skipped
|
||||
and drops it. The scan keeps the records beneath a dropped operand, unless
|
||||
another operand covers them.
|
||||
- An operand at or inside a directory named `.zfs` is never scanned, whether
|
||||
given alone or under another operand. The check looks at the path as given and
|
||||
at the path with its symlinks resolved. `scan` prints a warning naming the
|
||||
path, counts the operand as skipped and drops it. The records beneath it are
|
||||
kept, unless it lies under another operand. Then they are deleted, because
|
||||
that operand's walk never enters `.zfs` (see "Rules for the walk").
|
||||
given alone or under another operand. The check looks at the real path, so a
|
||||
symlink into a `.zfs` directory is not scanned either. `scan` prints a warning
|
||||
naming the real path, counts the operand as skipped and drops it. The records
|
||||
beneath it are kept, unless it lies under another operand. Then they are
|
||||
deleted, because that operand's walk never enters `.zfs` (see "Rules for the
|
||||
walk").
|
||||
- An operand on a filesystem that is not persistent is dropped too. `scan`
|
||||
prints a line naming it and its filesystem type, and does not count it as
|
||||
skipped.
|
||||
@@ -407,9 +416,9 @@ operands:
|
||||
is what makes a daily rescan cheap.
|
||||
- When such a file's size on disk differs from its recorded `disk_size`, `scan`
|
||||
updates `disk_size` alone. It still does not read the file, and counts it as
|
||||
unchanged. ZFS, for example, counts a new file's blocks only seconds after the
|
||||
write. A block is the unit in which a filesystem allocates disk space. A file
|
||||
scanned within those seconds gets its real `disk_size` at the next scan.
|
||||
unchanged. ZFS, for example, counts a new file's blocks up to seconds after
|
||||
the write. A block is the unit in which a filesystem allocates disk space. A
|
||||
file scanned within those seconds gets its real `disk_size` at the next scan.
|
||||
- `scan` hashes a file again when its size differs or its mtime is newer. The
|
||||
mtime is compared to the nanosecond, as finely as the filesystem records it. A
|
||||
same-size rewrite counts as a change whenever the filesystem gives it a later
|
||||
@@ -484,12 +493,12 @@ walk practical on large or busy storage pools.
|
||||
The walk knows them by device ID and inode number. The device ID is the
|
||||
number by which the kernel tells filesystems apart. The inode number tells
|
||||
the files of one filesystem apart. So the walk passes over them whatever
|
||||
path leads to them. The walk follows no symlink it finds. It does open each
|
||||
operand at the path given, even when that path runs through a symlinked
|
||||
directory. Such an operand can lead to the database files, and the walk
|
||||
passes over them there as well. The walk prints no line for the database
|
||||
files and does not count them as skipped. A record that an earlier scan left
|
||||
for one of them is deleted.
|
||||
path leads to them. The walk follows no symlink it finds. It opens each
|
||||
operand at its real path, not the path given. An operand given through a
|
||||
symlink can lead to the database files, and the walk passes over them there
|
||||
as well. The walk prints no line for the database files and does not count
|
||||
them as skipped. A record that an earlier scan left for one of them is
|
||||
deleted.
|
||||
|
||||
- By default the walk enters directories on other filesystems than its
|
||||
operand's. With `-x`, it never enters a directory on a different filesystem
|
||||
@@ -646,18 +655,19 @@ and `scan` leaves it ignored.
|
||||
|
||||
### `report` mode
|
||||
|
||||
`report` reads every record from the database. It takes no positional arguments.
|
||||
A positional argument is anything on the command line that is not a flag, such
|
||||
as the paths `scan` takes. `report` never touches the scanned filesystem. It
|
||||
neither stats nor opens any path in the records. It reads the database and
|
||||
writes to stdout and stderr, and nothing else. Its output is the same whether or
|
||||
not the scanned filesystem is still mounted.
|
||||
`report` reads every record from the database. It takes optional `PATH`
|
||||
operands. With none, it uses every record. With one or more, it uses only the
|
||||
records at or below them (see "Report paths"). `report` never touches the
|
||||
scanned filesystem. It neither stats nor opens any path in the records. It reads
|
||||
the database and writes to stdout and stderr, and nothing else. Its output is
|
||||
the same whether or not the scanned filesystem is still mounted.
|
||||
|
||||
Processing:
|
||||
|
||||
- Records are grouped by signature, the four values
|
||||
`(size, head, tail, content)`.
|
||||
- Every group with two or more paths is a duplicate group.
|
||||
- Empty files form no group, because removing one frees nothing.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
every other path is a `dupe`.
|
||||
- Groups are ordered by size, largest first, then by `first` path. The output is
|
||||
@@ -673,6 +683,26 @@ with one signature come together, and so do the records of one size. So `report`
|
||||
holds only the duplicate groups of the size it is reading. Once it reaches a
|
||||
smaller size, it writes them, ordered by `first` path.
|
||||
|
||||
#### Report paths
|
||||
|
||||
`report` and `trees` take the same `PATH` operands:
|
||||
|
||||
- A record is used when its path is an operand or lies below one. `/srv/photos`
|
||||
covers `/srv/photos/x` but not `/srv/photos2/x`.
|
||||
- The records used are grouped, ordered and totalled as they are without
|
||||
operands. A copy outside the operands is not reported. With copies `/a/f`,
|
||||
`/srv/photos/x` and `/srv/photos/y`, `sfdupes report /srv/photos` prints one
|
||||
row, `/srv/photos/x` and `/srv/photos/y`.
|
||||
- A relative operand is joined to the working directory as `pwd -P` prints it.
|
||||
- Each operand is then cleaned as text, as `realpath -ms` prints it. Its `.`
|
||||
elements and repeated or trailing slashes are dropped. Each `..` drops the
|
||||
element before it.
|
||||
- No operand is looked up on the filesystem. A symlink in an operand is not
|
||||
followed, and an operand need not exist. The records hold real paths (see
|
||||
"Operands"), so an operand that passes through a symlink matches no record.
|
||||
- An operand that matches no record is not an error. The report then has no
|
||||
rows.
|
||||
|
||||
#### Report output format
|
||||
|
||||
The report is TSV (tab-separated values) on stdout. It holds a header line, then
|
||||
@@ -700,22 +730,24 @@ Grouping and ordering use the stored path. The warnings about skipped paths that
|
||||
|
||||
The summary line on stderr gives:
|
||||
|
||||
- the records read;
|
||||
- the records read, which are all the records in the database, with or without
|
||||
operands;
|
||||
- the duplicate groups;
|
||||
- the dupe files, which is the number of rows;
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
|
||||
```
|
||||
report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable
|
||||
report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
|
||||
```
|
||||
|
||||
### `trees` mode
|
||||
|
||||
`trees` reads the same database as `report` and takes no positional arguments.
|
||||
It reports whole duplicate directory trees. Two directories are duplicate trees
|
||||
when they hold the same relative paths, and each path holds a file with the same
|
||||
signature in both. Like `report`, it never touches the scanned filesystem. It
|
||||
rebuilds the directory tree from the paths in the records, split on `/`.
|
||||
`trees` reads the same database as `report` and takes the same `PATH` operands
|
||||
(see "Report paths"). It reports whole duplicate directory trees. Two
|
||||
directories are duplicate trees when they hold the same relative paths, and each
|
||||
path holds a file with the same signature in both. Like `report`, it never
|
||||
touches the scanned filesystem. It rebuilds the directory tree from the paths in
|
||||
the records, split on `/`.
|
||||
|
||||
Definitions:
|
||||
|
||||
@@ -734,11 +766,18 @@ Processing:
|
||||
- `trees` builds the directory tree and computes every directory's digest. It
|
||||
then groups the directories by digest. Every group with two or more
|
||||
directories is a duplicate-tree group.
|
||||
- `trees` leaves out a group whose trees hold 0 bytes, because removing one
|
||||
frees nothing. An empty file still counts toward the digest of a tree that
|
||||
holds other files.
|
||||
- `trees` leaves out a group that a group of its members' parents already
|
||||
implies. A group is left out when its members' parents are distinct
|
||||
directories that all have the same digest. The parents' group, or an
|
||||
ancestor's, already implies it. A group is always reported when it holds
|
||||
sibling directories, or when its members' parents have different digests.
|
||||
- With operands, `trees` builds the tree from the records it uses. It reports
|
||||
only directories at or below an operand. A directory above every operand holds
|
||||
files that are not used, so its digest does not describe it. Such a directory
|
||||
is never reported and never implies a group of its children.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
every other path is a `dupe`.
|
||||
- Groups are ordered by total tree size, largest first, then by `first` path.
|
||||
@@ -788,7 +827,8 @@ directory's path is `/`.
|
||||
|
||||
The summary line on stderr gives:
|
||||
|
||||
- the records read;
|
||||
- the records read, which are all the records in the database, with or without
|
||||
operands;
|
||||
- the duplicate-tree groups;
|
||||
- the dupe trees, which is the number of rows;
|
||||
- the reclaimable bytes, the sum of `size` over all rows.
|
||||
@@ -797,6 +837,35 @@ The summary line on stderr gives:
|
||||
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
|
||||
```
|
||||
|
||||
### `dwim` mode
|
||||
|
||||
`dwim`, short for "do what I mean", scans a new machine and reports its
|
||||
duplicate trees in one command. It takes no positional arguments and no flags of
|
||||
its own. A positional argument is anything on the command line that is not a
|
||||
flag, such as the paths `scan` takes. Like `scan`, it should run as root, so
|
||||
that it can read every file.
|
||||
|
||||
`dwim` runs what `sfdupes scan /` runs with no flags. The walk skips the default
|
||||
excludes and crosses into other filesystems. `--workers` keeps its default. The
|
||||
database, its creation and the scan lock are those of `scan`. Then `dwim` runs
|
||||
what `sfdupes trees` runs. The largest trees come first.
|
||||
|
||||
The report goes to stdout, exactly as `trees` prints it. On stderr comes what
|
||||
`scan` prints, then what `trees` prints, then one last line:
|
||||
|
||||
```
|
||||
dwim: run "sfdupes trees" to print this report again
|
||||
```
|
||||
|
||||
The database keeps the scan, so `sfdupes trees` prints the same report later,
|
||||
even after the terminal is gone.
|
||||
|
||||
`SIGINT` or `SIGTERM` stops the scan as it stops `scan` (see "Stopping a scan").
|
||||
`dwim` then prints no report and exits 1. A fatal scan error also ends it before
|
||||
the report, with the same message and exit code as `scan`. Warnings for skipped
|
||||
files do not stop it. `dwim` stops catching signals when the scan ends, so a
|
||||
signal during the report ends it at once, as it ends `trees`.
|
||||
|
||||
### `info` mode
|
||||
|
||||
`info` takes no positional arguments and prints the database's details on
|
||||
@@ -837,7 +906,7 @@ scan is writing. `info` writes its lines after the `count` display has ended.
|
||||
|
||||
### Database
|
||||
|
||||
All four subcommands work on one SQLite database file.
|
||||
All five subcommands work on one SQLite database file.
|
||||
|
||||
#### Location
|
||||
|
||||
@@ -853,6 +922,8 @@ scan: new database /var/lib/sfdupes/db.sqlite (default)
|
||||
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
|
||||
```
|
||||
|
||||
`dwim` prints the `scan` line first, and the `trees` line after the scan.
|
||||
|
||||
A relative path is printed with the working directory in front of it. The
|
||||
working directory is printed as `pwd -P` prints it, with its symlinks resolved.
|
||||
The rest of the path is printed as set, so a `..` in it stays. The line names
|
||||
@@ -1136,9 +1207,9 @@ showing the count it stopped at.
|
||||
### Error handling and exit codes
|
||||
|
||||
- `0`: success, even when files were skipped with warnings.
|
||||
- `1`: a fatal error, or a `scan` stopped by `SIGINT` or `SIGTERM`. A fatal
|
||||
error is one of these:
|
||||
- an operand that does not exist, or whose details `scan` cannot read;
|
||||
- `1`: a fatal error, a `scan` stopped by `SIGINT` or `SIGTERM`, or a `dwim`
|
||||
stopped by either during its scan. A fatal error is one of these:
|
||||
- an operand that does not exist, or whose real path `scan` cannot find;
|
||||
- a list of mounted filesystems that `scan` cannot read;
|
||||
- another `scan` holding the lock (see "The scan lock");
|
||||
- a database, its directory or its lock file that cannot be created or
|
||||
@@ -1158,7 +1229,7 @@ showing the count it stopped at.
|
||||
`--workers abc`;
|
||||
- `scan` with no operand;
|
||||
- `--workers` below 1;
|
||||
- `report`, `trees` or `info` with a positional argument.
|
||||
- `info` or `dwim` with a positional argument.
|
||||
|
||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||
stdout, such as a full disk, is reported the same way. Two cases never reach
|
||||
@@ -1346,7 +1417,7 @@ All of the following, run in this directory, must pass:
|
||||
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
|
||||
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
|
||||
printf 'y' > "$d/tiny3" # 1-byte non-duplicate
|
||||
: > "$d/empty1"; : > "$d/empty2" # empty duplicates
|
||||
: > "$d/empty1"; : > "$d/empty2" # empty files
|
||||
# duplicate trees: t1 and t2 are identical; t3 differs by one filename
|
||||
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
|
||||
head -c 3000 /dev/urandom > "$d/t1/f1"
|
||||
@@ -1371,8 +1442,8 @@ All of the following, run in this directory, must pass:
|
||||
is the path that sorts first.
|
||||
- `t1/f1`, `t2/f1` and `t3/f1` form one group.
|
||||
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
|
||||
- `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`.
|
||||
- `unique.bin` and `tiny3` appear nowhere.
|
||||
- `tiny1` and `tiny2` form a pair.
|
||||
- `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
|
||||
- The groups come in order of size, largest first.
|
||||
|
||||
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
|
||||
@@ -1385,8 +1456,9 @@ All of the following, run in this directory, must pass:
|
||||
because its content changed. `copy.bin` and `copy2.bin` remain as one pair.
|
||||
`unique.bin` is gone from the database.
|
||||
|
||||
The test suite automates this scenario (see `scan_test.go`). It also checks
|
||||
that `report` and `trees` never touch the scanned filesystem.
|
||||
The test suite automates this scenario (see
|
||||
`internal/sfdupes/scan_test.go`). It also checks that `report` and `trees`
|
||||
never touch the scanned filesystem.
|
||||
|
||||
## TODO
|
||||
|
||||
|
||||
@@ -29,6 +29,46 @@
|
||||
Newest first. An entry ends with its date and its issue or branch, where those
|
||||
are known.
|
||||
|
||||
- `report` and `trees` take optional `PATH` operands. With operands, they use
|
||||
only the records at or below them. `trees` then reports only directories at or
|
||||
below an operand. Operands are made absolute and cleaned as text, never looked
|
||||
up on the filesystem (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/174)
|
||||
- `report` and `trees` leave out empty files. `report` writes no group of size
|
||||
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
|
||||
counts toward a tree's digest (2026-11-11,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/178)
|
||||
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
|
||||
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
|
||||
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
|
||||
path is now `sneak.berlin/go/sfdupes/cmd/sfdupes` (2026-11-10,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/157)
|
||||
|
||||
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
|
||||
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
|
||||
prints the report again (2026-11-06,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/154)
|
||||
|
||||
- `scan` resolves each operand to its real path, symlinks followed, before the
|
||||
walk. A symlink operand is scanned at its target instead of skipped. With
|
||||
`/srv/link` a symlink to a directory outside `/srv`, `scan /srv /srv/link/sub`
|
||||
walks `sub` on its own instead of dropping it and deleting its records
|
||||
(2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144)
|
||||
|
||||
- `README.md` and a comment in `internal/sfdupes/scan.go` no longer say ZFS
|
||||
counts a new file's blocks only seconds after the write. The count can land in
|
||||
under a second. Both now say up to seconds after the write, as the comment on
|
||||
`diskSizeOf` in `internal/sfdupes/scan_test.go` does (2026-11-03,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/152)
|
||||
|
||||
- The three disk size tests in `internal/sfdupes/scan_test.go` no longer fail at
|
||||
random on ZFS. `unix.Sync` can return before ZFS has counted a file's blocks,
|
||||
and one ZFS commit can take a file through several sizes. The test that needs
|
||||
the size to change waits for it, for at most 30 seconds. Each test reads the
|
||||
size before and after its scan. The rewrite test rescans until the two reads
|
||||
match, then compares exactly. The other two fail only on a `disk_size` outside
|
||||
the two reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150)
|
||||
|
||||
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
|
||||
sentence. Every completed step is kept, with its date and its issue or branch.
|
||||
The policy-compliance checklist of 2026-07-22 is folded into its step
|
||||
@@ -38,9 +78,13 @@ are known.
|
||||
it. It leads with what the program is and how to use it. A reference section
|
||||
covers every setting and every subcommand. Development and testing come last.
|
||||
Every statement was checked against the code on `next`. Five comments in
|
||||
`main_test.go` cite "Settings", where their rules now live. One comment each
|
||||
in `scan_test.go`, `trees_test.go` and `exclude_test.go` cites what the README
|
||||
now says (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
`internal/sfdupes/cli_test.go` cite "Settings", where their rules now live.
|
||||
One comment each in these files cites what the README now says:
|
||||
- `internal/sfdupes/scan_test.go`;
|
||||
- `internal/sfdupes/trees_test.go`;
|
||||
- `internal/sfdupes/exclude_test.go`.
|
||||
|
||||
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
|
||||
|
||||
- Each record holds `disk_size`, the bytes the file takes on disk. That is its
|
||||
allocated blocks, `st_blocks` × 512, from the `lstat` call the walk already
|
||||
@@ -113,12 +157,12 @@ are known.
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/109)
|
||||
|
||||
- The release rule was set. Until 1.0, each tip of `main` that passes the build
|
||||
gets the next `v0.0.N` tag. `go install sneak.berlin/go/sfdupes@latest` then
|
||||
installs it. `go.mod` retracts `v0.0.1`, the early build without a database.
|
||||
When the linker set no version, `sfdupes --version` prints the version Go
|
||||
recorded in the binary. That is the tag, or the pseudo-version of a
|
||||
`go install` build. It prints `dev` only when neither is known. README
|
||||
"Install" documented:
|
||||
gets the next `v0.0.N` tag.
|
||||
`go install sneak.berlin/go/sfdupes/cmd/sfdupes@latest` then installs it.
|
||||
`go.mod` retracts `v0.0.1`, the early build without a database. When the
|
||||
linker set no version, `sfdupes --version` prints the version Go recorded in
|
||||
the binary. That is the tag, or the pseudo-version of a `go install` build. It
|
||||
prints `dev` only when neither is known. README "Install" documented:
|
||||
- the rule;
|
||||
- what `@main` and `v0.0.1` give;
|
||||
- how to check an installed binary.
|
||||
@@ -369,8 +413,9 @@ are known.
|
||||
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/26)
|
||||
|
||||
- A test fails when either `hashWorker` cancellation check in `scan.go` is
|
||||
removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83)
|
||||
- A test fails when either `hashWorker` cancellation check in
|
||||
`internal/sfdupes/scan.go` is removed (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/83)
|
||||
|
||||
- A database path holding `?`, `#` or `%` opens exactly the file it names
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
|
||||
@@ -378,15 +423,15 @@ are known.
|
||||
- `scan` rejects `--workers` below 1 as a usage error instead of running
|
||||
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
|
||||
|
||||
- A test fails when either walk cancellation check in `scan.go` is removed
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
|
||||
- A test fails when either walk cancellation check in `internal/sfdupes/scan.go`
|
||||
is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
|
||||
|
||||
- A test checks that `scan` refuses a database with another schema version
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
|
||||
|
||||
- Four inaccurate comments in `cancel_test.go` were corrected, and
|
||||
`walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles` (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/33)
|
||||
- Four inaccurate comments in `internal/sfdupes/cancel_test.go` were corrected,
|
||||
and `walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles`
|
||||
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/33)
|
||||
|
||||
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/17)
|
||||
@@ -593,15 +638,15 @@ are known.
|
||||
- `loadRecords`;
|
||||
- `resolveRoots`.
|
||||
|
||||
The single exit point is `run` in `main.go`. It maps a `fatalError`,
|
||||
anything a subcommand returned, to exit 1, and cobra's own argument and flag
|
||||
errors to exit 2. So a runtime failure is never reported as a usage error,
|
||||
nor printed with the usage text. `main_test.go` drives the CLI in-process
|
||||
and asserts the exit codes from README "Error handling" and the
|
||||
stdout/stderr split. It also checks that a fatal error after the database is
|
||||
open leaves no `-wal` or `-shm` file behind, for `scan`, `report` or `trees`
|
||||
(2026-08-09, branch `db-close-on-fatal`,
|
||||
https://git.eeqj.de/sneak/sfdupes/issues/4)
|
||||
The single exit point is `run` in `internal/sfdupes/cli.go`. It maps a
|
||||
`fatalError`, anything a subcommand returned, to exit 1, and cobra's own
|
||||
argument and flag errors to exit 2. So a runtime failure is never reported
|
||||
as a usage error, nor printed with the usage text.
|
||||
`internal/sfdupes/cli_test.go` drives the CLI in-process and asserts the
|
||||
exit codes from README "Error handling" and the stdout/stderr split. It also
|
||||
checks that a fatal error after the database is open leaves no `-wal` or
|
||||
`-shm` file behind, for `scan`, `report` or `trees` (2026-08-09, branch
|
||||
`db-close-on-fatal`, https://git.eeqj.de/sneak/sfdupes/issues/4)
|
||||
|
||||
- `golangci-lint` was updated to v2.12.2 with the canonical config, merged as
|
||||
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and
|
||||
@@ -734,6 +779,4 @@ are known.
|
||||
|
||||
# Accepted Divergences
|
||||
|
||||
- Flat single-package layout, with the `.go` files in the repo root. That is
|
||||
fine for a small single-binary tool per the Go styleguide, and the tracker
|
||||
audit agrees.
|
||||
None.
|
||||
@@ -0,0 +1,29 @@
|
||||
// Command sfdupes quickly identifies candidate duplicate files across
|
||||
// very large filesystems without reading every byte of every file.
|
||||
// Files are considered duplicates when their sizes are equal and they
|
||||
// agree on three SHA-256 hashes: of the first 64 KiB, of the last 64
|
||||
// KiB, and of the content. The content hash covers the whole file under
|
||||
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
||||
// file is never read in full. scan maintains a persistent SQLite
|
||||
// database of file signatures (SFDUPES_DATABASE, default
|
||||
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
// sfdupes report > dupes.tsv
|
||||
// sfdupes trees > dupetrees.tsv
|
||||
// sfdupes info
|
||||
// sfdupes dwim
|
||||
// sfdupes --version
|
||||
//
|
||||
// See README.md for the complete specification.
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"sneak.berlin/go/sfdupes/internal/sfdupes"
|
||||
)
|
||||
|
||||
func main() { os.Exit(sfdupes.Main()) }
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // cancels the unexported runScan and hashWorker
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -1,23 +1,6 @@
|
||||
// Command sfdupes quickly identifies candidate duplicate files across
|
||||
// very large filesystems without reading every byte of every file.
|
||||
// Files are considered duplicates when their sizes are equal and they
|
||||
// agree on three SHA-256 hashes: of the first 64 KiB, of the last 64
|
||||
// KiB, and of the content. The content hash covers the whole file under
|
||||
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
||||
// file is never read in full. scan maintains a persistent SQLite
|
||||
// database of file signatures (SFDUPES_DATABASE, default
|
||||
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
|
||||
// sfdupes report > dupes.tsv
|
||||
// sfdupes trees > dupetrees.tsv
|
||||
// sfdupes info
|
||||
// sfdupes --version
|
||||
//
|
||||
// See README.md for the complete specification.
|
||||
package main
|
||||
// Package sfdupes implements the sfdupes command and the database its
|
||||
// subcommands share.
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -45,8 +28,16 @@ const (
|
||||
cmdReport = "report"
|
||||
cmdTrees = "trees"
|
||||
cmdInfo = "info"
|
||||
cmdDwim = "dwim"
|
||||
)
|
||||
|
||||
// reportPathsHelp explains the PATH operands of report and trees in
|
||||
// their help.
|
||||
const reportPathsHelp = "With PATH operands, only the records at or " +
|
||||
"below them are used. A relative PATH is joined to the working " +
|
||||
"directory. No PATH is looked up on the filesystem, so a symlink in " +
|
||||
"one is not followed."
|
||||
|
||||
// errNoSubcommand is returned by the root command when it is invoked
|
||||
// without a subcommand. That is a usage error, and the usage text
|
||||
// cobra prints for it is the whole message.
|
||||
@@ -84,20 +75,22 @@ func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
|
||||
return info.Main.Version
|
||||
}
|
||||
|
||||
func main() {
|
||||
// Main runs the command line in os.Args and returns the process exit
|
||||
// code.
|
||||
func Main() int {
|
||||
// Once the reader of a stdout pipe has gone, as in "sfdupes report |
|
||||
// head", the Go runtime ends the process with SIGPIPE on the next
|
||||
// write instead of returning an error (README "Error handling").
|
||||
// Registering for SIGPIPE with os/signal would change that.
|
||||
os.Exit(run(os.Args[1:], os.Stdout, os.Stderr))
|
||||
return run(os.Args[1:], os.Stdout, os.Stderr)
|
||||
}
|
||||
|
||||
// run executes args against the command tree and returns the process
|
||||
// exit code. It is the program's single exit point: the subcommands
|
||||
// return their errors instead of exiting, so every deferred cleanup —
|
||||
// above all closing the database, which checkpoints the SQLite WAL —
|
||||
// runs before the process ends. The report, trees and info subcommands
|
||||
// write their data to stdout.
|
||||
// runs before the process ends. The report, trees, info and dwim
|
||||
// subcommands write their data to stdout.
|
||||
func run(args []string, stdout, stderr io.Writer) int {
|
||||
// A nil slice makes cobra fall back to os.Args, which would let a
|
||||
// test binary's own flags reach the command tree.
|
||||
@@ -175,20 +168,24 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
"print the version to stdout")
|
||||
|
||||
reportCmd := &cobra.Command{
|
||||
Use: cmdReport,
|
||||
Use: cmdReport + " [PATH...]",
|
||||
Short: "Read the scan database and print the file-level duplicates report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runReport(ctx, stdout)
|
||||
Long: "Read the scan database and print the file-level duplicates " +
|
||||
"report.\n\n" + reportPathsHelp,
|
||||
Args: cobra.ArbitraryArgs,
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
return runReport(ctx, args, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
treesCmd := &cobra.Command{
|
||||
Use: cmdTrees,
|
||||
Use: cmdTrees + " [PATH...]",
|
||||
Short: "Read the scan database and print the duplicate-tree report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runTrees(ctx, stdout)
|
||||
Long: "Read the scan database and print the duplicate-tree " +
|
||||
"report.\n\n" + reportPathsHelp,
|
||||
Args: cobra.ArbitraryArgs,
|
||||
RunE: runE(func(ctx context.Context, args []string) error {
|
||||
return runTrees(ctx, args, stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
@@ -201,7 +198,16 @@ func newRootCommand(stdout, stderr io.Writer) *cobra.Command {
|
||||
}),
|
||||
}
|
||||
|
||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd)
|
||||
dwimCmd := &cobra.Command{
|
||||
Use: cmdDwim,
|
||||
Short: "Scan / with the default excludes, then print the duplicate-tree report",
|
||||
Args: cobra.NoArgs,
|
||||
RunE: runE(func(ctx context.Context, _ []string) error {
|
||||
return runDwim(ctx, "/", stdout)
|
||||
}),
|
||||
}
|
||||
|
||||
root.AddCommand(newScanCommand(), reportCmd, treesCmd, infoCmd, dwimCmd)
|
||||
|
||||
return root
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // drives the unexported run
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -14,6 +14,8 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"golang.org/x/sys/unix"
|
||||
)
|
||||
|
||||
// usageMarker is the first line of cobra's usage text, which a usage
|
||||
@@ -574,14 +576,14 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
||||
},
|
||||
cmdReport: {
|
||||
run: func() error {
|
||||
return runReport(cancelledContext(t), io.Discard)
|
||||
return runReport(cancelledContext(t), nil, io.Discard)
|
||||
},
|
||||
want: "report: database " + defaultDatabasePath + " (default)\n" +
|
||||
reportOpenLine + "\n" + reportOpenLine + "\n",
|
||||
},
|
||||
cmdTrees: {
|
||||
run: func() error {
|
||||
return runTrees(cancelledContext(t), io.Discard)
|
||||
return runTrees(cancelledContext(t), nil, io.Discard)
|
||||
},
|
||||
want: "trees: database " + defaultDatabasePath + " (default)\n" +
|
||||
treesOpenLine + "\n" + treesOpenLine + "\n",
|
||||
@@ -617,9 +619,8 @@ func TestRunUsageErrors(t *testing.T) {
|
||||
}{
|
||||
"no subcommand": {[]string{}, usageMarker},
|
||||
"scan without paths": {[]string{cmdScan}, usageMarker},
|
||||
"report with args": {[]string{cmdReport, "x"}, usageMarker},
|
||||
"trees with args": {[]string{cmdTrees, "x"}, usageMarker},
|
||||
"info with args": {[]string{cmdInfo, "x"}, usageMarker},
|
||||
"dwim with args": {[]string{cmdDwim, "x"}, usageMarker},
|
||||
"unknown flag": {[]string{cmdScan, "--nope", "/"}, usageMarker},
|
||||
"unknown subcommand": {[]string{"nope"}, "unknown command"},
|
||||
}
|
||||
@@ -797,7 +798,7 @@ func TestRunVersionWriteFailureIsFatal(t *testing.T) {
|
||||
func scanFixture(t *testing.T) []string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
|
||||
dupes := []string{
|
||||
writeFile(t, dir, "one/a.bin", pattern(1, 300)),
|
||||
@@ -867,7 +868,7 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
writeFile(t, dir, "a.bin", pattern(1, 300))
|
||||
|
||||
// Same size as a.bin, so the scan reads it, and the read fails.
|
||||
@@ -895,58 +896,278 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
||||
assertNoSidecars(t, path)
|
||||
}
|
||||
|
||||
func TestRunScanSkipsSymlinkOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
// realTempDir returns a new temporary directory by its real path, so
|
||||
// tests hold where the temporary directory is reached through a
|
||||
// symlink.
|
||||
func realTempDir(t *testing.T) string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||
|
||||
link := filepath.Join(dir, "link")
|
||||
|
||||
err := os.Symlink(filepath.Join(dir, "target"), link)
|
||||
dir, err := filepath.EvalSymlinks(t.TempDir())
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// Scanning a directory through the symlink stores a record beneath
|
||||
// the symlink's own path for a file beneath its target.
|
||||
scanOK(t, filepath.Join(link, "sub"))
|
||||
|
||||
assertOperandSkipped(t, path, link, "symlink",
|
||||
filepath.Join(link, "sub", "f"))
|
||||
return dir
|
||||
}
|
||||
|
||||
// symlink creates link pointing at target.
|
||||
func symlink(t *testing.T, target, link string) {
|
||||
t.Helper()
|
||||
|
||||
err := os.Symlink(target, link)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanFollowsSymlinkOperand scans link, a symlink to target.
|
||||
// The operand is scanned at its real path, and its file is recorded
|
||||
// there.
|
||||
func TestRunScanFollowsSymlinkOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||
|
||||
link := filepath.Join(dir, "link")
|
||||
symlink(t, filepath.Join(dir, "target"), link)
|
||||
|
||||
stderr := scanOK(t, link)
|
||||
|
||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 0 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanWalksOperandUnderSymlinkOperand scans link and link/sub,
|
||||
// with link a symlink to target. Both resolve under target, so
|
||||
// target/sub is walked once, as part of target.
|
||||
func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "target/sub/f", pattern(1, 10))
|
||||
|
||||
link := filepath.Join(dir, "link")
|
||||
symlink(t, filepath.Join(dir, "target"), link)
|
||||
|
||||
err := os.Symlink(filepath.Join(dir, "target"), link)
|
||||
stderr := scanOK(t, link, filepath.Join(link, "sub"))
|
||||
|
||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 0 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanWalksSymlinkedOperandBelowOperand is the case of
|
||||
// https://git.eeqj.de/sneak/sfdupes/issues/144: scan dir dir/link/sub,
|
||||
// with dir/link a symlink to a directory outside dir. The walk of dir
|
||||
// does not follow link, so dir/link/sub is walked on its own at its
|
||||
// real path, and the record an earlier scan stored there survives.
|
||||
func TestRunScanWalksSymlinkedOperandBelowOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
a := writeFile(t, dir, "a", pattern(1, 10))
|
||||
|
||||
outside := realTempDir(t)
|
||||
f := writeFile(t, outside, "sub/f", pattern(2, 10))
|
||||
|
||||
symlink(t, outside, filepath.Join(dir, "link"))
|
||||
|
||||
scanOK(t, filepath.Join(outside, "sub"))
|
||||
|
||||
stderr := scanOK(t, dir, filepath.Join(dir, "link", "sub"))
|
||||
|
||||
summary := "scan: 2 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"1 unchanged), 0 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
want := []string{a, f}
|
||||
slices.Sort(want)
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, want) {
|
||||
t.Errorf("records = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanDropsSymlinkOperandInsideOperand scans dir and link, with
|
||||
// link, outside dir, a symlink to dir/real. link resolves inside dir,
|
||||
// so it is dropped as covered by dir, and its file is recorded once.
|
||||
func TestRunScanDropsSymlinkOperandInsideOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "real/f", pattern(1, 10))
|
||||
|
||||
link := filepath.Join(realTempDir(t), "link")
|
||||
symlink(t, filepath.Join(dir, "real"), link)
|
||||
|
||||
stderr := scanOK(t, dir, link)
|
||||
|
||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 0 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanDanglingSymlinkOperandIsFatal scans a valid directory and
|
||||
// a symlink whose target does not exist. The scan fails with exit 1 and
|
||||
// an error naming the symlink as given, before it walks anything, so no
|
||||
// database is created.
|
||||
func TestRunScanDanglingSymlinkOperandIsFatal(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
writeFile(t, dir, "f", pattern(1, 10))
|
||||
|
||||
dangling := filepath.Join(realTempDir(t), "dangling")
|
||||
symlink(t, filepath.Join(dir, "missing"), dangling)
|
||||
|
||||
stdout := captureStdout(t)
|
||||
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := run([]string{cmdScan, dir, dangling}, os.Stdout, &stderr)
|
||||
if code != exitFatal {
|
||||
t.Errorf("run(scan %s %s) = %d, want %d", dir, dangling, code,
|
||||
exitFatal)
|
||||
}
|
||||
|
||||
assertFatalOutput(t, stderr.String(), stdout())
|
||||
|
||||
want := "sfdupes: resolve " + dangling + ": "
|
||||
if !strings.Contains(stderr.String(), want) {
|
||||
t.Errorf("stderr = %q, want %q", stderr.String(), want)
|
||||
}
|
||||
|
||||
_, err := os.Stat(path)
|
||||
if !errors.Is(err, fs.ErrNotExist) {
|
||||
t.Errorf("database: stat error %v, want it missing", err)
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath runs scan .
|
||||
// after a cd through home/proj, a symlink to data/proj. The file is
|
||||
// recorded under data/proj, its real path.
|
||||
func TestRunScanFromSymlinkedWorkingDirectoryRecordsRealPath(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
|
||||
|
||||
err := os.Mkdir(filepath.Join(dir, "home"), 0o750)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
// link is dropped as a symlink, but link/sub must still be scanned,
|
||||
// not dropped as lying under link.
|
||||
scanOK(t, link, filepath.Join(link, "sub"))
|
||||
symlink(t, filepath.Join(dir, "data", "proj"),
|
||||
filepath.Join(dir, "home", "proj"))
|
||||
|
||||
db, err := openDB(path, reportParams)
|
||||
// t.Chdir also sets PWD to the path through the symlink, as a
|
||||
// shell does after cd.
|
||||
t.Chdir(filepath.Join(dir, "home", "proj"))
|
||||
|
||||
scanOK(t, ".")
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanFollowsSymlinkBeforeDotDot scans a path with a ".." after
|
||||
// home/proj, a symlink to data/proj: home/proj/.. from dir, and .. after
|
||||
// a cd through home/proj. As with realpath, the ".." counts from
|
||||
// data/proj, so data is scanned and home is not.
|
||||
func TestRunScanFollowsSymlinkBeforeDotDot(t *testing.T) {
|
||||
for _, tc := range []struct{ wd, operand string }{
|
||||
{"", "home/proj/.."},
|
||||
{"home/proj", ".."},
|
||||
} {
|
||||
t.Run(tc.operand, func(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "data/proj/f", pattern(1, 10))
|
||||
g := writeFile(t, dir, "data/g", pattern(2, 10))
|
||||
writeFile(t, dir, "home/h", pattern(3, 10))
|
||||
symlink(t, filepath.Join(dir, "data", "proj"),
|
||||
filepath.Join(dir, "home", "proj"))
|
||||
|
||||
// t.Chdir also sets PWD to the path given, as a shell does
|
||||
// after cd.
|
||||
t.Chdir(filepath.Join(dir, tc.wd))
|
||||
|
||||
scanOK(t, tc.operand)
|
||||
|
||||
want := []string{f, g}
|
||||
slices.Sort(want)
|
||||
|
||||
if got := storedPaths(t, path); !slices.Equal(got, want) {
|
||||
t.Errorf("records = %q, want %q", got, want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunScanSkipsSymlinkToFIFOOperand scans link, a symlink to a FIFO.
|
||||
// The scan warns about the FIFO by its real path, counts it as skipped,
|
||||
// and exits 0.
|
||||
func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) {
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
dir := realTempDir(t)
|
||||
fifo := filepath.Join(dir, "fifo")
|
||||
|
||||
err := unix.Mkfifo(fifo, 0o600)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
link := filepath.Join(dir, "link")
|
||||
symlink(t, fifo, link)
|
||||
|
||||
recordByPath(t, dbRecords(t, db), filepath.Join(link, "sub", "f"))
|
||||
stderr := scanOK(t, link)
|
||||
|
||||
warning := "walk " + fifo + ": skipping FIFO operand\n"
|
||||
if !strings.Contains(stderr, warning) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||
}
|
||||
|
||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 1 skipped\n"
|
||||
if !strings.Contains(stderr, summary) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunScanSkipsZFSOperand(t *testing.T) {
|
||||
zfs := filepath.Join(t.TempDir(), ".zfs")
|
||||
zfs := filepath.Join(realTempDir(t), ".zfs")
|
||||
f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
|
||||
|
||||
// No scan records a file inside a .zfs directory, so the record
|
||||
@@ -997,7 +1218,7 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
|
||||
writeFile(t, hourly, "f", pattern(1, 10))
|
||||
f := writeFile(t, dir, "f", pattern(2, 10))
|
||||
@@ -1041,23 +1262,19 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
|
||||
|
||||
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
|
||||
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
|
||||
// by the path given, counted as skipped, and not walked.
|
||||
// by its real path, counted as skipped, and not walked.
|
||||
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
|
||||
|
||||
snaps := filepath.Join(dir, "snaps")
|
||||
symlink(t, filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
|
||||
|
||||
err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
daily := filepath.Join(snaps, "daily")
|
||||
stderr := scanOK(t, daily)
|
||||
stderr := scanOK(t, filepath.Join(snaps, "daily"))
|
||||
|
||||
daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily")
|
||||
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
|
||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 1 skipped\n"
|
||||
@@ -1069,24 +1286,19 @@ func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
|
||||
// a symlink to a directory outside any .zfs directory. The operand is
|
||||
// scanned as usual, and its file is recorded by the path given.
|
||||
// a symlink to pool/data. The operand is scanned as usual, and its file
|
||||
// is recorded by its real path.
|
||||
func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
|
||||
dir := realTempDir(t)
|
||||
f := writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
|
||||
|
||||
data := filepath.Join(dir, "data")
|
||||
symlink(t, filepath.Join(dir, "pool", "data"), data)
|
||||
|
||||
err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
daily := filepath.Join(data, "daily")
|
||||
stderr := scanOK(t, daily)
|
||||
stderr := scanOK(t, filepath.Join(data, "daily"))
|
||||
|
||||
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
|
||||
"0 unchanged), 0 skipped\n"
|
||||
@@ -1094,14 +1306,9 @@ func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
|
||||
t.Errorf("stderr = %q, want %q", stderr, summary)
|
||||
}
|
||||
|
||||
db, err := openDB(path, reportParams)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
if got := storedPaths(t, path); !slices.Equal(got, []string{f}) {
|
||||
t.Errorf("records = %q, want %q", got, []string{f})
|
||||
}
|
||||
|
||||
t.Cleanup(func() { _ = db.Close() })
|
||||
|
||||
recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
|
||||
}
|
||||
|
||||
func TestRunReportSucceeds(t *testing.T) {
|
||||
@@ -1517,9 +1724,13 @@ func TestStdoutWriteErrorPropagates(t *testing.T) {
|
||||
scanFixture(t)
|
||||
|
||||
cases := map[string]func(context.Context, io.Writer) error{
|
||||
cmdReport: runReport,
|
||||
cmdTrees: runTrees,
|
||||
cmdInfo: runInfo,
|
||||
cmdReport: func(ctx context.Context, w io.Writer) error {
|
||||
return runReport(ctx, nil, w)
|
||||
},
|
||||
cmdTrees: func(ctx context.Context, w io.Writer) error {
|
||||
return runTrees(ctx, nil, w)
|
||||
},
|
||||
cmdInfo: runInfo,
|
||||
}
|
||||
|
||||
for name, fn := range cases {
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -146,33 +146,44 @@ var errHashSize = errors.New("hash is not 32 bytes")
|
||||
|
||||
// databasePath resolves the database location: SFDUPES_DATABASE when
|
||||
// set and non-empty, the compiled-in default otherwise. It returns the
|
||||
// path, with the working directory, its symlinks resolved, put in front
|
||||
// of it when it is relative, and where the path came from, "default" or
|
||||
// "from SFDUPES_DATABASE", for the line each subcommand prints first on
|
||||
// stderr.
|
||||
// path, made absolute by absolutePath, and where the path came from,
|
||||
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
||||
// prints first on stderr.
|
||||
func databasePath() (string, string, error) {
|
||||
path, source := defaultDatabasePath, "default"
|
||||
if p := os.Getenv(databaseEnv); p != "" {
|
||||
path, source = p, "from "+databaseEnv
|
||||
}
|
||||
|
||||
if filepath.IsAbs(path) {
|
||||
return path, source, nil
|
||||
}
|
||||
|
||||
// The working directory as the kernel reports it, with no symlink
|
||||
// in it, as README "Database" says. os.Getwd may return PWD
|
||||
// instead, which can name it through a symlink.
|
||||
wd, err := unix.Getwd()
|
||||
abs, err := absolutePath(path)
|
||||
if err != nil {
|
||||
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
|
||||
}
|
||||
|
||||
// Not filepath.Abs or filepath.Join: they clean the path, which
|
||||
// drops each ".." together with the directory name before it. When
|
||||
// that directory is a symlink, the cleaned path names another file.
|
||||
return abs, source, nil
|
||||
}
|
||||
|
||||
// absolutePath returns path unchanged when it is absolute, and
|
||||
// otherwise the working directory, a "/" and path. The working
|
||||
// directory is the one the kernel reports, with no symlink in it;
|
||||
// os.Getwd may return PWD instead, which can name it through a symlink.
|
||||
// The result is not cleaned, as filepath.Abs and filepath.Join would
|
||||
// clean it. Cleaning drops each ".." together with the directory name
|
||||
// before it. When that name is a symlink, the cleaned path names
|
||||
// another file than the one the system reaches by following the symlink
|
||||
// first.
|
||||
func absolutePath(path string) (string, error) {
|
||||
if filepath.IsAbs(path) {
|
||||
return path, nil
|
||||
}
|
||||
|
||||
wd, err := unix.Getwd()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// The working directory ends in "/" only when it is the root.
|
||||
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
|
||||
return strings.TrimSuffix(wd, "/") + "/" + path, nil
|
||||
}
|
||||
|
||||
// createDatabaseDirectory creates the directory the database at path
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // calls the unexported database openers
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -0,0 +1,37 @@
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"runtime"
|
||||
)
|
||||
|
||||
// runDwim implements the dwim subcommand: it runs what "sfdupes scan
|
||||
// root" runs with no flags, then what "sfdupes trees" runs, and ends
|
||||
// with a line on stderr saying how to print the report again. The dwim
|
||||
// command passes "/" as root. It catches SIGINT and SIGTERM as scan
|
||||
// does, but only during the scan. An error from the scan, an interrupt
|
||||
// included, ends it before the report. During the report, a signal
|
||||
// ends it at once, as it ends trees.
|
||||
func runDwim(ctx context.Context, root string, stdout io.Writer) error {
|
||||
scanCtx, stop := interruptContext(ctx)
|
||||
err := runScan(scanCtx, []string{root}, runtime.NumCPU(), false, false)
|
||||
|
||||
stop()
|
||||
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
err = runTrees(ctx, nil, stdout)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
fmt.Fprintln(os.Stderr,
|
||||
`dwim: run "sfdupes trees" to print this report again`)
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,83 @@
|
||||
package sfdupes //nolint:testpackage // calls the unexported runDwim
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRunDwimPrintsLargestTreesFirst(t *testing.T) {
|
||||
// dwim scans its root, then prints the trees report, the larger of
|
||||
// two duplicate-tree pairs first, and ends by saying how to print
|
||||
// the report again.
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
root := realTempDir(t)
|
||||
|
||||
small := []string{
|
||||
filepath.Dir(writeFile(t, root, "a1/f.bin", pattern(1, 300))),
|
||||
filepath.Dir(writeFile(t, root, "a2/f.bin", pattern(1, 300))),
|
||||
}
|
||||
large := []string{
|
||||
filepath.Dir(writeFile(t, root, "b1/f.bin", pattern(2, 5000))),
|
||||
filepath.Dir(writeFile(t, root, "b2/f.bin", pattern(2, 5000))),
|
||||
}
|
||||
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
err := runDwim(t.Context(), root, &stdout)
|
||||
if err != nil {
|
||||
t.Fatalf("runDwim: %v; stderr: %s", err, stderr())
|
||||
}
|
||||
|
||||
lines := strings.Split(strings.TrimSuffix(stdout.String(), "\n"), "\n")
|
||||
if len(lines) != 3 {
|
||||
t.Fatalf("stdout = %q, want a header and two groups",
|
||||
stdout.String())
|
||||
}
|
||||
|
||||
wantFirst := large[0] + "\t" + large[1] + "\t1\t5000\t"
|
||||
if !strings.HasPrefix(lines[1], wantFirst) {
|
||||
t.Errorf("first group = %q, want it to start %q", lines[1], wantFirst)
|
||||
}
|
||||
|
||||
wantSecond := small[0] + "\t" + small[1] + "\t1\t300\t"
|
||||
if !strings.HasPrefix(lines[2], wantSecond) {
|
||||
t.Errorf("second group = %q, want it to start %q",
|
||||
lines[2], wantSecond)
|
||||
}
|
||||
|
||||
last := "\ndwim: run \"sfdupes trees\" to print this report again\n"
|
||||
if got := stderr(); !strings.HasSuffix(got, last) {
|
||||
t.Errorf("stderr = %q, want it to end %q", got, last)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunDwimScanErrorPrintsNoReport(t *testing.T) {
|
||||
// A fatal scan error ends dwim before the report: a root that does
|
||||
// not exist fails the scan as it fails scan itself.
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
missing := filepath.Join(realTempDir(t), "missing")
|
||||
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
err := runDwim(t.Context(), missing, &stdout)
|
||||
if err == nil {
|
||||
t.Fatal("runDwim on a missing root: error = nil, want it fatal")
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != "" {
|
||||
t.Errorf("stdout = %q, want no report", got)
|
||||
}
|
||||
|
||||
if got := stderr(); strings.Contains(got, "trees:") ||
|
||||
strings.Contains(got, "dwim:") {
|
||||
t.Errorf("stderr = %q, want nothing from trees or dwim", got)
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
// This file holds two rules for the walk (README "Rules for the walk"):
|
||||
// the default excludes, the paths a scan's walk does not enter below an
|
||||
@@ -1,6 +1,6 @@
|
||||
//go:build darwin
|
||||
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"slices"
|
||||
@@ -1,6 +1,6 @@
|
||||
//go:build !darwin
|
||||
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // checks the unexported defaultExcludes
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -286,7 +286,7 @@ func TestRunScanTempDirOperand(t *testing.T) {
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
|
||||
t.Skip(dir + " is not inside a default exclude")
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // seeds the database with unexported scanRec
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // drives the unexported startProgress
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"slices"
|
||||
"strings"
|
||||
"time"
|
||||
@@ -41,10 +42,11 @@ func reportPhase(name string) string {
|
||||
}
|
||||
|
||||
// runReport implements the report subcommand: it prints the file-level
|
||||
// duplicates report as TSV on stdout. It never touches the scanned
|
||||
// filesystem; its only I/O is the database, stdout, and stderr. It
|
||||
// names the database on stderr before it opens it. Any database
|
||||
// problem, including a missing database, is fatal.
|
||||
// duplicates report as TSV on stdout, from the records at or below the
|
||||
// given paths, or from every record when none is given. It never
|
||||
// touches the scanned filesystem; its only I/O is the database,
|
||||
// stdout, and stderr. It names the database on stderr before it opens
|
||||
// it. Any database problem, including a missing database, is fatal.
|
||||
//
|
||||
// The work runs in two phases, each under a progress display of its
|
||||
// own on stderr: open (open the database) and read (read the number of
|
||||
@@ -52,7 +54,12 @@ func reportPhase(name string) string {
|
||||
// order, writing each size's duplicate groups to stdout once that size
|
||||
// is read; see readReport). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
func runReport(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
paths, err := reportPaths(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -83,7 +90,7 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
label: reportPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db/out",
|
||||
})
|
||||
report, err := readReport(ctx, db, dbPath, stdout, prog)
|
||||
report, err := readReport(ctx, db, dbPath, paths, stdout, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -104,11 +111,12 @@ func runReport(ctx context.Context, stdout io.Writer) error {
|
||||
// dbPath, as its total, then reads every record in signature order,
|
||||
// counting each on prog, and writes the report to stdout: the header
|
||||
// line, then the rows of each size's duplicate groups once that size is
|
||||
// read. stdout may be the terminal prog is drawn on, so the lines go
|
||||
// read. Only the records that inReportPaths selects from paths join a
|
||||
// group. stdout may be the terminal prog is drawn on, so the lines go
|
||||
// through prog's writer. It returns what it wrote, with the totals for
|
||||
// the summary line.
|
||||
func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
stdout io.Writer, prog *progress,
|
||||
paths []string, stdout io.Writer, prog *progress,
|
||||
) (*reportWriter, error) {
|
||||
out := prog.writer(stdout)
|
||||
report := &reportWriter{out: out}
|
||||
@@ -123,6 +131,12 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
err = loadSignatureRows(ctx, db, prog.setTotal, func(r scanRec) error {
|
||||
prog.increment()
|
||||
|
||||
report.records++
|
||||
|
||||
if !inReportPaths(r.path, paths) {
|
||||
return nil
|
||||
}
|
||||
|
||||
writeErr = report.add(r)
|
||||
|
||||
return writeErr
|
||||
@@ -154,7 +168,7 @@ func readReport(ctx context.Context, db *sql.DB, dbPath string,
|
||||
// records of one size, so it holds only the duplicate groups of the
|
||||
// size being read, and writes them, ordered by first path, once a
|
||||
// record of a smaller size comes. It counts what the summary line
|
||||
// reports.
|
||||
// reports; readReport counts the records read.
|
||||
type reportWriter struct {
|
||||
out io.Writer
|
||||
|
||||
@@ -170,8 +184,6 @@ type reportWriter struct {
|
||||
// its paths; any other ends them, and one of a smaller size first
|
||||
// writes the groups of the size before.
|
||||
func (w *reportWriter) add(r scanRec) error {
|
||||
w.records++
|
||||
|
||||
if r.size == w.sig.size && r.head == w.sig.head &&
|
||||
r.tail == w.sig.tail && r.content == w.sig.content {
|
||||
w.paths = append(w.paths, r.path)
|
||||
@@ -195,10 +207,11 @@ func (w *reportWriter) add(r scanRec) error {
|
||||
}
|
||||
|
||||
// endGroup ends the paths read with w.sig: two or more are a duplicate
|
||||
// group, held with its paths sorted until writeGroups writes it.
|
||||
// group, held with its paths sorted until writeGroups writes it. Empty
|
||||
// files form no group, since removing one frees nothing.
|
||||
func (w *reportWriter) endGroup() {
|
||||
if len(w.paths) < minGroupSize {
|
||||
// A path with no duplicate is dropped, and its slice reused.
|
||||
if len(w.paths) < minGroupSize || w.sig.size == 0 {
|
||||
// The paths are dropped, and their slice reused.
|
||||
w.paths = w.paths[:0]
|
||||
|
||||
return
|
||||
@@ -245,6 +258,46 @@ func (w *reportWriter) finish() error {
|
||||
return w.writeGroups()
|
||||
}
|
||||
|
||||
// reportPaths returns the paths given to report or trees, each made
|
||||
// absolute by absolutePath and then cleaned. The records hold real
|
||||
// paths, with no "." or ".." element and no repeated or trailing
|
||||
// slash, so a given path is cleaned to match them. No path is looked up
|
||||
// on the filesystem.
|
||||
func reportPaths(args []string) ([]string, error) {
|
||||
paths := make([]string, 0, len(args))
|
||||
|
||||
for _, a := range args {
|
||||
abs, err := absolutePath(a)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve path %s: %w", escapePath(a), err)
|
||||
}
|
||||
|
||||
paths = append(paths, filepath.Clean(abs))
|
||||
}
|
||||
|
||||
return paths, nil
|
||||
}
|
||||
|
||||
// inReportPaths reports whether path is one of paths or lies below one
|
||||
// of them. Each of paths is clean and absolute, as reportPaths returns
|
||||
// it. With no paths, a report covers the whole database, so every path
|
||||
// is in it.
|
||||
func inReportPaths(path string, paths []string) bool {
|
||||
if len(paths) == 0 {
|
||||
return true
|
||||
}
|
||||
|
||||
for _, p := range paths {
|
||||
// Only the root directory's clean path ends in "/".
|
||||
dir := strings.TrimSuffix(p, "/") + "/"
|
||||
if path == p || strings.HasPrefix(path, dir) {
|
||||
return true
|
||||
}
|
||||
}
|
||||
|
||||
return false
|
||||
}
|
||||
|
||||
// escapePath returns a path as it is written in a report column (README
|
||||
// "Report output format"): a backslash, tab, newline or carriage return
|
||||
// becomes \\, \t, \n or \r, and every other byte is kept as it is.
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // checks the unexported dupeGroup results
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -80,7 +80,7 @@ func dupeGroups(t *testing.T, db *sql.DB) []dupeGroup {
|
||||
|
||||
var out bytes.Buffer
|
||||
|
||||
_, err := readReport(t.Context(), db, "", &out, nil)
|
||||
_, err := readReport(t.Context(), db, "", nil, &out, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
@@ -126,6 +126,98 @@ func dupeGroupsOf(t *testing.T, recs []scanRec) []dupeGroup {
|
||||
return dupeGroups(t, db)
|
||||
}
|
||||
|
||||
// TestRunReportPaths runs report with paths on three copies of one
|
||||
// file, two of them under /srv/photos, and a fourth copy in
|
||||
// /srv/photos2. Only the records at or below the given paths form the
|
||||
// groups and the totals, however each path is written.
|
||||
//
|
||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||
func TestRunReportPaths(t *testing.T) {
|
||||
h := testHash("h")
|
||||
a, x, y, z := "/a/copy", "/srv/photos/x", "/srv/photos/y", "/srv/photos2/z"
|
||||
|
||||
recs := make([]scanRec, 0, 4)
|
||||
for _, p := range []string{a, x, y, z} {
|
||||
recs = append(recs,
|
||||
scanRec{size: 7, head: h, tail: h, content: h, path: p})
|
||||
}
|
||||
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
header := "first\tdupe\tsize\n"
|
||||
row := func(first, dupe string) string {
|
||||
return first + "\t" + dupe + "\t7\n"
|
||||
}
|
||||
photos := header + row(x, y)
|
||||
|
||||
cases := map[string]struct {
|
||||
paths []string
|
||||
want string
|
||||
summary string
|
||||
}{
|
||||
"directory": {[]string{"/srv/photos"}, photos,
|
||||
"1 duplicate groups, 1 dupe files, 7 B reclaimable"},
|
||||
"trailing slash": {[]string{"/srv/photos/"}, photos, ""},
|
||||
"dot and dot-dot": {[]string{"/srv/./a/../photos"}, photos, ""},
|
||||
"files": {[]string{a, z}, header + row(a, z), ""},
|
||||
"overlapping": {[]string{"/srv", "/srv/photos"},
|
||||
header + row(x, y) + row(x, z), ""},
|
||||
"root": {[]string{"/"},
|
||||
header + row(a, x) + row(a, y) + row(a, z), ""},
|
||||
"no record": {[]string{"/srv/photo"}, header,
|
||||
"0 duplicate groups, 0 dupe files, 0 B reclaimable"},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
var stdout bytes.Buffer
|
||||
|
||||
stderr := captureStderr(t)
|
||||
|
||||
code := run(append([]string{cmdReport}, tc.paths...),
|
||||
&stdout, os.Stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(report %q) = %d, want %d; stderr: %s",
|
||||
tc.paths, code, exitOK, stderr())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != tc.want {
|
||||
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||
}
|
||||
|
||||
// The records read are all four, whatever the paths.
|
||||
summary := "report: 4 records read, " + tc.summary + "\n"
|
||||
if got := stderr(); tc.summary != "" &&
|
||||
!strings.HasSuffix(got, summary) {
|
||||
t.Errorf("stderr = %q, want it to end %q", got, summary)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
//nolint:paralleltest // t.Chdir changes the process-wide working directory
|
||||
func TestReportPathsMakesRelativePathsAbsolute(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
t.Chdir(dir)
|
||||
|
||||
wd, err := absolutePath(".")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
got, err := reportPaths([]string{"photos/", "a/../b", "/c//d"})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
want := []string{
|
||||
filepath.Join(wd, "photos"), filepath.Join(wd, "b"), "/c/d",
|
||||
}
|
||||
if !slices.Equal(got, want) {
|
||||
t.Errorf("reportPaths = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunReportEscapesPaths(t *testing.T) {
|
||||
t.Setenv(databaseEnv, seedDatabase(t, awkwardPairRecs()))
|
||||
|
||||
@@ -172,7 +264,7 @@ func TestReportStdoutFailsWhileReading(t *testing.T) {
|
||||
// Counts the records read, and draws nothing.
|
||||
prog := &progress{now: time.Now}
|
||||
|
||||
_, err = readReport(t.Context(), db, "", failingWriter{}, prog)
|
||||
_, err = readReport(t.Context(), db, "", nil, failingWriter{}, prog)
|
||||
if !errors.Is(err, errWriteFailed) ||
|
||||
!strings.HasPrefix(err.Error(), "write stdout: ") {
|
||||
t.Errorf("error = %v, want write stdout: %v", err, errWriteFailed)
|
||||
@@ -266,7 +358,7 @@ func TestReadReportOnTerminal(t *testing.T) {
|
||||
waitingFor: countingRecords,
|
||||
}, terminalWidth, time.Hour, time.Now)
|
||||
|
||||
_, err = readReport(t.Context(), db, "", os.Stderr, p)
|
||||
_, err = readReport(t.Context(), db, "", nil, os.Stderr, p)
|
||||
|
||||
p.finish()
|
||||
|
||||
@@ -434,6 +526,27 @@ func TestDupeGroups(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
|
||||
// duplicate group, while 1-byte files still do.
|
||||
func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
e, x := sha256.Sum256(nil), testHash("x")
|
||||
recs := []scanRec{
|
||||
{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
|
||||
{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
|
||||
{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
|
||||
}
|
||||
|
||||
groups := dupeGroupsOf(t, recs)
|
||||
if len(groups) != 1 || groups[0].size != 1 ||
|
||||
!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
|
||||
t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDupeGroupsContentSeparates(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"cmp"
|
||||
@@ -209,29 +209,30 @@ func interrupted(walked int) error {
|
||||
return errInterrupted
|
||||
}
|
||||
|
||||
// resolveRoots converts each PATH operand to an absolute, lexically
|
||||
// cleaned path (symlinks are not resolved) and verifies that it
|
||||
// exists. Database records are keyed by absolute path, so scan results
|
||||
// must not depend on the working directory.
|
||||
// resolveRoots converts each PATH operand to its real path: absolute,
|
||||
// with every symlink in it followed. Database records are keyed by
|
||||
// that path, so scan results depend neither on the working directory
|
||||
// nor on the symlinks an operand was given through. An operand that
|
||||
// cannot be resolved, because it or a symlink's target does not exist
|
||||
// or cannot be read, is a fatal error before any scanning.
|
||||
func resolveRoots(roots []string) ([]string, error) {
|
||||
abs := make([]string, 0, len(roots))
|
||||
resolved := make([]string, 0, len(roots))
|
||||
|
||||
for _, root := range roots {
|
||||
a, err := filepath.Abs(root)
|
||||
a, err := absolutePath(root)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
}
|
||||
|
||||
// A nonexistent operand is a fatal error before any scanning.
|
||||
_, err = os.Lstat(a)
|
||||
r, err := filepath.EvalSymlinks(a)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
return nil, fmt.Errorf("resolve %s: %w", root, err)
|
||||
}
|
||||
|
||||
abs = append(abs, a)
|
||||
resolved = append(resolved, r)
|
||||
}
|
||||
|
||||
return abs, nil
|
||||
return resolved, nil
|
||||
}
|
||||
|
||||
// pruneRoots drops operands already covered by another operand:
|
||||
@@ -417,14 +418,16 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
|
||||
|
||||
// walkableRoots returns the operands the walk can start from: regular
|
||||
// files, and directories not named .zfs, none of them inside a
|
||||
// directory named .zfs. Every other operand is warned about, counted as
|
||||
// skipped, and dropped. A dropped operand is no longer a root, so the
|
||||
// records stored beneath it are treated as if it had not been given:
|
||||
// those under no other root count as outside the roots and are not
|
||||
// deleted as unverified. An operand on a filesystem in rules that is not
|
||||
// persistent is dropped the same way, with a line saying so that is not
|
||||
// a warning, and is not counted as skipped. An operand that fails lstat
|
||||
// here is kept, and the walk warns about it unless it no longer exists.
|
||||
// directory named .zfs. The operands are real paths from resolveRoots,
|
||||
// so these checks apply to what a symlink operand points at. Every
|
||||
// other operand is warned about, counted as skipped, and dropped. A
|
||||
// dropped operand is no longer a root, so the records stored beneath it
|
||||
// are treated as if it had not been given: those under no other root
|
||||
// count as outside the roots and are not deleted as unverified. An
|
||||
// operand on a filesystem in rules that is not persistent is dropped
|
||||
// the same way, with a line saying so that is not a warning, and is not
|
||||
// counted as skipped. An operand that fails lstat here is kept, and the
|
||||
// walk warns about it unless it no longer exists.
|
||||
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
|
||||
kept := make([]string, 0, len(roots))
|
||||
|
||||
@@ -554,7 +557,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
||||
|
||||
// The space a file takes on disk can change while its size and
|
||||
// mtime stay the same. ZFS, for one, counts a new file's blocks
|
||||
// only seconds after the write.
|
||||
// up to seconds after the write.
|
||||
if old.diskSize != ev.rec.diskSize {
|
||||
s.diskSizes = append(s.diskSizes, ev.rec)
|
||||
}
|
||||
@@ -894,19 +897,13 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
|
||||
// operandWarning returns the one-line warning for an operand the walk
|
||||
// does not start from, naming the path and what it is, or "" for one it
|
||||
// does: a regular file, or a directory not named .zfs, that is not
|
||||
// inside a directory named .zfs, neither as given nor with its symlinks
|
||||
// resolved. Symlinks are never followed, including as operands.
|
||||
// inside a directory named .zfs. resolveRoots has already followed the
|
||||
// operand's symlinks, so a symlink here is an operand replaced by one
|
||||
// after it was resolved, and is not followed.
|
||||
func operandWarning(root string, fi fs.FileInfo) string {
|
||||
// The walk never enters a .zfs directory, so it starts from nothing
|
||||
// inside one either, not even through a symlink. An operand whose
|
||||
// symlinks cannot be resolved is checked as given.
|
||||
resolved, err := filepath.EvalSymlinks(root)
|
||||
if err != nil {
|
||||
resolved = root
|
||||
}
|
||||
|
||||
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
|
||||
slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
|
||||
// inside one either.
|
||||
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") {
|
||||
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
|
||||
root)
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
//go:build darwin
|
||||
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import "syscall"
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
//go:build !darwin
|
||||
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import "syscall"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // drives the unexported syncTree
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
@@ -43,10 +43,16 @@ func writeFile(t *testing.T, dir, name string, data []byte) string {
|
||||
}
|
||||
|
||||
// diskSizeOf returns the bytes the file at path takes on disk, from its
|
||||
// own lstat: st_blocks counts 512-byte units. ZFS counts a new file's
|
||||
// blocks only seconds after the write, and fsync does not hasten that.
|
||||
// A test therefore calls unix.Sync, which waits for it, before the scan
|
||||
// whose disk_size it compares with this.
|
||||
// own lstat: st_blocks counts 512-byte units. ZFS counts a write's
|
||||
// blocks only when it commits the write, up to seconds later, and one
|
||||
// commit can take the file through several growing sizes. So this value
|
||||
// can change at any moment. Neither fsync nor unix.Sync hastens that,
|
||||
// but a unix.Sync call often takes long enough that the blocks are
|
||||
// counted when it returns. A test that needs the value to change waits
|
||||
// for the change. A test that compares it with a scan's disk_size reads
|
||||
// it before and after the scan. The rewrite test rescans until the two
|
||||
// reads match, then compares exactly. The other two tests fail only on
|
||||
// a disk_size outside the two reads.
|
||||
func diskSizeOf(t *testing.T, path string) int64 {
|
||||
t.Helper()
|
||||
|
||||
@@ -1088,7 +1094,7 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
|
||||
func buildSmokeTree(t *testing.T) string {
|
||||
t.Helper()
|
||||
|
||||
dir := t.TempDir()
|
||||
dir := realTempDir(t)
|
||||
one := pattern(10, 2000)
|
||||
f1 := pattern(30, 3000)
|
||||
f2 := pattern(40, 100)
|
||||
@@ -1179,11 +1185,12 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
|
||||
t.Helper()
|
||||
|
||||
groups := dupeGroups(t, db)
|
||||
if len(groups) != 5 {
|
||||
t.Fatalf("len(groups) = %d, want 5", len(groups))
|
||||
if len(groups) != 4 {
|
||||
t.Fatalf("len(groups) = %d, want 4", len(groups))
|
||||
}
|
||||
|
||||
wantSizes := []int64{3000, 2000, 100, 1, 0}
|
||||
// The empty files empty1 and empty2 form no group.
|
||||
wantSizes := []int64{3000, 2000, 100, 1}
|
||||
for i, g := range groups {
|
||||
if g.size != wantSizes[i] {
|
||||
t.Errorf("groups[%d].size = %d, want %d",
|
||||
@@ -1448,20 +1455,26 @@ func TestScanStoresDiskSize(t *testing.T) {
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
|
||||
want := diskSizeOf(t, p)
|
||||
if want == 0 || want >= length {
|
||||
size := diskSizeOf(t, p)
|
||||
if size == 0 || size >= length {
|
||||
t.Fatalf("the file takes %d bytes on disk, want more than none "+
|
||||
"and fewer than its length %d", want, length)
|
||||
"and fewer than its length %d", size, length)
|
||||
}
|
||||
|
||||
for _, operand := range []string{filepath.Dir(p), p} {
|
||||
db := openTestDB(t)
|
||||
|
||||
// ZFS may count the blocks during the scan; see diskSizeOf.
|
||||
before := diskSizeOf(t, p)
|
||||
|
||||
syncTree(t, db, operand)
|
||||
|
||||
after := diskSizeOf(t, p)
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got != want {
|
||||
t.Errorf("scan of %s: disk_size = %d, want %d",
|
||||
operand, got, want)
|
||||
if got < before || got > after {
|
||||
t.Errorf("scan of %s: disk_size = %d, want %d to %d",
|
||||
operand, got, before, after)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1479,7 +1492,7 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
|
||||
|
||||
syncTree(t, db, dir)
|
||||
|
||||
before := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
recorded := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
|
||||
pokeAt(t, p, 512*1024, pattern(7, 64*1024))
|
||||
|
||||
@@ -1492,22 +1505,53 @@ func TestSyncScanRewriteUpdatesDiskSize(t *testing.T) {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
// Wait for ZFS to count the write; see diskSizeOf.
|
||||
deadline := time.Now().Add(30 * time.Second)
|
||||
for diskSizeOf(t, p) == recorded {
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("the file still takes %d bytes on disk 30 seconds "+
|
||||
"after the write into its hole", recorded)
|
||||
}
|
||||
|
||||
want := diskSizeOf(t, p)
|
||||
if want == before {
|
||||
t.Fatalf("the file still takes %d bytes on disk after the write "+
|
||||
"into its hole", want)
|
||||
unix.Sync()
|
||||
}
|
||||
|
||||
st := syncTree(t, db, dir)
|
||||
if st != (scanStats{walked: 1, updated: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
||||
}
|
||||
// ZFS may still be counting, so the size may change during a rescan.
|
||||
// Only a rescan with the same size on both sides of it can be compared
|
||||
// exactly. A rescan that keeps the old size still fails, because the
|
||||
// wait made before differ from it.
|
||||
for {
|
||||
before := diskSizeOf(t, p)
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got != want {
|
||||
t.Errorf("disk_size = %d, want %d", got, want)
|
||||
st := syncTree(t, db, dir)
|
||||
|
||||
after := diskSizeOf(t, p)
|
||||
|
||||
if st != (scanStats{walked: 1, updated: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 updated", st)
|
||||
}
|
||||
|
||||
if before == after {
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if got != after {
|
||||
t.Errorf("disk_size = %d, want %d", got, after)
|
||||
}
|
||||
|
||||
return
|
||||
}
|
||||
|
||||
if time.Now().After(deadline) {
|
||||
t.Fatalf("the size on disk was still changing 30 seconds after "+
|
||||
"the write into its hole: %d, then %d", before, after)
|
||||
}
|
||||
|
||||
// A later mtime makes the next rescan read the file again.
|
||||
later = later.Add(time.Second)
|
||||
|
||||
err = os.Chtimes(p, later, later)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1533,15 +1577,21 @@ func TestSyncScanUnchangedUpdatesDiskSize(t *testing.T) {
|
||||
|
||||
unix.Sync() // see diskSizeOf
|
||||
|
||||
// ZFS may count the blocks during the rescan; see diskSizeOf.
|
||||
before := diskSizeOf(t, p)
|
||||
|
||||
// A file read again would count as updated.
|
||||
st := syncTree(t, db, dir)
|
||||
|
||||
after := diskSizeOf(t, p)
|
||||
|
||||
if st != (scanStats{walked: 1, unchanged: 1}) {
|
||||
t.Fatalf("rescan stats = %+v, want 1 unchanged", st)
|
||||
}
|
||||
|
||||
got := recordByPath(t, dbRecords(t, db), p).diskSize
|
||||
if want := diskSizeOf(t, p); got != want {
|
||||
t.Errorf("disk_size = %d, want %d", got, want)
|
||||
if got < before || got > after {
|
||||
t.Errorf("disk_size = %d, want %d to %d", got, before, after)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2261,33 +2311,24 @@ func TestUnderRoot(t *testing.T) {
|
||||
}
|
||||
|
||||
// TestRunScanPassesOverDatabase scans a tree that holds the scan's own
|
||||
// database, reached directly and through a symlinked directory. The
|
||||
// walk passes over the database file, its -wal and -shm files and its
|
||||
// .lock file with no line and no count, and the record an earlier scan
|
||||
// left for the .lock file is deleted. A rescan of the unchanged tree
|
||||
// updates nothing, and report lists none of the database's files.
|
||||
// database, and in one case also a hard link to the database file. The
|
||||
// walk knows the database's files by device ID and inode number, so it
|
||||
// passes over the database file, its -wal and -shm files, its .lock
|
||||
// file and the hard link with no line and no count. The record an
|
||||
// earlier scan left for the .lock file is deleted. A rescan of the
|
||||
// unchanged tree updates nothing, and report lists none of the
|
||||
// database's files.
|
||||
func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||
for _, viaSymlink := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("via symlink %v", viaSymlink), func(t *testing.T) {
|
||||
base := t.TempDir()
|
||||
tree := filepath.Join(base, "real", "tree")
|
||||
operand := tree
|
||||
for _, viaHardLink := range []bool{false, true} {
|
||||
t.Run(fmt.Sprintf("via hard link %v", viaHardLink), func(t *testing.T) {
|
||||
tree := realTempDir(t)
|
||||
hardLink := filepath.Join(tree, "hardlink")
|
||||
|
||||
writeFile(t, tree, "a", pattern(1, 100))
|
||||
writeFile(t, tree, "b", pattern(1, 100))
|
||||
writeFile(t, tree, "empty1", nil)
|
||||
writeFile(t, tree, "empty2", nil)
|
||||
|
||||
if viaSymlink {
|
||||
err := os.Symlink(filepath.Join(base, "real"),
|
||||
filepath.Join(base, "link"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
operand = filepath.Join(base, "link", "tree")
|
||||
}
|
||||
|
||||
dbPath := filepath.Join(tree, "db.sqlite")
|
||||
t.Setenv(databaseEnv, dbPath)
|
||||
|
||||
@@ -2297,7 +2338,7 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||
}
|
||||
|
||||
err = applyChanges(t.Context(), db, []scanRec{
|
||||
{path: filepath.Join(operand, "db.sqlite.lock")},
|
||||
{path: filepath.Join(tree, "db.sqlite.lock")},
|
||||
}, nil, nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
@@ -2305,17 +2346,30 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
|
||||
|
||||
closeScanDatabase(t.Context(), db, dbPath)
|
||||
|
||||
assertScanSummary(t,
|
||||
if viaHardLink {
|
||||
err = os.Link(dbPath, hardLink)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
stderr := assertScanSummary(t,
|
||||
"4 files seen (4 added, 0 updated, 1 removed, 0 unchanged)",
|
||||
operand)
|
||||
assertScanSummary(t,
|
||||
tree)
|
||||
stderr += assertScanSummary(t,
|
||||
"4 files seen (0 added, 0 updated, 0 removed, 4 unchanged)",
|
||||
operand)
|
||||
tree)
|
||||
|
||||
if strings.Contains(stderr, hardLink) {
|
||||
t.Errorf("scan stderr = %q, want no line naming %s",
|
||||
stderr, hardLink)
|
||||
}
|
||||
|
||||
report := runStdout(t, cmdReport)
|
||||
if strings.Contains(report, "db.sqlite") ||
|
||||
!strings.Contains(report, filepath.Join(operand, "empty2")) {
|
||||
t.Errorf("report = %q, want the empty files and none of "+
|
||||
strings.Contains(report, hardLink) ||
|
||||
!strings.Contains(report, filepath.Join(tree, "b")) {
|
||||
t.Errorf("report = %q, want a and b and none of "+
|
||||
"the database's files", report)
|
||||
}
|
||||
})
|
||||
@@ -2339,8 +2393,9 @@ func TestRunScanPassesOverDatabaseOperands(t *testing.T) {
|
||||
|
||||
// assertScanSummary scans operands and fails the test unless the
|
||||
// summary line gives counts with nothing skipped, and the line naming
|
||||
// the database is the only one that names one of its files.
|
||||
func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
||||
// the database is the only one that names one of its files. It returns
|
||||
// everything the scan printed to stderr.
|
||||
func assertScanSummary(t *testing.T, counts string, operands ...string) string {
|
||||
t.Helper()
|
||||
|
||||
stderr := scanOK(t, operands...)
|
||||
@@ -2354,4 +2409,6 @@ func assertScanSummary(t *testing.T, counts string, operands ...string) {
|
||||
t.Errorf("scan %q: stderr = %q, want only the line naming the "+
|
||||
"database to name one of its files", operands, stderr)
|
||||
}
|
||||
|
||||
return stderr
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
package main
|
||||
package sfdupes
|
||||
|
||||
import (
|
||||
"context"
|
||||
@@ -37,8 +37,9 @@ func treesPhase(name string) string {
|
||||
|
||||
// runTrees implements the trees subcommand: it reads every record from
|
||||
// the database in path order, reconstructs the directory hierarchy from
|
||||
// the record paths, computes a Merkle-style digest per directory, and
|
||||
// prints maximal duplicate-tree groups as TSV on stdout. It never
|
||||
// the paths of the records at or below the given paths (of every record
|
||||
// when none is given), computes a Merkle-style digest per directory,
|
||||
// and prints maximal duplicate-tree groups as TSV on stdout. It never
|
||||
// touches the scanned filesystem; its only I/O is the database, stdout,
|
||||
// and stderr. It names the database on stderr before it opens it. Any
|
||||
// database problem, including a missing database, is fatal.
|
||||
@@ -49,7 +50,12 @@ func treesPhase(name string) string {
|
||||
// hierarchy), group (group the directories by digest) and write (write
|
||||
// the groups to stdout). Each display ends before the next phase
|
||||
// starts, and before an error is returned to be printed.
|
||||
func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
||||
paths, err := reportPaths(args)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
dbPath, source, err := databasePath()
|
||||
if err != nil {
|
||||
return err
|
||||
@@ -78,7 +84,7 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
label: treesPhase("read"), unit: "records", total: -1,
|
||||
waitingFor: countingRecords, stalledOn: "db",
|
||||
})
|
||||
records, super, allDirs, err := readTree(ctx, db, prog)
|
||||
records, super, allDirs, err := readTree(ctx, db, paths, prog)
|
||||
|
||||
prog.finish()
|
||||
|
||||
@@ -117,10 +123,17 @@ func runTrees(ctx context.Context, stdout io.Writer) error {
|
||||
}
|
||||
|
||||
// readTree gives prog the number of records in db as its total, then
|
||||
// reads every record in path order into the directory hierarchy,
|
||||
// counting each on prog. It returns the number of records read, the
|
||||
// super-root and every directory.
|
||||
func readTree(ctx context.Context, db *sql.DB,
|
||||
// reads every record in path order, counting each on prog. It adds the
|
||||
// records that inReportPaths selects from paths to the directory
|
||||
// hierarchy. It returns the number of records read, the super-root and
|
||||
// every directory at or below one of paths.
|
||||
//
|
||||
// A directory above every one of paths holds only the selected records
|
||||
// under it, so its digest does not describe it. It is left out of the
|
||||
// directories returned. A directory whose parent is left out gets the
|
||||
// super-root as its parent, so suppressed never judges its group by
|
||||
// that parent's digest.
|
||||
func readTree(ctx context.Context, db *sql.DB, paths []string,
|
||||
prog *progress,
|
||||
) (int, *treeNode, []*treeNode, error) {
|
||||
records := 0
|
||||
@@ -130,7 +143,10 @@ func readTree(ctx context.Context, db *sql.DB,
|
||||
records++
|
||||
|
||||
prog.increment()
|
||||
tree.add(r)
|
||||
|
||||
if inReportPaths(r.path, paths) {
|
||||
tree.add(r)
|
||||
}
|
||||
})
|
||||
if err != nil {
|
||||
return 0, nil, nil, err
|
||||
@@ -138,7 +154,21 @@ func readTree(ctx context.Context, db *sql.DB,
|
||||
|
||||
super, allDirs := tree.finish()
|
||||
|
||||
return records, super, allDirs, nil
|
||||
dirs := allDirs[:0]
|
||||
|
||||
for _, d := range allDirs {
|
||||
if !inReportPaths(d.path, paths) {
|
||||
continue
|
||||
}
|
||||
|
||||
if !inReportPaths(d.parent.path, paths) {
|
||||
d.parent = super
|
||||
}
|
||||
|
||||
dirs = append(dirs, d)
|
||||
}
|
||||
|
||||
return records, super, dirs, nil
|
||||
}
|
||||
|
||||
// writeTreeGroups writes dupes to stdout as TSV, a header line and then
|
||||
@@ -301,7 +331,9 @@ func fileEntry(name string, r scanRec) string {
|
||||
// collectTreeGroups groups directories by digest, counting each one on
|
||||
// prog as it is grouped, and returns every maximal group with two or
|
||||
// more members, each group's members sorted by path, groups ordered by
|
||||
// tree size descending then by first path ascending.
|
||||
// tree size descending then by first path ascending. A group whose
|
||||
// trees hold 0 bytes is left out, since removing one frees nothing. The
|
||||
// groups it implies hold 0 bytes too, so leaving it out hides nothing.
|
||||
func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||
prog *progress,
|
||||
) [][]*treeNode {
|
||||
@@ -315,7 +347,9 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
|
||||
var dupes [][]*treeNode
|
||||
|
||||
for _, g := range groups {
|
||||
if len(g) < minGroupSize || suppressed(g, super) {
|
||||
// The members share a digest, so they share a total size.
|
||||
if len(g) < minGroupSize || g[0].totalSize == 0 ||
|
||||
suppressed(g, super) {
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
package main
|
||||
package sfdupes //nolint:testpackage // checks the unexported treeNode results
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"database/sql"
|
||||
"os"
|
||||
"slices"
|
||||
@@ -211,6 +212,55 @@ func TestRunTreesEscapesPaths(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesPaths runs trees with paths. /x/a and /y/a are copies,
|
||||
// and so are /y/a and /srv/b. /x and /y also hold files that differ.
|
||||
// Given /x/a and /y/a, trees reads /x and /y as holding one copy each
|
||||
// of a, yet neither is reported: only the directories at or below a
|
||||
// given path are.
|
||||
func TestRunTreesPaths(t *testing.T) {
|
||||
f, g, k := testHash("f"), testHash("g"), testHash("k")
|
||||
recs := []scanRec{
|
||||
{size: 10, head: f, tail: f, content: f, path: "/srv/b/f"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/x/a/f"},
|
||||
{size: 20, head: g, tail: g, content: g, path: "/x/g"},
|
||||
{size: 10, head: f, tail: f, content: f, path: "/y/a/f"},
|
||||
{size: 30, head: k, tail: k, content: k, path: "/y/k"},
|
||||
}
|
||||
|
||||
header := "first\tdupe\tfiles\tsize\tdisk_size\n"
|
||||
|
||||
cases := map[string]struct {
|
||||
paths []string
|
||||
want string
|
||||
}{
|
||||
"none": {nil, header + "/srv/b\t/x/a\t1\t10\t0\n" +
|
||||
"/srv/b\t/y/a\t1\t10\t0\n"},
|
||||
"two copies": {[]string{"/x/a", "/y/a"},
|
||||
header + "/x/a\t/y/a\t1\t10\t0\n"},
|
||||
"parents": {[]string{"/x", "/y"}, header + "/x/a\t/y/a\t1\t10\t0\n"},
|
||||
"one copy": {[]string{"/srv/b"}, header},
|
||||
}
|
||||
|
||||
for name, tc := range cases {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Setenv(databaseEnv, seedDatabase(t, recs))
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := run(append([]string{cmdTrees}, tc.paths...),
|
||||
&stdout, &stderr)
|
||||
if code != exitOK {
|
||||
t.Fatalf("run(trees %q) = %d, want %d; stderr: %s",
|
||||
tc.paths, code, exitOK, stderr.String())
|
||||
}
|
||||
|
||||
if got := stdout.String(); got != tc.want {
|
||||
t.Errorf("stdout = %q, want %q", got, tc.want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// TestRunTreesDiskSize runs trees on two copies of a tree whose files
|
||||
// take different space on disk. The row's disk_size is the sum over the
|
||||
// dupe copy's files, those in its subdirectory included.
|
||||
@@ -435,6 +485,35 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
|
||||
// holding only empty files form no group, and that an empty file still
|
||||
// counts toward the digest of a tree that holds other files.
|
||||
func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
|
||||
sha256.Sum256(nil)
|
||||
recs := []scanRec{
|
||||
{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
|
||||
{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
|
||||
// Without the empty file, /p/x3 is no copy of /p/x1.
|
||||
{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
|
||||
}
|
||||
|
||||
super, dirs := treeOf(t, recs)
|
||||
|
||||
got := groupPaths(collectTreeGroups(dirs, super, nil))
|
||||
|
||||
want := [][]string{{"/p/x1", "/p/x2"}}
|
||||
if !slices.EqualFunc(got, want, slices.Equal) {
|
||||
t.Fatalf("groups = %v, want %v", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectTreeGroupsDifferingParents(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
Reference in new issue
Block a user